As you've noticed, the DelayWithFeedback does a smooth transition from the old to the new delay and as a side-effect, the distance between waveform peaks increases or decreases resulting in an apparent pitch change (same principle as a Doppler shift). If you remove the smooth transition, you drop the current contents of the delay line and instantly change delay time, but there can be a click due to the consequent discontinuity in the waveform.
One solution would be to use the smooth transition but to mute the audio output while the transition is occurring. For example, you could attenute the output at the same time you change the delay time, and you could bring the levels back up after the old signal has been cleared from the buffer and the new delay time is stable.
Another, related solution would be to turn off the smoothing and briefly attenuate whenever you change the delay time to smooth over the discontinuity when you do an instant delay change and empty the buffers. Here's an example.