The basic issue is that, when you are reading out of the memory at a different rate than you are writing into it, and the MemoryWriter is cyclic, there is a possibility for the write pointer to cross the read pointer resulting in a discontinuity.
Here's a Sound that can do clickless granular reverb even when you play back the grain cloud at a different !Rate. For a fun effect, try adjusting !LogFreq (to sweep the pitch of the input sine wave).