The input to output delay does not increase as a function of the number of modules in a patch. The entire signal flow graph is computed on each sample.
Some algorithms have a delay built into them as part of the algorithm, and this may be what you are hearing. For example, if you are doing a live spectral analysis and resynthesis, you may hear a delay due to the size of the analysis frame (set by the lowest analyzed frequency).
If you'd like to share your Sound, we would take a look at it and give you some tips for minimizing the input to output delay.