The best way I've found is to use FeedbackLoops and lots of scripting.
You can of course make a matrix mixer by hand with FeedbackLoopInput and FeedbackLoopOutput and lots of mixers on the input of each destination Sound. The issue I've found when I've done that is that the flow gets complicated quite quickly and there's not a neat way of laying out a matrix mixer in the flow editor. You have a lot of wires crossing from every possible source to every possible destination. Once I've built a Sound like that by hand I find that the things I'm creating feedback loops around, which are usually the things I want to change, are embedded in the middle of this web of connections. Also if I change the number of things in my matrix say from a 4x4 matrix to 5x5 it's a bit labourious adding all the extra connections in.
The scripted approach has the advantage of being flexible later but it's not the simpliest scripting.
I've built a couple of matrix mixers (stereo and mono) as re-usable encapsulation, which I'm just about to release (please excuse the shameless plug). They simplify the flow, make it easy to swap the Sounds in the middle of the matrix and create all the VCS controls for you.
Here's a sneak preview:
Here's a quick demo of me using it with a very simple setup of a looping cello sample, a delay and a low pass filter.