Hi Anders,
As you point out, in the SphericalPanner, elevation is based on an XYZ cube translated to lateral polar coordinates (like an Earth globe with a pole at each ear). Imagine a room where listener is at 0,0,0 and facing the screen. Y is vertical dimension, X horizontal, Z grows larger into the screen and smaller as it comes out of the screen.
If the HRTF recordings you want to use assume a vertical (rather than lateral) polar elevation then yes you would have to rotate the coordinates (perhaps simply by renaming the files? Not sure how they are formatted).