DSI confirmed, when you plug in anything in the B outputs, the B sounds get physically disconnected from A and the headphone only outputs what is on A. So, perhaps what you experience is the missing B layer sounds on the headphone.
Also, DSI confirmed, you will only hear 8 notes play back on a 16 voice unit when playing a single layer (Stack/Split not enabled), and with multi-mode disabled. In my experience this results in 8 notes using layer A and then the next 8 notes using layer B getting the effect of "dropped" notes on the A or B output. In other words... not sure how you can force 16 notes polyphony on one layer in non-stack/split mode.
Perhaps it would be a nice feature to be able to select the number of voices per layer AND to select to which output the layer A and B sounds go. (A only, B only, A+B)