I always thought that would be an artifact of the analogueness of the synth - once you change the parameters, it will immediately effect all voices. There is probably no memory for voice-specific parameters (it would need to be 16 times multitimbral for this?). But then I know near nothing about synth design.
Because I was as annoyed by the behavior, I once wrote code for the KnobKraft Orm that would switch a patch not by sending the sysex, but rather a flood of CC/NRPN messages that would morph the one sound into the other. Of course, not much control over the order of parameters changed, but it would actually be pretty quick and would avoid the nasty note off.
If you don't transfer the sequencer data, the synth has only about 160 parameters per layer IIRC, that would be about 4k of MIDI for both layers with 12 bytes per NRPN. This can be transmitted pretty quickly even over the MIDI jacks, and sounds nice.
Now that I talk about it I need to unearth that code...
https://github.com/christofmuc/KnobKraft-orm/issues/305