I'm assuming that you're using Osc B Amount as the source in Poly-Mod to bring Osc A to pitch. When I try this, there's a little bit of detuning, but it's totally based on keyboard position. I can tune the Poly-Mod Osc B Amount until Osc A stops beating, but it'll start beating a bit on any other note, and detuning increases in proportion to the distance from the originally-tuned note.
I think this would be expected for frequency modulation. Analog FM is never, never, never perfectly in tune. The Tune button has nothing to do with it because we're hearing side-bands (albeit close ones) from Osc A.
I do find it strange that you're hearing detuning based on the voice being used, and not solely on the pitch being played. I can't explain that at all.
If you'd like me (or anyone) to look at it more closely, attach a sysex file containing the actual patch you've made. I'd be happy to play around with it.
In general, across a five octave range, the detuning is noticeable but not unpleasant or extreme.