Analyzing the REV2 SysEx structure

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Analyzing the REV2 SysEx structure
« on: November 15, 2017, 11:51:20 AM »
I've begun on creating my own editor for the REV2 using the old program SoundDiver (I know... hardly anyone uses this I guess, but it's my SysEx workhorse and editor workhorse here, that is just how it is).

There is no information on this structure for newer DSI synths (a shame really), so I have to analyze every single data byte in the SysEx string to figure out what is what to be able to create both a memory handler, but also the editor itself.

I'm not sure as to why DSI decided to stop this... maybe because they are now version numbering their SysEx structures since the Prophet 12, so that it would take more than one list to cover them all... do not know, but I know that I need that list, so I'm doing it myself now.

This thread will serve as a blog for whatever encounters I find on my way to the end, and will end with a link to a file that describe every single byte in the dumps.

I can start out with an error in the Manual:

The descriptions for the headers of the SysEx strings are plain wrong.

1. The Global Data dump states that you get 50 nibbles (LS then MS) for 25 global parameters... that is not the case... it's in fact 29 MIDI bytes, and they are NOT nibbleized at all.

2. The Program Edit Buffer dump, and the Program Data Dump are also wrong. It states 1024 bytes expanded to 1171 MIDI bytes in "packed MS bit" format... the correct number is 2046 bytes expanded to 2339 MIDI bytes in "packed MS bit" format.

I remember these numbers also being wrong when I made my Prophet 12 editor back then... it seems that DSI are a little quick to just copy/paste these SysEx informations into new products without actually updating them correctly.

More info will follow as I stumble upon it...
If you need me, follow the shadows...

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #1 on: November 15, 2017, 12:30:04 PM »
This is a question for DSI:

I think your SysEx dump of both Program buffer and data have a bug... the size of 2046 bytes (unpacked) seems to not include the last byte needed for the Poly sequencer data when unpacked....

The whole dump is 2344 MIDI bytes long including F0, header, data, F7
The first four header MIDI bytes are: F0 01 2F 03 (for the edit buffer)
Then comes 2340 MIDI bytes (ending with a F7)

That gives 2339 data bytes. When these are unpacked from "packed MS bit" format, it gives 2046 bytes. The last velocity step of the poly sequencers 6th track is missing.

Can you please confirm if there is a bug or not?

It's rather important because if there is, then all the presets people dump will be missing the last velocity step of the 6th track of the polysequencer in Layer B.

Also I'm wondering if the correct size was actually meant to be 2048 bytes? ... this fits more logically when thinking abot usueal memory allocation sizes... the previous (and wrong statement in the manual) was 1024... the double of that would be 2048... so I find it strange that 2047 would be chosen really... also I wonder if this bug is also present for the Global data dump... it looks like one of those bugs where the counter for how much needs to be dumped, is off by one... might be a general bug for all SysEx?
« Last Edit: November 15, 2017, 01:29:11 PM by Razmo »
If you need me, follow the shadows...

dslsynth

  • ***
  • 1040
Re: Analyzing the REV2 SysEx structure
« Reply #2 on: November 15, 2017, 12:39:14 PM »
Interesting! Any chance you could post the edit buffer dump here as a binary file (possibly in a zip file)?
#!/bin/sh
cp -f $0 $HOME/.signature

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #3 on: November 15, 2017, 12:52:44 PM »
Interesting! Any chance you could post the edit buffer dump here as a binary file (possibly in a zip file)?

Sure... it'll be here shortly
If you need me, follow the shadows...

Re: Analyzing the REV2 SysEx structure
« Reply #4 on: November 15, 2017, 01:24:08 PM »
Just write to DSI support and they provide you information about program offsets. As I did when developed the synthet

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #5 on: November 15, 2017, 01:32:41 PM »
Just write to DSI support and they provide you information about program offsets. As I did when developed the synthet

Nope they do not... just ask dslsynth... he just tried and got the cold shoulder... this is obviously something we need to do ourselves because of too little time for DSI to create it.... I wonder how Dave handled all that by himself when he did the Evolver and Prophet08 as a one-man company because these manuals have all the needed information in them... but what'a'ya'know... I'll decode the data, I did it with the Prophet 12, and I'll do it with the REV2 as well... I need the info ... other editors use only the NRPN to create VSTs... but those editors that need to control the memory slots as well will need this SysEx structure data somehow, when editing the structures...
« Last Edit: November 15, 2017, 01:36:06 PM by Razmo »
If you need me, follow the shadows...

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #6 on: November 15, 2017, 01:41:19 PM »
Oh... I probably should mention, that I'm on the latest beta OS... ALWAYS!!! ... :)
If you need me, follow the shadows...

Re: Analyzing the REV2 SysEx structure
« Reply #7 on: November 15, 2017, 01:43:41 PM »
Just write to DSI support and they provide you information about program offsets. As I did when developed the synthet


Nope they do not... just ask dslsynth... he just tried and got the cold shoulder... this is obviously something we need to do ourselves because of too little time for DSI to create it.... I wonder how Dave handled all that by himself when he did the Evolver and Prophet08 as a one-man company because these manuals have all the needed information in them... but what'a'ya'know... I'll decode the data, I did it with the Prophet 12, and I'll do it with the REV2 as well... I need the info ... other editors use only the NRPN to create VSTs... but those editors who need to control the memory slots as well will need to have access to this data...
My editors VST and standalone uses sysex messaging of course. And I requested undocumented information include program offsets in sysex dumps from DSI without any problems.

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #8 on: November 15, 2017, 01:58:05 PM »
Just write to DSI support and they provide you information about program offsets. As I did when developed the synthet


Nope they do not... just ask dslsynth... he just tried and got the cold shoulder... this is obviously something we need to do ourselves because of too little time for DSI to create it.... I wonder how Dave handled all that by himself when he did the Evolver and Prophet08 as a one-man company because these manuals have all the needed information in them... but what'a'ya'know... I'll decode the data, I did it with the Prophet 12, and I'll do it with the REV2 as well... I need the info ... other editors use only the NRPN to create VSTs... but those editors who need to control the memory slots as well will need to have access to this data...
My editors VST and standalone uses sysex messaging of course. And I requested undocumented information include program offsets in sysex dumps from DSI without any problems.


Whom exactly did you talk to about this info? ... I also received it when I made an editor for the programs on Tempest... it was just a cut/paste of the code itself with the list, but it was nonetheless very useful... I just talked to dslsynth from in here... he mailed DSI support to get this info on REV2, and even wrote that it could wait until after NAMM he told me, if they were busy... but he got a VERY definite "NO!" from Carson on support... so I suppose it depends on who you have a talk with at DSI, if you have any luck in this...

Point is that I KNEW this before even making this thread... it's the reason I'm doing it myself instead because I do not want the hassle of asking for this and just get a cold shoulder.... this is info that according to the MIDI specification should be mandatory in any product manual... all other major companies do this... MOOG, Roland, Yamaha, Korg, Waldorf etc... and if not, they usually give you the info if you ask politely about it.

But my history with DSI and bug-reports-fixing-stuff is anyway pretty legendary... just look at my posts on the old 3rd party DSI forum... and I do not want to start another bickering about it, so I'll just do this myself... it's a waste of time really... so when I'm done, the info I get from the analyzing will be freely available for others seeking the information, and spare them the hassle of trying to get it... I'll also post eventual bugs I find in here, and write them to DSI support at the same time to make sure they see them... but I do not expect anything to be fixed... I learned my lesson on this the hard way earlier with especially the Evolver bugs...
« Last Edit: November 15, 2017, 02:01:27 PM by Razmo »
If you need me, follow the shadows...

dslsynth

  • ***
  • 1040
Re: Analyzing the REV2 SysEx structure
« Reply #9 on: November 15, 2017, 02:33:58 PM »
It have to be said that I asked for more than just the parameter offsets. But I think that my request was partially misunderstood as being much more labor intensive than it really would be. My estimate is that no more than five to ten minutes per instrument for the essential information (program offsets, CC/NRPN tables, sources, destinations). And that may even include the nice-to-have information which is the other enum types used in the program format.

The real challenge is documenting the sysex versioning which requires a few bits of information about the older sysex format versions including what version numbers are in use. But it would probably not take that much longer.

However, I was told that their reply was "non negotiable and final" so I will just leave it there. Had they offered the information requested I would have offered to maintain a set of complete and precise sysex documents for all of their current instruments (except for Tempest). Especially as I have to tooling to do it fast, precise and well.
#!/bin/sh
cp -f $0 $HOME/.signature

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #10 on: November 15, 2017, 04:05:16 PM »
I have looked a bit more into this... I have downloaded the factory presets from DSI's website and compared it to the dump I get from the REV2 when I request preset number 1 of the first bank "LosVangelis2041"...

Both the dump and the program in the file are the same length (2046 bytes unpacked), and both is missing not ONE but TWO bytes from the end of the structure (the last two velocity steps of track 6 I think they are).

This also explains my wondering about why the full dump is not 2048, but 2046 instead... the last two bytes is simply missing for some reason.

There are one more difference in the dump and from the file i downloaded... in the file, the last 3 bytes is 00 00 00.... in the dump of the same program from the REV2, they are 0C 00 00

this is strange too because it is exactly the same preset "LosVangelis2041" that I compared... they should be identical... I even dumped the whole file of programs that I compared with to the REV2, before I compared the dump and file...
I think something is wrong... I doubt anyone would have noticed it since we're talking about the last two velocity steps of track 6 in the poly sequencer... it would require that you use six note polyphony in your sequence, and play all the way to the 63rd and 64th step to hear it, and it would only be the velocities... on top you'd have to dump the preset out, and back again to actually loose the two bytes, since they are probably inside the REV2 when you create the sequence...

but still... it must be a bug that should be fixed.
« Last Edit: November 15, 2017, 04:21:36 PM by Razmo »
If you need me, follow the shadows...

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #11 on: November 16, 2017, 12:03:57 AM »
OK... it have come to my knowledge, that the last four bytes in the poly sequencers data are "bad" as soon as it is sent out of the REV2, and that DSI for some reason are unable to fix this... this means that the poly sequencer is litterally broken as soon as you send a patch dump of any kind out of the REV2... even the factory banks from the DSI site are broken like this.

You get 2046 bytes in a dump (when unpacked), but the dump should have been 2048 to fit the whole program... why the last four bytes is damaged, I do not know, only DSI knows this obviously.

I just think that the users should know this fact, that if you decide to dump anything out of the REV2, the last four velocity steps of the polysequencers 6th track in Layer B will be destroyed in the process and are forever gone.

You could say though, that this is probably not so big a deal, as it will require that the last four steps is actually used (steps 61-64), and that it will only apply for the sixth key pressed, and only the velocities of this. And then all this only applies to Layer B.

This is what I've learned... if some of it is wrong, I expect that DSI will correct me.
« Last Edit: November 16, 2017, 12:19:46 AM by Razmo »
If you need me, follow the shadows...

dslsynth

  • ***
  • 1040
Re: Analyzing the REV2 SysEx structure
« Reply #12 on: November 16, 2017, 01:42:44 AM »
OK... it have come to my knowledge, that the last four bytes in the poly sequencers data are "bad" as soon as it is sent out of the REV2, and that DSI for some reason are unable to fix this...

This made me wonder about the reasons for this problem to persist. One explanation could be that the internal code architecture prevents this problem from being fixed or being way too much work for them. Another is that their OS can only handle one size of program vectors in sysex messages and that it would be too much work to support both 2046 and 2048 length sysex messages as a way to make a workaround.

In either case the poly sequencer in layer B is a bit challenged. But at least the MIDI requests seems to be working. So while its plain annoying it could be a lot worse than this. Have you tried how well layer B poly sequencer works on shorter sequence lengths?
#!/bin/sh
cp -f $0 $HOME/.signature

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #13 on: November 16, 2017, 01:52:27 AM »
OK... it have come to my knowledge, that the last four bytes in the poly sequencers data are "bad" as soon as it is sent out of the REV2, and that DSI for some reason are unable to fix this...

This made me wonder about the reasons for this problem to persist. One explanation could be that the internal code architecture prevents this problem from being fixed or being way too much work for them. Another is that their OS can only handle one size of program vectors in sysex messages and that it would be too much work to support both 2046 and 2048 length sysex messages as a way to make a workaround.

In either case the poly sequencer in layer B is a bit challenged. But at least the MIDI requests seems to be working. So while its plain annoying it could be a lot worse than this. Have you tried how well layer B poly sequencer works on shorter sequence lengths?

I'm not even aware that the poly sequencer on Layer B was challenged!? ... I've not used the Polysequencer at all yet, as I've only reached the point in my editor where the memory handling is finished... I've not made any editor yet, and have only now begun playing with the interface of the REV2 today for editing (where I also found some serious weird LFO behavior, that I've written about in another thread, and sent to support).

I will probably not be using the polysequencer much, as I usually prefer to play sequences via me DAW sequencer... but I'll look into it when I reach that stage in the editor creation.
If you need me, follow the shadows...

dslsynth

  • ***
  • 1040
Re: Analyzing the REV2 SysEx structure
« Reply #14 on: November 16, 2017, 02:04:39 AM »
So maybe its an imperfection that is rightfully annoying but not really crippling the instrument in most practical uses? That would of cause be good news! In any case: do check out how things work in practice and see if things work as they should.
#!/bin/sh
cp -f $0 $HOME/.signature

Re: Analyzing the REV2 SysEx structure
« Reply #15 on: November 16, 2017, 02:14:24 AM »
I confirm that the last 4 positions of the sequencer of layer B are not saving in program buffer. I think this is not a bug but restriction of some software design reasons and restrictions like memory limits etc.

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #16 on: November 16, 2017, 02:21:59 AM »
So maybe its an imperfection that is rightfully annoying but not really crippling the instrument in most practical uses? That would of cause be good news! In any case: do check out how things work in practice and see if things work as they should.

Sure... it's a bug that you will only run into in very VERY rare circumstances... sure I can live with it, but I do think that it's best that the users know about it, so that they can avoid it.

To me it's not really an issue (except for the knowledge, so that I can take it into acount when doing the editor of course)... hitting those four missing bytes with the Poly Stepsequencer will be very rare... length would have to be larger than 60 steps... and those steps would have to have six keys pressed at the same time to happen (five or lover should not be a problem), and only the velocity of those four last steps is the problem... and only happening on Layer B... it's obvious that it's a very rare bug to run into.

Besides, I suppose that you can edit these four bytes via NRPN correctly (have not tried this yet, but I see no reason it should not work as it has nothing to do with the SysEx data)... so if you REALLY needed those to be right when editing from an editor, you could simply store the edit buffer from the REV2 frontpanel instead, saving it to a location, and it should be fine I guess...

You could also correct the bug with an editor actually... if you have the right data for the four bytes in the structure inside your editors edit buffer, just send this with the flaw to the REV2, and after this, send out four NRPNs with the right data for those four missing bytes... problem solved.... the only real problem is when you request a program... you'd have to edit those four missing bytes on every dump yourself (logically).

But all in all... yes... a small bug that DSI obviously chose not to bother fixing because it's too complicated... fair enough.
If you need me, follow the shadows...

dslsynth

  • ***
  • 1040
Re: Analyzing the REV2 SysEx structure
« Reply #17 on: November 16, 2017, 02:30:58 AM »
But of cause it would have been nice to know up front as a documented limitation so that users are aware of it. That would be my recommended solution.
#!/bin/sh
cp -f $0 $HOME/.signature

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #18 on: November 16, 2017, 02:36:11 AM »
But of cause it would have been nice to know up front as a documented limitation so that users are aware of it. That would be my recommended solution.

Exactly... it's easily circumvented as long as you know the bug is there, and how to work around it. I'm actually more worried about the weird behavior I'm experiencing with the LFO's right now...
If you need me, follow the shadows...

dslsynth

  • ***
  • 1040
Re: Analyzing the REV2 SysEx structure
« Reply #19 on: November 16, 2017, 02:43:02 AM »
I'm actually more worried about the weird behavior I'm experiencing with the LFO's right now...

Understandable! However, R2 a recent and current product so I would be very surprised if this bug is not resolved at some point in the future. Makes me wonder if its cause is algorithmic or voice CPU overload.
#!/bin/sh
cp -f $0 $HOME/.signature