The Official Sequential/Oberheim Forum

SEQUENTIAL/DSI => Prophet => Prophet Rev2 => Topic started by: Razmo on November 15, 2017, 11:51:20 AM

Title: Analyzing the REV2 SysEx structure
Post by: Razmo on November 15, 2017, 11:51:20 AM
I've begun on creating my own editor for the REV2 using the old program SoundDiver (I know... hardly anyone uses this I guess, but it's my SysEx workhorse and editor workhorse here, that is just how it is).

There is no information on this structure for newer DSI synths (a shame really), so I have to analyze every single data byte in the SysEx string to figure out what is what to be able to create both a memory handler, but also the editor itself.

I'm not sure as to why DSI decided to stop this... maybe because they are now version numbering their SysEx structures since the Prophet 12, so that it would take more than one list to cover them all... do not know, but I know that I need that list, so I'm doing it myself now.

This thread will serve as a blog for whatever encounters I find on my way to the end, and will end with a link to a file that describe every single byte in the dumps.

I can start out with an error in the Manual:

The descriptions for the headers of the SysEx strings are plain wrong.

1. The Global Data dump states that you get 50 nibbles (LS then MS) for 25 global parameters... that is not the case... it's in fact 29 MIDI bytes, and they are NOT nibbleized at all.

2. The Program Edit Buffer dump, and the Program Data Dump are also wrong. It states 1024 bytes expanded to 1171 MIDI bytes in "packed MS bit" format... the correct number is 2046 bytes expanded to 2339 MIDI bytes in "packed MS bit" format.

I remember these numbers also being wrong when I made my Prophet 12 editor back then... it seems that DSI are a little quick to just copy/paste these SysEx informations into new products without actually updating them correctly.

More info will follow as I stumble upon it...
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on November 15, 2017, 12:30:04 PM
This is a question for DSI:

I think your SysEx dump of both Program buffer and data have a bug... the size of 2046 bytes (unpacked) seems to not include the last byte needed for the Poly sequencer data when unpacked....

The whole dump is 2344 MIDI bytes long including F0, header, data, F7
The first four header MIDI bytes are: F0 01 2F 03 (for the edit buffer)
Then comes 2340 MIDI bytes (ending with a F7)

That gives 2339 data bytes. When these are unpacked from "packed MS bit" format, it gives 2046 bytes. The last velocity step of the poly sequencers 6th track is missing.

Can you please confirm if there is a bug or not?

It's rather important because if there is, then all the presets people dump will be missing the last velocity step of the 6th track of the polysequencer in Layer B.

Also I'm wondering if the correct size was actually meant to be 2048 bytes? ... this fits more logically when thinking abot usueal memory allocation sizes... the previous (and wrong statement in the manual) was 1024... the double of that would be 2048... so I find it strange that 2047 would be chosen really... also I wonder if this bug is also present for the Global data dump... it looks like one of those bugs where the counter for how much needs to be dumped, is off by one... might be a general bug for all SysEx?
Title: Re: Analyzing the REV2 SysEx structure
Post by: dslsynth on November 15, 2017, 12:39:14 PM
Interesting! Any chance you could post the edit buffer dump here as a binary file (possibly in a zip file)?
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on November 15, 2017, 12:52:44 PM
Interesting! Any chance you could post the edit buffer dump here as a binary file (possibly in a zip file)?

Sure... it'll be here shortly
Title: Re: Analyzing the REV2 SysEx structure
Post by: codeKnobs on November 15, 2017, 01:24:08 PM
Just write to DSI support and they provide you information about program offsets. As I did when developed the synthet
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on November 15, 2017, 01:32:41 PM
Just write to DSI support and they provide you information about program offsets. As I did when developed the synthet

Nope they do not... just ask dslsynth... he just tried and got the cold shoulder... this is obviously something we need to do ourselves because of too little time for DSI to create it.... I wonder how Dave handled all that by himself when he did the Evolver and Prophet08 as a one-man company because these manuals have all the needed information in them... but what'a'ya'know... I'll decode the data, I did it with the Prophet 12, and I'll do it with the REV2 as well... I need the info ... other editors use only the NRPN to create VSTs... but those editors that need to control the memory slots as well will need this SysEx structure data somehow, when editing the structures...
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on November 15, 2017, 01:41:19 PM
Oh... I probably should mention, that I'm on the latest beta OS... ALWAYS!!! ... :)
Title: Re: Analyzing the REV2 SysEx structure
Post by: codeKnobs on November 15, 2017, 01:43:41 PM
Just write to DSI support and they provide you information about program offsets. As I did when developed the synthet


Nope they do not... just ask dslsynth... he just tried and got the cold shoulder... this is obviously something we need to do ourselves because of too little time for DSI to create it.... I wonder how Dave handled all that by himself when he did the Evolver and Prophet08 as a one-man company because these manuals have all the needed information in them... but what'a'ya'know... I'll decode the data, I did it with the Prophet 12, and I'll do it with the REV2 as well... I need the info ... other editors use only the NRPN to create VSTs... but those editors who need to control the memory slots as well will need to have access to this data...
My editors VST and standalone uses sysex messaging of course. And I requested undocumented information include program offsets in sysex dumps from DSI without any problems.
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on November 15, 2017, 01:58:05 PM
Just write to DSI support and they provide you information about program offsets. As I did when developed the synthet


Nope they do not... just ask dslsynth... he just tried and got the cold shoulder... this is obviously something we need to do ourselves because of too little time for DSI to create it.... I wonder how Dave handled all that by himself when he did the Evolver and Prophet08 as a one-man company because these manuals have all the needed information in them... but what'a'ya'know... I'll decode the data, I did it with the Prophet 12, and I'll do it with the REV2 as well... I need the info ... other editors use only the NRPN to create VSTs... but those editors who need to control the memory slots as well will need to have access to this data...
My editors VST and standalone uses sysex messaging of course. And I requested undocumented information include program offsets in sysex dumps from DSI without any problems.


Whom exactly did you talk to about this info? ... I also received it when I made an editor for the programs on Tempest... it was just a cut/paste of the code itself with the list, but it was nonetheless very useful... I just talked to dslsynth from in here... he mailed DSI support to get this info on REV2, and even wrote that it could wait until after NAMM he told me, if they were busy... but he got a VERY definite "NO!" from Carson on support... so I suppose it depends on who you have a talk with at DSI, if you have any luck in this...

Point is that I KNEW this before even making this thread... it's the reason I'm doing it myself instead because I do not want the hassle of asking for this and just get a cold shoulder.... this is info that according to the MIDI specification should be mandatory in any product manual... all other major companies do this... MOOG, Roland, Yamaha, Korg, Waldorf etc... and if not, they usually give you the info if you ask politely about it.

But my history with DSI and bug-reports-fixing-stuff is anyway pretty legendary... just look at my posts on the old 3rd party DSI forum... and I do not want to start another bickering about it, so I'll just do this myself... it's a waste of time really... so when I'm done, the info I get from the analyzing will be freely available for others seeking the information, and spare them the hassle of trying to get it... I'll also post eventual bugs I find in here, and write them to DSI support at the same time to make sure they see them... but I do not expect anything to be fixed... I learned my lesson on this the hard way earlier with especially the Evolver bugs...
Title: Re: Analyzing the REV2 SysEx structure
Post by: dslsynth on November 15, 2017, 02:33:58 PM
It have to be said that I asked for more than just the parameter offsets. But I think that my request was partially misunderstood as being much more labor intensive than it really would be. My estimate is that no more than five to ten minutes per instrument for the essential information (program offsets, CC/NRPN tables, sources, destinations). And that may even include the nice-to-have information which is the other enum types used in the program format.

The real challenge is documenting the sysex versioning which requires a few bits of information about the older sysex format versions including what version numbers are in use. But it would probably not take that much longer.

However, I was told that their reply was "non negotiable and final" so I will just leave it there. Had they offered the information requested I would have offered to maintain a set of complete and precise sysex documents for all of their current instruments (except for Tempest). Especially as I have to tooling to do it fast, precise and well.
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on November 15, 2017, 04:05:16 PM
I have looked a bit more into this... I have downloaded the factory presets from DSI's website and compared it to the dump I get from the REV2 when I request preset number 1 of the first bank "LosVangelis2041"...

Both the dump and the program in the file are the same length (2046 bytes unpacked), and both is missing not ONE but TWO bytes from the end of the structure (the last two velocity steps of track 6 I think they are).

This also explains my wondering about why the full dump is not 2048, but 2046 instead... the last two bytes is simply missing for some reason.

There are one more difference in the dump and from the file i downloaded... in the file, the last 3 bytes is 00 00 00.... in the dump of the same program from the REV2, they are 0C 00 00

this is strange too because it is exactly the same preset "LosVangelis2041" that I compared... they should be identical... I even dumped the whole file of programs that I compared with to the REV2, before I compared the dump and file...
I think something is wrong... I doubt anyone would have noticed it since we're talking about the last two velocity steps of track 6 in the poly sequencer... it would require that you use six note polyphony in your sequence, and play all the way to the 63rd and 64th step to hear it, and it would only be the velocities... on top you'd have to dump the preset out, and back again to actually loose the two bytes, since they are probably inside the REV2 when you create the sequence...

but still... it must be a bug that should be fixed.
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on November 16, 2017, 12:03:57 AM
OK... it have come to my knowledge, that the last four bytes in the poly sequencers data are "bad" as soon as it is sent out of the REV2, and that DSI for some reason are unable to fix this... this means that the poly sequencer is litterally broken as soon as you send a patch dump of any kind out of the REV2... even the factory banks from the DSI site are broken like this.

You get 2046 bytes in a dump (when unpacked), but the dump should have been 2048 to fit the whole program... why the last four bytes is damaged, I do not know, only DSI knows this obviously.

I just think that the users should know this fact, that if you decide to dump anything out of the REV2, the last four velocity steps of the polysequencers 6th track in Layer B will be destroyed in the process and are forever gone.

You could say though, that this is probably not so big a deal, as it will require that the last four steps is actually used (steps 61-64), and that it will only apply for the sixth key pressed, and only the velocities of this. And then all this only applies to Layer B.

This is what I've learned... if some of it is wrong, I expect that DSI will correct me.
Title: Re: Analyzing the REV2 SysEx structure
Post by: dslsynth on November 16, 2017, 01:42:44 AM
OK... it have come to my knowledge, that the last four bytes in the poly sequencers data are "bad" as soon as it is sent out of the REV2, and that DSI for some reason are unable to fix this...

This made me wonder about the reasons for this problem to persist. One explanation could be that the internal code architecture prevents this problem from being fixed or being way too much work for them. Another is that their OS can only handle one size of program vectors in sysex messages and that it would be too much work to support both 2046 and 2048 length sysex messages as a way to make a workaround.

In either case the poly sequencer in layer B is a bit challenged. But at least the MIDI requests seems to be working. So while its plain annoying it could be a lot worse than this. Have you tried how well layer B poly sequencer works on shorter sequence lengths?
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on November 16, 2017, 01:52:27 AM
OK... it have come to my knowledge, that the last four bytes in the poly sequencers data are "bad" as soon as it is sent out of the REV2, and that DSI for some reason are unable to fix this...

This made me wonder about the reasons for this problem to persist. One explanation could be that the internal code architecture prevents this problem from being fixed or being way too much work for them. Another is that their OS can only handle one size of program vectors in sysex messages and that it would be too much work to support both 2046 and 2048 length sysex messages as a way to make a workaround.

In either case the poly sequencer in layer B is a bit challenged. But at least the MIDI requests seems to be working. So while its plain annoying it could be a lot worse than this. Have you tried how well layer B poly sequencer works on shorter sequence lengths?

I'm not even aware that the poly sequencer on Layer B was challenged!? ... I've not used the Polysequencer at all yet, as I've only reached the point in my editor where the memory handling is finished... I've not made any editor yet, and have only now begun playing with the interface of the REV2 today for editing (where I also found some serious weird LFO behavior, that I've written about in another thread, and sent to support).

I will probably not be using the polysequencer much, as I usually prefer to play sequences via me DAW sequencer... but I'll look into it when I reach that stage in the editor creation.
Title: Re: Analyzing the REV2 SysEx structure
Post by: dslsynth on November 16, 2017, 02:04:39 AM
So maybe its an imperfection that is rightfully annoying but not really crippling the instrument in most practical uses? That would of cause be good news! In any case: do check out how things work in practice and see if things work as they should.
Title: Re: Analyzing the REV2 SysEx structure
Post by: codeKnobs on November 16, 2017, 02:14:24 AM
I confirm that the last 4 positions of the sequencer of layer B are not saving in program buffer. I think this is not a bug but restriction of some software design reasons and restrictions like memory limits etc.
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on November 16, 2017, 02:21:59 AM
So maybe its an imperfection that is rightfully annoying but not really crippling the instrument in most practical uses? That would of cause be good news! In any case: do check out how things work in practice and see if things work as they should.

Sure... it's a bug that you will only run into in very VERY rare circumstances... sure I can live with it, but I do think that it's best that the users know about it, so that they can avoid it.

To me it's not really an issue (except for the knowledge, so that I can take it into acount when doing the editor of course)... hitting those four missing bytes with the Poly Stepsequencer will be very rare... length would have to be larger than 60 steps... and those steps would have to have six keys pressed at the same time to happen (five or lover should not be a problem), and only the velocity of those four last steps is the problem... and only happening on Layer B... it's obvious that it's a very rare bug to run into.

Besides, I suppose that you can edit these four bytes via NRPN correctly (have not tried this yet, but I see no reason it should not work as it has nothing to do with the SysEx data)... so if you REALLY needed those to be right when editing from an editor, you could simply store the edit buffer from the REV2 frontpanel instead, saving it to a location, and it should be fine I guess...

You could also correct the bug with an editor actually... if you have the right data for the four bytes in the structure inside your editors edit buffer, just send this with the flaw to the REV2, and after this, send out four NRPNs with the right data for those four missing bytes... problem solved.... the only real problem is when you request a program... you'd have to edit those four missing bytes on every dump yourself (logically).

But all in all... yes... a small bug that DSI obviously chose not to bother fixing because it's too complicated... fair enough.
Title: Re: Analyzing the REV2 SysEx structure
Post by: dslsynth on November 16, 2017, 02:30:58 AM
But of cause it would have been nice to know up front as a documented limitation so that users are aware of it. That would be my recommended solution.
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on November 16, 2017, 02:36:11 AM
But of cause it would have been nice to know up front as a documented limitation so that users are aware of it. That would be my recommended solution.

Exactly... it's easily circumvented as long as you know the bug is there, and how to work around it. I'm actually more worried about the weird behavior I'm experiencing with the LFO's right now...
Title: Re: Analyzing the REV2 SysEx structure
Post by: dslsynth on November 16, 2017, 02:43:02 AM
I'm actually more worried about the weird behavior I'm experiencing with the LFO's right now...

Understandable! However, R2 a recent and current product so I would be very surprised if this bug is not resolved at some point in the future. Makes me wonder if its cause is algorithmic or voice CPU overload.
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on November 16, 2017, 02:47:53 AM
I'm actually more worried about the weird behavior I'm experiencing with the LFO's right now...

Understandable! However, R2 a recent and current product so I would be very surprised if this bug is not resolved at some point in the future. Makes me wonder if its cause is algorithmic or voice CPU overload.

Hard to say, but I THINK it's just algorithmic... it works right on some keys, and switch around a few keys... the keys it happens on are connected, and it's about 2-4 keys in a row that will cause this... it does this even when only pressing one key at a time... so if the majority works ok this way and a few do not, I believe it cannot be a CPU power problem... but I cannot know... DSI will have to solve this, because it's darn annoying.
Title: Re: Analyzing the REV2 SysEx structure
Post by: dslsynth on November 16, 2017, 02:56:24 AM
This is a perfect time for DSI staff to reassure their customers that this problem will be resolved eventually!
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on November 20, 2017, 06:44:25 AM
Current analysis here... there are quite a few indexes that I either have not populated yet, or they are simply "reserved" spaces because I think I've covered all of the parameters now (all of the NRPN parameters in the manual is covered here, so I doubt more are hiding in the non populated indexes in my list)... there may be some errors in there, as I've not made my editor yet, it's simply a "current state of analysis"... especially the poly sequencer steps bit fields may be wrong... need to experiment as the editor finishes...

Also, mod source and destination tables have not yet been made... they should match the lists in the manual I guess...

And also, note that I have NO IDEAR if there is a byte in one of the unused indexes, that designate a versioning of the dump as is the case with the Prophet 12... I do not even know if there are more than one version at this very moment... it has been made with the presets dumped thru OS 1.1.0.1.

-----------------------------------------

PROGRAM DATA DUMP (SYSEX):

LAYER A:
0      OSC1 Freq (0-120)
1      OSC1 Freq (0-120)
2      OSC1 Finetune (0-100)
3      OSC2 Finetune (0-100)
4      OSC1 Shape (0-4, off/Saw/SawTri/Tri/Pul)
5      OSC2 Shape (0-4, off/Saw/SawTri/Tri/Pul)
6      OSC1 ShapeMod (0-99)
7      OSC2 ShapeMod (0-99)
8      OSC1 Glide (0-127)
9      OSC2 Glide (0-127)
10      OSC1 Key on/off (0-1, off/on)
11      OSC2 Key on/off (0-1, off/on)
12      OSC1 Note Reset (0-1, off/on)
13      OSC2 Note Reset (0-1, off/on)
14      OSC Mix (0-127)
15      SubOct Level (0-127)
16      Noise Level (0-127)
17      Sync (0-1, off/on)
18      Glide Mode (0-3, Fixed Rate/Fixed Rate A/Fixed Time/Fixed Time A)
19      Glide on/off (0-1, off/on)
20      Pitch Bend Range (0-12)
21      OSC Slop (0-127)
22      Cutoff Freq (0-164)
23      Resonance (0-127)
24      Filter Key Amount (0-127)
25      Filter Audio Mod (0-127)
26      Filter Poles (0-1, 2pole/4pole)
27      VCA Level (0-127)
28      Program Volume (0-127)
29      Pan Spread (0-127)
30      Env3 Destination (0-52, *see table below)
31      Env3 Repeat (0-1, off/on)
32      EnvF Amount (0-254)
33      EnvA Amount (0-127)
34      Env3 Amount (0-254)
35      EnvF Velocity (0-127)
36      EnvA Velocity (0-127)
37      Env3 Velocity (0-127)
38      EnvF Delay (0-127)
39      EnvA Delay (0-127)
40      Env3 Delay (0-127)
41      EnvF Attack (0-127)
42      EnvA Attack (0-127)
43      Env3 Attack (0-127)
44      EnvF Decay (0-127)
45      EnvA Decay (0-127)
46      Env3 Decay (0-127)
47      EnvF Sustain (0-127)
48      EnvA Sustain (0-127)
49      Env3 Sustain (0-127)
50      EnvF Release (0-127)
51      EnvA Release (0-127)
52      Env3 Release (0-127)
53      LFO1 Rate (0-150)
54      LFO2 Rate (0-150)
55      LFO3 Rate (0-150)
56      LFO4 Rate (0-150)
57      LFO1 Shape (0-5, off/Tri/Saw/RevSaw/Square/Random)
58      LFO2 Shape (0-5, off/Tri/Saw/RevSaw/Square/Random)
59      LFO3 Shape (0-5, off/Tri/Saw/RevSaw/Square/Random)
60      LFO4 Shape (0-5, off/Tri/Saw/RevSaw/Square/Random)
61      LFO1 Amount (0-127)
62      LFO2 Amount (0-127)
63      LFO3 Amount (0-127)
64      LFO4 Amount (0-127)
65      LFO1 Destination (0-52, *see table below)
66      LFO2 Destination (0-52, *see table below)
67      LFO3 Destination (0-52, *see table below)
68      LFO4 Destination (0-52, *see table below)
69      LFO1 ClkSync (0-1, 0ff/on)
70      LFO2 ClkSync (0-1, 0ff/on)
71      LFO3 ClkSync (0-1, 0ff/on)
72      LFO4 ClkSync (0-1, 0ff/on)
73      LFO1 KeySync (0-1, off/on)
74      LFO2 KeySync (0-1, off/on)
75      LFO3 KeySync (0-1, off/on)
76      LFO4 KeySync (0-1, off/on)
77      Mod1 Source (0-22, *see table below)
78      Mod2 Source (0-22, *see table below)
79      Mod3 Source (0-22, *see table below)
80      Mod4 Source (0-22, *see table below)
81      Mod5 Source (0-22, *see table below)
82      Mod6 Source (0-22, *see table below)
83      Mod7 Source (0-22, *see table below)
84      Mod8 Source (0-22, *see table below)
85      Mod1 Amount (0-254)
86      Mod2 Amount (0-254)
87      Mod3 Amount (0-254)
88      Mod4 Amount (0-254)
89      Mod5 Amount (0-254)
90      Mod6 Amount (0-254)
91      Mod7 Amount (0-254)
92      Mod8 Amount (0-254)
93      Mod1 Destination (0-52, *see table below)
94      Mod2 Destination (0-52, *see table below)
95      Mod3 Destination (0-52, *see table below)
96      Mod4 Destination (0-52, *see table below)
97      Mod5 Destination (0-52, *see table below)
98      Mod6 Destination (0-52, *see table below)
99      Mod7 Destination (0-52, *see table below)
100      Mod8 Destination (0-52, *see table below)
101      Mod Wheel Amount (0-254)
102      Mod Wheel Destination (0-52, *see table below)
103      Pressure Mod Amount (0-254)
104      Pressure Mod Destination (0-52, *see table below)
105      Breath Mod Amount (0-254)
106      Breath Mod Destination (0-52, *see table below)
107      Velocity Mod Amount (0-254)
108      Velocity Mod Destination (0-52, *see table below)
109      MIDI Foot Mod Amount (0-254)
110      MIDI Foot Mod Destination (0-52, *see table below)
111      Gated Seq1 Destination (0-52, *see table below)
112      Gated Seq2 Destination (0-53, *see table below)
113      Gated Seq3 Destination (0-52, *see table below)
114      Gated Seq4 Destination (0-53, *see table below)
115      FX Select (0-13, Off/DM/DDS/BBD/Ch/PH/PL/PM/F1/F2/Rvb/RM/Dst/HPF)
116      FX on/off (0-1, off/on)
117      FX Mix (0-127)
118      FX Parameter 1 (0-255)
119      FX Parameter 2 (0-127)
120      FX Clock Sync on/off (0-1, off/on)
121      -
122      Key Mode (0-5, Low/Hi/Last/LowR/HiR/LastR)
123      Unison on/off (0-1, off/on)
124      Unison Mode (0-15/Chord)
125      -
126      -
127      -
128      -
129      -
130      BPM (30-250)
131      Divide (0-12, H/Q/8th/8H/8S/8T/16th/16H/16S/16T/32nd/32T/64T
132      Arp Mode (0-4, Up/Down/Up+Down/Random/assign)
133      Arp Range (0-2, 1Oct/2Oct/3Oct)
134      Arp Repeats (0-3)
135      Arp Relatch on/off (0-1, off/on)
136      Arp on/off (0-1, off/on)
137      -
138      Sequencer Mode (0-4, Normal/NoReset/NoGate/NoGateReset/KeyStep)
139      Sequencer Type (0-1, Gated/Poly)
140-155   Gated Seq1 Step 1-16 (0-125/reset/rest)
156-171   Gated Seq2 Step 1-16 (0-125/reset/rest)
172-187   Gated Seq3 Step 1-16 (0-125/reset/rest)
188-203   Gated Seq4 Step 1-16 (0-125/reset/rest)
204      -
205      -
206      -
207      -
208      Unison Detune (0-16)
209      Pan Mod Mode (0-1, Alternate/Fixed)
210      -
211      -
212      -
213      -
214      -
215      -
216      -
217      -
218      -
219      -
220      -
221      -
222      -
223      -
224      -
225      -
226      -
227      -
228      -
229      -
230      -
231      Layer Mode (0-2, LayerA/SplitAB/StackAB)
232      Split Point (0-120)
233      -
234      -
235-254   Layer A Name
255      -
256-319   Poly Seq Track1 Notes (bit0-7;note, bit8;tie)
320-383   Poly Seq Track1 Velocities (bit0-7;velocity, bit8;rest)
384-447   Poly Seq Track2 Notes (bit0-7;note, bit8;tie)
448-511   Poly Seq Track2 Velocities (bit0-7;velocity, bit8;rest)
512-575   Poly Seq Track3 Notes (bit0-7;note, bit8;tie)
576-639   Poly Seq Track3 Velocities (bit0-7;velocity, bit8;rest)
640-703   Poly Seq Track4 Notes (bit0-7;note, bit8;tie)
704-767   Poly Seq Track4 Velocities (bit0-7;velocity, bit8;rest)
768-831   Poly Seq Track5 Notes (bit0-7;note, bit8;tie)
832-895   Poly Seq Track5 Velocities (bit0-7;velocity, bit8;rest)
896-959   Poly Seq Track6 Notes (bit0-7;note, bit8;tie)
960-1023   Poly Seq Track6 Velocities (bit0-7;velocity, bit8;rest)

LAYER B:
Same as above, just add 1024 to the index value.
Last four bytes (2044-2047) does not exist in SysEx dump.
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on November 20, 2017, 07:12:37 AM
a few errors in the manual regarding NRPNs:

NRPN no. 9 is missing (OSC2 KBD on/off)
Description of NRPN no. 37 is a copy of the above (VCA Env Release), should be LFO1 Frequency
NRPN 52 description... the number "4" is missing.
 
Title: Re: Analyzing the REV2 SysEx structure
Post by: shiihs on March 25, 2018, 03:10:27 PM
Thanks for posting the sysex analysis. It has been tremendously helpful in my experiments so far.
It's a pity this is not documented officially, but I guess there's no guarantee for sysex to remain compatible in future OS revisions, so it might be too much work to maintain the documentation. Anyway I wanted to contribute my 5 cents by adding a sysex analysis of the global parameters in OS version 1.1.0.3. Note that the data must NOT be midi-unpacked in order to analyze the contents!

SYSEX_BYTEPOS   NRPN    NAME                    VALUES      DISPLAY_VALUES
0               4097    "Master Coarse Tune"    0-24        -12 - +12
1               4096    "Master Fine Tune"      0-100       -50 - +50
2               4098    "MIDI Channel"          0-16        0=ALL,1-16
3               4099    "MIDI Clock Mode"       0-4         0=OFF,1=MASTER,2=SLAVE,3=SLAVE THRU,4=SLAVE NO S/S
4               4100    "MIDI Clock Cable"      0-1         0=MIDI,1=USB
5               4101    "MIDI Param Send"       0-2         0=OFF,1=CC,2=NRPN
6               4102    "MIDI Param Receive"    0-2         0=OFF,1=CC,2=NRPN
7               4103    "MIDI Control Enable"   0-1         0=OFF,1=ON
8               4104    "MIDI SysEx Cable"      0-1         0=MIDI,1=USB
9               4105    "MIDI Out Select"       0-1         0=MIDI,1=USB
10               --     "UNKNOWN"
11               --     "UNKNOWN"
12              4107    "Local Control"         0-1         0=OFF,1=ON
13              4111    "Seq Pedal Mode"        0-3         0=NORMAL,1=TRIGGER,2=GATE,3=TRIG-GATE
14              4109    "Pot Mode"              0-2         0=RELATIVE,1=PASSTHRU,2=JUMP
15              4112    "Sustain Polarity"      0-1         0=NORMAL,1=REVERSE
16              4116    "Alt. Tunings"          0-16        (see list in manual)
17              4113    "Velocity Curve"        0-7         velocity curve 1,2,..8
18              4114    "Pressure Curve"        0-3         pressure curve 1,2,..4
19              4115    "Mono/Stereo"           0-1         0=stereo,1=mono
20              4120    "Screen Saver"          0-1         0=OFF,1=ON
21              4119    "Multi Mode"            0-1         0=OFF,1=ON
22              4118    "MIDI Prog Enable"      0-1         0=OFF,1=ON
23              4121    "Sustain/Arp"           0-1         0=ARP HOLD,1=SUSTAIN
24              4122    "Foot Assign"           0-5         0=Breath CC2,1=Foot CC4,2=Exp CC11,3=Volume,4=LPF Full,5=LPF Half
25               --     "UNKNOWN"
26               --     "UNKNOWN"
27               --     "UNKNOWN"
28               --     "UNKNOWN"

Title: Re: Analyzing the REV2 SysEx structure
Post by: dslsynth on March 26, 2018, 02:07:41 PM
It's a pity this is not documented officially, but I guess there's no guarantee for sysex to remain compatible in future OS revisions, so it might be too much work to maintain the documentation.

Tooling exist to ease the documentation of the sysex format and and that tooling is quite robust to changes caused by software updates. So its indeed possible to document the sysex formats precisely. However, its also a question of resources and priorities.
Title: Re: Analyzing the REV2 SysEx structure
Post by: musicmaker on April 20, 2018, 12:23:44 AM
Although it not a critical issue it just gives a feeling the REV2 is not "complete". Apparently there are unused Sysex fields that  could be used to add the data for this step and such change would not break backward compatibility with the current patches.


Title: Re: Analyzing the REV2 SysEx structure
Post by: extempo on April 20, 2018, 11:48:09 AM
In development you leave space for parameters you may or may not need. In the end there are bytes that don't get used. This in no way makes the instrument incomplete.
Title: Re: Analyzing the REV2 SysEx structure
Post by: chapelier fou on July 26, 2018, 01:18:37 PM
I've never used Sysex messages other than to dump or save stuff.
But does this mean that i can remotely change a Gated Pattern sequence by sending sysex messages ?
Title: Re: Analyzing the REV2 SysEx structure
Post by: RonGerrist on July 26, 2018, 01:37:08 PM
I've never used Sysex messages other than to dump or save stuff.
But does this mean that i can remotely change a Gated Pattern sequence by sending sysex messages ?

It seems like it. I would go for just using NRPN for that; much easier.
Title: Re: Analyzing the REV2 SysEx structure
Post by: chapelier fou on July 26, 2018, 01:47:15 PM
Great, thanks, that's exciting !
I'd like to send patterns from Max For Live to the Prophet, I don't know much about either Sysex or NRPN, but i'll check. There is a brand new free max for live NRPN message sender, so I can grab some knowledge from there. But Sysex might be more convenient to format with Max/MSP [sxformat] object ? I'm open to suggestions !
Title: Re: Analyzing the REV2 SysEx structure
Post by: chapelier fou on July 26, 2018, 01:57:09 PM
So, I you don't mind helping a Sysex noob, what would a sysex message to set up seq 1 step 1 look like ?
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on July 26, 2018, 01:58:03 PM
I've never used Sysex messages other than to dump or save stuff.
But does this mean that i can remotely change a Gated Pattern sequence by sending sysex messages ?

Yes you can... you can change everything with the SysEx commands... With only NRPNs you do not have full control of the entire memory, you do with SysEx... you can store presets (even in the factory ones, and this only works via SysEx dumps). Also all global parameters can be changed... A SysEx for the REV2 has all the parameters of both layers in it, including the gated sequencers and the polysequencers data as well... unfortunately the SysEx of the REV2 is broken... the last four values of the sixth tracks velocity in layer B are not in the dump...

I've made an editor that control all with the VEry old SoundDiver program from Emagic, and it works fine:
Title: Re: Analyzing the REV2 SysEx structure
Post by: chapelier fou on July 26, 2018, 02:02:48 PM
I've made an editor that control all with the VEry old SoundDiver program from Emagic, and it works fine:
Wow, looks nice ! Congrats ! Out of curiosity, do you share the editor ? (I'm on Mac).
As for the broken sysex messages, can we hope for a firmware update ?
Anyway, I'd be happy if someone could help me with how to format the messages to change the Seq values.
Title: Re: Analyzing the REV2 SysEx structure
Post by: shiihs on July 26, 2018, 02:10:16 PM
So, I you don't mind helping a Sysex noob, what would a sysex message to set up seq 1 step 1 look like ?

If you want to change only a few parameters you are probably much better off sending NRPN messages.
Every step has its own NRPN parameter. These are documented in the manual. In the gated sequencer you can also use NRPN to set up the destinations.

You cannot - to the best of my knowledge - use sysex to update only some parameters selectively, you always have to send all values for all parameters in bulk. Add to that that the sysex structure at times is a bit tricky, with e.g. sign bits of modulation values hiding inside bytes that encode other (enumeration) parameters. I assume this complexity was introduced to keep backward compatibility with prophet 8, but that's only a wild guess as I've never studied it.
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on July 26, 2018, 02:19:12 PM
I've made an editor that control all with the VEry old SoundDiver program from Emagic, and it works fine:
Wow, looks nice ! Congrats ! Out of curiosity, do you share the editor ? (I'm on Mac).
As for the broken sysex messages, can we hope for a firmware update ?
Anyway, I'd be happy if someone could help me with how to format the messages to change the Seq values.

Yes, i don't mind sharing the adaption if anyone want it, but i don't think that many are using SoundDiver anymore to be honest... if anyone want it, just write me a Private Message and I'll send it...

The broken SysEx is on the bug list at DSI... I've reported the bug twice, and the second time they asked me to submit a report again, to make sure it was put on the bug list... so i asume it'll be fixed soon... but it still does not work in the current beta, so I'm still waiting for the fix.

The NRPNs are just plain NRPNs... the values for the sequencer is all in the manual, but for Layer A, Sequence value one it is:

$B0, $63, $01 (top 7 bits of the parameter number)
$B0, $62, $40 (bottom 7 bits of the parameter number)

$B0, $06, $00 (top 7 bits of the parameter value)
$B0, $26, $00 (bottom 7 bits of the parameter value)

remember that MIDI bytes are 7 bits, not 8! ... both values are 14 bits in length, split over two 7bit MIDI bytes.
Title: Re: Analyzing the REV2 SysEx structure
Post by: chapelier fou on July 26, 2018, 02:21:22 PM

If you want to change only a few parameters you are probably much better off sending NRPN messages.
Every step has its own NRPN parameter. These are documented in the manual. In the gated sequencer you can also use NRPN to set up the destinations.

You cannot - to the best of my knowledge - use sysex to update only some parameters selectively, you always have to send all values for all parameters in bulk. Add to that that the sysex structure at times is a bit tricky, with e.g. sign bits of modulation values hiding inside bytes that encode other (enumeration) parameters. I assume this complexity was introduced to keep backward compatibility with prophet 8, but that's only a wild guess as I've never studied it.

Ok that's super helpful. I'll try to figure out how to send NRPNs from Max. Thanks a lot, I appreciate the spirit of this place !
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on July 26, 2018, 02:21:37 PM
So, I you don't mind helping a Sysex noob, what would a sysex message to set up seq 1 step 1 look like ?

If you want to change only a few parameters you are probably much better off sending NRPN messages.
Every step has its own NRPN parameter. These are documented in the manual. In the gated sequencer you can also use NRPN to set up the destinations.

You cannot - to the best of my knowledge - use sysex to update only some parameters selectively, you always have to send all values for all parameters in bulk. Add to that that the sysex structure at times is a bit tricky, with e.g. sign bits of modulation values hiding inside bytes that encode other (enumeration) parameters. I assume this complexity was introduced to keep backward compatibility with prophet 8, but that's only a wild guess as I've never studied it.

True... all single parameter changes must be done via NRPNs... even the global ones, plus all sequencer data... the advantage to the SysEx is that it allow you to control the program memories which is nice for creating librarians and managing your programs (and also overwrite the factory programs... as this is only doable via SysEx dumps)..
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on July 26, 2018, 02:25:23 PM
By the way... it's not totally true that you can only overwrite the factory programs via SysEx... you can copy all user banks to the factory banks and vice versa from the REV2 itself, you just have to enter service mode and choose that function... hold down the two transpose buttons, and at the same time press "global"... you can find the feature in the service mode.
Title: Re: Analyzing the REV2 SysEx structure
Post by: chapelier fou on July 26, 2018, 02:29:31 PM
True... all single parameter changes must be done via NRPNs... even the global ones, plus all sequencer data... the advantage to the SysEx is that it allow you to control the program memories which is nice for creating librarians and managing your programs (and also overwrite the factory programs... as this is only doable via SysEx dumps)..
Yeah, but my point is just to be able to use any sound in the synth and use M4L to send a pre-made sequence to the sequencers.
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on July 26, 2018, 02:32:58 PM
True... all single parameter changes must be done via NRPNs... even the global ones, plus all sequencer data... the advantage to the SysEx is that it allow you to control the program memories which is nice for creating librarians and managing your programs (and also overwrite the factory programs... as this is only doable via SysEx dumps)..
Yeah, but my point is just to be able to use any sound in the synth and use M4L to send a pre-made sequence to the sequencers.

Then you'll only need NRPN functionality... there are a few parameters in the program structure that you can only get at via SysEx dumps though, but they are not really needed for normal use... one I recall is an "Editor Number"... a stored value that is used to signal what editor was used to create a program, and then a "Version Number" that designates what version the dump structure is... Prophet 12 for example has more than one revision of the dump... but this is of course irrelevant for your use :)
Title: Re: Analyzing the REV2 SysEx structure
Post by: shiihs on July 26, 2018, 02:44:25 PM
Yeah, but my point is just to be able to use any sound in the synth and use M4L to send a pre-made sequence to the sequencers.

I implemented that (and a whole lot more) in supercollider (available on github).

Sending an NRPN is not so difficult, basically you have to send four midi control change messages:

on CC 99, send the most significant 7 bits of the NRPN number (NRPN >> 7)
on CC 98, send the least significant 7 bits of the NRPN number (NRPN & 127)
on CC 6, send the most significant 7 bits of the desired value (value >> 7)
on CC 38, send the least significant 7 bits of the desired value (value & 127)

and you have to make sure that your synth is set up to receive NRPN parameters (it's in the global options somewhere).
Title: Re: Analyzing the REV2 SysEx structure
Post by: chapelier fou on July 27, 2018, 03:56:00 AM
Many thanks to you. I've got it working and it's promising !
Title: Re: Analyzing the REV2 SysEx structure
Post by: cbx on September 19, 2018, 11:07:10 PM
Did he ever figure it out? All the manufacturers no longer provide this information for new products. My guess is they expect MIDI to finally go away. But, Korg and Dave Smith always kept to the tradition so I was surprised they stopped. Considering Dave Smith invented MIDI.

If only you could read NRPN controller values from the machine, there would be little need for SYSEX anyways.
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on September 20, 2018, 02:38:55 AM
SysEx still would be the only way to do memory management, even if NRPN's could be read out... SysEx has no limits in size either... With NRPN you have a maximum number of 16384 which will be way too little if you were to address a lot of programs this way. You would have to set the current memory program slot by defining an NRPN as a program pointer in that case... It is doable, but highly controversial i would say.
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on September 20, 2018, 08:16:06 AM
Did he ever figure it out? All the manufacturers no longer provide this information for new products. My guess is they expect MIDI to finally go away. But, Korg and Dave Smith always kept to the tradition so I was surprised they stopped. Considering Dave Smith invented MIDI.

If only you could read NRPN controller values from the machine, there would be little need for SYSEX anyways.

The change with not listing the SysEx structure started with the Prophet 12 as far as I'm aware, and my bet on why they have started dropping this info is because since the Prophet 12, they began doing "versioning" of the SysEx structure if they wanted to change something... The Prophet 12 is at version 3 of it's SysEx, and when that happened, my own editor stopped working correctly with some mod matrix sources/destination... that was because they put in more, and also sorted them, meaning that a source that had a specific index before, now suddenly had a new index... they did not simply "add to the indexes"...

This means that if the SysEx of a device is getting a new version, any information already out would be wrong... they would have to give out a new document with each new version of the SysEx... even the SysEx length could be changed this way because the synth is internally accepting older versions of the SysEx dump... the dump has a dedicated "version byte" in it, so the OS can always see what version the program was made in, and adapt to it, without the user even noticing it.

This versioning though gives editors a hard time following up if they have to support all the versions...

Rev2 is actually a "version 2" ... the version byte in the SysEx structure says "2"... I suspect that version 1 is in fact simply the old Prophet 8 format because the REV2 can load the older Prophet 08 programs directly...
Title: Re: Analyzing the REV2 SysEx structure
Post by: cbx on September 20, 2018, 01:38:59 PM
SysEx still would be the only way to do memory management, even if NRPN's could be read out... SysEx has no limits in size either... With NRPN you have a maximum number of 16384 which will be way too little if you were to address a lot of programs this way. You would have to set the current memory program slot by defining an NRPN as a program pointer in that case... It is doable, but highly controversial i would say.

True. I need the sysex for sound design algorithms. I don't think I'll get any new syntheziers anyways, but if I can't program the Rev 2 at all algorithmically I just will pass on it.
Title: Re: Analyzing the REV2 SysEx structure
Post by: cbx on September 20, 2018, 03:47:54 PM
Well it looks cool I hope you get it working. The more I think about it, the less I really need the sysex. I would just be starting with a patch from the prophet 08 and then transfer them to the Rev 2 using NRPN. While it would be useful to have the information for the sysex I could still program the machine with just NRPN controls and a patch from another machine as a start.
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on September 20, 2018, 04:22:54 PM
Well it looks cool I hope you get it working. The more I think about it, the less I really need the sysex. I would just be starting with a patch from the prophet 08 and then transfer them to the Rev 2 using NRPN. While it would be useful to have the information for the sysex I could still program the machine with just NRPN controls and a patch from another machine as a start.

I analyzed the SysEx structure... you can find it earlier in this thread :)
Title: Re: Analyzing the REV2 SysEx structure
Post by: cbx on September 20, 2018, 07:32:30 PM
Well it looks cool I hope you get it working. The more I think about it, the less I really need the sysex. I would just be starting with a patch from the prophet 08 and then transfer them to the Rev 2 using NRPN. While it would be useful to have the information for the sysex I could still program the machine with just NRPN controls and a patch from another machine as a start.

I analyzed the SysEx structure... you can find it earlier in this thread :)

Ok, sorry for bumping, I was unsure if you had finished it, because it said you would update when you built your editor.

Does it use the same algorithm the Prophet 08 used for packing the high bits?

Thanks for doing this, I don't know if I will get one as I am very happy with my Mopho-10.
Title: Re: Analyzing the REV2 SysEx structure
Post by: cbx on September 21, 2018, 12:23:24 AM
I think they don't publish them anymore because most people are wireless now, and maybe studios complained about this. But I'd think with bluetooth it should not be possible to hack someone's keyboard when they are on stage. Otherwise, I can't think of any darn good reason why they no longer give you this information. It's like buying a car and keeping how to change a mystery. Kind of annoying me.
Title: Re: Analyzing the REV2 SysEx structure
Post by: cbx on September 21, 2018, 12:33:23 AM
Anyhow, sorry for so many posts but, it is all manufacturers that do this now. DSI was the last to stop publishing them I think. Korg always made everything available, which is why I liked them. Now, nothing. So, it is the entire industry, not just Sequential.
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on September 21, 2018, 01:02:02 AM
Well it looks cool I hope you get it working. The more I think about it, the less I really need the sysex. I would just be starting with a patch from the prophet 08 and then transfer them to the Rev 2 using NRPN. While it would be useful to have the information for the sysex I could still program the machine with just NRPN controls and a patch from another machine as a start.

I analyzed the SysEx structure... you can find it earlier in this thread :)

Ok, sorry for bumping, I was unsure if you had finished it, because it said you would update when you built your editor.

Does it use the same algorithm the Prophet 08 used for packing the high bits?

Thanks for doing this, I don't know if I will get one as I am very happy with my Mopho-10.

Yes, DSI has used the same packing scheme since the Evolver, so it's exactly the same... the list should be ok, but please notice that if DSI suddenly change the structure it would not be correct anymore... I had a private message with support regarding the current bug with the LFO time divisions (in sync mode), and I was told that they have this on the bug lidt, and will do some "versioning" to fix it with the next OS beta... I'm not sure if they mean a version 3 of the REV2 SysEx, but if it is, my list on page 2 of this thread will probably not match anymore... only time will tell...
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on September 21, 2018, 01:06:38 AM
Anyhow, sorry for so many posts but, it is all manufacturers that do this now. DSI was the last to stop publishing them I think. Korg always made everything available, which is why I liked them. Now, nothing. So, it is the entire industry, not just Sequential.

As long as they publish the SysEx dump and request messages, I can live with it because it's not that hard to analyze what bytes is what... but it is definitely tiresome to do... some companies have them but won't give them unless you ask for them... I've had a table from MOOG every time I asked for several of their products, and Waldorf usually end up with a document at some point, though they're rather slow at it... Korg (as you say) always has the info, so does Roland and Yamaha... still... it seems to be the smaller niche companies that for some reason think that this info is not important anymore... and yes, it's irritating to say the least, but it's in vain to try and ask for them at DSI... they have not even explained why they do not create these lists anymore... just plain silence...
Title: Re: Analyzing the REV2 SysEx structure
Post by: extempo on September 21, 2018, 09:38:34 AM
We absolutely provide parameter offset lists to customers who ask. Combined with the program dump sysex structure outlined in the MIDI implementation section of our manuals, this fills in the blanks for anyone looking to build an editor, or something of the sort. Interested parties are few and far between, but we do provide the info. Razmo, I believe we provided you with precisely such a document for the Rev2 at one point, did we not?
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on September 21, 2018, 12:30:21 PM
We absolutely provide parameter offset lists to customers who ask. Combined with the program dump sysex structure outlined in the MIDI implementation section of our manuals, this fills in the blanks for anyone looking to build an editor, or something of the sort. Interested parties are few and far between, but we do provide the info. Razmo, I believe we provided you with precisely such a document for the Rev2 at one point, did we not?

No you did not... if you did, there would have been no need to analyze everything mysself ;) ... in fact I did not ask for it either, as I decided to just do it myself instead... you did provide me with data for the Tempest though...

I can see now that my comment above could be read as if you at DSI do not give out the SysEx specs... that was not what i meant actually... I meant that it would be in vain trying to make you include this in the manuals again... many have shown an interest in this info over the last many years in different threads in here, but you at DSI never comment on it, so I assume you have no intention of supplying this info anymore (in the manuals)...

Would you care to tell us, why you have decided not to include this info in the manual, as it was in earlier manuals for your products?
Title: Re: Analyzing the REV2 SysEx structure
Post by: dslsynth on September 21, 2018, 02:23:00 PM
Most likely its just too much work to document manually. Especially as new versions of the sysex format shows up.

If the full sysex spec were readily available it would be used a lot more because people would just download it from the site and start working/playing with it. Much like what happened with the older and now discontinued DSI instruments. So in that way the currently observed demand at the support office may not reflect the actual interest in having this information available.
Title: Re: Analyzing the REV2 SysEx structure
Post by: cbx on September 21, 2018, 03:00:37 PM
Most likely its just too much work to document manually. Especially as new versions of the sysex format shows up.

If the full sysex spec were readily available it would be used a lot more because people would just download it from the site and start working/playing with it. Much like what happened with the older and now discontinued DSI instruments. So in that way the currently observed demand at the support office may not reflect the actual interest in having this information available.

Beat's me, but they use to to do this with keyboards that had far more parameters than these ones. Korg and Roland workstations had massive sysex, all documented. Now nobody documents it. It makes me upset because you need that information for get the data for the patch, so you can use it for sound design. I was very disappointed Moog never documented their keyboards either, all you have is NRPN which isn't useful for sound design because you can't poll the keyboard for the control values.
Title: Re: Analyzing the REV2 SysEx structure
Post by: extempo on September 21, 2018, 03:11:50 PM
With the exception of the Evolver series, which used sysex for real-time parameter changes rather than NRPNs, all of our instruments include the same basic information in the MIDI implementation section of the manuals. We can provide a list of the packed parameter offsets for program dumps to those who write support and ask. This information is not listed in our manuals because it takes up a lot of space and it's relevant only to a tiny portion of users. I understand that it's extremely relevant to this portion of users, but the fact remains that the vast majority never do more with sysex than perhaps backup their programs. I don't mean to be pedantic, but if you totaled the number of people who expressed interest in detailed sysex information for our instruments in forum posts across the web, it would equal a tiny portion of the instruments in the field. In any case, write support to ask and ye shall receive :)

Regarding changes to the parameter data included in a program dump, when we develop a new instrument, we will do some rough calculations as to how large a program needs to be to hold all of the parameters we anticipate it containing. There are always empty bytes included to allow for parameters to be added as an OS develops and matures. If new parameters are added in an OS update, this doesn't alter the order of the structure from previous versions--new parameters will just occupy bytes that were previously unused.

I hope this helps!
Title: Re: Analyzing the REV2 SysEx structure
Post by: cbx on September 21, 2018, 03:36:29 PM
With the exception of the Evolver series, which used sysex for real-time parameter changes rather than NRPNs, all of our instruments include the same basic information in the MIDI implementation section of the manuals. We can provide a list of the packed parameter offsets for program dumps to those who write support and ask. This information is not listed in our manuals because it takes up a lot of space and it's relevant only to a tiny portion of users. I understand that it's extremely relevant to this portion of users, but the fact remains that the vast majority never do more with sysex than perhaps backup their programs. I don't mean to be pedantic, but if you totaled the number of people who expressed interest in detailed sysex information for our instruments in forum posts across the web, it would equal a tiny portion of the instruments in the field. In any case, write support to ask and ye shall receive :)

Regarding changes to the parameter data included in a program dump, when we develop a new instrument, we will do some rough calculations as to how large a program needs to be to hold all of the parameters we anticipate it containing. There are always empty bytes included to allow for parameters to be added as an OS develops and matures. If new parameters are added in an OS update, this doesn't alter the order of the structure from previous versions--new parameters will just occupy bytes that were previously unused.

I hope this helps!

Thank you very much!
Title: Re: Analyzing the REV2 SysEx structure
Post by: dslsynth on September 21, 2018, 03:54:06 PM
There are always empty bytes included to allow for parameters to be added as an OS develops and matures. If new parameters are added in an OS update, this doesn't alter the order of the structure from previous versions--new parameters will just occupy bytes that were previously unused.

Thanks for the answer! :-)

To be ultra pedantic there two types of changes to the program vector: parameter additions and parameter value changes. Value changes such as adding new sources or destinations will not change the program vector itself but will require a version number in the sysex format to tell older program versions from newer program versions. Similar actions may be needed for new parameters if zero is not the neutral state of a parameter.

The advantage of versioned programs is that it allows for better ordering of sources/destinations in the user interface. The disadvantage of versioned programs is that not all generic editors support versioned programs and it take extra effort to document it precisely.
Title: Re: Analyzing the REV2 SysEx structure
Post by: cbx on September 21, 2018, 05:05:14 PM
The Prophet Rev 2 looks really cool I've been reading the manual. I have two Tetras I use in various ways, and I think the Rev2 can do almost everything that they can do. I use the tetras in different ways, sometimes with my Mopho keyboards to extend the voices to 5. But, you can also layer programs on the tetra, so I will use it with (the mopho) to stack patches. Lately I have had it set up in series in multimode giving me 8 sequencers and a midi looper. I do all percussion and rhthym on one and the other is for melody and harmonies. So, I have been very happy with my setup. But, the Rev 2 can do some thing interesting things that this can not. Even if I don't have sysex I can always start with a Prophet 08/mopho patch and fine tune the NRPN, so it wasn't a huge problem but still, I do use the sysex data when designing sounds. On my moog little phatty, allI can do is randomize the CC controllers as a starting point. I can do that with Mopho/Prophet 08 too, but it's easier if I can just load sysex and then apply meta patch as starting point.
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on September 22, 2018, 02:37:03 AM
With the exception of the Evolver series, which used sysex for real-time parameter changes rather than NRPNs, all of our instruments include the same basic information in the MIDI implementation section of the manuals. We can provide a list of the packed parameter offsets for program dumps to those who write support and ask. This information is not listed in our manuals because it takes up a lot of space and it's relevant only to a tiny portion of users. I understand that it's extremely relevant to this portion of users, but the fact remains that the vast majority never do more with sysex than perhaps backup their programs. I don't mean to be pedantic, but if you totaled the number of people who expressed interest in detailed sysex information for our instruments in forum posts across the web, it would equal a tiny portion of the instruments in the field. In any case, write support to ask and ye shall receive :)

Regarding changes to the parameter data included in a program dump, when we develop a new instrument, we will do some rough calculations as to how large a program needs to be to hold all of the parameters we anticipate it containing. There are always empty bytes included to allow for parameters to be added as an OS develops and matures. If new parameters are added in an OS update, this doesn't alter the order of the structure from previous versions--new parameters will just occupy bytes that were previously unused.

I hope this helps!

This is not completely true... you never moved any entries in your SysEx structures around, that is correct, but on the P12 you changed the indexes of either the Mod Matrix sources or destinations (cannot remember which it was), so that the index numbers did not match previous versions of the SysEx (it happened as of SysEx version 3) ... I know this because I had to change my editor in SoundDiver to support this version (3)... that old program does not allow for support of versioning.

Does this mean, that REV2 SysEx version (the version byte in the structure) is now version 3?

Anyway... I'm really happy to see that you now officially say that you DO hand out the structure lists, if anyone needs them... I'll be sure to contact you once I get a Prophet X ;)
Title: Re: Analyzing the REV2 SysEx structure
Post by: cbx on September 22, 2018, 03:45:20 AM
I am really sorry brought this thread back to life, but I am glad we can get the information. It can not be any worse than a Yamaha TX81Z manual really.
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on September 22, 2018, 03:58:51 AM
About my question on SysEx revision above... no need to answer that, I just checked, and the REV2 is now at SysEx version 3... I'm not sure if there are any real change to the indexes, or if it's only to see the difference of what OS was used to create a program that this new version is used for... anyway I'm happy that this LFO bug has finaly been fixed... THANK YOU SCI!
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on September 22, 2018, 04:03:08 AM
With the exception of the Evolver series, which used sysex for real-time parameter changes rather than NRPNs, all of our instruments include the same basic information in the MIDI implementation section of the manuals. We can provide a list of the packed parameter offsets for program dumps to those who write support and ask. This information is not listed in our manuals because it takes up a lot of space and it's relevant only to a tiny portion of users. I understand that it's extremely relevant to this portion of users, but the fact remains that the vast majority never do more with sysex than perhaps backup their programs. I don't mean to be pedantic, but if you totaled the number of people who expressed interest in detailed sysex information for our instruments in forum posts across the web, it would equal a tiny portion of the instruments in the field. In any case, write support to ask and ye shall receive :)

Regarding changes to the parameter data included in a program dump, when we develop a new instrument, we will do some rough calculations as to how large a program needs to be to hold all of the parameters we anticipate it containing. There are always empty bytes included to allow for parameters to be added as an OS develops and matures. If new parameters are added in an OS update, this doesn't alter the order of the structure from previous versions--new parameters will just occupy bytes that were previously unused.

I hope this helps!

I have a simple question that I hope you will answer:

Will you ever consider fixing the problem with the REV2 SysEx not carrying the last four bytes of the poly sequencer? ... or will the SysEx always be crippled with this problem?
Title: Re: Analyzing the REV2 SysEx structure
Post by: cbx on September 22, 2018, 06:06:49 AM
What I like to do is lay down tracks on my midi looper with my mopho and boogy to them. The midi looper is the best thing ever made. You can make your own the MidiSizer MidiREX. The coolest thing ever
Title: Re: Analyzing the REV2 SysEx structure
Post by: Gomjab on September 22, 2018, 09:02:07 AM
About my question on SysEx revision above... no need to answer that, I just checked, and the REV2 is now at SysEx version 3... I'm not sure if there are any real change to the indexes, or if it's only to see the difference of what OS was used to create a program that this new version is used for... anyway I'm happy that this LFO bug has finaly been fixed... THANK YOU SCI!
I sent you a PM regarding the LFO bug fix but I don’t see it in the sent folder so I must have screwed up.

I was wondering what impact the bug fix had on your legacy patches.  I know you make extreme use of modulations and wondered if it broke a bunch.

Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on September 22, 2018, 12:03:44 PM
About my question on SysEx revision above... no need to answer that, I just checked, and the REV2 is now at SysEx version 3... I'm not sure if there are any real change to the indexes, or if it's only to see the difference of what OS was used to create a program that this new version is used for... anyway I'm happy that this LFO bug has finaly been fixed... THANK YOU SCI!
I sent you a PM regarding the LFO bug fix but I don’t see it in the sent folder so I must have screwed up.

I was wondering what impact the bug fix had on your legacy patches.  I know you make extreme use of modulations and wondered if it broke a bunch.

It came just fine... i just forgot to reply back to you (sorry)... according to what the change-list say, some things may not sound right, while other will (regarding older programs)... I had especially three programs that was screwed up by OS .18... one of them sounded like it should again, but the two others sound somewhat different for some reason, and i cannot make them sound like they used to anymore, so I simply just changed those two a bit until I found something that sounded alright... I agree with SCI in this matter; it should rather be fixed, than kept like an old P08 bug :)
Title: Re: Analyzing the REV2 SysEx structure
Post by: Gomjab on September 22, 2018, 03:36:53 PM
Great to hear.  I wasn’t questioning fix, just hoping it wasn’t too painful for you. 
Title: Re: Analyzing the REV2 SysEx structure
Post by: malkav30 on September 26, 2018, 08:49:55 AM
I just stumbled upon this thread, and HAD to say a big thank you Razmo for this thread !
I see you use Soundiver, I use Genedit on Atari myself, and rely heavily on sysex structures being documented to fool around with my synths. I have been very disapointed by some manufacturers that just don't provide this info even when asking support, so I'm glad you did the analysis, and I'm very pleased to read that Sequential is still providing this kind of information for their synths!
Title: Re: Analyzing the REV2 SysEx structure
Post by: keston on March 04, 2020, 07:57:25 PM
This analysis of the REV2 SysEx structure is really helpful! However, does anyone know how values greater than 127 are handled in the SysEx program dump? For example when I observe the value of the Filter ENV Amount (0-254 or -127 to 127 in the interface) the sysex value is 1/2 the range of the NRPN (0-126 versus 0-254). Is it as simple as multiplying by 2 to get an approximation (+/- 1) of the actual value?
Title: Re: Analyzing the REV2 SysEx structure
Post by: shiihs on March 05, 2020, 03:25:17 PM
This analysis of the REV2 SysEx structure is really helpful! However, does anyone know how values greater than 127 are handled in the SysEx program dump? For example when I observe the value of the Filter ENV Amount (0-254 or -127 to 127 in the interface) the sysex value is 1/2 the range of the NRPN (0-126 versus 0-254). Is it as simple as multiplying by 2 to get an approximation (+/- 1) of the actual value?

No. Actually it's a bit trickier than that. In the case of the filter env amount (which is encoded in the 32nd byte of the sysex stream for layer A) e.g. there's an extra bit which is the high bit of the 30th byte in the sysex stream (which holds the env 3 destination).

All parameters with either signed or >127 values use this technique of hiding an extra bit in another field in the sysex stream. I quite exhaustively reverse engineered all of them to create my book disassembling presets in the Rev2 (pinned post on this forum). All my code is available on github https://github.com/shimpe/sc-prophet-rev2 which - if you manage to read some supercollider code - should give you more than a head start on exploring the depths of the rev2. I've already used it for all kinds of exotic experiments like automorphing patches, creating a polyphonic gated sequencer allowing to automate all parameters, live bidirectional editor for patches etc

Title: Re: Analyzing the REV2 SysEx structure
Post by: keston on March 06, 2020, 11:44:20 AM
This analysis of the REV2 SysEx structure is really helpful! However, does anyone know how values greater than 127 are handled in the SysEx program dump? For example when I observe the value of the Filter ENV Amount (0-254 or -127 to 127 in the interface) the sysex value is 1/2 the range of the NRPN (0-126 versus 0-254). Is it as simple as multiplying by 2 to get an approximation (+/- 1) of the actual value?

No. Actually it's a bit trickier than that. In the case of the filter env amount (which is encoded in the 32nd byte of the sysex stream for layer A) e.g. there's an extra bit which is the high bit of the 30th byte in the sysex stream (which holds the env 3 destination).

All parameters with either signed or >127 values use this technique of hiding an extra bit in another field in the sysex stream. I quite exhaustively reverse engineered all of them to create my book disassembling presets in the Rev2 (pinned post on this forum). All my code is available on github https://github.com/shimpe/sc-prophet-rev2 which - if you manage to read some supercollider code - should give you more than a head start on exploring the depths of the rev2. I've already used it for all kinds of exotic experiments like automorphing patches, creating a polyphonic gated sequencer allowing to automate all parameters, live bidirectional editor for patches etc

Amazing project, Stefaan! So are the property values under PatchDumper.sc in the patch_explainer object the sequence (minus the offset) that the parameters are sent from the REV2 program dump? I've been discovering a similar but significantly different sequence in OS 1.1.5.9. Hopefully it's consistent per OS and patch.
Title: Re: Analyzing the REV2 SysEx structure
Post by: Razmo on March 07, 2020, 03:18:56 AM
Please note, the rev2 has a sysex bug that is unfixable... It actually misses a few bytes at the end of the message that happen to be a few sequencer parameters in the end of track 6... There is no way to fix this, and any sequence recorded on the rev2 alone will have these few values truncated as soon as you send them via SysEx... Rev2 will perfectly recieve the SysEx, but those last few sequencer parameters are just not there anymore.

It is not a big problem since using the last step and the few last step parameters there is highly unlikely to happen... But it did confuse me in the beginning.
Title: Re: Analyzing the REV2 SysEx structure
Post by: shiihs on March 08, 2020, 02:39:35 PM
So are the property values under PatchDumper.sc in the patch_explainer object the sequence (minus the offset) that the parameters are sent from the REV2 program dump? I've been discovering a similar but significantly different sequence in OS 1.1.5.9. Hopefully it's consistent per OS and patch.

PatchDumper was one of my very first experiments. It's still there but it's not used a lot anymore. You'd better watch elsewhere:

In the file ScProphetRev2.sc, there's a variable assignment "this.rev2_nrpn=...." which lists for every nrpn number in the rev2 what is the sysex position, the min/max values, the name, the unit (if applicable), the sysex position for an extra sign or range extension bit (if applicable), a lookup table to translate values in the sysex stream to symbolic values (if applicable).

There's also a variable sysex_bytepos, which does the opposite: it maps from sysex byte pos to nrpn number and name

There's a variable rev2_nrpn_globals, which lists the locations of all global parameters in its sysex stream, and the reverse mapping from sysex position to global parameter (and nrpn) in global_sysex_bytepos.

This hasn't changed between OS versions (the globals were extended once or twice, and the current information is valid for the latest available beta version of the OS)
Title: Re: Analyzing the REV2 SysEx structure
Post by: keston on March 09, 2020, 06:05:04 PM
It took some doing but I managed to decipher what I needed out of the REV2 sysex program dump. The main difficulty was figuring out how values > 127 are encoded in the MOD slots. For example, it turns out that the additional data for MOD amounts 1 through 6 is stored in a single byte representing 64 possible combinations of values higher than 127 or lower than 128.

I ended up creating the attached tables to document exactly what values in which bytes represent a parameter value greater than 127. In the table the first (numbered) column is the position in the sysex stream of the additional byte. Then 1 means > 127 and 0 means < 128. If you're coding something that needs to manage this simply add 128 if the additional byte's value matches a 1 in the column.
Title: Re: Analyzing the REV2 SysEx structure
Post by: shiihs on March 10, 2020, 09:25:46 AM
The main difficulty was figuring out how values > 127 are encoded in the MOD slots. For example, it turns out that the additional data for MOD amounts 1 through 6 is stored in a single byte representing 64 possible combinations of values higher than 127 or lower than 128.

Hmmm... Your approach could be more difficult than strictly needed because (I'm making wild assumptions here) perhaps you didn't unpack the sysex data as (very briefly) described in the manual (and done in the supercollider program). In my findings (tested quite extensively) there's never more than one sign bit per byte in the unpacked sysex midi data.


Title: Re: Analyzing the REV2 SysEx structure
Post by: keston on March 11, 2020, 08:35:21 PM
Hmmm... Your approach could be more difficult than strictly needed because (I'm making wild assumptions here) perhaps you didn't unpack the sysex data as (very briefly) described in the manual (and done in the supercollider program). In my findings (tested quite extensively) there's never more than one sign bit per byte in the unpacked sysex midi data.

I am unpacking the sysex in Max. I think my description was confusing. It's not technically parameter data that's encoded in the additional byte, but a group of bits that indicate if the parameter value needs to be increased by 128 or not. In the tables I shared I decoded the additional byte values into 0s or 1s that signal if 128 needs to be added to the value.

For example if the 102nd byte in the sysex dump has a value of 78 then the amount values for MOD1, MOD2, MOD3, and MOD6 all need to be increased by 128. I've tested this approach in Max and it is working great in OS 1.1.5.9, and now it's rolled into my REV2 Degrader for retrieving and degrading Layer A.