Analyzing the REV2 SysEx structure

dslsynth

  • ***
  • 1040
Re: Analyzing the REV2 SysEx structure
« Reply #60 on: September 21, 2018, 03:54:06 PM »
There are always empty bytes included to allow for parameters to be added as an OS develops and matures. If new parameters are added in an OS update, this doesn't alter the order of the structure from previous versions--new parameters will just occupy bytes that were previously unused.

Thanks for the answer! :-)

To be ultra pedantic there two types of changes to the program vector: parameter additions and parameter value changes. Value changes such as adding new sources or destinations will not change the program vector itself but will require a version number in the sysex format to tell older program versions from newer program versions. Similar actions may be needed for new parameters if zero is not the neutral state of a parameter.

The advantage of versioned programs is that it allows for better ordering of sources/destinations in the user interface. The disadvantage of versioned programs is that not all generic editors support versioned programs and it take extra effort to document it precisely.
#!/bin/sh
cp -f $0 $HOME/.signature

cbx

  • **
  • 145
Re: Analyzing the REV2 SysEx structure
« Reply #61 on: September 21, 2018, 05:05:14 PM »
The Prophet Rev 2 looks really cool I've been reading the manual. I have two Tetras I use in various ways, and I think the Rev2 can do almost everything that they can do. I use the tetras in different ways, sometimes with my Mopho keyboards to extend the voices to 5. But, you can also layer programs on the tetra, so I will use it with (the mopho) to stack patches. Lately I have had it set up in series in multimode giving me 8 sequencers and a midi looper. I do all percussion and rhthym on one and the other is for melody and harmonies. So, I have been very happy with my setup. But, the Rev 2 can do some thing interesting things that this can not. Even if I don't have sysex I can always start with a Prophet 08/mopho patch and fine tune the NRPN, so it wasn't a huge problem but still, I do use the sysex data when designing sounds. On my moog little phatty, allI can do is randomize the CC controllers as a starting point. I can do that with Mopho/Prophet 08 too, but it's easier if I can just load sysex and then apply meta patch as starting point.

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #62 on: September 22, 2018, 02:37:03 AM »
With the exception of the Evolver series, which used sysex for real-time parameter changes rather than NRPNs, all of our instruments include the same basic information in the MIDI implementation section of the manuals. We can provide a list of the packed parameter offsets for program dumps to those who write support and ask. This information is not listed in our manuals because it takes up a lot of space and it's relevant only to a tiny portion of users. I understand that it's extremely relevant to this portion of users, but the fact remains that the vast majority never do more with sysex than perhaps backup their programs. I don't mean to be pedantic, but if you totaled the number of people who expressed interest in detailed sysex information for our instruments in forum posts across the web, it would equal a tiny portion of the instruments in the field. In any case, write support to ask and ye shall receive :)

Regarding changes to the parameter data included in a program dump, when we develop a new instrument, we will do some rough calculations as to how large a program needs to be to hold all of the parameters we anticipate it containing. There are always empty bytes included to allow for parameters to be added as an OS develops and matures. If new parameters are added in an OS update, this doesn't alter the order of the structure from previous versions--new parameters will just occupy bytes that were previously unused.

I hope this helps!

This is not completely true... you never moved any entries in your SysEx structures around, that is correct, but on the P12 you changed the indexes of either the Mod Matrix sources or destinations (cannot remember which it was), so that the index numbers did not match previous versions of the SysEx (it happened as of SysEx version 3) ... I know this because I had to change my editor in SoundDiver to support this version (3)... that old program does not allow for support of versioning.

Does this mean, that REV2 SysEx version (the version byte in the structure) is now version 3?

Anyway... I'm really happy to see that you now officially say that you DO hand out the structure lists, if anyone needs them... I'll be sure to contact you once I get a Prophet X ;)
« Last Edit: September 22, 2018, 03:04:54 AM by Razmo »
If you need me, follow the shadows...

cbx

  • **
  • 145
Re: Analyzing the REV2 SysEx structure
« Reply #63 on: September 22, 2018, 03:45:20 AM »
I am really sorry brought this thread back to life, but I am glad we can get the information. It can not be any worse than a Yamaha TX81Z manual really.

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #64 on: September 22, 2018, 03:58:51 AM »
About my question on SysEx revision above... no need to answer that, I just checked, and the REV2 is now at SysEx version 3... I'm not sure if there are any real change to the indexes, or if it's only to see the difference of what OS was used to create a program that this new version is used for... anyway I'm happy that this LFO bug has finaly been fixed... THANK YOU SCI!
If you need me, follow the shadows...

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #65 on: September 22, 2018, 04:03:08 AM »
With the exception of the Evolver series, which used sysex for real-time parameter changes rather than NRPNs, all of our instruments include the same basic information in the MIDI implementation section of the manuals. We can provide a list of the packed parameter offsets for program dumps to those who write support and ask. This information is not listed in our manuals because it takes up a lot of space and it's relevant only to a tiny portion of users. I understand that it's extremely relevant to this portion of users, but the fact remains that the vast majority never do more with sysex than perhaps backup their programs. I don't mean to be pedantic, but if you totaled the number of people who expressed interest in detailed sysex information for our instruments in forum posts across the web, it would equal a tiny portion of the instruments in the field. In any case, write support to ask and ye shall receive :)

Regarding changes to the parameter data included in a program dump, when we develop a new instrument, we will do some rough calculations as to how large a program needs to be to hold all of the parameters we anticipate it containing. There are always empty bytes included to allow for parameters to be added as an OS develops and matures. If new parameters are added in an OS update, this doesn't alter the order of the structure from previous versions--new parameters will just occupy bytes that were previously unused.

I hope this helps!

I have a simple question that I hope you will answer:

Will you ever consider fixing the problem with the REV2 SysEx not carrying the last four bytes of the poly sequencer? ... or will the SysEx always be crippled with this problem?
If you need me, follow the shadows...

cbx

  • **
  • 145
Re: Analyzing the REV2 SysEx structure
« Reply #66 on: September 22, 2018, 06:06:49 AM »
What I like to do is lay down tracks on my midi looper with my mopho and boogy to them. The midi looper is the best thing ever made. You can make your own the MidiSizer MidiREX. The coolest thing ever

Gomjab

  • **
  • 110
Re: Analyzing the REV2 SysEx structure
« Reply #67 on: September 22, 2018, 09:02:07 AM »
About my question on SysEx revision above... no need to answer that, I just checked, and the REV2 is now at SysEx version 3... I'm not sure if there are any real change to the indexes, or if it's only to see the difference of what OS was used to create a program that this new version is used for... anyway I'm happy that this LFO bug has finaly been fixed... THANK YOU SCI!
I sent you a PM regarding the LFO bug fix but I don’t see it in the sent folder so I must have screwed up.

I was wondering what impact the bug fix had on your legacy patches.  I know you make extreme use of modulations and wondered if it broke a bunch.

« Last Edit: September 22, 2018, 10:20:25 AM by Gomjab »

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #68 on: September 22, 2018, 12:03:44 PM »
About my question on SysEx revision above... no need to answer that, I just checked, and the REV2 is now at SysEx version 3... I'm not sure if there are any real change to the indexes, or if it's only to see the difference of what OS was used to create a program that this new version is used for... anyway I'm happy that this LFO bug has finaly been fixed... THANK YOU SCI!
I sent you a PM regarding the LFO bug fix but I don’t see it in the sent folder so I must have screwed up.

I was wondering what impact the bug fix had on your legacy patches.  I know you make extreme use of modulations and wondered if it broke a bunch.

It came just fine... i just forgot to reply back to you (sorry)... according to what the change-list say, some things may not sound right, while other will (regarding older programs)... I had especially three programs that was screwed up by OS .18... one of them sounded like it should again, but the two others sound somewhat different for some reason, and i cannot make them sound like they used to anymore, so I simply just changed those two a bit until I found something that sounded alright... I agree with SCI in this matter; it should rather be fixed, than kept like an old P08 bug :)
If you need me, follow the shadows...

Gomjab

  • **
  • 110
Re: Analyzing the REV2 SysEx structure
« Reply #69 on: September 22, 2018, 03:36:53 PM »
Great to hear.  I wasn’t questioning fix, just hoping it wasn’t too painful for you. 

Re: Analyzing the REV2 SysEx structure
« Reply #70 on: September 26, 2018, 08:49:55 AM »
I just stumbled upon this thread, and HAD to say a big thank you Razmo for this thread !
I see you use Soundiver, I use Genedit on Atari myself, and rely heavily on sysex structures being documented to fool around with my synths. I have been very disapointed by some manufacturers that just don't provide this info even when asking support, so I'm glad you did the analysis, and I'm very pleased to read that Sequential is still providing this kind of information for their synths!
Prophet REV2-8, PRO3, OB 6 Keyboard, lots of old stuff...

Re: Analyzing the REV2 SysEx structure
« Reply #71 on: March 04, 2020, 07:57:25 PM »
This analysis of the REV2 SysEx structure is really helpful! However, does anyone know how values greater than 127 are handled in the SysEx program dump? For example when I observe the value of the Filter ENV Amount (0-254 or -127 to 127 in the interface) the sysex value is 1/2 the range of the NRPN (0-126 versus 0-254). Is it as simple as multiplying by 2 to get an approximation (+/- 1) of the actual value?

shiihs

  • **
  • 103
  • phasing in and out of reality
Re: Analyzing the REV2 SysEx structure
« Reply #72 on: March 05, 2020, 03:25:17 PM »
This analysis of the REV2 SysEx structure is really helpful! However, does anyone know how values greater than 127 are handled in the SysEx program dump? For example when I observe the value of the Filter ENV Amount (0-254 or -127 to 127 in the interface) the sysex value is 1/2 the range of the NRPN (0-126 versus 0-254). Is it as simple as multiplying by 2 to get an approximation (+/- 1) of the actual value?

No. Actually it's a bit trickier than that. In the case of the filter env amount (which is encoded in the 32nd byte of the sysex stream for layer A) e.g. there's an extra bit which is the high bit of the 30th byte in the sysex stream (which holds the env 3 destination).

All parameters with either signed or >127 values use this technique of hiding an extra bit in another field in the sysex stream. I quite exhaustively reverse engineered all of them to create my book disassembling presets in the Rev2 (pinned post on this forum). All my code is available on github https://github.com/shimpe/sc-prophet-rev2 which - if you manage to read some supercollider code - should give you more than a head start on exploring the depths of the rev2. I've already used it for all kinds of exotic experiments like automorphing patches, creating a polyphonic gated sequencer allowing to automate all parameters, live bidirectional editor for patches etc

--
gear: prophet rev2 16 voice, kawai NV10, casio wk-7600, Roland Integra-7, supercollider, ardour

links:

https://www.youtube.com/stefaanhimpe
https://soundcloud.com/stefaanhimpe
https://technogems.blogspot.com
https://a-touch-of-music.blogspot.com/

Re: Analyzing the REV2 SysEx structure
« Reply #73 on: March 06, 2020, 11:44:20 AM »
This analysis of the REV2 SysEx structure is really helpful! However, does anyone know how values greater than 127 are handled in the SysEx program dump? For example when I observe the value of the Filter ENV Amount (0-254 or -127 to 127 in the interface) the sysex value is 1/2 the range of the NRPN (0-126 versus 0-254). Is it as simple as multiplying by 2 to get an approximation (+/- 1) of the actual value?

No. Actually it's a bit trickier than that. In the case of the filter env amount (which is encoded in the 32nd byte of the sysex stream for layer A) e.g. there's an extra bit which is the high bit of the 30th byte in the sysex stream (which holds the env 3 destination).

All parameters with either signed or >127 values use this technique of hiding an extra bit in another field in the sysex stream. I quite exhaustively reverse engineered all of them to create my book disassembling presets in the Rev2 (pinned post on this forum). All my code is available on github https://github.com/shimpe/sc-prophet-rev2 which - if you manage to read some supercollider code - should give you more than a head start on exploring the depths of the rev2. I've already used it for all kinds of exotic experiments like automorphing patches, creating a polyphonic gated sequencer allowing to automate all parameters, live bidirectional editor for patches etc

Amazing project, Stefaan! So are the property values under PatchDumper.sc in the patch_explainer object the sequence (minus the offset) that the parameters are sent from the REV2 program dump? I've been discovering a similar but significantly different sequence in OS 1.1.5.9. Hopefully it's consistent per OS and patch.

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #74 on: March 07, 2020, 03:18:56 AM »
Please note, the rev2 has a sysex bug that is unfixable... It actually misses a few bytes at the end of the message that happen to be a few sequencer parameters in the end of track 6... There is no way to fix this, and any sequence recorded on the rev2 alone will have these few values truncated as soon as you send them via SysEx... Rev2 will perfectly recieve the SysEx, but those last few sequencer parameters are just not there anymore.

It is not a big problem since using the last step and the few last step parameters there is highly unlikely to happen... But it did confuse me in the beginning.
If you need me, follow the shadows...

shiihs

  • **
  • 103
  • phasing in and out of reality
Re: Analyzing the REV2 SysEx structure
« Reply #75 on: March 08, 2020, 02:39:35 PM »
So are the property values under PatchDumper.sc in the patch_explainer object the sequence (minus the offset) that the parameters are sent from the REV2 program dump? I've been discovering a similar but significantly different sequence in OS 1.1.5.9. Hopefully it's consistent per OS and patch.

PatchDumper was one of my very first experiments. It's still there but it's not used a lot anymore. You'd better watch elsewhere:

In the file ScProphetRev2.sc, there's a variable assignment "this.rev2_nrpn=...." which lists for every nrpn number in the rev2 what is the sysex position, the min/max values, the name, the unit (if applicable), the sysex position for an extra sign or range extension bit (if applicable), a lookup table to translate values in the sysex stream to symbolic values (if applicable).

There's also a variable sysex_bytepos, which does the opposite: it maps from sysex byte pos to nrpn number and name

There's a variable rev2_nrpn_globals, which lists the locations of all global parameters in its sysex stream, and the reverse mapping from sysex position to global parameter (and nrpn) in global_sysex_bytepos.

This hasn't changed between OS versions (the globals were extended once or twice, and the current information is valid for the latest available beta version of the OS)
« Last Edit: March 08, 2020, 02:41:34 PM by shiihs »
--
gear: prophet rev2 16 voice, kawai NV10, casio wk-7600, Roland Integra-7, supercollider, ardour

links:

https://www.youtube.com/stefaanhimpe
https://soundcloud.com/stefaanhimpe
https://technogems.blogspot.com
https://a-touch-of-music.blogspot.com/

Re: Analyzing the REV2 SysEx structure
« Reply #76 on: March 09, 2020, 06:05:04 PM »
It took some doing but I managed to decipher what I needed out of the REV2 sysex program dump. The main difficulty was figuring out how values > 127 are encoded in the MOD slots. For example, it turns out that the additional data for MOD amounts 1 through 6 is stored in a single byte representing 64 possible combinations of values higher than 127 or lower than 128.

I ended up creating the attached tables to document exactly what values in which bytes represent a parameter value greater than 127. In the table the first (numbered) column is the position in the sysex stream of the additional byte. Then 1 means > 127 and 0 means < 128. If you're coding something that needs to manage this simply add 128 if the additional byte's value matches a 1 in the column.

shiihs

  • **
  • 103
  • phasing in and out of reality
Re: Analyzing the REV2 SysEx structure
« Reply #77 on: March 10, 2020, 09:25:46 AM »
The main difficulty was figuring out how values > 127 are encoded in the MOD slots. For example, it turns out that the additional data for MOD amounts 1 through 6 is stored in a single byte representing 64 possible combinations of values higher than 127 or lower than 128.

Hmmm... Your approach could be more difficult than strictly needed because (I'm making wild assumptions here) perhaps you didn't unpack the sysex data as (very briefly) described in the manual (and done in the supercollider program). In my findings (tested quite extensively) there's never more than one sign bit per byte in the unpacked sysex midi data.


--
gear: prophet rev2 16 voice, kawai NV10, casio wk-7600, Roland Integra-7, supercollider, ardour

links:

https://www.youtube.com/stefaanhimpe
https://soundcloud.com/stefaanhimpe
https://technogems.blogspot.com
https://a-touch-of-music.blogspot.com/

Re: Analyzing the REV2 SysEx structure
« Reply #78 on: March 11, 2020, 08:35:21 PM »
Hmmm... Your approach could be more difficult than strictly needed because (I'm making wild assumptions here) perhaps you didn't unpack the sysex data as (very briefly) described in the manual (and done in the supercollider program). In my findings (tested quite extensively) there's never more than one sign bit per byte in the unpacked sysex midi data.

I am unpacking the sysex in Max. I think my description was confusing. It's not technically parameter data that's encoded in the additional byte, but a group of bits that indicate if the parameter value needs to be increased by 128 or not. In the tables I shared I decoded the additional byte values into 0s or 1s that signal if 128 needs to be added to the value.

For example if the 102nd byte in the sysex dump has a value of 78 then the amount values for MOD1, MOD2, MOD3, and MOD6 all need to be increased by 128. I've tested this approach in Max and it is working great in OS 1.1.5.9, and now it's rolled into my REV2 Degrader for retrieving and degrading Layer A.