Analyzing the REV2 SysEx structure

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #40 on: July 26, 2018, 02:32:58 PM »
True... all single parameter changes must be done via NRPNs... even the global ones, plus all sequencer data... the advantage to the SysEx is that it allow you to control the program memories which is nice for creating librarians and managing your programs (and also overwrite the factory programs... as this is only doable via SysEx dumps)..
Yeah, but my point is just to be able to use any sound in the synth and use M4L to send a pre-made sequence to the sequencers.

Then you'll only need NRPN functionality... there are a few parameters in the program structure that you can only get at via SysEx dumps though, but they are not really needed for normal use... one I recall is an "Editor Number"... a stored value that is used to signal what editor was used to create a program, and then a "Version Number" that designates what version the dump structure is... Prophet 12 for example has more than one revision of the dump... but this is of course irrelevant for your use :)
If you need me, follow the shadows...

shiihs

  • **
  • 103
  • phasing in and out of reality
Re: Analyzing the REV2 SysEx structure
« Reply #41 on: July 26, 2018, 02:44:25 PM »
Yeah, but my point is just to be able to use any sound in the synth and use M4L to send a pre-made sequence to the sequencers.

I implemented that (and a whole lot more) in supercollider (available on github).

Sending an NRPN is not so difficult, basically you have to send four midi control change messages:

on CC 99, send the most significant 7 bits of the NRPN number (NRPN >> 7)
on CC 98, send the least significant 7 bits of the NRPN number (NRPN & 127)
on CC 6, send the most significant 7 bits of the desired value (value >> 7)
on CC 38, send the least significant 7 bits of the desired value (value & 127)

and you have to make sure that your synth is set up to receive NRPN parameters (it's in the global options somewhere).
--
gear: prophet rev2 16 voice, kawai NV10, casio wk-7600, Roland Integra-7, supercollider, ardour

links:

https://www.youtube.com/stefaanhimpe
https://soundcloud.com/stefaanhimpe
https://technogems.blogspot.com
https://a-touch-of-music.blogspot.com/

Re: Analyzing the REV2 SysEx structure
« Reply #42 on: July 27, 2018, 03:56:00 AM »
Many thanks to you. I've got it working and it's promising !

cbx

  • **
  • 145
Re: Analyzing the REV2 SysEx structure
« Reply #43 on: September 19, 2018, 11:07:10 PM »
Did he ever figure it out? All the manufacturers no longer provide this information for new products. My guess is they expect MIDI to finally go away. But, Korg and Dave Smith always kept to the tradition so I was surprised they stopped. Considering Dave Smith invented MIDI.

If only you could read NRPN controller values from the machine, there would be little need for SYSEX anyways.

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #44 on: September 20, 2018, 02:38:55 AM »
SysEx still would be the only way to do memory management, even if NRPN's could be read out... SysEx has no limits in size either... With NRPN you have a maximum number of 16384 which will be way too little if you were to address a lot of programs this way. You would have to set the current memory program slot by defining an NRPN as a program pointer in that case... It is doable, but highly controversial i would say.
« Last Edit: September 20, 2018, 02:41:41 AM by Razmo »
If you need me, follow the shadows...

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #45 on: September 20, 2018, 08:16:06 AM »
Did he ever figure it out? All the manufacturers no longer provide this information for new products. My guess is they expect MIDI to finally go away. But, Korg and Dave Smith always kept to the tradition so I was surprised they stopped. Considering Dave Smith invented MIDI.

If only you could read NRPN controller values from the machine, there would be little need for SYSEX anyways.

The change with not listing the SysEx structure started with the Prophet 12 as far as I'm aware, and my bet on why they have started dropping this info is because since the Prophet 12, they began doing "versioning" of the SysEx structure if they wanted to change something... The Prophet 12 is at version 3 of it's SysEx, and when that happened, my own editor stopped working correctly with some mod matrix sources/destination... that was because they put in more, and also sorted them, meaning that a source that had a specific index before, now suddenly had a new index... they did not simply "add to the indexes"...

This means that if the SysEx of a device is getting a new version, any information already out would be wrong... they would have to give out a new document with each new version of the SysEx... even the SysEx length could be changed this way because the synth is internally accepting older versions of the SysEx dump... the dump has a dedicated "version byte" in it, so the OS can always see what version the program was made in, and adapt to it, without the user even noticing it.

This versioning though gives editors a hard time following up if they have to support all the versions...

Rev2 is actually a "version 2" ... the version byte in the SysEx structure says "2"... I suspect that version 1 is in fact simply the old Prophet 8 format because the REV2 can load the older Prophet 08 programs directly...
If you need me, follow the shadows...

cbx

  • **
  • 145
Re: Analyzing the REV2 SysEx structure
« Reply #46 on: September 20, 2018, 01:38:59 PM »
SysEx still would be the only way to do memory management, even if NRPN's could be read out... SysEx has no limits in size either... With NRPN you have a maximum number of 16384 which will be way too little if you were to address a lot of programs this way. You would have to set the current memory program slot by defining an NRPN as a program pointer in that case... It is doable, but highly controversial i would say.

True. I need the sysex for sound design algorithms. I don't think I'll get any new syntheziers anyways, but if I can't program the Rev 2 at all algorithmically I just will pass on it.

cbx

  • **
  • 145
Re: Analyzing the REV2 SysEx structure
« Reply #47 on: September 20, 2018, 03:47:54 PM »
Well it looks cool I hope you get it working. The more I think about it, the less I really need the sysex. I would just be starting with a patch from the prophet 08 and then transfer them to the Rev 2 using NRPN. While it would be useful to have the information for the sysex I could still program the machine with just NRPN controls and a patch from another machine as a start.

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #48 on: September 20, 2018, 04:22:54 PM »
Well it looks cool I hope you get it working. The more I think about it, the less I really need the sysex. I would just be starting with a patch from the prophet 08 and then transfer them to the Rev 2 using NRPN. While it would be useful to have the information for the sysex I could still program the machine with just NRPN controls and a patch from another machine as a start.

I analyzed the SysEx structure... you can find it earlier in this thread :)
If you need me, follow the shadows...

cbx

  • **
  • 145
Re: Analyzing the REV2 SysEx structure
« Reply #49 on: September 20, 2018, 07:32:30 PM »
Well it looks cool I hope you get it working. The more I think about it, the less I really need the sysex. I would just be starting with a patch from the prophet 08 and then transfer them to the Rev 2 using NRPN. While it would be useful to have the information for the sysex I could still program the machine with just NRPN controls and a patch from another machine as a start.

I analyzed the SysEx structure... you can find it earlier in this thread :)

Ok, sorry for bumping, I was unsure if you had finished it, because it said you would update when you built your editor.

Does it use the same algorithm the Prophet 08 used for packing the high bits?

Thanks for doing this, I don't know if I will get one as I am very happy with my Mopho-10.

cbx

  • **
  • 145
Re: Analyzing the REV2 SysEx structure
« Reply #50 on: September 21, 2018, 12:23:24 AM »
I think they don't publish them anymore because most people are wireless now, and maybe studios complained about this. But I'd think with bluetooth it should not be possible to hack someone's keyboard when they are on stage. Otherwise, I can't think of any darn good reason why they no longer give you this information. It's like buying a car and keeping how to change a mystery. Kind of annoying me.

cbx

  • **
  • 145
Re: Analyzing the REV2 SysEx structure
« Reply #51 on: September 21, 2018, 12:33:23 AM »
Anyhow, sorry for so many posts but, it is all manufacturers that do this now. DSI was the last to stop publishing them I think. Korg always made everything available, which is why I liked them. Now, nothing. So, it is the entire industry, not just Sequential.

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #52 on: September 21, 2018, 01:02:02 AM »
Well it looks cool I hope you get it working. The more I think about it, the less I really need the sysex. I would just be starting with a patch from the prophet 08 and then transfer them to the Rev 2 using NRPN. While it would be useful to have the information for the sysex I could still program the machine with just NRPN controls and a patch from another machine as a start.

I analyzed the SysEx structure... you can find it earlier in this thread :)

Ok, sorry for bumping, I was unsure if you had finished it, because it said you would update when you built your editor.

Does it use the same algorithm the Prophet 08 used for packing the high bits?

Thanks for doing this, I don't know if I will get one as I am very happy with my Mopho-10.

Yes, DSI has used the same packing scheme since the Evolver, so it's exactly the same... the list should be ok, but please notice that if DSI suddenly change the structure it would not be correct anymore... I had a private message with support regarding the current bug with the LFO time divisions (in sync mode), and I was told that they have this on the bug lidt, and will do some "versioning" to fix it with the next OS beta... I'm not sure if they mean a version 3 of the REV2 SysEx, but if it is, my list on page 2 of this thread will probably not match anymore... only time will tell...
If you need me, follow the shadows...

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #53 on: September 21, 2018, 01:06:38 AM »
Anyhow, sorry for so many posts but, it is all manufacturers that do this now. DSI was the last to stop publishing them I think. Korg always made everything available, which is why I liked them. Now, nothing. So, it is the entire industry, not just Sequential.

As long as they publish the SysEx dump and request messages, I can live with it because it's not that hard to analyze what bytes is what... but it is definitely tiresome to do... some companies have them but won't give them unless you ask for them... I've had a table from MOOG every time I asked for several of their products, and Waldorf usually end up with a document at some point, though they're rather slow at it... Korg (as you say) always has the info, so does Roland and Yamaha... still... it seems to be the smaller niche companies that for some reason think that this info is not important anymore... and yes, it's irritating to say the least, but it's in vain to try and ask for them at DSI... they have not even explained why they do not create these lists anymore... just plain silence...
If you need me, follow the shadows...

Re: Analyzing the REV2 SysEx structure
« Reply #54 on: September 21, 2018, 09:38:34 AM »
We absolutely provide parameter offset lists to customers who ask. Combined with the program dump sysex structure outlined in the MIDI implementation section of our manuals, this fills in the blanks for anyone looking to build an editor, or something of the sort. Interested parties are few and far between, but we do provide the info. Razmo, I believe we provided you with precisely such a document for the Rev2 at one point, did we not?
SEQUENTIAL

Razmo

  • ***
  • 2168
  • I am shadow...
    • Kaleidoscopic Artworks
Re: Analyzing the REV2 SysEx structure
« Reply #55 on: September 21, 2018, 12:30:21 PM »
We absolutely provide parameter offset lists to customers who ask. Combined with the program dump sysex structure outlined in the MIDI implementation section of our manuals, this fills in the blanks for anyone looking to build an editor, or something of the sort. Interested parties are few and far between, but we do provide the info. Razmo, I believe we provided you with precisely such a document for the Rev2 at one point, did we not?

No you did not... if you did, there would have been no need to analyze everything mysself ;) ... in fact I did not ask for it either, as I decided to just do it myself instead... you did provide me with data for the Tempest though...

I can see now that my comment above could be read as if you at DSI do not give out the SysEx specs... that was not what i meant actually... I meant that it would be in vain trying to make you include this in the manuals again... many have shown an interest in this info over the last many years in different threads in here, but you at DSI never comment on it, so I assume you have no intention of supplying this info anymore (in the manuals)...

Would you care to tell us, why you have decided not to include this info in the manual, as it was in earlier manuals for your products?
« Last Edit: September 21, 2018, 12:34:08 PM by Razmo »
If you need me, follow the shadows...

dslsynth

  • ***
  • 1040
Re: Analyzing the REV2 SysEx structure
« Reply #56 on: September 21, 2018, 02:23:00 PM »
Most likely its just too much work to document manually. Especially as new versions of the sysex format shows up.

If the full sysex spec were readily available it would be used a lot more because people would just download it from the site and start working/playing with it. Much like what happened with the older and now discontinued DSI instruments. So in that way the currently observed demand at the support office may not reflect the actual interest in having this information available.
#!/bin/sh
cp -f $0 $HOME/.signature

cbx

  • **
  • 145
Re: Analyzing the REV2 SysEx structure
« Reply #57 on: September 21, 2018, 03:00:37 PM »
Most likely its just too much work to document manually. Especially as new versions of the sysex format shows up.

If the full sysex spec were readily available it would be used a lot more because people would just download it from the site and start working/playing with it. Much like what happened with the older and now discontinued DSI instruments. So in that way the currently observed demand at the support office may not reflect the actual interest in having this information available.

Beat's me, but they use to to do this with keyboards that had far more parameters than these ones. Korg and Roland workstations had massive sysex, all documented. Now nobody documents it. It makes me upset because you need that information for get the data for the patch, so you can use it for sound design. I was very disappointed Moog never documented their keyboards either, all you have is NRPN which isn't useful for sound design because you can't poll the keyboard for the control values.

Re: Analyzing the REV2 SysEx structure
« Reply #58 on: September 21, 2018, 03:11:50 PM »
With the exception of the Evolver series, which used sysex for real-time parameter changes rather than NRPNs, all of our instruments include the same basic information in the MIDI implementation section of the manuals. We can provide a list of the packed parameter offsets for program dumps to those who write support and ask. This information is not listed in our manuals because it takes up a lot of space and it's relevant only to a tiny portion of users. I understand that it's extremely relevant to this portion of users, but the fact remains that the vast majority never do more with sysex than perhaps backup their programs. I don't mean to be pedantic, but if you totaled the number of people who expressed interest in detailed sysex information for our instruments in forum posts across the web, it would equal a tiny portion of the instruments in the field. In any case, write support to ask and ye shall receive :)

Regarding changes to the parameter data included in a program dump, when we develop a new instrument, we will do some rough calculations as to how large a program needs to be to hold all of the parameters we anticipate it containing. There are always empty bytes included to allow for parameters to be added as an OS develops and matures. If new parameters are added in an OS update, this doesn't alter the order of the structure from previous versions--new parameters will just occupy bytes that were previously unused.

I hope this helps!
SEQUENTIAL

cbx

  • **
  • 145
Re: Analyzing the REV2 SysEx structure
« Reply #59 on: September 21, 2018, 03:36:29 PM »
With the exception of the Evolver series, which used sysex for real-time parameter changes rather than NRPNs, all of our instruments include the same basic information in the MIDI implementation section of the manuals. We can provide a list of the packed parameter offsets for program dumps to those who write support and ask. This information is not listed in our manuals because it takes up a lot of space and it's relevant only to a tiny portion of users. I understand that it's extremely relevant to this portion of users, but the fact remains that the vast majority never do more with sysex than perhaps backup their programs. I don't mean to be pedantic, but if you totaled the number of people who expressed interest in detailed sysex information for our instruments in forum posts across the web, it would equal a tiny portion of the instruments in the field. In any case, write support to ask and ye shall receive :)

Regarding changes to the parameter data included in a program dump, when we develop a new instrument, we will do some rough calculations as to how large a program needs to be to hold all of the parameters we anticipate it containing. There are always empty bytes included to allow for parameters to be added as an OS develops and matures. If new parameters are added in an OS update, this doesn't alter the order of the structure from previous versions--new parameters will just occupy bytes that were previously unused.

I hope this helps!

Thank you very much!