SYSEX vs NRPN Program Parameter Numbers

blewis · « **on:** November 14, 2015, 08:58:57 AM »

I'm messing about with pulling out some SYSEX stuff from my new Prophet-6. I'm running into issues not being able to determine which bytes in the SYSEX payload are what Program Parameters. Since the SYSEX Program Parameters are not specifically documented in the manual, I have started by assuming they must be specified by the NRPN numbers (otherwise I would have expected another SYSEX specific table in the manual)

I've started by trying to extract the Program Names, as the 3 digit display is one hurdle I want to overcome. The NRPN Program Parameter number for the first byte of the Program Name is 236, but I'm finding I can obtain the Program Name starting at byte 107! Way different.

I believe I'm unpacking the data correctly, but I'm open to being shown how I've messed up. I'm using a utility function from ctrlr.org from an existing Mopho panel and it seems to be built into the application's library. (ctrlr.org is cool as sh!t BTW). Besides ctrlr.org saying the Program Names starts at 107, my MIDI Monitor application is also showing the Program Name in ASICII starting _way_ before byte 236 - evening considering raw packing bytes.

I've e-mailed DSI support and the response, I assume, is coming, but is slower than normal. So here is more of what I've done:

I don't think this is a packing vs unpacking problem.

I've done SYSEX dumps changing the first 7 Program Parameters listed in the NRPN table.

Byte 0 of the SYSEX payload is the packing byte (byte 0 after the F0 01 2D 03 request I'm making to dump the edit buffer). I can "flip bits" in it by adjusting the parameters with > 127 values (OSC 1 shape, PW, OSC 2 fine), and it would be "the start of the 8 byte packet" (per the manual)

The NRPN numbers suggest the Program Parameters are, from 0 (zero here being the 0th byte _after_ the packing byte):

0 Osc1 Freq
1 Osc 1 Sync
2 Osc 1 Level
3 Osc 1 Shape
4 Osc 1 PW
5 Osc 2 Freq
6 Osc 2 Fine Freq

But I'm seeing: (again zero being the 0th byte _after_ the packing byte)

0 Osc 1 Freq
12 Osc 1 Sync (there's going to be an extra packing byte in there, but 12 != 1 for sure, so it's probably byte 11)
8 Osc 1 Level (there's a packing byte in there, so really this is 7. 7 != 2 the NRPN)
3 Osc 1 Shape
5 Osc 1 PW
1 Osc 2 Freq
2 Osc 2 Fine

I'm pretty sure I can't use the NRPN Program Parameter ordering for this job. Do the programmers have a handy list they can send me please?

dslsynth · « **Reply #1 on:** November 14, 2015, 02:51:22 PM »

Oh its a well known problem that have existed since Prophet-12. Same thing with Tempest which is almost not documented at all on the sysex front except for some friendly leaks made via Pym.

Anyway, Prophet 12 is the first "normal" DSI instrument where the program vector layout is not documented at all. Furthermore it was recently revealed that the Prophet 12 program format uses a version number as there exists multiple versions of it. One can assume that the old versions are converted into the new version once loaded as the current sound.

What you are seeing is that Prophet-6 have the same problem. After skimming the Prophet-6 manual it seems that its even worse documented than Prophet 12 as the value ranges of the NRPN parameters are not even documented as well as in the previous instruments.

Am I surprised? Not really! DSI have been very busy in recent years putting out new highly priced instruments and seems mostly interested in moving forward (same old story).

Also DSI have hired a new techdoc writer which did indeed progress the quality of the main body of the manual quite visibly. But it seems that the technical sides of the manual - the very detail oriented program vector layout - have been given lower priority. At the same time it seemed that Pym wanted to use a different way of accessing program parameters inside the code which caused wildly diverging NRPN and program vector indexes.

What I am saying is essentially that the only way to determine the program vector layouts and associated NRPN message value mappings are by reverse engineering it which requires direct access to the instrument.

Given these conditions the only way forward is to write a piece of software to automatically detect the program vector indexes for NRPN parameter. I have such a piece of code though I need to update it for program layout versions and Prophet-6. Only trouble is that it is written in C++11 and works only on Linux.

So: do you have access to a Linux machine?

Also, for DSI: Are program vector layout versions used for Prophet-6 and Pro 2?

Pym · « **Reply #2 on:** November 14, 2015, 04:38:50 PM »

Actually the program pack shouldn't change very much at all. I try to make minimal changes when I change ranges or add a param, I'd rather not move stuff around. Gets too confusing.

It's pretty much the same packing mechanism in the P12 and P6 as it was in our older instruments. The only one that uses a bit packing template is the Tempest due to block sizes in the internal file system. Was not our first choice to do that

Quote from: dslsynth on November 14, 2015, 02:51:22 PM

Oh its a well known problem that have existed since Prophet-12. Same thing with Tempest which is almost not documented at all on the sysex front except for some friendly leaks made via Pym.

Anyway, Prophet 12 is the first "normal" DSI instrument where the program vector layout is not documented at all. Furthermore it was recently revealed that the Prophet 12 program format uses a version number as there exists multiple versions of it. One can assume that the old versions are converted into the new version once loaded as the current sound.

What you are seeing is that Prophet-6 have the same problem. After skimming the Prophet-6 manual it seems that its even worse documented than Prophet 12 as the value ranges of the NRPN parameters are not even documented as well as in the previous instruments.

Am I surprised? Not really! DSI have been very busy in recent years putting out new highly priced instruments and seems mostly interested in moving forward (same old story).

Also DSI have hired a new techdoc writer which did indeed progress the quality of the main body of the manual quite visibly. But it seems that the technical sides of the manual - the very detail oriented program vector layout - have been given lower priority. At the same time it seemed that Pym wanted to use a different way of accessing program parameters inside the code which caused wildly diverging NRPN and program vector indexes.

What I am saying is essentially that the only way to determine the program vector layouts and associated NRPN message value mappings are by reverse engineering it which requires direct access to the instrument.

Given these conditions the only way forward is to write a piece of software to automatically detect the program vector indexes for NRPN parameter. I have such a piece of code though I need to update it for program layout versions and Prophet-6. Only trouble is that it is written in C++11 and works only on Linux.

So: do you have access to a Linux machine?

Also, for DSI: Are program vector layout versions used for Prophet-6 and Pro 2?

blewis · « **Reply #3 on:** November 14, 2015, 05:43:55 PM »

Thanks for dropping by Pym.

What reference should I be using to determine the SYSEX Program Parameter ordering? Is the ordering the same as the NRPN Parameters?

I'm really puzzled here because the 0th character of the Program Name NRPN is listed as NRPN #236, but the 0th character of the Program Name is showing up in the 107th byte of the SYSEX payload (after being unpacked).

Pym, am I off in the weeds with mapping those first 7 Program Parameters? They don't seem to match the NRPN numbering very well at all.

dslsynth · « **Reply #4 on:** November 15, 2015, 03:37:45 PM »

Quote from: Pym on November 14, 2015, 04:38:50 PM

Actually the program pack shouldn't change very much at all. I try to make minimal changes when I change ranges or add a param, I'd rather not move stuff around. Gets too confusing.

It's pretty much the same packing mechanism in the P12 and P6 as it was in our older instruments. The only one that uses a bit packing template is the Tempest due to block sizes in the internal file system. Was not our first choice to do that

Thanks a lot for responding, Pym!

And thanks for clarifying the packing part by which I assume means the encoding of the 8-bit byte program vector into a stream of 7-bit bytes in the sysex messages. All that is left then is the location of each parameter within the program vector, what controller messages change this parameter, what range of values this parameter can take and how they are interpreted.

The original posters question is what the location of the program parameters are as the manual only lists the NRPN and CC messages affecting the parameters. A similar lack of information exists for Prophet 12 and most likely Pro2 too.

Anyway, how do we get this information out to the developers who wants to write software for your instruments?

Would it be possible that DSI at some point could find the resources to get this information documented? Or will the developers have to figure it out for themselves? Or maybe we could find a way to collaborate on getting this information documented and made available?

Also, do DSI have access to a Linux machine at their HQ?

blewis · « **Reply #5 on:** November 19, 2015, 01:42:04 PM »

Quote from: dslsynth on November 14, 2015, 02:51:22 PM

So: do you have access to a Linux machine?

I'm on Mac OS which has its own flavor of UNIX. Maybe I could compile it?

I may wind up having to rig up what you suggested in Ctrlr if I can't get anymore help.

Are you/would this work:
1) zeroing out all NRPNs
2) request edit buffer dump
3) store SYSEX vector result
4) max out first NRPN
5) request another edit buffer dump
6) compare SYSEX vector to see which byte changed
7) continue for all NRPNs?

That might make the first part of a visual unit test for my Ctrlr panel anyway and be reusable if I wanted to mess with the Pro2.

I have a lot of DSI stuff and was thinking my past positive support experience would carry through on this request too. But I can understand why this stuff might be hidden

dslsynth · « **Reply #6 on:** November 19, 2015, 02:55:52 PM »

Quote from: blewis on November 19, 2015, 01:42:04 PM

I'm on Mac OS which has its own flavor of UNIX. Maybe I could compile it?

The trouble is that the MIDI I/O API on Linux and Mac OSX is different. The C++11 program I wrote is Linux only for exactly that reason. The rest of the code is portable.

DSI have in the past been very open about their sysex format. My feeling the current documentation level is dictated by company resources rather than wanting to keep the sysex formats secret. Which is why I wrote the C++11 program to determine the program vector indexes from NRPN messages. Still have to make it work with the newer instruments as undocumented features such as versions got to be resolved precisely for each instrument and that requires both access to the instruments and information from DSI.

blewis · « **Reply #7 on:** November 19, 2015, 07:27:23 PM »

Quote from: dslsynth on November 19, 2015, 02:55:52 PM

My feeling the current documentation level is dictated by company resources rather than wanting to keep the sysex formats secret.

I think so too. I hope I'm not coming off as having any conspiracy theories.

dslsynth · « **Reply #8 on:** November 20, 2015, 05:33:35 AM »

I got an idea this morning. If DSI wants to collaborate a good part of the program vector documentation could be generated automatically via various inputs and a some qualified scripting.

blewis · « **Reply #9 on:** November 22, 2015, 04:22:04 PM »

Okay. Gave up on getting any response. I'm perplexed why this was something hard to answer, but I'm moving forward from here.

Spent some time today getting my Ctrlr panel to send NRPNs. I wrote a function to compare the previous SYSEX edit buffer dump from the currently obtained SYSEX buffer dump.

For Osc1 and Osc2, I can now fiddle with the modulators one at a time, then I set a break point and in the ctrlr.org Lua debugger, then I can watch it stop on the unpacked byte that differs. This way I don't have to eyeball the raw SYSEX message and count bytes. I do have to hit Play on the debugger an annoying amount of times... :-)

Doing that, I can manually work my way through the NRPN table and get the corresponding SYSEX byte and I'm working with unpacked data. Hint, for OSC1 and OSC2, they don't match the NRPN number very regularly.

A large number of the 1024 bytes are sequencer data, so I only have to work through maybe 200+ manually. I'd be more grouchy about it, but the truth is if I want to do this panel, I have to tediously go through the panel modulator knobs one by one anyway. I could automate this more, but again, since I have a bunch of panel knob variables to adjust manually, doing it one at a time is reasonable anyway.

blewis · « **Reply #10 on:** November 22, 2015, 04:26:43 PM »

screen shot

dslsynth · « **Reply #11 on:** November 23, 2015, 09:02:35 AM »

Screen shot looking really neat... *BANG*

I got some Prophet 12 data the other day that I am in the process of scripting the processing of so that a sysex document can be made. Quite a journey ahead though. Expect Prophet-6 to follow once Prophet 12 is up and running.

Pym · « **Reply #12 on:** November 23, 2015, 10:56:35 AM »

Message me your email address and I'll send you the enums and const arrays, should help speed it up

Quote from: blewis on November 22, 2015, 04:22:04 PM

Okay. Gave up on getting any response. I'm perplexed why this was something hard to answer, but I'm moving forward from here.

Spent some time today getting my Ctrlr panel to send NRPNs. I wrote a function to compare the previous SYSEX edit buffer dump from the currently obtained SYSEX buffer dump.

For Osc1 and Osc2, I can now fiddle with the modulators one at a time, then I set a break point and in the ctrlr.org Lua debugger, then I can watch it stop on the unpacked byte that differs. This way I don't have to eyeball the raw SYSEX message and count bytes. I do have to hit Play on the debugger an annoying amount of times... :-)

Doing that, I can manually work my way through the NRPN table and get the corresponding SYSEX byte and I'm working with unpacked data. Hint, for OSC1 and OSC2, they don't match the NRPN number very regularly.

A large number of the 1024 bytes are sequencer data, so I only have to work through maybe 200+ manually. I'd be more grouchy about it, but the truth is if I want to do this panel, I have to tediously go through the panel modulator knobs one by one anyway. I could automate this more, but again, since I have a bunch of panel knob variables to adjust manually, doing it one at a time is reasonable anyway.

blewis · « **Reply #13 on:** November 23, 2015, 11:43:42 AM »

Quote from: dslsynth on November 23, 2015, 09:02:35 AM

Screen shot looking really neat... *BANG*

I got some Prophet 12 data the other day that I am in the process of scripting the processing of so that a sysex document can be made. Quite a journey ahead though. Expect Prophet-6 to follow once Prophet 12 is up and running.

Good progress!

LA Keys · « **Reply #14 on:** November 23, 2015, 05:21:29 PM »

Quote from: blewis on November 22, 2015, 04:26:43 PM

screen shot

Ohhh Yessss!

LA

goodweather · « **Reply #15 on:** March 01, 2016, 12:09:25 AM »

Quote from: blewis on November 22, 2015, 04:26:43 PM

screen shot

Nice! FYI, I'm busy to build a DSI Pro2 panel...
I'll make a separate topic inside the Pro2 forum and on Ctrlr forum soon.

Wolfram · « **Reply #16 on:** May 24, 2016, 12:49:09 PM »

Quote from: blewis on November 14, 2015, 08:58:57 AM

I'm messing about with pulling out some SYSEX stuff from my new Prophet-6. I'm running into issues not being able to determine which bytes in the SYSEX payload are what Program Parameters. Since the SYSEX Program Parameters are not specifically documented in the manual, I have started by assuming they must be specified by the NRPN numbers (otherwise I would have expected another SYSEX specific table in the manual)

I've started by trying to extract the Program Names, as the 3 digit display is one hurdle I want to overcome. The NRPN Program Parameter number for the first byte of the Program Name is 236, but I'm finding I can obtain the Program Name starting at byte 107! Way different.

I believe I'm unpacking the data correctly, but I'm open to being shown how I've messed up. I'm using a utility function from ctrlr.org from an existing Mopho panel and it seems to be built into the application's library. (ctrlr.org is cool as sh!t BTW). Besides ctrlr.org saying the Program Names starts at 107, my MIDI Monitor application is also showing the Program Name in ASICII starting _way_ before byte 236 - evening considering raw packing bytes.

I've e-mailed DSI support and the response, I assume, is coming, but is slower than normal. So here is more of what I've done:

I don't think this is a packing vs unpacking problem.

I've done SYSEX dumps changing the first 7 Program Parameters listed in the NRPN table.

Byte 0 of the SYSEX payload is the packing byte (byte 0 after the F0 01 2D 03 request I'm making to dump the edit buffer). I can "flip bits" in it by adjusting the parameters with > 127 values (OSC 1 shape, PW, OSC 2 fine), and it would be "the start of the 8 byte packet" (per the manual)

The NRPN numbers suggest the Program Parameters are, from 0 (zero here being the 0th byte _after_ the packing byte):

0 Osc1 Freq
1 Osc 1 Sync
2 Osc 1 Level
3 Osc 1 Shape
4 Osc 1 PW
5 Osc 2 Freq
6 Osc 2 Fine Freq

But I'm seeing: (again zero being the 0th byte _after_ the packing byte)

0 Osc 1 Freq
12 Osc 1 Sync (there's going to be an extra packing byte in there, but 12 != 1 for sure, so it's probably byte 11)
8 Osc 1 Level (there's a packing byte in there, so really this is 7. 7 != 2 the NRPN)
3 Osc 1 Shape
5 Osc 1 PW
1 Osc 2 Freq
2 Osc 2 Fine

I'm pretty sure I can't use the NRPN Program Parameter ordering for this job. Do the programmers have a handy list they can send me please?

I have also this problem. The order of the parameter in the NRPN list is different to the unpacked data of a dump.
I must find out the parameter with try and error.
And the big shame is the DSI support is unable to help. That's very poor.

cbmd · « **Reply #17 on:** May 24, 2016, 01:01:17 PM »

Hi Wolfram,

the issue you are encountering is due to your data not being unpacked. All the relevant information is provided in the manual for unpacking the data.

cbmd · « **Reply #18 on:** May 24, 2016, 03:17:18 PM »

Actually, that's my bad. Wolfram was successful in unpacking the data. The packed parameter list does differ than the list given on page 77 of the manual. For those interested, I've attached a PDF of the packed parameter assignments to this message.

chipaudette · « **Reply #19 on:** May 25, 2016, 06:25:53 AM »

Quote from: cbmd on May 24, 2016, 03:17:18 PM

Actually, that's my bad. Wolfram was successful in unpacking the data. The packed parameter list does differ than the list given on page 77 of the manual. For those interested, I've attached a PDF of the packed parameter assignments to this message.

Sweet!