System Exclusive Data Packing

chysn

  • *****
  • 1812
System Exclusive Data Packing
« on: September 30, 2015, 12:16:44 PM »
On the old forums, there used to be a lot of questions about the packed data format for DSI system exclusive files. The manuals give a bare minimum of information. The information is usable, but easily misinterpreted. So, I'm going to give it a treatment here. Feel free to ask questions or add additional info. There will also be some code in C that performs the packing and unpacking operations.

Why do they "pack" the data in the first place?

MIDI is an eight-bit protocol. It differentiates between commands and data with the high bit (bit 7). If the high bit is 1, the byte is a MIDI command. If the high bit is 0, the byte is data. With 8 bits, there are 256 possible values. Since the high bit is spoken for, there are only 128 possible values available for data bytes. Many of DSI's parameters have more than 128 possible values (cutoff on a Mopho/P08/Tetra, for example), so they need to somehow represent 8-bit data in a protocol that allows only 7-bit data. That's the problem that "packing" solves.

The procedure

The data for DSI parameters is split into eight-byte packets. The first byte in each packet contains the high-bit values for the next seven parameters. The second through eighth bytes contain the low seven bits of these parameters. Let's say that a DSI synthesizer has data for the following parameters (note: this does not represent a real synth, it's just to illustrate how data is packed):

Parameter #1 - Attack: 0
Parameter #2 - Decay: 200
Parameter #3 - Sustain: 220
Parameter #4 - Release: 80
Parameter #5 - Waveform: 1 (where 1 represents a square wave)
Parameter #6 - Cutoff Freq: 170
Parameter #7 - Resonance: 32


These values can be represented in binary. So:

Attack = 0000 0000
Decay = 1100 1000
Sustain = 1101 1100
Release = 0101 0000
Waveform = 0000 0001
Cutoff = 1010 1010
Resonance = 0010 0000


Note how several of the parameters (Decay, Sustain, Cutoff) have their high bits set. If these were sent as-is via SysEx, those would be interpreted as command status bytes, and the dump wouldn't work. So, we take the high bit of each parameter in order, and put it into a seven-bit data byte, starting with bit 0.

0100110
      ^ High bit for Attack (bit 0)
     ^  High bit for Decay (bit 1)
    ^  High bit for Sustain (bit 2), etc...


And we send that as the first data byte in the packet. Then, we send the seven parameters with the lowest seven bits, starting with Attack:

0010 0110 (high bits in pack byte)
0000 0000 (Attack)
0100 1000 (Decay)
0101 1100 (Sustain)
0101 0000 (Release)
0000 0001 (Waveform)
0010 1010 (Cutoff)
0010 0000 (Resonance)


That's packed data. Repeat for the next set of seven parameters until you're done. The last packet probably won't contain exactly seven parameters. That's okay, though, you just put 0 in unused places in the pack byte.

Going both ways

Seeing how this is done, it's pretty easy to see how unpacking is done. You simply grab the first data byte in each packet of eight bytes, and use bitwise OR to set the high byte of your data, shifting the whole pack byte right by one as you go.

Code is in the next post.
« Last Edit: October 01, 2015, 02:58:59 AM by chysn »
Prophet 5 Rev 4 #2711

MPC One+ ∙ MuseScore 4

www.wav2pro3.comwww.soundcloud.com/beige-mazewww.github.com/chysnwww.beigemaze.com

he/him/his

chysn

  • *****
  • 1812
Re: System Exclusive Data Packing
« Reply #1 on: September 30, 2015, 01:15:17 PM »
All right, for now it's time to give up on putting code in a post. When I try doing that, the forum pops me back to the index page without saving. Sorry.

The header file is attached to this post, and was given the .txt extension.
« Last Edit: October 01, 2015, 02:50:24 AM by chysn »
Prophet 5 Rev 4 #2711

MPC One+ ∙ MuseScore 4

www.wav2pro3.comwww.soundcloud.com/beige-mazewww.github.com/chysnwww.beigemaze.com

he/him/his

dslsynth

  • ***
  • 1040
Re: System Exclusive Data Packing
« Reply #2 on: September 30, 2015, 01:27:29 PM »
When looking at the table describing the encoding in the manual its important to understand that its a bit pattern being described. Of particular importance is to understand where the top bit for each data byte is placed in the header byte.

As chysn says the last encoded packet will be partial with the header byte followed by the remaining data bytes. In fact I am not aware of any DSI product where the program vector length is a multiple of seven. So the last encoded package will be partial for all MIDI controllable products made by DSI.

PS: Actually it would be great if DSI at some point could use their excellent techdoc writing talents to make an official easily readable and technical precise document/webpage describing this encoding.
« Last Edit: September 30, 2015, 01:41:00 PM by dslsynth »
#!/bin/sh
cp -f $0 $HOME/.signature

blewis

  • ***
  • 258
Re: System Exclusive Data Packing
« Reply #3 on: November 13, 2015, 08:34:14 AM »
That was a very helpful post! Thanks for doing that!

Re: System Exclusive Data Packing
« Reply #4 on: May 22, 2018, 10:03:59 AM »
All right, for now it's time to give up on putting code in a post. When I try doing that, the forum pops me back to the index page without saving. Sorry.

The header file is attached to this post, and was given the .txt extension.

Thank you very much! This is just what I needed for my Teensy (Arduino compatible) based hardware controller. I was able to use parts of your code to unpack an incoming program dump and store it to SD card.

chysn

  • *****
  • 1812
Re: System Exclusive Data Packing
« Reply #5 on: May 22, 2018, 07:01:22 PM »
Glad to hear it!
Prophet 5 Rev 4 #2711

MPC One+ ∙ MuseScore 4

www.wav2pro3.comwww.soundcloud.com/beige-mazewww.github.com/chysnwww.beigemaze.com

he/him/his