Krog Kronos Intro Videos

Timing in iOS

See this link: http://atastypixel.com/blog/experiments-with-precise-timing-in-ios/

The bottom line is that no form of NSTimer/RunLoop/etc. is going to provide sufficiently accurate timing for anything musical....

Misery (Continued)

So I move the project over to the iPhone.

Everything works perfectly - no problems with NSLog() or breakpoints flaking out...

What the hell!

Misery (Continued)

So I found this link.  A way to reset the debugger to some degree:

1. Close XCode

2. Delete the $project/build/*

3. Delete /Developer/Platforms/iPhoneOS.platform/DeviceSupport/4.2.1

4. Restart XCode

5. Go to organizer and agree to let it download "collect" what it wants (symbols) from the device

6. In Organizer, again right click -> Add device to provisioning portal


Doing this I now have a project which ignores breakpoints and/or NSLog statements about 10% or so of the time - which is almost tolerable.  I set a breakpoint at the start of the project - if it doesn't stop I just restart it.


Its really hard to imagine what these folders have to do with NSLog and breakpoints...

Misery (Continued)

So the best I can do to fix the NSLog() and breakpoint problem on this particular project (or a project using the same file, i.e., another independently constructed duplicate project) is to diddle the Code Signing settings for "Any iOS" between "iPhone Developer" and "iPhone Developer: me XXXXXX".

Good news on the audio front...

It turns out that when you instantiate an AudioUnit the settings of "[ AVAudioSession sharedInstance] setCategory:" matter. 

For example, with the category set to "AVAudioSessionCategoryPlayback" you can instantiate an AudioUnit with input but the render loop will not be called.  Changing the category to "AVAudioSessionCategoryPlayAndRecord" after instantiation does not change this - even if you stop and restart the AudioUnit. 

So if you don't use the Record category your render loop will simply not be called.

I have been looking around at various documentation for AudioUnits wonder what you are allowed to do and not do with multiple AudioUnits simultaneously.  Certainly it seems you should be able to have more than one at any given time and in fact as long as they don't overlap on input and output and you have the right category set on instantiation this seems to be true from my experiments.

But there is little documentation of this - at least I could not find it.

AudioUnits and AVAudioPlayer do not work well together either.  However, I probably did not have the categories set properly so this might be part of the cause though some other testing leads me to believe there are other incompatibilities.

Basically I eliminated AVAudioPlayer as well as some other MixerHostAudio code I was using.  I now do my own mixing and sound triggering in a pair of AudioUnits and it seems far simpler and more efficient.

Misery (Continued)

So I have still not been able to figure out completely what's going on with the NSLog() and breakpoints.

I tried Googling this 'xcode gdb "sharedlibrary apply-load-rules all" ' where I did find this post (here) on various "code signing" issues.

It seems like the gdb debugger does not always attach to the remote iOS process unless some voodoo in the "code signing" section of the "Build" configuration is set to the proper phase of the moon.

Diddling this seems to make NSLog() and breakpoints work again:

I changed the 'Any OS' tag to some default.

This Apple software is totally flaky and unreliable in this regard.

[ UPDATE ]  While this worked a few times in a row on XCode 3 no dice in XCode 4.

Diddling the same parameter does not cause NSLog() or breakpoints to work.

[ UPDATE ] I regenerated the provisioning profiles and upgraded a different iPad to 4.3.5 (I was on 4.3.3 for the problems above).  After reinstalling all of this it seems to be working ("seems to be working" basically means I have several time connected and disconnect the iPad over an hour or so and each time breakpoints continue to work.

Now that I think about it I probably updated iTunes to 10.4.1 (10) - but I cannot recall if the problems started before that or not - I think not in this case.

Is it fixed?

The jury is still out...

XCode, AudioUnits, Breakpoints and NSLog Misery...

So I am debugging an iOS AudioUnit render callback function.

I have the exact same code working in a standalone project so I know that everything is correct.  The code in question looks something like this:

OSStatus recordingCallback(void *inRefCon,
                 AudioUnitRenderActionFlags * ioActionFlags,
                 const AudioTimeStamp *inTimeStamp,
                 UInt32 inBusNumber,
                 UInt32 inNumberFrames,
                  AudioBufferList *ioData)
{   
  AudioBuffer buffer;
  AudioBufferList bufferList;
      ...
 
   buffer.mNumberChannels = 1;
  buffer.mDataByteSize = inNumberFrames * 2;
  buffer.mData = malloc( inNumberFrames * 2 );
  bufferList.mNumberBuffers = 1;
  bufferList.mBuffers[0] = buffer;

  ...

  status = AudioUnitRender(audioUnit,
                           ioActionFlags,
                           inTimeStamp,
                           inBusNumber,
                           inNumberFrames,
                           &bufferList);

  if (status != noErr)
  {
    NSLog(@"An AudioUnitRender error occurred (%d).", status);
  }
  ...
}
 
The buffer arrangement is such that only a single buffer is being used.

In the stand alone project status is always equal to noErr and the NSLog function is never called and the render works fine.

When I copy this code without change into the main project suddenly all hell breaks loose.

First off, status starts reliably returning -50 (which indicates a bad parameter is being passed to AudioUnitRender).  

Bad parameter? 

How on earth can that be as the exact same parameters work perfectly in the standalone project.  After endless diddling to ensure that I am not doing something stupid I start to look for other reasons this might be failing.

Eventually I discover a post on some Apple developer list where someone else has a similar problem.  They trace this to the use of AVAudioPlayer and its associated start method.  (Basically this means if you have some other part of the project playing background music the above code ceases to work correctly and indicates bad parameters are being passed.)

Commenting out this code (the AVAudioPlayer code along with some MixerHostAudio-derived code) appears to make the project work but I notice that I am no longer seeing NSLog output (the project barfs out all kinds of status as it boots up).  Then breakpoints seem to no longer work so I cannot tell if removing the use of AVAudioPlayer is really working or not.

Now I become extremely frustrated. 

How can NSLog simply cease to work?

So I think - well, this must be an XCode 4 problem...

I open the project in the latest version of XCode 3 and NSLog does not work there either nor do breakpoints...!!!

What the hell?  I curse Apple, Jobs, and the iOS development staff.

(This is made worse by recently discovering that Adobe CS 2 will not work on Lion, Lion being required to have the newest XCode 4 and no doubt iOS 5.  I want a law passed that requires Apple to compensate me for taking away functionality with required software releases...)

Now what?

More hours of Googling and diddling until if find this on StackOverflow (with some changes):

    1. Quit XCode 3/4.
    2. Open a terminal window and cd to the project directory
    3. cd into the .xcodeproj directory
    4. Delete everything except the .pbxproj file.

Or, in Finder Right/Option-Clicking on the .xcodeproj bundle, pick "Show Package Contents," and remove all but the .pbxproj file.

This seems to work - at least some of the time and sort of magically the breakpoints and NSLog output reappear and work as expected!

But then it flakes out again - sometimes working - sometime not.

I guess at this point the only things I can come up with are either

A) Somehow the render function is messing things up.  Everything works save for the microphone callback just fine - or at least it did - no flakies until I added in that extra AudioUnit code.  It seems as if this addition permanently wacked out XCode4 for this project (others at least appear to still work correctly including the standalone version of the AudioUnit.

B) Mixing to much AudioUnit stuff is a problem or flakey...  Others report the AVAudioPlayer problem with AudioUnit and other things (like playing movies).

More to come...

iOS 4, Voip

I have a VOIP application.  It runs in the background as per the Apple docs.

On thing is interesting.  In the handler for my input and output streams I have this code:

     if ([ UIApplication sharedApplication ].applicationState == UIApplicationStateActive)
      {
        NSLog(@"APPSTATE: UIApplicationStateActive");
      }
      else if ([ UIApplication sharedApplication ].applicationState == UIApplicationStateInactive)
      {
        NSLog(@"APPSTATE: UIApplicationStateInactive");
      }
      else if ([ UIApplication sharedApplication ].applicationState == UIApplicationStateInactive)
      {
        NSLog(@"APPSTATE: UIApplicationStateInactive");
      }
      else
      {
        NSLog(@"APPSTATE: Unknown: %d", [ UIApplication sharedApplication ].applicationState);
      }

According to the headers the only allowable values for "applicationState" are  UIApplicationStateActive, UIApplicationStateInactive, and UIApplicationStateBackground.

So when my app is in background with TCP/IP running it prints out

  APPSTATE: Unknown: 2

So it seems like the background VOIP state is 2.  An undocumented state at that!

MIDI Polyphony (Part 1)

In working on the 3i controller my experience with the iConnectMIDI has led me down an interesting path.

One of the things the 3i needs to be able to do is detect "movement nuance" and to transmit information to an associated sampler or synth about what to do with that movement nuance.

I define "movement nuance" to be a continuously variable parameter that is associated with a given playing note.  MIDI Aftertouch is an example of this - typically used to determine how much "pressure" is being used to hold down a key after a note is sounded, i.e., once the piano key is down I can push "harder" to generate a different Aftertouch value.

MIDI Pitchbend is similar as well except that it affects all the notes, not just the one you are currently touching.

Normal MIDI keyboards are "note polyphonic" which means that you can play a chord.  But the "effects" that MIDI allows in terms of "movement nuance" are limited.

Traditionally MIDI is used with keyboards and to a degree with wind instruments where its limited ability to handle "nuance movement" is not really an issue or where things like Aftertouch is sufficient to deal with whatever the musician is trying to do.

But if you think about other kinds of instruments, for example string instruments, then there is a lot more nuance associated with playing the note.  Some examples of this include "hammer on" and "hammer off", "bending" as on a guitar or violin, sliding "into a note" on a violin, the notion that some other element such as a bow is sounding a note at one pitch and the pitch now needs to change, and so on.

Then there is the effect after releasing the note, e.g., "plucking" a string or sounding a piano note with the damper up.  The note sounds in response to the initial note event.  The event is release (MIDI Note Off) but the note still sounds.

While this is fine on a piano its much different on a stringed instrument.   For something like a violin you may be holding down a string which is plucked.  After the pluck is released you may wish to slide your finger that's holding the note to change pitch.

Here MIDI does not off significant help.

The notion of a note sounding beyond your interaction with a keyboard is also very non-MIDI.  MIDI is designed around a basic "note on"/"note off" model where everything else revolves around that.

Certainly its possible to try and "extend" MIDI with a notion of CC values or other tricks in order to get more information per note - but MIDI wasn't really designed for that.  (Some MIDI guitars run one note per channel - but that's about all the innovation they can muster.)

From my perspective with the 3i really the only thing left is to reconsider the whole notion of what a MIDI keyboard really is.

So my idea is that, in a very general sense, when you touch a playing surface you create a sort of MIDI "virtual channel" that is associated with whatever that touch (and corresponding sound) is supposed to do.  That "virtual channel" lives on through the sounding of the note through whatever "note on" activities are required, through the "note off" and on through any sort of "movement nuance" that continues after the "note off" events, e.g., plucking a violin note on E and sliding your finger up to E#.

This model gives the controller a full MIDI channel to interact with a synth or sampler for the entire duration (life time) of the "note event" - which I define as lasting from the initial note sounding through note off through the "final silence".

So instead of having a rack of virtual instruments for an entire "keyboard" you now have the idea of a full MIDI channel of virtual instruments for a single "touch" that lasts the lifetime of a note.  The virtual rack being conjured up at the initiation of the sound and being "run" until "final silence" - at which point the virtual rack is recycled for another note.

Each "virtual rack" associated with a note has to last as long as the note is sounding.

This allows a controller to manipulate MIDI data on its "virtual channel" as the life of the note progresses.

Now, since a note can last a long time (seconds) and the player can continue to pluck or play other notes you find that you can quickly run out of channels on a standard 16 channel MIDI setup because a MIDI channel can be tied up with that note for quite a while (also seconds) - particularly when you factor in having two different instruments (say bass and guitar) set up (as a split) or when you have layers.

This is effectively MIDI polyphony in the sense that you need to figure out the maximum number of channels that could be active at a given time.  On "hardwired" synths or samplers this is typically something like 64 or 128 or more. (Of course they are managing simultaneous sound samples, not MIDI channels.  But in my case we need MIDI via the 3i to do the job of the hardware sampler so we need MIDI channels instead of (or to act as) "voices".)

Which leave us with 16 MIDI channels is simply not enough.

And this means that using a MIDI device to move data from the 3i to the synth or sampler is inherently limited because at most it can support only 16 channels.

This if further compounded by the fact that once the MIDI reaches outside the host OS it must be throttled down to 32.5K baud in order to be compliant with the MIDI standard (see this PDF).

I believe this is what I am seeing with iConnectMIDI - as fast as the device is it cannot move data off of an iOS device faster than 32.5K baud because iOS is throttling it.

So the only way to get data off the device is via a high speed USB cable (WiFi being no good for reasons mentioned before).

Avoiding iOS "Sync in Progress"...

As a developer I am often in a position of connecting an iOS device to my Mac for various reasons.

It is vastly annoying that every time I connect the iPad iTunes tries to sync with it - even if the iPad or iPhone is synced with another computer (and even when you tell iTunes not to bother with this particular iOS device). 

So far I have not figured out how to turn this off via iTunes in a suitable way.  I want my devices that are ones I use for business to sync when connected - but not devices for development - which might be synced to other Macs.  I seem to be able to disable it for devices synched with a given Mac but not with other Macs.

However, you can defeat it by simply being lazy!

Simply wait for a new version if iTunes to show up and then don't launch it.

Each new version of iTunes insists you agree to the vast legalese is presents the first time it runs.  However, until you agree (or disagree) no syncing occurs when you attach an iOS device.  Thus as long as you don't need to run iTunes on your development machine you can safely ignore it and its legalese and happily connect and disconnect to your hearts content without "Sync in Progress".

The Apple 30-Pin Misery Plan...

I have been struggling to get high-speed MIDI information out of iOS devices.  Last time I wrote about how the iConnectMIDI failed to give me what I was looking for (I have an email into their support group but so far no follow up response).  Since then - and though I am still playing with that device - I have found another device that I thought would help.

But first I'd like to talk about what I think is going on with iOS and MIDI. 

First off, I believe that the ubiquitous 30-pin Apple USB connector is filled with magic.  In particular there is a "chip" of some sort that resides on one end or the other of the 30-pin that does a lot of magical things for connecting devices.  It is said that there is a patent involved and that anyone wanting to create a cable that is plug-compatible with the iOS devices must use that patented or licensed technology.

There is basically a single USB 2.0 output channel on a given iOS device available via the 30-pin connector.  USB is a high-speed (500 Mb) frame-based architecture and requires two pins on the 30-pin (see this).  Since MIDI output is basically serial output the 30-pin has to share the USB pins and hence the USB.

So what Apple appears to have done was make the iOS devices "look" for MIDI class-compliant devices on the USB line via the Camera Connection kit, if it finds one then it talks to it.

(Note that the Camera Connection Kit only works on iPad at this time.)

However, the USB coming out of the iOS device does not seem to work in the typical USB-daisy-chain mode where you can have multiple devices on a USB line (at least I have not been able to get it to work that way though more testing is required here).  iOS also wants to power the MIDI USB device and, with its battery situation tightly controlled by Apple, the device cannot draw too much power or its rejected.

Next up there is the Apple iOS software talking to the USB port.  The USB line is limited by the iOS drivers available.  The key is here is "class compliant" - if your MIDI device is "class compliant" and draws a minimum of power - it works.  Ditto for various audio interfaces as documented here (I did not try this myself but it makes sense that "class compliant" audio would work as well).  So if some USB device is not known to iOS then it simply isn't recognized or processed.  Add to this the fact that you cannot load drivers into iOS and you end up quite limited.

So the bottom line is that if you want MIDI out of an iOS device you have to use either 1) a Camera Connection Kit on iPad, 2) an iConnectMIDI on any iOS device, or 3) your own solution.

All this leads to something I had hoped to use to solve this: dockStubz from cableJive.

dockStubz is a 30-pin pass through with a mini USB port on the side.  My hope was that MIDI devices connected through the dockStubz via the Camera Connector would work and I could run the debugger.

The device has its own male 30-pin on one end and a female 30-pin on the other.

The mini USB port is on the side.

It connects to the iPhone or iPad and the USB cable works just as if you were using a standard out-of-the-box Apple 30-pin to USB.

It works in XCode just fine.  There's only one rub.  You cannot plug anything else into the 30-pin and have the USB connection work.  So installing the Camera Connection Kit disables and disconnects the mini USB.

Rats!  Some much for believing internet posts about debugging MIDI on the iPad. 

Though now that I have done my research I can see why it doesn't work.  The mini-USB is physically connected to the iOS USB pins on the 30-pin so long as nothing is plugged into the 30-pin.  Plugging in another device breaks that connection.

The other thing that seems to be happening is that when a MIDI device is attached there is some form of throttling going on to ensure that the attached MIDI device is not overrun.  I found this out when a bug in my code had me dumping MIDI CC messages out in a near-tight-loop.

Running these out via my own USB line happens at full USB speed and no throttling occurs (however I am not going through the iOS MIDI driver and instead performing other trickery).

The test will be to determine if MIDI port to MIDI port inside the iOS device is also throttled.  At least on a Mac running 10.6.8 this is not the case - MIDI port to MIDI port seems to run full bore.  My guess is there is some sort of magic happening in the conversion from iOS to USB.

Only more time will tell...

iConnectMIDI - Product and Analysis

For the last several months I have been awaiting a product call iConnectMIDI.

The reason for this is that I have been doing work on iOS and would like very much for their to be a supported way to communicate with the corresponding devices (iPhone, iPad) at speeds that are suitable for live gigging.  Since, as I wrote in previous posts, I don't believe WiFi offers that performance I have been on sort of a quest to get the best technology available to help me.

Until iOS 4.2 there was no MIDI support at all in iOS save for the Line 6 MIDI Mobilizer which I believe has its own special software.  With 4.2 the basic Mac OS X MIDI interface API became available for iOS.  Along with this, on the iPad at least, came the Camera Connect 30-pin Apple to USB conversion adapter that allowed some USB MIDI devices to be connected directly to the iPad (the Camera unit does not work on the iPhone - I suspect because the iOS device battery is driving the MIDI unit and for iPhones the batter would die too quickly to suit Apple).

Not wanting limitations with my development effort, including performance limitations in getting data from the iPad to elsewhere either as MIDI or otherwise I found that I had to create my own, "Gold Standard" solution.  My solution, which so far out performs all other USB/MIDI apps available for iOS allows me to play MIDI (or do virtually anything else I can do over a wire) at full USB speeds.  Its Mac OS X (though it would work on Windows or Linux as well) software that allows me to simply connect my iOS devices to my Mac via USB and talk to them at full USB speed.

So far the best benchmark of performance has been this: generating continuously variable MIDI CC messages on an iOS device in a tight loop, i.e., while a finger is dragging.  For virtually all devices save my "Gold Standard" USB solution the devices cannot keep up.

Which brings me to the iConnectMIDI.  Mine arrive a few days ago from Sweetwater.

The first thing I did was tear it apart to see what makes it tick.  Here it is out of the box:

It has three USB port - two minis (D1 and D2) and one regular "A" USB along with a power light and an activity light for each of the type of different connectors. 

There are two full MIDI ports on the back.  It uses a non-standard 5.2V 800 ma power system (musicians prefer 9V) also on the back:

Inside the guts the iConnectMIDI uses a TI Stellaris (LM3S3749) ARM programmable controller and a set of three SILabs F321 USB 2.0 controllers (link) for each of its three USB ports (one regular, two mini).

Mine is serial #0x1B (27 if you don't speak hex).

The web site talks about the huge performance boost this device has over standard MIDI (which runs at 30K baud or about 3 Kb/second).  The 30K baud speed is based on ancient UART technology used in modems - save that MIDI runs a 0..5V signal.  The web site claims 12 Mb/second of transfer speed (see this) - significantly faster.

So I've played around with this for a day or so trying to see what it will do and not do.

Here are my results so far.

USB port (A wude) connector does not meet the USB MIDI specs for the iPad to talk directly to the iConnectMIDI like E-mu X-midi 2x2 (see this site for a full set of compatibility issues and devices).  This means that the iPad via the Camera Connector Kit does not recognize this device as a MIDI device.

(It would be nice if the iConnectMIDI did present its USB ports as iPad compatible.  This has to do with some sort of standard for MIDI USB devices and whether a particular device follows that standard or not.  The key here is "class compliant" USB MIDI.)

The iConnectMIDI (iCM) does, however, come with its own special Apple 30-pin to USB mini cable that magically works when plugged between the iConnectMIDI and the iPad.  (I think this cable is were all the real magic is happening but I am not sure how at this point.)

I have been using Native Instruments Kontakt 4.0 quite a bit for testing the 3i (for reason which will become clear in later posts).

Connecting multiple different MIDI-producing apps to the iCM via USB does not drive Kontakt (iPad -> iCM D2 mini-USB, iCM D1 mini-USB to Mac, Kontakt set to use iCM D1 port which appears in the Kontakt options area).

However, by renaming the iCM D1 port on the Mac with MIDI Patch bay to something else, e.g., "iConnectMIDI ICM USB D1 - >  Frank", does work if Kontakt is set to use "Frank" as a source of MIDI. 

Most interestingly, if I change the iPad over to use the Apple Camera Connector and something like the E-MU which is known to support the iPad USB MIDI then it does work. This implies that when the MIDI commands go through two SILABS F321 chips something is getting lost or munged along the way.  If the MIDI goes through only one by sending it through a MIDI connector (I do not think the F321s are involved in that signal chain) it does work.

(iCM - if you are listening this is a relatively serious issue...)

The same connection that does not work with Kontakt 4.0 does indeed drive Apple Logic correctly as well as MIDI Monitor.  This tells me that its probably at least in part a Kontakt issue but I don't see them fixing it any time soon.

The 3i names its MIDI ports on the Mac similarly to that of the iCM so I am not sure what this issues is - it does not to me appear to be name related though name length could be involved.

A more interesting test was to hook things up using full USB and then blast away with MIDI CC messages at full iPad bore.

As I said above my USB solution works sending these messages at full blast.  However, the iCM, for which I expected to see the same results, did not perform nearly as well.  It seemed like the iPad overran the iCM causing it to store a few seconds of CC commands and flow them out at a much slower pace.

However, after some thought I suspect that actually this is an iPad MIDI issue and not an issue with the iCM.  I think what's really happening is that the iCM is somehow magically tapping into raw iOS MIDI on the 30-pin port with its magical connector.  The iPad MIDI driver is what's actually throttling out the MIDI CC's based on some sort of internal mechanism.

The iCM (and I base this on observing the lights) is just getting them at this much slower rate.

(MIDI has not flow control so I suppose this makes sense.)

From my perspective this is very unfortunate because it means that the iPad is limiting the speed with which it can send MIDI to around 30K baud - no doubt to conserve battery life.

Why Apple chooses to inject these interesting little limitations on these devices is beyond me.  It would be nice if the MIDI were really fast - it would be more flexible that way.  Even nicer if this was a API controllable issue.

But this limitation really means that even if the iCM has the 12 Mb/sec of bandwidth the marketing literature talks about you're just not going to see it in the signal chain from you iPad.  So if I were iCM I would be bitching up a storm at Apple because clearly they are in at Apple to know the magic for the 30-pin connector yet the iPad and iPhone (though I haven't tested it yet) are throttled!

(Again, I understand that older equipment has limitations but this is just ugly.  I guess too that the iCM probably does not have much memory and hence is happy that the iPad ensures that there never will be a buffer overrun.)

Imagine, for example, I want a volume slider on the iPad and I want all the volume messages to go through because I am using it like whammy bar or something where the results need to be relatively continuous.  Certainly this prevents that and yes I know I can quantize my volume messages so there are fewer but that's not the point. 

Why create an iPad that's clearly very fast computationally and supports USB 2.0 very well and then nail on a fifth performance limiting MIDI fifth wheel made out of old, rotting timber?  Sure the MIDI standard limits speed to 30K baud - but don't hard code it because some devices might work faster - like your own!

For the money this kind of makes the iCM a pricey fifth wheel, especially when I can buy things like this USB MIDI at Amazon for $5.00 USD - for $10 bucks I get as much bandwidth.

I was really hoping for that 12Mb/sec of MIDI throughput to include the iOS devices...

Foiled again!

I guess its back to the Gold Standard....

Synthodeon, MIDI and the Future... (Part 2)

So far I have not said anything remotely interesting, at least technically, but hopefully that will change now... 

There are in fact products just like I described in my last post: ipMIDI, a C# MIDI code set, a Cycling 74 patch, QmidiNet, and many others.  All of these products, though, treat MIDI as a simple end-point-to-end-point topology that does nothing beyond the basic MIDI capabilities.

But my interest here goes beyond this basic ability.

First off, when you are working with software samplers, e.g., Kontakt, you have the ability to organize multiple samplers on the same MIDI channel or duplicate samplers on a set of MIDI channels.  This ability allows you to have much finer control over what is played for a given set of MIDI values as well as allows you to do things that you otherwise could not.

In terms of something like, say, Guitar samples, its often the case that there are a lot of articulations available for a given sound, e.g., strumming up versus strumming down, starting on one string, ending on another, mutes, harmonics, and so on.  Many software samplers today cram these sounds into chord sequences, e.g., I hold down "C" in on section of keys and I use "A#" in another to strum a C chord.  Some samplers allow you to control what chords are being sounded (on the fly or with setups) and others do not.

What happens is that the complexity of what is being (or needs to be played) quickly and exponentially gets out of hand - there are too many ways (inversions, where on the "guitar neck" your hand is, and so on) to play a "C" chord, which form of "C" is appropriate for the piece I am playing, and so on.

At a low level MIDI is getting the "note up & down" events to the sampler but the "sound" the musician is asking for is somewhat beyond this basic ability.

While things like MIDI CC values could be used to address this issue they are not generally because picking you own choices for specific CC values may be "non-standard" and would render any software or hardware that did so incompatible with a lot of existing MIDI equipment.

Even something somewhat standard, like breath control (velocity) MIDI values, are not always handled correctly.  For example, I've played an EWI 4000s live for several years using either Mr. Sax (from samplemodelling.com) or Roland XV-5050s with the Patchman Sound banks (BTW, I cannot say enough good things about these sound banks).  Some out-of-the-box 5050 patches works with a breath controller, some do not (patchman ones always do, even on the XV-5050, as they were designed for this purpose).

Mr. Sax uses a variety of CC controls for special things while playing - which is inconvenient if you do not have a way to generate them.  This is particularly true live - I have enough to haul around as it is without yet more knobs and pedals to munge CC values.

For the more sophisticated guitar functionality you would like a controller to be able to do "more" for the player in terms of triggering samples in response to the player's hand movements.  There are many cases where one MIDI channel is simply not enough to do this.

Take, for example, note bends.  On a synth keyboard you typically bend the whole keyboard with the pitch wheel, i.e., all the notes change in response to the wheel.  On a faux sax or clarinet with the EWI you bend a single note (by biting the mouth piece).  On a guitar you can bend multiple notes differently and simultaneously.  MIDI does not help you in this regard.  (Not to beat up MIDI - it was developed before the things I am talking about were even possible.)

Now throw into the mix your typical keyboard layering and/or splits.  Now I really need multiple MIDI channels per split to do anything beyond basic notes.

Then there are devices like the iOS ones from Apple (and well, possibly, as Android).  Playing them wireless in the context of a guitar or keyboard where exact millisecond  timing is required is a show stopper - its simply not reliable enough.

(Yes you can create a private network on your Mac, etc. but I've not had satisfactory results much less taken it into a paid live gig alone or with other musicians.)

You really need a wire to get this done.

(At least Macs work live - I've played a variety of samplers and Logic live without problems.  I guess the same is true for PCs but I personally would not rely on one...  BTW, I am no PC bigot and what I will be posting going forward will certainly apply to PCs as well.)

So the bottom line here is that we need more than just a single MIDI channel between a controller for modern samplers if we want to get anything done that gives us much closer control that we get for a single keyboard, etc.

My solution to this is very, very simple: combine collections of MIDI channels into (sub-)groups, or what I will refer to as simply groups and sets of MIDI virtual ports into what I will call Multis.

Now MIDI makes this easy on a computer because ports have names, e.g., Kontakt Virtual Output.  These names are derived from the software or, in the case of hardware, from the device itself as part of the USB standard for modern MIDI devices.

Devices are often, even in the wild, named sequentially with simple numbering, e.g., E-MU Xmidi 2x2 Midi Out 1, E-MU Xmidi 2x2 Midi Out 2, ..., so it makes sense to think of sequentially numbered devices as a Multi.

But fortunately we don't have to worry about actual devices for this abstract model.  If we have software that simply creates groups of ports through the local OS we can create and name them any way we like, e.g., Port 1, Port 2, Port 3.

Secondly, because in the world I describe we can support any number of MIDI port plus have a fast, reliable means to talk between the we can have a relatively large number of MIDI channels in a Multi, say four, and still achieve very high performance.

So, for example, in the diagram below we have "A MIDI 1" a (Single) logical OS-based MIDI port we've created on Node 1 that's linked to Node 2 (or to other random Nodes).  We also have a set of MIDI ports "B MIDI 1", "B MIDI 2", and "B MIDI 3" that we want to think about as a Multi.  We name this Multi by dropping off the space and number to get "B MIDI".  Any port the follows the template of "B MIDI x" where x is a sequential number is included.

We divide "A MIDI 1" up into groups of, for example, four MIDI channels - groups being named 1, 2, 3, and 4.  We also divide the Multi "B MIDI" into twelve groups of four MIDI channels.  Groups are numbered from one on the "first" MIDI port and consume units of n channels per group until you run out of channels (n is always less than 16 - at least for now - though I suppose there is no reason n couldn't be 32 or 18 or something else like that).

So what we have done is create logical groups of MIDI channels into a simple grouping and naming system.  There's no requirement as to how many or few channels fit into a group, or how many ports into a Multi - those are decisions based on the application.

So what?

You might love MIDI as it is and hate me for what I am saying - so be it.

But face it, as I said in the last post, MIDI is like a Platypus - its here, it works, we love it but its not the best we can do and its certainly not the latest design...

For one thing when we talk to a sampler it makes more sense to talk about a sound using a group rather than a channel.  (Certainly one sound can be one group which is one channel - which is what MIDI is today.)  But if I want to create a more complex controller that works like some of the guitar-based sampler keyboard inputs I can (no pun intended) re-group into a set of, say four or eight MIDI channels, and use the extra bandwidth to do interesting things in the controller to diddle the sampler in interesting ways.

For another we can send metadata along a given Multi between apps that speak the same MIDI protocol at the Multi level.  Because I have good hard bandwidth between, say, Node 1 and Node 2, I can encode things into MIDI and use it as a transport layer - either in a group or as a Multi.  (There are other, more serious reasons to want to do this which will be revealed as the '3i' gets closer to debut.)  This gives me a reason not to invent yet another MIDI over UDP/TCP protocol - though I may release tools that make of this simple to do.

After all this is basically kind of like what's going on with MIDI-controlled DAW control surfaces so why not buy into the same concept for actual MIDI controllers?

In the reality of the 3i what this all really means is that the 3i thinks about the world as groups of MIDI channels - not just one.  Which makes effects beyond what one channel can do possible if not down right easy.   The 3i sort of kind of only cares about output groups and its set involves knowledge of the target sampler to some degree in order to take advantage of what the sampler can really do.

Synthodeon, MIDI and the Future... (Part 1)

As a developer of software for the last 35 odd years and a gigging musician for the last 10 or so years I have come to know MIDI quite well.

Recently Synthodeon has been working on something called the '3i'.  This is, among other things, a MIDI controller that will allow new types of playing that currently impossible due to the fundamental limitations of MIDI.  While I cannot yet talk about exactly what the 3i is and does I can say a few things about what it needs to do MIDI-wise.

First of all MIDI has been tied to the notion of a MIDI cable and to some extent a "keyboard" - after all this is what it was originally designed for.  There are sixteen channels on the cable and each channel supports things like pitch wheels, notes and all the standard MIDI stuff.   Most of today's keyboard controllers support other things like CC messages for knobs and sliders.

But on the whole the "concept" is still very simple: Channels are tied to sound controllers (maybe with splits using extra MIDI channels to support two different sounds or to overlay sounds).  Keys pressed operate within a given channel using Note On/Off messages, and so on.

So what don't I like about MIDI?

First off I don't like the basic model because its so very limited - particularly when you look at what can be done with things like Kontakt or Logic or any other MIDI-able DAW-type software.  These apps have the ability to tie different samplers or functions to specific MIDI channels, i.e., I can have the same sampler on three consecutive MIDI channels.  But, for example, most keyboards can only access a single sampler.

Secondly certain types of MIDI software in Kontakt, e.g., guitars in Orange Tree or Vir2, have funky multi-key controller options for their samples, e.g., you have keys to select a chord or note and separate "strum" or other note-sounding keys, reminiscent an AutoHarp.  This is somewhat awkward and pushes the notion of MIDI away from keyboards - but there is no alternative place to go currently.

Third the MIDI bend (and in general the CC) model is very limited - I can only perform functions at the channel level.  What happens if I want to bend a single note?  Controllers I am aware of don't handle this at all.

Fourth, with the advent of network software (wireless, wired, USB, etc.) MIDI can run over a virtual channel from point A to point B.  However there is no standard model for this.  There is stuff like OSC and others but they are not standardized.

Fifth, things like iOS devices have limited asymmetric MIDI functionality.  What I mean by this is that on an iPad, for example, you can plug in a MIDI device but only through a "Camera Connector" - and then your device must meet a variety of vague requirements (USB and power) in order to work.

Sixth MIDI is just not reliable enough for serious live music over WiFi regardless of what you do - even on local, private WiFi's with Mac's.

Seventh, I have a lot of devices - computers, iOS devices, keyboards, wind instruments.  I want a MIDI model that represents each of these "places" on some sort of virtual "MIDI backbone" so that I can plug something into one device on the backbone and have that connection available at other points on the backbone.

Now I am sure there are a lot of arguments about why I am wrong, or why such-and-such a specific tool can overcome one of the limitations I listed above, but the bottom line is MIDI is twenty-some years old and really hasn't kept up with modern times.  I want something that works in a modern way.

(Can you think of anything else you use that old on a daily computer or music basis save something like a physical guitar?)

I can't.

The oldest stuff I have that I use all the time are things like studio monitors and vintage synths - and most of that is not twenty years old.

So I've been looking around and found this site (OpenMuse) which talks somewhat about this in general.  There are links to some standards efforts which mostly appear to no longer exist.  There are also some patent links to US Patents claiming some form of jurisdiction over MIDI over networks.

Irrespective of any "standards" efforts there are a couple of patents to note.  First there is U.S. Patent 5,983,280 "System Using Standard Ethernet Frame Format for Communicating MIDI Information Over an Ethernet Network" and U.S. Patent 6,353,169 "Universal Audio Communications and Control System and Method."

The first patent ('280) has a UK assignee.  The patent discusses MIDI using Ethernet broadcast of various forms, multicast, and so on and it appears to be protecting some proprietary hardware and protocols.  (See this for a reasonable discussion with which I would concur.)

The second patent, owned by Gibson, uses MIDI to control audio.  Again I do not see this as relevant here.

So neither of these patents from my reading address what I will be describing here (though I am not an attorney and my opinions are worth exactly what you have paid to read article). 

Are there more patents in this area?  Its hard to say (and this is a reason the US Patent system is flawed - there is no way to discover what else is out there for a given topic).

The fact that there are numerous software apps (various OSC things, etc.) out there doing this sort of thing in one fashion or another leads me to believe that if there are they are probably not valid or everyone out there would be getting stomped on.

All that said what I want to see a MIDI software topology, at least at an abstract level, is something like this:

Here each "Node" (Node 1, Node 2, etc.) is a physical device (an iOS device, a computer, etc.).  The MIDI ports listed on each device are virtual, i.e., not physical.  If I need to link things to the outside world on a given device I can use some sort of MIDI port connector or bridge software to do that.

I want fast, clean delivery of MIDI messages - plain and simple.  No other fancy nonsense to get in the way, no wizards, no autoconnect please press okay.  (Who wants their shit to go dead during their epic solo?  Worse, who want's to thrash through menus and crap trying to fix it afterword while the audience and your band mates stare at you with hate-filled grimaces.)

Basically I want to create MIDI endpoints between devices easily, reliably, deterministically and cleanly.

I want a physical wire (green arcs) between the Nodes (USB speed or better) because I want sub-millisecond response (WiFi is okay for testing and noodling around but not for live).  To play live you need this along with the headroom it offers (for really fast pickers).

(I measured my response via MIDI on moderately fast piano piece - I can hear down to about 2.5ms - 3ms between chord strikes.  I am not a pro and I cannot play really fast but my ears are pretty good.  Sub-millisecond leaves a factor of about 2x which should be enough.)

I don't want automatic or automagic connectivity via something like Bonjour because for live gigs I don't want software "helping me" set things up.  I don't really want to (accidentally) be on the venue WiFi (where some clown will start downloading porn videos to his iPhone just as I hit the first note in my solo).

Instead I want hard coded/wired links via fixed device-based TCP/IP settings over deterministic, private connections that stay the way I set them up and come back up spot-on. (Who hasn't had the circuit breaker or generator go out in the middle of a gig at a festival?)

I don't know if a local BlueTooth network is fast enough for this - especially outside of a "computer deveice", e.g., iOS.  I have done a lot of testing and so far all I can say is "I am not sure."  I am also concerned about BlueTooth interference (audience members with lots of active BlueTooth gear on).

From a device perspective Mac OS X works reliably on stage - at least in terms of no OS, audio or midi glitches and no battery fails - networking and iOS wise this all remains to be seen.  I do not yet know about Android for this type of application though from what I have been reading its still immature in the developer area.