Synthia - A 2 player light and sound instrument

Synthia

Synthia is a 2 (or more) player light and music instrument.



It was shown at the Santa Cruz Museum of Art and History GLOW event:

Concept

It all started when a friend asked me to collaborate on a Burning Man art project. She had an idea for a cool octagonal gazebo with a roof made out of research day-glow orange solar panels. The problem was, what on earth are we going to power with this?

"I can fix that problem." I told her :) Before this, I'd considered doing some interactive light projects, but was feeling that light alone isn't very engaging. I'd wanted to do some kind of free form simon says thing or light-pad sound instrument.

The gazebo was going to be octagonal, and the number 8 struck me, and I thought of 8 directional stations, each with 8 buttons. Each station could control a different instrument, encouraging collaboration and cooperation. Each station could send pulse of light along an LED strip, perhaps simulating gravity and falling back to earth.

We didn't end up getting the grant for the project, but I had this idea stuck in my head and wanted to build at least part of it!

Structure

I scaled it down to 2 stations, facing each other across a play field. I'm no sketchup pro, but this is roughly what I was thinking: 
(for scale, consider the cross intersects around 3' up.)

The solar panels ended up being glass; they are heavy and kinda dangerous. A little to dangerous for being overhead and had a large enough surface area that BRC winds would have been a serious problem.

I ditched the roof, moved the solar panels to their own structure (basically a lean-to), and reinforced the play field structure to the point where drunk people couldn't cause harm and people could lean on it, park a bike, etc.

I wanted it to be more visible when viewed from the side (e.g. from far away), and thought it would be good to bow the play field, giving some extra vertical dimension from the side. I also liked the idea of clear plastic over the strips.

This lead to changing the X pattern a bit, and used it to bow out the play field. This is what it looked like under construction: 

Component overview

Brains

Synthia is powered by a Rasbery Pi and a pair of Arduni Pro Minis.

Why Pi? There was something attractive about using a Raspberry Pi and the challenge of getting everything to work on it. Also I wouldn't mind as much if the playa killed it than a laptop.
With a little bit of microcontroller glue, the Pi's serial port could be used to drive the 960 NeoPixels. Unfortunately NeoPixels max out around 800kbps and 960 pixels, the maximum framerate would be around 35 fps and with some noticeable tearing as each pixel was drawn successively.

A low refresh rate meant I'd need to combine frames or the light would look like it skipped over some pixels when things were moving fast. Experimentation showed that these look much better at high framerates.

I ended up grabbing a pair of FadeCandy controllers. Internally they drive something like 400fps. They also present a really easy to use network protocol.

Here's an early prototype of working sound and light:

Input

There are 16 buttons and 6 pots to scan, and the Raspberry Pi model B doesn't have that kind of IO.
I delegate this to a pair of 3.3v Arduni Pro Minis. They are connected to the Pi via the UART, creating a bus so that both stations could transmit over the same line. Also, daisy chaining the button pixels off of the end of the strips didn't work well - there was too much distance and they were unreliable and would flicker. So the arduino also controls the pixels in the buttons.
A resistor pulls the Pi's RX bus line up to 3.3v. Each station may pull the line low via the diode while transmitting.

Each station needed a good number of keys, and should light up as well. I wanted something decidedly not like a piano keyboard, and ended up with an ergonomic arcade layout. There needed to be at least 3 dials for pitch, scale, and instrument. I found some awesome 10-turn pots for the putch and instrument dials which helps when there are 70 pitch settings and over 100 instruments.

It took quite a bit to get the stations assembled. The plastic was easy enough to drill, but took some dremeling to get the buttons to fit. I also learned that soldering the buttons while they are pressed can damage them (I had the station face down, working on the underside, which pressed all the buttons).

The clear arcade buttons from adafruit worked perfectly. With a quick drill, the 5mm neopixels fit perfectly and light up the buttons beautifully.

Assembly

To get everything right, I created stencils in OmniGrafle that I could print at 1:1 and use for drill guides.

Small holes in the play field plastic allow zip ties to hold the strips up.

The main structure wasn't complete before it was time to leave. At least I had got all of the googling out of the way first :)

To keep things simple and timely, it's mostly a 2x4 frame with some plywood.

Software

What language to use? Python was my first choice.

Music synthesis with fluidsynth was fairly cpu intensive on the Pi with around 60-70% cpu and spikes to 100%. I couldn't play some midi files without stuttering, but I found that I could downgrade audio to 22KHz and save some cycles. Next up was sending data to fadecandy. This is where the wheels fell off; it ended up taking WAY to much cpu just to draw pixels at modest framerates. There wasn't enough CPU to play music and draw pixels.

The FadeCandy server itself wasn't taking much CPU at all, and I've really come to appreciate the optimizations they've made there.

I tried node, and it was better but still took a lot of cpu trying to drive 960 pixels at 100fps. I dusted off my old standby, Java. I got 100 fps and CPU use was less than 25%, leaving almost enough for music synthesis if Java took about the same CPU as fluidsynth.

Now to figure out a way to get a software synth running in Java and loading a soundfont (the built-in instruments are low quality). Turns out thats not so hard, but the documentation is sparse. I spent a lot of time reading old Gervill source code.

The station input needs to be polled at a fairly fast rate to avoid missing short keypresses or introducing delays.

For serial, first I tried Pi4J, but found that there's some issues. It doesn't have an interupt or blocking based serial input and polls on 100ms intervals. That kind of input latency/jitter would be really distracting.

I tried RxTx, which worked, but found CPU use was quite high.

I was digging around in Pi forums and someone mentioned ser2net - wow, ser2net is amazing. CPU was was very low, and it was very easy to set up. I would recommend this to anyone doing serial on a Pi (or any linux based system).

The Pi polls each station one at a time so that they do not talk over each other. Each station waits for it's ID, grabs the pixel data, and sends back current button state and pot measurements.

This was easier than wiring an enable pin to each station, both in terms of hardware and software. On the arduino side, it's as simple as calling Serial.find(ID). Some care was taken to make the ID unlikely to occur in the pixel data stream in case a packet becomes misaligned.

There is a simple checksum (xor) for input data, but none for the pixel data. I may add that in the future, but so far I have not observed any errors. If there are they will be corrected by the next packet.

Simulator

When developing, it was helpful to be able to hack on my laptop without having to go back and forth to try new things.
The hardware interfaces are abstracted away, and there are 3 modes that Synthia can run in:

• Emulated - A window is shown with sliders instead of knobs. I tryed to lay out the keys so 2 people could share a keyboard.

• Partial - main play area is sent to FadeCandy server, the rest is still in a window. Good for testing play field visuals.

• Hardware - runs headless, using both FadeCandy server and ser2net connection. This can also be run on a laptop if the FadeCandy server and ser2net are remotely accessible!

TODO

Right now Synthia is a fun instrument. I think there's some opportunity to add game or interaction-less modes.