On Monday I performed a live version of the Don’t Hold Back Mashup at the Update conference in Brighton (which is part of the excellent Brighton Digital Festival).
Following the performance there were a lot of questions about the setup I used. This post will aim to run through the different hardware and software used, some of the problems that arise mixing multiple videos in real time and a few of the solutions.
The Setup
- AKAI APC40
- Laptop
- MIDI Yoke
- Monomulator (optional extra)
- Ableton Live 8
- Arkaos GrandVJ
- Microsoft LifeCam Cinema HD
AKAI APC40
First of all, some kind of hardware controller is a good idea. While you can trigger clips with your computer keyboard a dedicated controller like the AKAI APC40 is far more engaging to perform with and much more interesting for an audience to watch. The APC40 benefits from a great visual feedback mechanism so you and the audience can easily tell which clips are currently active and which are available to trigger. It uses the MIDI protocol to both communicate with the software used and to trigger the lights within the pads.
There are a few similar hardware options available but the APC40 has a great mix of clip launching pads along with rotary and slider controls.
Also worth considering are:
- AKAI APC20 (the same as the APC40 but without the rotary controllers)
- Novation Launchpad
- Monome (uses OSC rather than MIDI)
- Livecontrol for iPad
- FaderFox Controllers
- Native Instruments Maschine
Laptop
While you could carry your desktop to a gig, it’s probably not the best idea.
Laptops are pretty powerful these days but if you are going to be running high quality multi-track audio along with multiple video then you will need something with a reasonable spec. The laptop I use is a Dell Studio 1558:
- Intel Core i3 2.13Ghz
- 3GB RAM
- Windows 64-bit
Midi Yoke
Midi Yoke is a great MIDI routing driver for Windows (IAC does this on Mac) allowing you to send MIDI between applications. While both Ableton Live and GrandVJ are capable of communicating via ReWire I found using Midi Yoke to be a little more straight forward as there is no need to send transport information such as track position etc. between the two applications. It also allows you to use the excellent Monomulator application.
Monomulator (optional)
Note: While the Monomulator was not really used in the performance at Update it is part of my setup and gives you some great additional options if you own an AKAI APC40 or Novation LaunchPad.
I’m a big fan of the Brian Crabtree’s Monome projects, especially the original 40h seen here:
The Monomulator is a great, free Max/MSP application by TrackTeam Audio that takes the input from the APC40 and remaps the MIDI messages to increase the ACP40′s functionality, run Monome apps and output OSC.
In standard mode the Monomulator takes the MIDI messages and simply passes them though onto Ableton Live (as if the Monomulator was not there). Pressing combinations of the Shift and Send buttons on the APC40 selects different modes, which remaps the APC40 controls giving you a fader mode, a keyboard mode and a Monome mode (allowing you to run monome apps). This can be a little tricky to set up but Banginclude on YouTube has recorded a couple of very useful videos showing the setup and use of the Monomulator.
Ableton Live 8
Live 8 is fantastic for audio performance and as it’s name suggests it is heavily geared towards live use so it is stable, fast and inspiring to use. In this setup Ableton Live deals with all the audio in the performance and also sends MIDI messages on (using MIDI Yoke) to GrandVJ (which deals with all the video).
Click the image below to see a screenshot of the Update performance file:
The track is arranged into scenes (horizontal strips) with each scene containing the audio file to be played on the left and a series of MIDI files that will then send data on to GrandVJ via MIDI Yoke on the right. All clips simply send a MIDI note matching the length of the audio clip they represent. The red clips send a MIDI note to GrandVJ which stops one of the layers from displaying video.
Triggering the scene on the APC40 will play the audio in Ableton Live and also the MIDI clips which are then routed like this:
Each of the MIDI tracks outputs to MIDI Yoke (in this case MIDI Yoke 7, but you can just pick any unused one) and importantly outputs on MIDI channel 2. This is because the APC40 uses MIDI note on and off signals on channel 1 to communicate with Ableton Live. This helps stop any accidental triggering of clips as the MIDI data going from the APC40 to Ableton Live will always be on Channel 1 and the data going from Ableton Live to GrandVJ will be on channel 2.
Arkaos GrandVJ
There are a few options out there for controlling video in realtime. GrandVJ is nice as it has two performance modes, is very stable and has great MIDI integration.
Click the image below to see the Arkaos File for the Update conference:
Each video is assigned to one of 8 layers that can then be controlled in the right hand panel. This works much in the same way as layers in Photoshop. All the clips assigned to the top layer are image files set up as masks, creating the white borders around the videos.
You can take a look at the MIDI mapping of the video files by clicking this image:
As you can see each of the video files is MIDI mapped to activate when it receives the corresponding MIDI note on channel 2 from MIDI Yoke (sent from Ableton Live). In the layers panel the ‘Kill Layer’ buttons each have a MIDI assignment that comes from the red clips in Ableton Live. This stops videos from a previous part of the track continuing to loop when a new section is triggered.
You will notice 3 additional MIDI mappings on channel 1. These are set up to take MIDI data directly from the APC40, in this case the rotary controllers. This way you can control both Ableton Live and GrandVJ at the same time with one controller.
Two of the rotary dials are mapped to control the opacity of a layer in GrandVJ as well as control the cutoff on a low pass filter on a track in Ableton Live. Turning the dial will fade in the video and also filter in the audio (which is a slightly nicer effect than just fading in the audio).
One rotary dial is mapped to control the audio cross fader in Ableton Live and the video cross fader in GrandVJ allowing very quick flipping between the two videos and corresponding audio.
Microsoft LifeCam Cinema HD
Having some kind of visual indication of the actions you are taking during a performance greatly increases the audiences understanding of the process and the music being played. GrandVJ can take a variety of video sources and mix them with video files and a webcam is a really simple way of doing this.
For the performance I just positioned the webcam on the APC40 so that the audience could see what I was doing with the control surface and also in the background could see themselves (helping to prove that it is actually all being performed in real-time, which can be a problem). GrandVJ treats this just as it would a video file and you can assign MIDI to trigger the webcam input.
There are tons of webcams out there. The LifeCam is very straightforward and outputs at a high resolution (if you want it to) so is a good choice for this kind of thing.
System Drawbacks
This setup is definately a work in progress and there are a number of drawbacks including:
Scene trigger only
As each audio files needs to trigger a corresponding MIDI file in Ableton Live you can only control the flow of the track through triggering scene rather than individual clips.
Video layout
The videos have to be layed out manually before the performance so if a video loop is playing full screen in one section of the track and in the bottom quater of the screen later on you will need two video clips in GrandVJ and assign there position and size properties in advance. This is another reason for scene triggering as each scene has videos at different sizes and a border mask to match those videos. Ideally the system would automatically detect the number of videos triggered and assign them in real-time to a place on the screen.
MIDI mapping
While the MIDI mapping system works well it is extremely time intensive to create all the MIDI clips in Live them assign them in GrandVJ and test it all. You need to have a fairly set idea of the track arrangement before you start.
Other options
There are lots of different ways of doing this kind of thing and one to diffinitely look into is JAM by fellow Brightonian Jonathan Hammond. This is a Mac only application that allows you to control visuals through Ableton Live and Serato.
If you know of different solutions for live A/V setups then let me know in the comments.
Also it you have any questions about the setup described then just ask below or contact me on Twitter