The S+V Interview: Rock Band Blitz Audio Lead Joe Kelly

Harmonix Music Systems redefined the peripheral-based rhythm genre with Rock Band in 2007. Five years, two numbered sequels and a few notable band-specific spin-offs later, their fans have a huge music library at their disposal. Thanks to smart thinking at the outset, almost all of the material from each successive game release has been playable within the Rock Band ecosystem (The Beatles Rock Band notwithstanding). There's somewhere in the neighborhood of three thousand songs in the Rock Band catalog if you count the community-authored tracks available on the Rock Band Network store.

The problem is, no one wants to buy peripheral-based games anymore. The novelty's over and isn't coming back anytime soon. Despite this, Harmonix has been keeping Rock Band fresh by releasing new songs and albums each week for players to download. This is great for hardcore fans, but not everyone wants to go through the hassle of getting the band back together because they want to play a few new tracks. That's where Rock Band Blitz comes in.

Almost every song from the past half-decade works within this new game, but there's a twist. Blitz isn't played with a peripheral, it's played with a regular controller and you only need two buttons to rock out. I wanted to know how this even works (since retooling the gigantic back catalogue of music just couldn't possibly be feasible), so I called Joe Kelly, who's a composer and sound designer at Harmonix. On Blitz, he was the audio lead.

The Blitz released the other week on Xbox Live Arcade and the PSN store is much different than the early prototypes (which used custom-authored content and had three-note, rather than two-note lanes), but I'm happy how the final product turned out. Kelly told us all about the process that led to the release version.

Was cross-compatibility with all the music the plan from the beginning with the Rock Band series?

Not outside of the instrument-specific games. Blitz is a whole new thing that was thrown together and we really had to struggle a lot with the concept of using the music we authored for a completely different purpose. There were definitely no plans with the original Rock Band games for this.

How does Blitz decode all information from the previously released music if it wasn't planned for it from the beginning?

In the very beginning we had a prototype (which [Harmonix Design Director] Chris Foster refers to as Pre-Blitz,) where everything was custom authored. Instead of a two-note lane, we had a three-note lane and everything had to be specifically authored for that game.

Specifically for drums, the analogy we made was tapping on your steering wheel to a beat with your fingers. We custom authored some songs that would reflect that. We have our left and right lanes based on which hand you would be tapping with on the steering wheel. We authored those lanes for the music. Once we had to convert all of the songs, we took those original custom-authored ones and tried to see how we could get that out of what we already had in Rock Band. That went through tons and tons of iteration and experimenting with exactly what beats needed to go where.

For the drums, there's a lot of kick and snare. For right-handed drummers, we'd take the kick and put it on the right and the snare would be on the left [of the note highway]. That was the most rudimentary concept of it. From there we had to use a lot of different information. For the drums, we're actually taking the animation data. With Rock Band, whether the drummer is playing with his right or left hand for any particular drum, it was pretty easy to take that and convert it over to every one of those hand hits being used for left and right drum gems in Blitz.

So with every drummer, did you encode the information for each song with what hand someone was using? That sounds like a ridiculous amount of work.

When Rock Band songs are authored, we do the expert, hard, medium and easy note charts. But then on top of that, the animation data is authored as what hand the drummer's going to play in the game. All that information was custom done for each song already. That was one of the ways we tried to take what we had already done and use it in a way that was intuitive for Blitz.

The one issue there is that we didn't have any kind of information like that for the guitars. We didn't have left and right hand information for that, so the analogy I made from the beginning when we were custom authoring things was an up or down strum on a guitar, or up and down picking. That's the most basic way you can represent rhythm for the guitar. What we did was we looked at the expert and the medium difficulty tracks because they're all authored on the same MIDI track. You'll have notes that exist in both the expert and medium difficulty and ones that only exist in expert because we basically take the expert track when we're authoring for Rock Band and we'll strip it down to its more basic or more important elements for the medium difficulty.

Those more accented beats are what go into the medium difficulty and everything else kind of gets left out. What we did was those more accented notes was put them on the right lane and the less accented notes that don't exist in the medium difficulty get put in the left lane. Then there's a little more that goes into how many notes are in that left lane in a row to make sure it's not just a huge string of left notes that go on for too long.

All of the calculations and the gems and note placement, that's done on-the-fly?

Yeah. Once you actually select your power-ups and go to start that song, that loading screen that takes about three seconds is where all that processing is done.

What's been the biggest challenge in keeping the DLC working across each game or keeping it working within Blitz?

It was mostly more a matter of taste than anything. It was a matter of keeping it fun and making it feel musical because in our original prototype and very, very early in Blitz we had three lanes instead of two. There were a lot of reasons for the switch to two, mostly because it doesn't make as much sense for the controllers. It makes sense for us to just have a left or right lane and you can use a thumb or finger for each. The concern with going to two lanes is it would feel less musical and feel more arbitrary. But once we found something that represented each of the instruments with two lanes it ended up feeling a lot better. The biggest challenge was simply making the two lanes feel musical and feel relevant to the song and feel unique to each instrument.

Instrument uniqueness was a big problem we struggled with for a long time. I think we probably did achieve it once we went to our single difficulty scheme because originally with Blitz, you could choose your difficulty and because of that, we weren't able to use the medium difficulty, the medium and expert difficulty technique for note placement we use now. Once we made that switch and said you're always playing on expert, we were able to do a lot more and it opened up more possibilities for how we're converting these tracks and made it feel a lot more musical.

Was the decision to always play on expert difficulty a function of not having to worry about failing out of a song from too many missed notes? It's not about survival anymore, it's about getting a high score.

Those worked well together, but that wasn't really the intention. The no-fail was basically another kind of diversion from Rock Band, where we wanted to say it wasn't about playing accurately enough to just be scraping by to the end. It was more about just trying to get a high score. It's kept a little bit more casual by the fact that you never feel particularly bad about your performance and it just wouldn't make you restart if you weren't doing well. It was just about doing better each time you played.

With the new songs that come with the game, were those chosen more with making them more fun in mind, or with them making more sense for Blitz, or to make them work better for the instrument-based games?

Definitely the instrument peripherals were the focus and will remain the focus for Rock Band DLC. Because we had such a huge library to begin with once we started Blitz, we can pretty much guarantee anything that comes forward will not be such a huge diversion from that. We can guarantee that because we knew going in we had thousands of songs to work with that any future ones, as long as they make sense for Rock Band 3 then they'll make sense for Blitz.

If somebody bought just the pro versions of the songs for Rock Band 3 does Blitz read those differently when it's building the note charts?

Pro isn't taken into consideration for Blitz. When you have that separate DLC to upgrade those songs, those songs aren't loaded at all, actually.

What was it like keeping the balance of being able to keep the players informed without overwhelming them? There's a lot going on onscreen at once.

The biggest trick with all our games is to keep all the sound effects from being too musical while you're in the song. We couldn't have very specific pitches and durations because they'd conflict with the music so it's really a matter of keeping everything synthesized and non-musical and non-pitched just to be able to give you feedback and have it not interfere with the song.

When you miss a note it's very obvious, and a very unique sound says you've missed the note. It's kind of like a different aspect of the sound wave, like a dissonant sound compared to what's being played on the song.

It's as different from the instrument [as we could make them]. The "miss" sounds are instrument specific, so when you're playing the drums you hear these really powerful booming drum sounds, but then those miss sounds are more like sticks clicking together and are very obvious mistake sounds to keep that dissonance, yeah.

You said you were using MIDI for the tracks. Is there a specific codec? What are the limitations of working with MIDI as opposed to a .WAV file?

The audio itself is still encoded as a form of .WAV file, but the MIDI information is only for the actual authoring of the tracks. No real information is taken from the audio itself for what you're playing. It's all custom MIDI files that specifically place notes on the tracks.

Was there anything you weren't able to do with the audio in Blitz that you did in the instrument games?

Not too much. It's branded as a Rock Band game, we were able to do most of the stuff, the crowds. Crowds are a good example that actually were not in the original [prototype]. That came in halfway through. It's definitely mostly the same, I would say there's less focus on the crowd because that's not the environment that the game's in. It's not like the instrument Rock Band and we're not going for that simulation of you're playing on a stage, the whole metaphor.

The environment is completely different, there's different visual and audio [directions] we're going for. With the crowd [reacts] at the checkpoints, for instance, wasn't in for a long time. There was some contention, some back and forth whether it should be in there at all. Eventually it went in to give those little reinforcements that you're playing a Rock Band game and you do have this crowd that's cheering for you. At the end of a song, you make it to that crowd at the finish line.