Hackable Audio: Crafting Sonic Puzzles for Google

In the summer of 2022, Antfood worked with Google’s Creative Lab to develop the sound of Hacking Google: a six-part series of never-before-told stories delving into the work of Google’s five elite security teams and how they keep us safe online.

Hacking Google is not just a series for watching’s sake. It is simultaneously an urgent PSA, notifying us about the myriad ways we are vulnerable online, as well as a way to drive interest in the information security area.

To further engage budding “ethical hackers”, the Creative Lab wove a number of puzzles and challenges into the Hacking Google series, giving rewards to the first people or teams to find the answer.

In addition to composing an original score for the series and devising a unique sonic language for the cyber world, Antfood was given a special task from the Creative Lab:

Incorporate hidden messages into the soundtrack in unconventional ways.

Game on.

Though Hacking Google focuses primarily on recent technology, a number of scenes delve into the early days of the internet and the tech of that era, including landline phones.

All phones use a consistent language for dial tones, known as dual tone multi frequency (dual tone because the sounds that emanate from a dial are always the combination of two tones).

For example, when you press number 1 on a dial pad, the phone simultaneously plays out 697 Hz and 1209 Hz, resulting in the sound we’re all too familiar with.

This language of matching numbers with sound on the dial pad is global, so when a dial tone sound effect was needed in Episode 0, we took the opportunity to hide a phone number in the sound design, which was discernible and translatable by anyone with a functioning phone. Our audio led viewers to a toll free number, which, when called, revealed the next step in the puzzle.

Musical cryptography, encoding messages with the notes of a song, is nothing new. It can be traced back to the 1600s, as a technique that Bach would employ to cheekily sign his name in compositions. He would use the notes B, A, C, and H (in Germany at that time, our present day B was known as an H, and our present day B♭ was known as B) in that order in key moments, though, true to form, Bach tended to reverse and invert this motif in his compositions as well. While Bach would use the literal letters of the notes to sign his name, not all composers were so lucky to have names with only the first eight letters of the alphabet.

French composers in the 19th century devised a more repeatable system of musical cryptography, where each of the 7 musical notes (A-G) has 3-4 different possible letter equivalents, similar to how the keypad on a telephone has 3-4 letters for each number. This gives each combination of notes multiple possibilities, leaving the exact message up to interpretation, but solves the problem of musical notes only referring to a few letters of the alphabet.

Google asked us to hide the password “One Hook” – a theme that surfaces in the series – in Episode 2 using musical cryptography.

Inspired by the history of cryptography in Western classical music, but interested in pushing our method into the future, we asked ourselves how we could encode messages into the notes of our compositions in a way that 1.) connects to the subject of the film and 2.) is decipherable by nonmusicians.

In the system we concocted, called QWERTY Pentatonic, we took notes from the D minor pentatonic scale and assigned them to keys as oriented on a traditional QWERTY keyboard, in ascending order starting on Q. QWERTY Pentatonic has strong ties to the subject matter of Hacking Google, incorporating the computer keyboard as the mode of deciphering the hidden message.

Using the pentatonic scale as the source of our musical notes functioned on two levels. First, the pentatonic scale is a particularly consonant scale, meaning that notes generally work well with other notes in the scale, so no matter which word we spelled, we knew that the resulting melody would sound nice.

Secondly, a pentatonic scale is more spaced out than a traditional major or minor scale, with five notes per octave rather than seven. Because there is more distance between notes, a listener can more easily distinguish between pitches in the scale.

Antfood chose to employ Morse Code as a method for embedding messages into the sound design because it is not only a sound-based pattern language used around the globe, but it also provides interesting rhythmic possibilities with dots and dashes giving us short and long notes, respectively.

Google tasked us with hiding a URL in Morse Code with the series’ sound design, but the thing about this system of communication is that it is inefficient. Each letter is denoted by multiple dots and dashes. For example, J is ᐧ — — —, which takes a lot of time to get through (1.4 seconds if we’re going by the book). For just one letter, that is a lot of time investment, especially in the context of a series, so we divided the URL into six pieces and snuck a piece into the sound design of each episode after the opening titles.

Perhaps the most challenging and best concealed audio puzzle we created was a QR code, which could only be revealed through analyzing the audio track from Episode 2 through a spectrograph. A spectrogram (the image that a spectrograph produces after analyzing audio) functions in a similar way to infrared — but for sound — where a visual depiction of the frequency content of a signal indicates which sounds on the frequency spectrum are most intense.

We were inspired by some recent instances of hidden messages revealed through spectrograms in popular music and video games. For example, the soundtrack to the video game Doom encodes satanic messages in its score, which some astute listeners were able to spot.

In the popular music world, Aphex Twin hid various images, including the face of their lead singer Richard D. James, in their song which is now known as Equation.

We built a tool in Max MSP through which we could take an image and convert it into sound. With this tool, we could have encoded any image into the sound design, but Google chose to display a QR code which, when revealed and scanned, would lead participants to the next step in their hunt.

The challenge with encoding images through a spectrogram is that in order to get a clear depiction, you need to use audio that is oftentimes non-music and even abrasive to listen to. So instead of trying to weave a QR code into the musical score, we incorporated it into our sound design, using this particular moment to accompany a moment of scrolling on a screen and flickering numbers and symbols.

Incorporating messages and puzzles into our score for Hacking Google gave Antfood the opportunity to connect with our audience on a deeper level that we are usually able to. We not only threw down a challenge for viewers to solve, but we also communicated literal messages to them.

As we saw, the practice of encoding secret messages within music is not a new phenomenon, but with audio technology improving and becoming more accessible by the day, we have more opportunities to drop surprise-and-delight moments into our music and audio, which allows us to convey even more information than what is initially audible in our score, treat our most engaged listeners to unexpected easter eggs, and send literal messages that lyric-less music often cannot.