Musical Experiments Inspire New Video Game Music Ensemble

An image of a Pokemon spooky face and rail tracks from the Minecraft musical video "Lavender Town" by Tess McCumber Image by Tess McCumber
An image from "Lavender Town," a Minecraft musical experience created by SRT major Tess McCumber that won an award at the 2022 Maker Music Festival.

06/07/2022
By Katharine Webster

Sound recording technology (SRT) major Tess McCumber ’22 grew up playing Pokémon games on her Nintendo DS, and then also became obsessed with the massively popular game Minecraft. 

So when she heard that students in the Music Department’s Contemporary Electronic Ensemble, led by Assoc. Teaching Prof. Ramon Castillo, were creating music videos using Minecraft, she joined up. 

She says the practical experience of synchronizing sounds to video games built on what she had learned in classes required for her major – and gave her a better understanding of electric circuits that will help her in her new career as a sound technician at AudioLink, a company that provides voiceover recording and production services. 

“We had to take electrical engineering courses about sound technology, but I didn’t really understand circuitry until we did this hands-on project in Minecraft,” McCumber says. “Seeing it visually, seeing how wiring and electronics and machines work, was the biggest takeaway.”

An image of falling creatures from the Minecraft musical video "Chicken Hero, Villager Hero, Cow Hero" created by Assoc. Teaching Prof. Ramon Castillo Image by Ramon Castillo
An image of falling chickens, villagers and cows from a Minecraft musical performance by Music Assoc. Teaching Prof. Ramon Castillo. He directs the Contemporary Electronic Ensemble and is starting the new Video Game Ensemble in the fall.
In her first semester in the ensemble, in fall 2021, she watched, learned and helped with a project. During her second and final semester, she created the video performance “Lavender Town,” which combines elements of her two favorite games – a Pokémon soundtrack linked within Minecraft to a mine cart traveling on rail tracks that trigger different game actions.

“Lavender Town” was among 18 UMass Lowell projects featured at the all-video, online 2022 Maker Music Festival, which drew participants from around the world. While most create their own instruments, both analog and electronic, organizer Sherry Huss was happy to accept digital performances this year because the festival was virtual, due to the COVID-19 pandemic.

Castillo encouraged students in his Contemporary Electronic Ensemble class to participate in the festival, which debuted the weekend of May 14-15. Nearly a dozen students did, and Castillo ended up organizing an entire virtual UML building. He spoke about several of the projects in a real-time session during the festival, and he will speak and perform live at A New HOPE hackers' conference in New York City in July, as well.

In addition to the Minecraft videos, the UML building at the Maker Music Festival features a video about the EcoSonic Playground Project, a pair of live performances on original or reconstructed instruments, and two video projects by students from Prof. Gena Greher’s Technology in Music Education class, as well as several of Castillo’s original compositions and collaborations with his 8-year-old daughter, Luna. 

Four of the UML projects won awards at the festival, including “Lavender Town” and the EcoSonic Playground.

An image from the Minecraft music video Motorbike, with connected sound and control blocks, by Kyle Kashiwabara and Matt Makuc Image by Kyle Kashiwabara and Matt Makuc
An image from "Motorbike," showing arrays of connected sound and control blocks in Minecraft that play when the motorbike travels by them.
Castillo says he turned to Minecraft during the COVID-19 pandemic to bring an element of live play and performance back into the Contemporary Electronic Ensemble. He was inspired in part by a musical video game that guitar performance major Sean Lavigne ’21 created in fall 2020.

Castillo was also inspired by his daughter, who plays the educational, multiplayer version of Minecraft, and the depth of electronic music possibilities the game offers.

“You could see the other person and interact with them in real time, and that’s exactly what we lost during remote learning,” Castillo says. “So using Minecraft in the ensemble seemed like a way we could get that back, at least to a degree. Every project the students created had to have an element of live performance, too.”

The live performances took place at the end of every semester on the social media platform Twitch.

Castillo had already begun some musical Minecraft experiments of his own, a group of works he calls “void loop ().” In “Chicken Hero, Villager Hero, Cow Hero,” chickens, villagers and cows fall from different heights onto virtual pressure plates that respond to their “weight” by triggering in-game sounds and actions, via circuits constructed of “redstone,” an electrically conductive element within the game. 

Performance major Tristan Fruzzetti performs live, using video images from Minecraft and Logic software as a green screen and accompaniment Image by Tristan Fruzzetti
Performance major Tristan Fruzzetti used Minecraft video clips and an audio track to accompany his live guitar performance.
Castillo’s ensemble students took a variety of approaches in their Minecraft musical works. Some, like McCumber, created visual accompaniments to existing music, while others created the music itself using different machines within Minecraft. 

Vocal performance major Kyle Kashiwabara and composition for new media major Matt Makuc used stop-motion animation in “Motorbike,” another award winner, to simulate a motorcycle traveling along a road. As it travels, it activates layers of redstone-connected sound and command blocks. (Command blocks contain a line of code that can make a more complex sound or prompt another action within the game.)

Some students created both visual and musical accompaniments to live performance. In “Pillager Jam,” which also won an award, SRT major Tristan Fruzzetti created a video with an audio track and visuals from Minecraft and Logic digital recording software to accompany his solo performance on electric guitar.

The Minecraft musical projects have proved so popular that Castillo is starting a new ensemble in the fall, the Video Game Ensemble. The Music Department is funding a dedicated Minecraft server that Castillo and his students can use and modify. Castillo has already built a virtual replica of UML’s Durgin Hall inside it.

A good friend of Castillo’s, Alex Rigopulos, the co-founder and creative director of Harmonix Music Systems who helped design the original Guitar Hero and Rock Band games, has made a generous donation to fund some add-ons and other technology for the new ensemble.

Image of The Meteorologist, an instrument that responds to atmospheric conditions and human touch, created by SRT major Ryan Katz Image by Ryan Katz
The Meteorologist, an original electronic instrument created by SRT major Ryan Katz, responds to the weather and human touch.
Meanwhile, now that ensembles are live again, the Contemporary Electronic Ensemble will return to its original purpose, Castillo says. Typically, students in the ensemble learn how to use electronic music software and synthesizers expressively in live performance. Some even create their own electronic instruments.

Ryan Katz, an SRT major and rising senior who has played in the ensemble for four semesters, stuck with creating and adapting electronic instruments and sound banks during the pandemic, rather than migrating to Minecraft.

One of his videos in the 2022 Maker Music Festival, “Interstellar Bone Collector,” features a live, improvised piece he performed with two friends, using a Yamaha keyboard he’d rewired to create new sounds, a practice known as “circuit bending.”

His other video features “The Meteorologist,” an instrument he invented to respond to the weather, using sensors that measure atmospheric and ground conditions including light, temperature and soil moisture. A human player adds more chaotic and percussive sounds by manipulating a joystick, knob and trackpad.

Although “The Meteorologist” starts by playing a simple chord progression, it never plays the same piece twice, Katz says, because it responds to the weather and human touch at that time and location.

“I wanted an intersection between what we've created and what the environment creates,” he says.