Audio engineering student Lluis Estape has released Unwrapped, a browser-based game that transforms any song into a navigable 3D world where instruments are physical objects. Built for Tripothon S1, the project bridges the gap between abstract sound design and tangible interaction, allowing users to walk through a beach landscape where drums, bass, and vocals exist as distinct 3D models. The application runs entirely in the browser using three.js for rendering and the Web Audio API for spatial processing, eliminating the need for heavy game engines or local installations.

The AI-Driven Asset Pipeline

The core innovation lies in how Unwrapped generates its 3D assets. Instead of using pre-made models, the system analyzes the input song using Demucs to separate stems into drums, bass, vocals, guitar, piano, and other instruments. It then extracts audio descriptors like brightness, density, and noisiness using librosa. These acoustic features are mapped to a vocabulary that constructs specific text prompts for the Tripo API. For example, a bright, clean guitar generates a different 3D object than a dark, distorted one, ensuring the visual representation reflects the actual sonic character of the track.

Interactive Spatial Mixing Mechanics

Once the world is generated, the interaction model mimics a physical mixing desk. Users can approach an instrument to increase its volume or move away to silence it, leveraging HRTF-based spatial audio via PannerNodes. The project implements a 'table' mechanic where users place effect pucks next to instruments to apply processing. The underlying Web Audio graph is rebuilt dynamically every frame based on the proximity of these pucks. This allows for real-time chaining of effects like reverb, delay, and distortion, with visual feedback showing impulse responses and frequency curves directly in the 3D space.

Robust Asynchronous Generation Strategy

Handling the asynchronous nature of AI model generation required careful engineering to avoid wasted credits. Estape implemented a retrying HTTP adapter with a poll loop that tolerates network blips, preventing charges for models that fail to download. Furthermore, the system writes the world.json state after every successful model generation. This checkpointing mechanism ensures that if a generation run fails halfway through, the previously paid-for assets are preserved rather than lost. Parallel processing via a thread pool significantly reduces the time required to generate all instrument models for a single song.

Key Takeaways

  • The project uses a custom prompt builder that maps audio features to visual descriptors, creating unique 3D assets per song.
  • Web Audio graph chains are torn down and rebuilt every frame to handle dynamic effect chaining without audio cuts.
  • Spatial audio is central to the gameplay, requiring headphones to experience the directional mixing mechanics.
  • The pipeline is currently local, but the developer plans to host it for public song uploads.

The Bottom Line

Unwrapped is a masterclass in creative tooling, proving that you don't need a AAA engine to build immersive spatial audio experiences. By tightly coupling audio analysis with generative AI, Estape has created a powerful educational tool that makes mixing intuitive and visually engaging.