I had this idea for in game immersion. What if I could have objects in the game respond to music the user is playing with another application or respond to the in game music? Could the trees sway to the beat? The grass move with a high pitched sound? What about showing an audio trace texture? Well that is what spawned this next project. It was Windows OS only as it used the loopback feature available in Windows Vista and later Windows OSes.
The practical application was simple. Create a loopback monitoring thread in a game engine (Torque 3D). This thread would gather the data of any audio currently being played on the system. From there the data would be passed to other threaded objects for processing. One such object was an FFT object. Once the data was processed it was fed to a texture target that allowed rendering based on the audio data. This was done in the main graphics loop of the engine. So the processor intensive stuff was threaded and didn’t slow down the engine. The graphics were displayed by the engines normal graphics routines. The result is really fun.
The code for this is available here:
Below are some videos showcasing this in action: