Audio programming in actionscript 3, it’s awesome. This article will get you started and will guide you past the pitfalls any novice will encounter. I should know, I experienced them all!
For “Beter Horen” (Better Hearing), we at Mangrove created a fun viral called “Twitter in Concert”. The intention of this site is to create awareness for ear protection. If you want to continue enjoying concerts for years to come, protect your ears and wear some plugs!
Why is this article penned in English?
Because Flash audio programming is a tough cookie and I think not only Dutch speaking developers should benefit from the knowledge I’m sharing.
The concept
The idea was to create a virtual concert. And because you want to enjoy music for as long as you live we made it a never ending one. Each visitor can create a short piece of music and add it to the stream to become part of this initiative.
Where does twitter come in?
We chose twitter as our platform for distribution, and did so for a number of reasons; when you create something mash-up-like using twitter it has media value. Although most of us web developers are used to Twitter integration by now, a large part of the general public will regard it as a novelty. Twitter has a great API allowing for easy integration. By choosing Twitter as a requirement, the participants need to have an account including some private data. This makes abusing the application less attractive.
Many parts create the whole
Let’s take a look at how the Flash application fits into the site scheme.
The above gives you an idea of what the Flash application talks and listens to.
For clarity I’ve left out our mobile version, but I assure you it’s there and it rocks.
We’re getting visitors from twitter.com, via online and offline promotion, and from links on blogs, forums and so on. The application searches the Twitter timeline for tweets that contain a “Twitter in Concert” hash tag. When these tweets are collected and screened, the application asks our pH8 cms to provide the music these users made. After all data is loaded, the application shows our visitors a timeline of tweets and starts playing the music that goes with it.
If you want to stop listening and start participating, you can switch to composing mode and create your very own piece of music by sequencing ten different percussion instruments on a default beat loop. For those who (think they) can’t make music, we have three presets to use or get inspired by.
When done composing, you can move on to twitter.com to post your creation and share it with your followers. Of course we hope these followers will come have a listen and in turn create their very own addition to the “Twitter in Concert” stream!
Flash dynamic audio
Now that you know what the application is all about, let’s open it up and take a look inside. Flash’s Sound object was closed for a very long time, and many developers complained about it to Adobe. So much that Adobe started listening! From Flash player 10 and upwards the Sound object is wide open, meaning we can finally create or modify its content dynamically. Want to load a piece of music and make it play at twice the speed? Or create your own synthesizer? Maybe add effects like reverb and echo? Create sounds for your game on the fly? Well now you can.
The application I’ve built uses this new Sound API in the following way; the library contains ten instrument samples and one looping 16 bar beat. I load all of these into their own Sound instance, extract the audio data, and feed this data to the audio stream at user defined intervals, creating the music you hear from your speakers.
What is sound and how is it digitized?
Sound is waves produced by a vibrating object, interpreted by our brain as an auditory sensation. For example, when you hit a guitar string, it will start to vibrate. This movement makes the air around it move. When these air waves reach your ears they are interpreted by your brain and you experience sound. If you want to learn a bit more about digital audio check out this nice primer PDF by Adobe. Here’s the gist of it for Flash: Sound waves can be represented graphically by using an X-axis for time and Y-axis for amplitude. Flash uses an amplitude range from -1.0 to 1.0. Any values below or above will be truncated.
Now that we know this, making sound digital is trivial. Mono sound translates to an array of values ranging from -1.0 to 1.0. For stereo, you use an array storing value pairs of the same range for both channels (left and right). For my project, translating this data back to what I wanted to hear from my speakers was a different matter altogether; keeping tempo requires more magic.
Flash audio playback
Let’s take a look at how Flash plays back our beautiful audio data.
Flash uses an audio stream to send a signal to your speakers. This audio stream uses data from an audio buffer which is a fixed size container. You can set the buffer size, using a range of 2048 to 8192 bytes. When the buffer is close to empty and needs more data, a sample data event is fired. The listener you’ve set for this can then fill up the buffer with new data from your audio data source. This source can be data from an mp3 you’ve loaded, better yet, it can be filled dynamically.
How to handle keeping tempo
In our sequencer we have 8 measures, each counting 4 beats. The tempo of the music is set to 110 beats per minute. Timing instruments to play at user defined intervals (beats), means we need to know how much sample data fits into a beat. We can calculate the number of samples per beat using the following formula: int(44100*60/4/110).
44100 is the sample rate Flash uses. This is a fixed value, we can’t change it. Sample rate means the number of samples used for 1 second of audio. In our formula we multiply by 60 to see how much data is used per minute. We then divide by 4 because of the number of beats we use in a measure and finally divide by 110 which is the number of beats per minute. To round it we cast to integer, which gives us the result we need: 6014 samples per beat! Now isn’t that exciting?
Have a look at how that compares to our audio buffer, which in my case is set at 4096 samples.
As you can see, buffer refills don’t run in sync with the beat. This poses the following challenge; when we refill the buffer we need to take into account the offset number of samples between the buffer and the current beat. I’ve indicated this difference above in red. The first buffer fill and beat are in sync. In the second buffer fill however, the exemplary kick-drum indicated above in gray needs to start playing after 6014 – 4096 = 1918 samples to stick with the beat.
Final challenge; syncing animations with music
Finally my music was keeping tempo, but my animations weren’t! It took me quite a while to find out why, but after some field research (read: googling), I found out about a thing called latency. This is the term used for the time it takes to convert a digital signal to the desired analog signal, namely the sound from the speakers. Luckily, this latency can be easily calculated like so:
var latency:int = int(sampleDataEvent.position/44.1 – soundChannel.position/44.1);
This gives us the latency in milliseconds which can be used to time animations using, for instance, setTimeout:
setTimeout(playAnimation, latency);
Wrapping up
Well, that’s about it I guess. I hope this article will inspire you to do some audio programming. And of course, if you find any errors in the above, let me know. Thanks for your time! Now go visit twitterinconcert.nl and make some tunes.