how-to-create-a-webcam-audio-visualizer-with-three.js

A tutorial on how to create a Three.js powered audio visualizer that takes input from the user’s webcam.

WebcamAudioVisualizer_featured

From our monthly sponsor: Take WordPress to a whole new level with Divi’s incredibly advanced visual builder technology. Try it for free.

In this tutorial you’ll learn how to create an interesting looking audio visualizer that also takes input from the web camera. The result is a creative visualizer with a depth distortion effect. Although the final result looks complex, the Three.js code that powers it is straightforward and easy to understand.

So let’s get started.

Processing flow

The processing flow of our script is going to be the following:

  1. Create a vertex from every pixel of the image we get from the web camera input
  2. Use the image data from the web camera and apply the magnitude value of the sound frequency to the Z coordinate of each particle
  3. Draw
  4. Repeat point 2 and 3

Now, let’s have a look at how we can get and use the data from the web camera.

Web camera

First of all, let’s see how to access the web camera and get an image from it.

Camera access

For camera access in the browser, simply use getUserMedia().

video = document.getElementById("video");

const option = {
    video: true,
    audio: false
};

// Get image from camera
navigator.getUserMedia(option, (stream) => {
    video.srcObject = stream;  // Load as source of video tag
    video.addEventListener("loadeddata", () => {
        // ready
    });
}, (error) => {
    console.log(error);
});

Draw camera image to canvas

After camera access succeeded, we’ll get the image from the camera and draw it on the canvas.

const getImageDataFromVideo = () => {
    const w = video.videoWidth;
    const h = video.videoHeight;
    
    canvas.width = w;
    canvas.height = h;
    
    // Reverse image like a mirror
    ctx.translate(w, 0);
    ctx.scale(-1, 1);

    // Draw to canvas
    ctx.drawImage(image, 0, 0);

    // Get image as array
    return ctx.getImageData(0, 0, w, h);
};

About acquired imageData

ctx.getImageData() returns an array which RGBA is in order.

[0]  // R
[1]  // G
[2]  // B
[3]  // A

[4]  // R
[5]  // G
[6]  // B
[7]  // A...

And this is how you can access the color information of every pixel.

for (let i = 0, len = imageData.data.length; i < len; i =4) {
    const index = i * 4;  // Get index of "R" so that we could access to index with 1 set of RGBA in every iteration.?0, 4, 8, 12...?
    const r = imageData.data[index];
    const g = imageData.data[index   1];
    const b = imageData.data[index   2];
    const a = imageData.data[index   3];
}

Accessing image pixels

We are going to calculate the X and Y coordinates so that the image can be placed in the center.

const imageData = getImageDataFromVideo();
for (let y = 0, height = imageData.height; y < height; y  = 1) {
    for (let x = 0, width = imageData.width; x < width; x  = 1) {
        const vX = x - imageData.width / 2;  // Shift in X direction since origin is center of screen
        const vY = -y   imageData.height / 2;  // Shift in Y direction in the same way (you need -y)
    }
}

Create particles from image pixels

For creating a particle, we can use THREE.Geometry() and THREE.PointsMaterial().

Each pixel is added to the geometry as a vertex.

const geometry = new THREE.Geometry();
geometry.morphAttributes = {};
const material = new THREE.PointsMaterial({
    size: 1,
    color: 0xff0000,
    sizeAttenuation: false
});

const imageData = getImageDataFromVideo();
for (let y = 0, height = imageData.height; y < height; y  = 1) {
    for (let x = 0, width = imageData.width; x < width; x  = 1) {
        const vertex = new THREE.Vector3(
            x - imageData.width / 2,
            -y   imageData.height / 2,
            0
        );
        geometry.vertices.push(vertex);
    }
}
particles = new THREE.Points(geometry, material);
scene.add(particles);

Draw

In the drawing stage, the updated image is drawn using particles by getting the image data from the camera and calculating a grayscale value from it.

By calling this process on every frame, the screen visual is updated just like a video.

const imageData = getImageDataFromVideo();
for (let i = 0, length = particles.geometry.vertices.length; i < length; i  ) {
    const particle = particles.geometry.vertices[i];
    let index = i * 4;

    // Take an average of RGB and make it a gray value.
    let gray = (imageData.data[index]   imageData.data[index   1]   imageData.data[index   2]) / 3;

    let threshold = 200;
    if (gray < threshold) {
        // Apply the value to Z coordinate if the value of the target pixel is less than threshold.
        particle.z = gray * 50;
    } else {
        // If the value is greater than threshold, make it big value.
        particle.z = 10000;
    }
}
particles.geometry.verticesNeedUpdate = true;

Audio

In this section, let’s have a look at how the audio is processed.

Loading of the audio file and playback

For audio loading, we can use THREE.AudioLoader().

const audioListener = new THREE.AudioListener();
audio = new THREE.Audio(audioListener);

const audioLoader = new THREE.AudioLoader();
// Load audio file inside asset folder
audioLoader.load('asset/audio.mp3', (buffer) => {
    audio.setBuffer(buffer);
    audio.setLoop(true);
    audio.play();  // Start playback
});

For getting the average frequency analyser.getAverageFrequency() comes in handy.

By applying this value to the Z coordinate of our particles, the depth effect of the visualizer is created.

Getting the audio frequency

And this is how we get the audio frequency:

// About fftSize https://developer.mozilla.org/en-US/docs/Web/API/AnalyserNode/fftSize
analyser = new THREE.AudioAnalyser(audio, fftSize);

// analyser.getFrequencyData() returns array of half size of fftSize.
// ex. if fftSize = 2048, array size will be 1024.
// data includes magnitude of low ~ high frequency.
const data = analyser.getFrequencyData();

for (let i = 0, len = data.length; i < len; i  ) {
    // access to magnitude of each frequency with data[i].
}

Combining web camera input and audio

Finally, let’s see how the drawing process works that uses both, the camera image and the audio data.

Manipulate the image by reacting to the audio

By combining the techniques we’ve seen so far, we can now draw an image of the web camera with particles and manipulate the visual using audio data.

const draw = () => {
    // Audio
    const data = analyser.getFrequencyData();
    let averageFreq = analyser.getAverageFrequency();

    // Video
    const imageData = getImageData();
    for (let i = 0, length = particles.geometry.vertices.length; i < length; i  ) {
        const particle = particles.geometry.vertices[i];
    
        let index = i * 4;
        let gray = (imageData.data[index]   imageData.data[index   1]   imageData.data[index   2]) / 3;
        let threshold = 200;
        if (gray < threshold) {
            // Apply gray value of every pixels of web camera image and average value of frequency to Z coordinate of particle.
            particle.z = gray * (averageFreq / 255);
        } else {
            particle.z = 10000;
        }
    }
    particles.geometry.verticesNeedUpdate = true;  // Necessary to update

    renderer.render(scene, camera);

    requestAnimationFrame(draw);
};

And that’s all. Wasn’t that complicated, was it? Now you know how to create your own audio visualizer using web camera and audio input.

We’ve used THREE.Geometry and THREE.PointsMaterial here but you can take it further and use Shaders. Demo 2 shows an example of that.

We hope you enjoyed this tutorial and get inspired to create something with it.

81 comments

  1. My spouse and I stumbled over here different
    web page and thought I should check things out. I like what I see
    so i am just following you. Look forward to finding out about your web page for a second time.

    0mniartist asmr

  2. Excellent post. I was checking continuously this blog and
    I am impressed! Very helpful info specially the last part :
    ) I care for such info much. I was seeking this certain info
    for a long time. Thank you and best of luck. 0mniartist asmr

  3. Everything posted made a bunch of sense. However, what
    about this? what if you added a little information? I mean,
    I don’t wish to tell you how to run your
    website, but suppose you added something that makes people desire more?

    I mean How to Create a Webcam Audio Visualizer with Three.js – Pavvy Designs is a little boring.
    You ought to glance at Yahoo’s front page and watch how they create news headlines
    to get people to open the links. You might try adding a video
    or a related picture or two to get people excited about everything’ve got to say.
    Just my opinion, it would make your blog a little bit more interesting.

  4. Please let me know if you’re looking for
    a article author for your blog. You have some really great posts and I believe I would be
    a good asset. If you ever want to take some of the load
    off, I’d really like to write some material for
    your blog in exchange for a link back to mine.
    Please send me an e-mail if interested. Many thanks!

  5. Hi there, its nice paragraph on the topic of media print, we all be familiar with media is a wonderful source of facts.

  6. I was very happy to discover this page. I need to to thank you for ones time due to this wonderful read!!
    I definitely loved every little bit of it and I have you saved
    to fav to check out new things in your web site.

  7. Hello, after reading this remarkable paragraph i am as well glad to share my familiarity here with
    colleagues.

  8. It’s the best time to make a few plans for the long run and it’s time
    to be happy. I have read this put up and if I may just I want
    to recommend you few attention-grabbing issues
    or tips. Perhaps you could write subsequent articles relating to this article.
    I desire to learn even more things about it!

  9. Today, I went to the beach with my kids.
    I found a sea shell and gave it to my 4 year old daughter and said “You can hear the ocean if you put this to your ear.” She placed the shell
    to her ear and screamed. There was a hermit crab inside and it pinched her ear.
    She never wants to go back! LoL I know this is totally off topic but I had to tell someone!

  10. scoliosis
    I think what you said made a great deal of sense.
    But, what about this? suppose you were to create a killer headline?
    I ain’t suggesting your content is not good,
    however what if you added something to possibly grab people’s
    attention? I mean How to Create a Webcam Audio Visualizer
    with Three.js – Pavvy Designs is kinda boring. You might peek at Yahoo’s front page and watch how they create news titles to grab
    people interested. You might add a video or a related pic or two to grab readers interested
    about everything’ve written. Just my opinion, it would
    make your website a little livelier. scoliosis

  11. scoliosis
    I would like to thank you for the efforts you have put in writing this
    site. I really hope to see the same high-grade
    content from you in the future as well. In fact, your creative writing abilities has inspired me to get
    my own site now 😉 scoliosis

  12. scoliosis
    Pretty great post. I just stumbled upon your blog and wanted to say that I have truly enjoyed surfing around your blog posts.
    After all I will be subscribing on your rss feed and I hope you
    write once more very soon! scoliosis

  13. free dating sites
    Having read this I believed it was very enlightening.
    I appreciate you spending some time and effort to put this informative article together.
    I once again find myself spending way too much time both reading
    and commenting. But so what, it was still worthwhile! https://785days.tumblr.com/ free dating sites

  14. Greetings! Very helpful advice in this particular article!

    It’s the little changes that will make the biggest
    changes. Many thanks for sharing!

  15. Hello! I could have sworn I’ve been to this site before but after browsing through some of the post I realized
    it’s new to me. Anyhow, I’m definitely glad I found it and I’ll be book-marking and checking back often!

  16. Pretty component to content. I just stumbled upon your website and in accession capital to claim that I acquire actually loved
    account your blog posts. Any way I will be subscribing to your
    augment or even I achievement you get admission to constantly fast.

  17. Good day! I could have sworn I’ve been to this website before but after checking through some of the post
    I realized it’s new to me. Nonetheless, I’m definitely glad I found it and I’ll
    be book-marking and checking back frequently!

  18. Hi! I’m at work browsing your blog from my new iphone!
    Just wanted to say I love reading through
    your blog and look forward to all your posts! Keep up the outstanding work!

  19. I will right away grab your rss feed as I can not in finding your email subscription hyperlink or e-newsletter service.

    Do you’ve any? Please permit me recognise so that I may subscribe.

    Thanks.

  20. I was curious if you ever considered changing
    the layout of your site? Its very well written; I love what youve got to
    say. But maybe you could a little more in the way of content so people could connect
    with it better. Youve got an awful lot of text
    for only having one or two images. Maybe you could
    space it out better?

  21. Pingback: cvs cialis cost
  22. Pingback: viagra discounts
  23. Pingback: tadalafil otc
  24. Pingback: uses for cialis
  25. Pingback: viagra 25mg cost
  26. Pingback: neurontin 800
  27. Pingback: carvedilol dosage
  28. Pingback: viagra ingredients
  29. Pingback: levitra from india
  30. Pingback: 30 mg sildenafil
  31. Pingback: levitra directions
  32. Pingback: cialis works
  33. Pingback: cialis ingredient
  34. Pingback: drugs like viagra

Leave a Reply

Your email address will not be published.