How to Create a Webcam Audio Visualizer with Three.js

• Create, Webcam

5 min. (1077 words)

how-to-create-a-webcam-audio-visualizer-with-three.js

A tutorial on how to create a Three.js powered audio visualizer that takes input from the user’s webcam.

WebcamAudioVisualizer_featured

From our monthly sponsor: Take WordPress to a whole new level with Divi’s incredibly advanced visual builder technology. Try it for free.

In this tutorial you’ll learn how to create an interesting looking audio visualizer that also takes input from the web camera. The result is a creative visualizer with a depth distortion effect. Although the final result looks complex, the Three.js code that powers it is straightforward and easy to understand.

So let’s get started.

Processing flow

The processing flow of our script is going to be the following:

  1. Create a vertex from every pixel of the image we get from the web camera input
  2. Use the image data from the web camera and apply the magnitude value of the sound frequency to the Z coordinate of each particle
  3. Draw
  4. Repeat point 2 and 3

Now, let’s have a look at how we can get and use the data from the web camera.

Web camera

First of all, let’s see how to access the web camera and get an image from it.

Camera access

For camera access in the browser, simply use getUserMedia().

video = document.getElementById("video");

const option = {
    video: true,
    audio: false
};

// Get image from camera
navigator.getUserMedia(option, (stream) => {
    video.srcObject = stream;  // Load as source of video tag
    video.addEventListener("loadeddata", () => {
        // ready
    });
}, (error) => {
    console.log(error);
});

Draw camera image to canvas

After camera access succeeded, we’ll get the image from the camera and draw it on the canvas.

const getImageDataFromVideo = () => {
    const w = video.videoWidth;
    const h = video.videoHeight;
    
    canvas.width = w;
    canvas.height = h;
    
    // Reverse image like a mirror
    ctx.translate(w, 0);
    ctx.scale(-1, 1);

    // Draw to canvas
    ctx.drawImage(image, 0, 0);

    // Get image as array
    return ctx.getImageData(0, 0, w, h);
};

About acquired imageData

ctx.getImageData() returns an array which RGBA is in order.

[0]  // R
[1]  // G
[2]  // B
[3]  // A

[4]  // R
[5]  // G
[6]  // B
[7]  // A...

And this is how you can access the color information of every pixel.

for (let i = 0, len = imageData.data.length; i < len; i =4) {
    const index = i * 4;  // Get index of "R" so that we could access to index with 1 set of RGBA in every iteration.?0, 4, 8, 12...?
    const r = imageData.data[index];
    const g = imageData.data[index   1];
    const b = imageData.data[index   2];
    const a = imageData.data[index   3];
}

Accessing image pixels

We are going to calculate the X and Y coordinates so that the image can be placed in the center.

const imageData = getImageDataFromVideo();
for (let y = 0, height = imageData.height; y < height; y  = 1) {
    for (let x = 0, width = imageData.width; x < width; x  = 1) {
        const vX = x - imageData.width / 2;  // Shift in X direction since origin is center of screen
        const vY = -y   imageData.height / 2;  // Shift in Y direction in the same way (you need -y)
    }
}

Create particles from image pixels

For creating a particle, we can use THREE.Geometry() and THREE.PointsMaterial().

Each pixel is added to the geometry as a vertex.

const geometry = new THREE.Geometry();
geometry.morphAttributes = {};
const material = new THREE.PointsMaterial({
    size: 1,
    color: 0xff0000,
    sizeAttenuation: false
});

const imageData = getImageDataFromVideo();
for (let y = 0, height = imageData.height; y < height; y  = 1) {
    for (let x = 0, width = imageData.width; x < width; x  = 1) {
        const vertex = new THREE.Vector3(
            x - imageData.width / 2,
            -y   imageData.height / 2,
            0
        );
        geometry.vertices.push(vertex);
    }
}
particles = new THREE.Points(geometry, material);
scene.add(particles);

Draw

In the drawing stage, the updated image is drawn using particles by getting the image data from the camera and calculating a grayscale value from it.

By calling this process on every frame, the screen visual is updated just like a video.

const imageData = getImageDataFromVideo();
for (let i = 0, length = particles.geometry.vertices.length; i < length; i  ) {
    const particle = particles.geometry.vertices[i];
    let index = i * 4;

    // Take an average of RGB and make it a gray value.
    let gray = (imageData.data[index]   imageData.data[index   1]   imageData.data[index   2]) / 3;

    let threshold = 200;
    if (gray < threshold) {
        // Apply the value to Z coordinate if the value of the target pixel is less than threshold.
        particle.z = gray * 50;
    } else {
        // If the value is greater than threshold, make it big value.
        particle.z = 10000;
    }
}
particles.geometry.verticesNeedUpdate = true;

Audio

In this section, let’s have a look at how the audio is processed.

Loading of the audio file and playback

For audio loading, we can use THREE.AudioLoader().

const audioListener = new THREE.AudioListener();
audio = new THREE.Audio(audioListener);

const audioLoader = new THREE.AudioLoader();
// Load audio file inside asset folder
audioLoader.load('asset/audio.mp3', (buffer) => {
    audio.setBuffer(buffer);
    audio.setLoop(true);
    audio.play();  // Start playback
});

For getting the average frequency analyser.getAverageFrequency() comes in handy.

By applying this value to the Z coordinate of our particles, the depth effect of the visualizer is created.

Getting the audio frequency

And this is how we get the audio frequency:

// About fftSize https://developer.mozilla.org/en-US/docs/Web/API/AnalyserNode/fftSize
analyser = new THREE.AudioAnalyser(audio, fftSize);

// analyser.getFrequencyData() returns array of half size of fftSize.
// ex. if fftSize = 2048, array size will be 1024.
// data includes magnitude of low ~ high frequency.
const data = analyser.getFrequencyData();

for (let i = 0, len = data.length; i < len; i  ) {
    // access to magnitude of each frequency with data[i].
}

Combining web camera input and audio

Finally, let’s see how the drawing process works that uses both, the camera image and the audio data.

Manipulate the image by reacting to the audio

By combining the techniques we’ve seen so far, we can now draw an image of the web camera with particles and manipulate the visual using audio data.

const draw = () => {
    // Audio
    const data = analyser.getFrequencyData();
    let averageFreq = analyser.getAverageFrequency();

    // Video
    const imageData = getImageData();
    for (let i = 0, length = particles.geometry.vertices.length; i < length; i  ) {
        const particle = particles.geometry.vertices[i];
    
        let index = i * 4;
        let gray = (imageData.data[index]   imageData.data[index   1]   imageData.data[index   2]) / 3;
        let threshold = 200;
        if (gray < threshold) {
            // Apply gray value of every pixels of web camera image and average value of frequency to Z coordinate of particle.
            particle.z = gray * (averageFreq / 255);
        } else {
            particle.z = 10000;
        }
    }
    particles.geometry.verticesNeedUpdate = true;  // Necessary to update

    renderer.render(scene, camera);

    requestAnimationFrame(draw);
};

And that’s all. Wasn’t that complicated, was it? Now you know how to create your own audio visualizer using web camera and audio input.

We’ve used THREE.Geometry and THREE.PointsMaterial here but you can take it further and use Shaders. Demo 2 shows an example of that.

We hope you enjoyed this tutorial and get inspired to create something with it.

141 comments

  7. My spouse and I stumbled over here different
    web page and thought I should check things out. I like what I see
    so i am just following you. Look forward to finding out about your web page for a second time.

    0mniartist asmr

  8. Excellent post. I was checking continuously this blog and
    I am impressed! Very helpful info specially the last part :
    ) I care for such info much. I was seeking this certain info
    for a long time. Thank you and best of luck. 0mniartist asmr

  9. Everything posted made a bunch of sense. However, what
    about this? what if you added a little information? I mean,
    I don’t wish to tell you how to run your
    website, but suppose you added something that makes people desire more?

    I mean How to Create a Webcam Audio Visualizer with Three.js – Pavvy Designs is a little boring.
    You ought to glance at Yahoo’s front page and watch how they create news headlines
    to get people to open the links. You might try adding a video
    or a related picture or two to get people excited about everything’ve got to say.
    Just my opinion, it would make your blog a little bit more interesting.

  10. Please let me know if you’re looking for
    a article author for your blog. You have some really great posts and I believe I would be
    a good asset. If you ever want to take some of the load
    off, I’d really like to write some material for
    your blog in exchange for a link back to mine.
    Please send me an e-mail if interested. Many thanks!

  12. Hi there, its nice paragraph on the topic of media print, we all be familiar with media is a wonderful source of facts.

  13. I was very happy to discover this page. I need to to thank you for ones time due to this wonderful read!!
    I definitely loved every little bit of it and I have you saved
    to fav to check out new things in your web site.

  14. Hello, after reading this remarkable paragraph i am as well glad to share my familiarity here with
    colleagues.

  15. It’s the best time to make a few plans for the long run and it’s time
    to be happy. I have read this put up and if I may just I want
    to recommend you few attention-grabbing issues
    or tips. Perhaps you could write subsequent articles relating to this article.
    I desire to learn even more things about it!

  16. Today, I went to the beach with my kids.
    I found a sea shell and gave it to my 4 year old daughter and said “You can hear the ocean if you put this to your ear.” She placed the shell
    to her ear and screamed. There was a hermit crab inside and it pinched her ear.
    She never wants to go back! LoL I know this is totally off topic but I had to tell someone!

  18. scoliosis
    I think what you said made a great deal of sense.
    But, what about this? suppose you were to create a killer headline?
    I ain’t suggesting your content is not good,
    however what if you added something to possibly grab people’s
    attention? I mean How to Create a Webcam Audio Visualizer
    with Three.js – Pavvy Designs is kinda boring. You might peek at Yahoo’s front page and watch how they create news titles to grab
    people interested. You might add a video or a related pic or two to grab readers interested
    about everything’ve written. Just my opinion, it would
    make your website a little livelier. scoliosis

  19. scoliosis
    I would like to thank you for the efforts you have put in writing this
    site. I really hope to see the same high-grade
    content from you in the future as well. In fact, your creative writing abilities has inspired me to get
    my own site now 😉 scoliosis

  20. scoliosis
    Pretty great post. I just stumbled upon your blog and wanted to say that I have truly enjoyed surfing around your blog posts.
    After all I will be subscribing on your rss feed and I hope you
    write once more very soon! scoliosis

  21. free dating sites
    Having read this I believed it was very enlightening.
    I appreciate you spending some time and effort to put this informative article together.
    I once again find myself spending way too much time both reading
    and commenting. But so what, it was still worthwhile! https://785days.tumblr.com/ free dating sites

  23. Greetings! Very helpful advice in this particular article!

    It’s the little changes that will make the biggest
    changes. Many thanks for sharing!

  24. Hello! I could have sworn I’ve been to this site before but after browsing through some of the post I realized
    it’s new to me. Anyhow, I’m definitely glad I found it and I’ll be book-marking and checking back often!

  27. Pretty component to content. I just stumbled upon your website and in accession capital to claim that I acquire actually loved
    account your blog posts. Any way I will be subscribing to your
    augment or even I achievement you get admission to constantly fast.

  28. Good day! I could have sworn I’ve been to this website before but after checking through some of the post
    I realized it’s new to me. Nonetheless, I’m definitely glad I found it and I’ll
    be book-marking and checking back frequently!

  29. Hi! I’m at work browsing your blog from my new iphone!
    Just wanted to say I love reading through
    your blog and look forward to all your posts! Keep up the outstanding work!

  30. I will right away grab your rss feed as I can not in finding your email subscription hyperlink or e-newsletter service.

    Do you’ve any? Please permit me recognise so that I may subscribe.

    Thanks.

  32. I was curious if you ever considered changing
    the layout of your site? Its very well written; I love what youve got to
    say. But maybe you could a little more in the way of content so people could connect
    with it better. Youve got an awful lot of text
    for only having one or two images. Maybe you could
    space it out better?

  36. I like the helpful info you provide in your articles. I’ll
    bookmark your weblog and check again here regularly. I am quite certain I will learn lots of new stuff right here!
    Good luck for the next!

  37. Very nice post. I just stumbled upon your blog and wished
    to say that I’ve truly enjoyed browsing your blog posts. In any case I will be subscribing to your feed and I hope you write again soon!

  39. Excellent blog! Do you have any hints for aspiring writers?
    I’m planning to start my own website soon but I’m a little lost on everything.

    Would you advise starting with a free platform like WordPress or go for a paid option? There are so many choices out there that I’m totally confused
    .. Any recommendations? Kudos!

  41. Simply wish to say your article is as amazing. The clarity in your post is just
    excellent and i can assume you’re an expert on this subject.

    Well with your permission allow me to grab your RSS feed to keep updated with forthcoming post.
    Thanks a million and please continue the rewarding work.

  42. We are a gaggle of volunteers and starting a new scheme
    in our community. Your website provided us with helpful information to work on. You have performed a formidable activity and our whole community can be thankful to you.

  45. Unquestionably believe that that you stated. Your favorite reason appeared to be
    at the internet the easiest factor to remember of. I say
    to you, I definitely get annoyed even as other folks
    think about issues that they just don’t realize about.
    You controlled to hit the nail upon the top
    as smartly as outlined out the whole thing without having side effect ,
    other people can take a signal. Will likely be again to get more.
    Thank you

  47. Heya i’m for the primary time here. I found this board and I in finding It truly helpful
    & it helped me out much. I’m hoping to give something again and help others like you aided me.

  50. Hey just wanted to give you a quick heads up. The text in your post seem to be
    running off the screen in Chrome. I’m not sure if this is a formatting
    issue or something to do with internet browser compatibility but
    I thought I’d post to let you know. The style and design look great though!
    Hope you get the issue fixed soon. Cheers

  53. I truly love your blog.. Pleasant colors & theme. Did you build this website yourself?
    Please reply back as I’m attempting to create my own personal site and would
    like to know where you got this from or what the theme is called.
    Many thanks!

  54. Do you mind if I quote a few of your posts as long as I provide credit and sources back to your weblog?
    My blog is in the exact same area of interest as yours and my users would certainly benefit from some of the information you provide here.

    Please let me know if this alright with you. Cheers!

  56. Pretty section of content. I just stumbled upon your blog and in accession capital to assert that I get actually enjoyed account your blog posts.
    Any way I’ll be subscribing to your feeds and even I achievement you access consistently fast.

  57. I used to be suggested this web site via my cousin. I’m
    no longer positive whether or not this put up is written via him as nobody else know such certain about my trouble.
    You’re amazing! Thank you!

  59. Hey there! I just wanted to ask if you ever have any problems
    with hackers? My last blog (wordpress) was
    hacked and I ended up losing a few months of hard work due to no back up.
    Do you have any methods to protect against hackers?

  64. Hey I am so excited I found your website, I really found
    you by error, while I was browsing on Google for something
    else, Anyways I am here now and would just
    like to say many thanks for a tremendous post and a all round thrilling blog (I also love the theme/design), I don’t have time to browse it all at the moment but I have
    saved it and also included your RSS feeds, so when I have
    time I will be back to read a great deal more, Please do keep up the superb work.
    cheap flights http://1704milesapart.tumblr.com/ cheap flights

  66. Do you have a spam issue on this website; I also
    am a blogger, and I was curious about your situation; many of us have
    developed some nice methods and we are looking to swap solutions with other folks, please shoot me an e-mail if interested.
    scoliosis surgery https://0401mm.tumblr.com/ scoliosis surgery

  67. This is really interesting, You’re a very skilled blogger.

    I have joined your rss feed and look forward to seeking more of your wonderful post.
    Also, I’ve shared your site in my social networks!

  70. Whoa! This blog looks exactly like my old one!
    It’s on a completely different topic but it has pretty
    much the same layout and design. Wonderful choice of colors!

  71. I do not know whether it’s just me or if everybody else encountering issues with your site.

    It appears as if some of the text in your posts are running off
    the screen. Can someone else please provide feedback and let
    me know if this is happening to them too? This
    could be a issue with my internet browser because I’ve had this happen before.

    Thank you

  81. Hey! This post couldn’t be written any better!
    Reading through this post reminds me of my previous room mate!
    He always kept talking about this. I will forward this article to
    him. Fairly certain he will have a good read. Thank you for
    sharing!

  83. I’m not sure where you are getting your information, but
    great topic. I needs to spend some time learning much
    more or understanding more. Thanks for great information I
    was looking for this information for my mission.

  87. I know this website provides quality depending articles or reviews and additional
    material, is there any other website which offers
    these kinds of stuff in quality?

  88. That is a really good tip especially to those new to the blogosphere.

    Brief but very precise information… Many thanks for sharing this one.
    A must read post!

  89. Hey there, You have done an incredible job. I will certainly digg it
    and personally suggest to my friends. I’m confident
    they’ll be benefited from this site.

  94. This is very interesting, You’re a very skilled blogger.

    I have joined your rss feed and look forward to seeking more
    of your fantastic post. Also, I’ve shared your site in my
    social networks!

  95. Pingback: low cost ordering tablet viagra
  96. Pingback: cialis for sale in india
  97. Pingback: viagra discount card australia
  98. Pingback: viagra canada purchase
  99. Pingback: sildenafil tablet price
  100. Pingback: how to get cialis australia
  101. Pingback: how long are amoxicillin pills good for
  102. Pingback: amoxicillin for acne dosage
  103. Pingback: azithromycin dosage
  104. Pingback: daily dosage of celebrex
  105. Pingback: can i buy cialis in mexico
  106. Pingback: dick on viagra australia
  107. Pingback: generic cialis 5mg australia
  108. Pingback: cheap cialis fast shipping
  109. Pingback: cialis soft donde comprar paypal
  110. Pingback: viagra generico 25 mg dimensioni
  111. Pingback: cvs cialis cost
  112. Pingback: viagra discounts
  113. Pingback: tadalafil otc
  114. Pingback: coupons for sildenafil
  115. Pingback: uses for cialis
  116. Pingback: viagra 25mg cost
  117. Pingback: neurontin 800
  118. Pingback: hydrochlorothiazide potassium
  119. Pingback: interactions for clonidine
  120. Pingback: what does finasteride do
  121. Pingback: carvedilol dosage
  122. Pingback: flagyl 400mg tablets
  123. Pingback: cialis non prescription
  124. Pingback: viagra ingredients
  125. Pingback: levitra from india
  126. Pingback: acyclovir side effects
  127. Pingback: levitra vs cialis review
  128. Pingback: generic vardenafil 20mg india
  129. Pingback: 30 mg sildenafil
  130. Pingback: sildenafil viagra 100mg
  131. Pingback: levitra directions
  132. Pingback: finasteride medication
  133. Pingback: fda label naltrexone
  134. Pingback: valtrex dose for outbreak
  135. Pingback: pronunciation of tizanidine
  136. Pingback: no prescription cialis
  137. Pingback: prescription ciprofloxacin hcl
  138. Pingback: cialis works
  139. Pingback: cialis ingredient
  140. Pingback: drugs like viagra
  141. Pingback: keto diet results 12 weeks

Leave a Reply

Your email address will not be published.