A tutorial on how to create a Three.js powered audio visualizer that takes input from the user’s webcam.
From our monthly sponsor: Take WordPress to a whole new level with Divi’s incredibly advanced visual builder technology. Try it for free.
In this tutorial you’ll learn how to create an interesting looking audio visualizer that also takes input from the web camera. The result is a creative visualizer with a depth distortion effect. Although the final result looks complex, the Three.js code that powers it is straightforward and easy to understand.
So let’s get started.
Processing flow
The processing flow of our script is going to be the following:
- Create a vertex from every pixel of the image we get from the web camera input
- Use the image data from the web camera and apply the magnitude value of the sound frequency to the Z coordinate of each particle
- Draw
- Repeat point 2 and 3
Now, let’s have a look at how we can get and use the data from the web camera.
Web camera
First of all, let’s see how to access the web camera and get an image from it.
Camera access
For camera access in the browser, simply use getUserMedia().
video = document.getElementById("video");
const option = {
video: true,
audio: false
};
// Get image from camera
navigator.getUserMedia(option, (stream) => {
video.srcObject = stream; // Load as source of video tag
video.addEventListener("loadeddata", () => {
// ready
});
}, (error) => {
console.log(error);
});Draw camera image to canvas
After camera access succeeded, we’ll get the image from the camera and draw it on the canvas.
const getImageDataFromVideo = () => {
const w = video.videoWidth;
const h = video.videoHeight;
canvas.width = w;
canvas.height = h;
// Reverse image like a mirror
ctx.translate(w, 0);
ctx.scale(-1, 1);
// Draw to canvas
ctx.drawImage(image, 0, 0);
// Get image as array
return ctx.getImageData(0, 0, w, h);
};About acquired imageData
ctx.getImageData() returns an array which RGBA is in order.
[0] // R
[1] // G
[2] // B
[3] // A
[4] // R
[5] // G
[6] // B
[7] // A...And this is how you can access the color information of every pixel.
for (let i = 0, len = imageData.data.length; i < len; i =4) {
const index = i * 4; // Get index of "R" so that we could access to index with 1 set of RGBA in every iteration.?0, 4, 8, 12...?
const r = imageData.data[index];
const g = imageData.data[index 1];
const b = imageData.data[index 2];
const a = imageData.data[index 3];
}Accessing image pixels
We are going to calculate the X and Y coordinates so that the image can be placed in the center.
const imageData = getImageDataFromVideo();
for (let y = 0, height = imageData.height; y < height; y = 1) {
for (let x = 0, width = imageData.width; x < width; x = 1) {
const vX = x - imageData.width / 2; // Shift in X direction since origin is center of screen
const vY = -y imageData.height / 2; // Shift in Y direction in the same way (you need -y)
}
}Create particles from image pixels
For creating a particle, we can use THREE.Geometry() and THREE.PointsMaterial().
Each pixel is added to the geometry as a vertex.
const geometry = new THREE.Geometry();
geometry.morphAttributes = {};
const material = new THREE.PointsMaterial({
size: 1,
color: 0xff0000,
sizeAttenuation: false
});
const imageData = getImageDataFromVideo();
for (let y = 0, height = imageData.height; y < height; y = 1) {
for (let x = 0, width = imageData.width; x < width; x = 1) {
const vertex = new THREE.Vector3(
x - imageData.width / 2,
-y imageData.height / 2,
0
);
geometry.vertices.push(vertex);
}
}
particles = new THREE.Points(geometry, material);
scene.add(particles);Draw
In the drawing stage, the updated image is drawn using particles by getting the image data from the camera and calculating a grayscale value from it.
By calling this process on every frame, the screen visual is updated just like a video.
const imageData = getImageDataFromVideo();
for (let i = 0, length = particles.geometry.vertices.length; i < length; i ) {
const particle = particles.geometry.vertices[i];
let index = i * 4;
// Take an average of RGB and make it a gray value.
let gray = (imageData.data[index] imageData.data[index 1] imageData.data[index 2]) / 3;
let threshold = 200;
if (gray < threshold) {
// Apply the value to Z coordinate if the value of the target pixel is less than threshold.
particle.z = gray * 50;
} else {
// If the value is greater than threshold, make it big value.
particle.z = 10000;
}
}
particles.geometry.verticesNeedUpdate = true;Audio
In this section, let’s have a look at how the audio is processed.
Loading of the audio file and playback
For audio loading, we can use THREE.AudioLoader().
const audioListener = new THREE.AudioListener();
audio = new THREE.Audio(audioListener);
const audioLoader = new THREE.AudioLoader();
// Load audio file inside asset folder
audioLoader.load('asset/audio.mp3', (buffer) => {
audio.setBuffer(buffer);
audio.setLoop(true);
audio.play(); // Start playback
});For getting the average frequency analyser.getAverageFrequency() comes in handy.
By applying this value to the Z coordinate of our particles, the depth effect of the visualizer is created.
Getting the audio frequency
And this is how we get the audio frequency:
// About fftSize https://developer.mozilla.org/en-US/docs/Web/API/AnalyserNode/fftSize
analyser = new THREE.AudioAnalyser(audio, fftSize);
// analyser.getFrequencyData() returns array of half size of fftSize.
// ex. if fftSize = 2048, array size will be 1024.
// data includes magnitude of low ~ high frequency.
const data = analyser.getFrequencyData();
for (let i = 0, len = data.length; i < len; i ) {
// access to magnitude of each frequency with data[i].
}Combining web camera input and audio
Finally, let’s see how the drawing process works that uses both, the camera image and the audio data.
Manipulate the image by reacting to the audio
By combining the techniques we’ve seen so far, we can now draw an image of the web camera with particles and manipulate the visual using audio data.
const draw = () => {
// Audio
const data = analyser.getFrequencyData();
let averageFreq = analyser.getAverageFrequency();
// Video
const imageData = getImageData();
for (let i = 0, length = particles.geometry.vertices.length; i < length; i ) {
const particle = particles.geometry.vertices[i];
let index = i * 4;
let gray = (imageData.data[index] imageData.data[index 1] imageData.data[index 2]) / 3;
let threshold = 200;
if (gray < threshold) {
// Apply gray value of every pixels of web camera image and average value of frequency to Z coordinate of particle.
particle.z = gray * (averageFreq / 255);
} else {
particle.z = 10000;
}
}
particles.geometry.verticesNeedUpdate = true; // Necessary to update
renderer.render(scene, camera);
requestAnimationFrame(draw);
};And that’s all. Wasn’t that complicated, was it? Now you know how to create your own audio visualizer using web camera and audio input.
We’ve used THREE.Geometry and THREE.PointsMaterial here but you can take it further and use Shaders. Demo 2 shows an example of that.
We hope you enjoyed this tutorial and get inspired to create something with it.
81 comments
avana tablet buy avanafil uk
priligy without a doctor prescription buy avanafil usa
http://vslevitrav.com/ – levitra 20mg best price
tadalafil 40 mg from india https://elitadalafill.com/ tadalafil daily use
Yes! Finally something about asmr https://0mniartist.tumblr.com. asmr 0mniartist
I think the admin of this website is in fact working hard in support of his web page,
since here every information is quality based information. asmr 0mniartist
My spouse and I stumbled over here different
web page and thought I should check things out. I like what I see
so i am just following you. Look forward to finding out about your web page for a second time.
0mniartist asmr
Excellent post. I was checking continuously this blog and
I am impressed! Very helpful info specially the last part :
) I care for such info much. I was seeking this certain info
for a long time. Thank you and best of luck. 0mniartist asmr
Everything posted made a bunch of sense. However, what
about this? what if you added a little information? I mean,
I don’t wish to tell you how to run your
website, but suppose you added something that makes people desire more?
I mean How to Create a Webcam Audio Visualizer with Three.js – Pavvy Designs is a little boring.
You ought to glance at Yahoo’s front page and watch how they create news headlines
to get people to open the links. You might try adding a video
or a related picture or two to get people excited about everything’ve got to say.
Just my opinion, it would make your blog a little bit more interesting.
Please let me know if you’re looking for
a article author for your blog. You have some really great posts and I believe I would be
a good asset. If you ever want to take some of the load
off, I’d really like to write some material for
your blog in exchange for a link back to mine.
Please send me an e-mail if interested. Many thanks!
Peculiar article, totally what I needed.
Hi there, its nice paragraph on the topic of media print, we all be familiar with media is a wonderful source of facts.
I was very happy to discover this page. I need to to thank you for ones time due to this wonderful read!!
I definitely loved every little bit of it and I have you saved
to fav to check out new things in your web site.
Hello, after reading this remarkable paragraph i am as well glad to share my familiarity here with
colleagues.
It’s the best time to make a few plans for the long run and it’s time
to be happy. I have read this put up and if I may just I want
to recommend you few attention-grabbing issues
or tips. Perhaps you could write subsequent articles relating to this article.
I desire to learn even more things about it!
Today, I went to the beach with my kids.
I found a sea shell and gave it to my 4 year old daughter and said “You can hear the ocean if you put this to your ear.” She placed the shell
to her ear and screamed. There was a hermit crab inside and it pinched her ear.
She never wants to go back! LoL I know this is totally off topic but I had to tell someone!
This is my first time go to see at here and i am in fact
impressed to read all at single place.
scoliosis
I think what you said made a great deal of sense.
But, what about this? suppose you were to create a killer headline?
I ain’t suggesting your content is not good,
however what if you added something to possibly grab people’s
attention? I mean How to Create a Webcam Audio Visualizer
with Three.js – Pavvy Designs is kinda boring. You might peek at Yahoo’s front page and watch how they create news titles to grab
people interested. You might add a video or a related pic or two to grab readers interested
about everything’ve written. Just my opinion, it would
make your website a little livelier. scoliosis
scoliosis
I would like to thank you for the efforts you have put in writing this
site. I really hope to see the same high-grade
content from you in the future as well. In fact, your creative writing abilities has inspired me to get
my own site now 😉 scoliosis
scoliosis
Pretty great post. I just stumbled upon your blog and wanted to say that I have truly enjoyed surfing around your blog posts.
After all I will be subscribing on your rss feed and I hope you
write once more very soon! scoliosis
free dating sites
Having read this I believed it was very enlightening.
I appreciate you spending some time and effort to put this informative article together.
I once again find myself spending way too much time both reading
and commenting. But so what, it was still worthwhile! https://785days.tumblr.com/ free dating sites
dating sites
Do you have any video of that? I’d want to find out more details.
dating sites
Greetings! Very helpful advice in this particular article!
It’s the little changes that will make the biggest
changes. Many thanks for sharing!
Hello! I could have sworn I’ve been to this site before but after browsing through some of the post I realized
it’s new to me. Anyhow, I’m definitely glad I found it and I’ll be book-marking and checking back often!
Ahaa, its nice discussion on the topic of this piece of writing here at this blog, I
have read all that, so now me also commenting at this
place.
If you would like to increase your know-how just
keep visiting this web site and be updated with the newest
news posted here.
Pretty component to content. I just stumbled upon your website and in accession capital to claim that I acquire actually loved
account your blog posts. Any way I will be subscribing to your
augment or even I achievement you get admission to constantly fast.
Good day! I could have sworn I’ve been to this website before but after checking through some of the post
I realized it’s new to me. Nonetheless, I’m definitely glad I found it and I’ll
be book-marking and checking back frequently!
Hi! I’m at work browsing your blog from my new iphone!
Just wanted to say I love reading through
your blog and look forward to all your posts! Keep up the outstanding work!
I will right away grab your rss feed as I can not in finding your email subscription hyperlink or e-newsletter service.
Do you’ve any? Please permit me recognise so that I may subscribe.
Thanks.
It’s great that you are getting thoughts from this post as well as from our dialogue made
at this time.
I was curious if you ever considered changing
the layout of your site? Its very well written; I love what youve got to
say. But maybe you could a little more in the way of content so people could connect
with it better. Youve got an awful lot of text
for only having one or two images. Maybe you could
space it out better?
tadalafil generic usa tadalafil price comparison cialis price australia
hydroxycholorquin chloroquine side effects what is hydroxychloride
erectile booster method reviews https://www.pharmaceptica.com/