<aside> 🔐 Pick one of the models (Face api, Face mesh), and following the examples and ml5.js documentation experiment with controlling elements of a p5.js sketch (color, geometry, sound, text) with the output of the model. (You may also choose a ml5.js model not covered here if you like!)

</aside>

Opera Singer

For this week’s assignment, I decided to create a project that detects the audience’s mouth and plays an opera song when it is widely opened. I used p5.js and Face.api to build it.

Screen Recording 2023-10-08 at 8.36.29 AM.mov

There are three cases in this project, depending on how much the audience’s mouth is opened.

  1. When audience close their mouth: No effect

  2. When audience open their mouth small: Instruction saying “Open you mouth wider”

  3. When audience open their mouth big: Play the opera song on loop with particle effect.

I wanted to create a project where users can feel like they are singing with their facial features. When the music is played, I added the particle system of music notes to pop within a specific duration to add visual interactivity.

Untitled

Audio Source: https://freesound.org/people/mickel11/sounds/90803/

P5. js Sketch

https://editor.p5js.org/[email protected]/sketches/wG7APu9bh

Future Tasks

If I have more time to iterate on this project, I would like to update the following things.

  1. Create more visual effects when opening the mouth. Drawing a stage or changing the lighting could make the project dramatic.
  2. Create an introduction in the beginning to help audiences interact with the project.
  3. Instead of playing the same audio file in a loop, play an opera vocal with one tone but a randomized pitch. Thorugh this, audiences can compose their music.