Machine Learning

GUI example/base sketch

This little sketch has been set up to have the best practices for configuring p5js for apps and small installations - it includes ml5

Example repository from Tensor-flow - easy to try and experiment with

Basic linear regression (the essence of Machine Learning)

In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables. Watch youtube video here

Creating a regression extracting features of MobileNet


Teach the machine to detect different elements (teachablemachine)

This example uses the teachable machine toturial from google.

  • Go to the webpage and press get started.

  • Chose image classification.

  • Name the two classes. E.g Cup and noCup.

  • Record a bunch of images with either a cup or no cup.

  • Ask it to train your classifier.

  • When done press export model.

  • Press the upload model button.

  • Copy paste the sharable link ind to the example sketch for the "imageModelURL" at the top of the sketch.

  • Run your code.

Their pose classification system is not really working for p5JS at the moment so use the example above for that

Hand tracking

Handpose is a machine-learning model that allows for palm detection and hand-skeleton finger tracking in the browser. It can detect a maximum of one hand at a time and provides 21 3D hand keypoints that describe important locations on the palm and fingers.

Documentation

FaceMesh

Facemesh is a machine-learning model that allows for facial landmark detection in the browser. It can detect multiple faces at once and provides 486 3D facial landmarks that describe the geometry of each face. Facemesh works best when the faces in view take up a large percentage of the image or video frame and it may struggle with small/distant faces.

documentation - Inspirational video

Pose tracking for bodily interaction

Uses the camera to make a skeleton of a person and track different parts of the person. This is similar to Kinect tracking, but only uses the webcamera. To get one of the points it tracking use:

getLimpPosition(0, 0);

The first 0 is the person id and the second is the point on that person skeleton.


Face and mood detection

Artificial intelligence can be used to recognize faces and assess their mood. This can e.g. be used to make an Instagram filter.

The diagram to the right gives you the different points. For example, you can find the nose by using point number 41. You can get the individual positions and draw an ellipse with this code:

ellipse(positions[42][0], positions[42][1], 20, 20);

You can get how angry a person is with this code:

predictedEmotions[0].value




From: https://github.com/stc/face-tracking-p5js

Based on this library: https://github.com/auduno/clmtrackr

Based on this paper: https://dl.acm.org/doi/10.1007/s11263-010-0380-4



Read more about face mesh here: https://google.github.io/mediapipe/solutions/face_mesh.html

Object detection library

Object detection libraries have ben taught to recognize object through a hupe library of images of different objects - the cocossd method has the following syntax:

for (let i = 0; i < detections.length; i++) {

let object = detections[i];

rect(object.x, object.y, object.width, object.height);

text(object.label, object.x + 10, object.y + 24);

}

The for loop run through the objects detected in the array detections. For each object, it draws a rectangle around the object and adds a text label.

Be aware that the library takes a while to load.

Speech to text

Speech to text will listen to the audio input and convert it into text. It is not precise, but surprisingly good. You can then try to detect words and use them to do things. Right now it is responding to "kage". The small text is interim results and the large text is the final result.

Text to speech

This example plays a string of text. This way you can make interactions that are based on voice-based language instead of visuals etc. To have the voice say "Husk at spise fisk" you would need to write:

playText("Husk at spise fisk");

Speech to text & Text to speech

This is a combined sketch which allows you to listen to voices and return a string based on keyword matches.

Teach the machine to detect different poses

In this example you record different kinds of poses and through machine learning it will recognize which poses you are in.

  • Start the program.

  • Press add pose as many times as you want poses

  • Go into the physical pose in front of the camera and press the pose that should match.

  • Do this with all the poses

  • Then you can get the best match through findBestMatch().id

  • If you want to keep the learned poses - press download and you will get a file that you can reupload. Set the variable classifierName to the name of the file in the top of the program.

Style transfer model

Use styletransfer to generate different styles from runway ml. You need to created your public hosted styletransfer url.

https://app.runwayml.com/model-collection/stylization

Mask person and/or background

This example uses bodypix to mask a person. The internal resolution, output stride and model have the largest effects on accuracy/speed. A higher internal resolution results in higher accuracy but lower the speed. A higher output stride results in lower accuracy but higher speed. A larger model, indicated by both architecture and multiplier dropdown, results in higher accuracy but lower speed.

NLP Compromise

Compromise tries its best to turn text into data. It makes limited and sensible decisions.

Create a classic chatbot

RiveScript exposes a simple plain text scripting language that's easy to learn and begin writing in quickly. No need to mess with complicated opaque XML structures or random symbols and line noise like with some other chatbot languages!

To develop the bot you need to edit its brain in the bot.txt file.

Read more here - Good article about chatbots


Create an ML chatbot

This is a much more advanced machine learning based chatbot based on nlp.js. Read more here - look at the corpus.json file for modification.

USE ML TO DRAW

Sketch that draws birds - based on https://quickdraw.withgoogle.com/


Generate text with CHARNN

ML5 Example

Interactive LSTM Text Generation Example using p5.js

This uses a pre-trained model on a corpus of Virginia Woolf

For more models see: https://github.com/ml5js/ml5-data-and-training/tree/master/models/charRNN

Parametric body generating example

Simple online parametric body generating example.

Make human

A large application to parametrically design human bodies.

Hand tracking v2 - 3d

https://www.youtube.com/watch?v=BX8ibqq0MJU&ab_channel=KazukiUmeda

Demo - Description

https://editor.p5js.org/hobye/sketches/cyVVlAcZu

Koncepts:

KNN, regression, gan, nlp, http://www.wekinator.org/, Text generation with an RNN, Large language models . large language models (LLMs)