Machine Learning
GUI example/base sketch
This little sketch has been set up to have the best practices for configuring p5js for apps and small installations - it includes ml5
Example repository from Tensor-flow - easy to try and experiment with
Basic linear regression (the essence of Machine Learning)
In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables. Watch youtube video here
Creating a regression extracting features of MobileNet
Teach the machine to detect different elements (teachablemachine)
This example uses the teachable machine toturial from google.
Go to the webpage and press get started.
Chose image classification.
Name the two classes. E.g Cup and noCup.
Record a bunch of images with either a cup or no cup.
Ask it to train your classifier.
When done press export model.
Press the upload model button.
Copy paste the sharable link ind to the example sketch for the "imageModelURL" at the top of the sketch.
Run your code.
Their pose classification system is not really working for p5JS at the moment so use the example above for that
Hand tracking
Handpose is a machine-learning model that allows for palm detection and hand-skeleton finger tracking in the browser. It can detect a maximum of one hand at a time and provides 21 3D hand keypoints that describe important locations on the palm and fingers.
FaceMesh
Facemesh is a machine-learning model that allows for facial landmark detection in the browser. It can detect multiple faces at once and provides 486 3D facial landmarks that describe the geometry of each face. Facemesh works best when the faces in view take up a large percentage of the image or video frame and it may struggle with small/distant faces.
Pose tracking for bodily interaction
Uses the camera to make a skeleton of a person and track different parts of the person. This is similar to Kinect tracking, but only uses the webcamera. To get one of the points it tracking use:
getLimpPosition(0, 0);
The first 0 is the person id and the second is the point on that person skeleton.
The drawing example uses a Graphics element to draw on read more here.
Face and mood detection
Artificial intelligence can be used to recognize faces and assess their mood. This can e.g. be used to make an Instagram filter.
The diagram to the right gives you the different points. For example, you can find the nose by using point number 41. You can get the individual positions and draw an ellipse with this code:
ellipse(positions[42][0], positions[42][1], 20, 20);
You can get how angry a person is with this code:
predictedEmotions[0].value
From: https://github.com/stc/face-tracking-p5js
Based on this library: https://github.com/auduno/clmtrackr
Based on this paper: https://dl.acm.org/doi/10.1007/s11263-010-0380-4
Read more about face mesh here: https://google.github.io/mediapipe/solutions/face_mesh.html
Object detection library
Object detection libraries have ben taught to recognize object through a hupe library of images of different objects - the cocossd method has the following syntax:
for (let i = 0; i < detections.length; i++) {
let object = detections[i];
rect(object.x, object.y, object.width, object.height);
text(object.label, object.x + 10, object.y + 24);
}
The for loop run through the objects detected in the array detections. For each object, it draws a rectangle around the object and adds a text label.
Be aware that the library takes a while to load.
Speech to text
Speech to text will listen to the audio input and convert it into text. It is not precise, but surprisingly good. You can then try to detect words and use them to do things. Right now it is responding to "kage". The small text is interim results and the large text is the final result.
Text to speech
This example plays a string of text. This way you can make interactions that are based on voice-based language instead of visuals etc. To have the voice say "Husk at spise fisk" you would need to write:
playText("Husk at spise fisk");
Speech to text & Text to speech
This is a combined sketch which allows you to listen to voices and return a string based on keyword matches.
Teach the machine to detect different poses
In this example you record different kinds of poses and through machine learning it will recognize which poses you are in.
Start the program.
Press add pose as many times as you want poses
Go into the physical pose in front of the camera and press the pose that should match.
Do this with all the poses
Then you can get the best match through findBestMatch().id
If you want to keep the learned poses - press download and you will get a file that you can reupload. Set the variable classifierName to the name of the file in the top of the program.
Style transfer model
Use styletransfer to generate different styles from runway ml. You need to created your public hosted styletransfer url.
CHAT GPT API
Basic example of using chat GPT.
OBS - The key in the sketch is encrypted. You need to create an account and use your own api key, To encrypt your own key:
Set apiKeyEncryptedGpt = "" in the top of the sketch and
It will prompt you to enter your own key and a password.
Then copy the new apiKeyEncryptedGpt in from the console.
Restart the sketch and you will prompted for the password for the new key.
Mask person and/or background
This example uses bodypix to mask a person. The internal resolution, output stride and model have the largest effects on accuracy/speed. A higher internal resolution results in higher accuracy but lower the speed. A higher output stride results in lower accuracy but higher speed. A larger model, indicated by both architecture and multiplier dropdown, results in higher accuracy but lower speed.
NLP Compromise
Compromise tries its best to turn text into data. It makes limited and sensible decisions.
Create a classic chatbot
RiveScript exposes a simple plain text scripting language that's easy to learn and begin writing in quickly. No need to mess with complicated opaque XML structures or random symbols and line noise like with some other chatbot languages!
To develop the bot you need to edit its brain in the bot.txt file.
Read more here - Good article about chatbots
Create an ML chatbot
This is a much more advanced machine learning based chatbot based on nlp.js. Read more here - look at the corpus.json file for modification.
Generate text with CHARNN
ML5 Example
Interactive LSTM Text Generation Example using p5.js
This uses a pre-trained model on a corpus of Virginia Woolf
For more models see: https://github.com/ml5js/ml5-data-and-training/tree/master/models/charRNN
Parametric body generating example
Simple online parametric body generating example.
Make human
A large application to parametrically design human bodies.
Hand tracking v2 - 3d
https://www.youtube.com/watch?v=BX8ibqq0MJU&ab_channel=KazukiUmeda
https://editor.p5js.org/hobye/sketches/cyVVlAcZu
Koncepts:
KNN, regression, gan, nlp, http://www.wekinator.org/, Text generation with an RNN, Large language models . large language models (LLMs)