ml4a-ofx + osc-modules

Binaries for all applications found in ml4a-ofx including the apps folder and osc-modules. Please refer to the documentation in each app for individual instructions.

Installation

In order to complete installation, run the setup script to download necessary large data files. If you're on OSX 10.12+, this script also disables Mac's translocation of the apps so they can access the data folder.

sh setup.sh

If you wish to use KinectOSC, you must also download the zip file openni.zip and place the folder openni inside the data folder.

Apps

More documentation can be found in the individual READMEs. This release includes:

ml4a-ofx apps

• AudioClassifier : Train a classifier on microphone audio recordings. AudioClassifier2 is w.i.p., will merge soon.
• AudioTSNEViewer : View a collection of audio samples organized via t-SNE on extracted audio features. Works both on a collection (folder) of individual files, or a single audio file which can is segmented by onsets or beats.
• ConvnetClassifier : (deprecated in favor of ConvnetPredictor) Train an image classifier on a webcam feed.
• ConvnetRegressor : (deprecated in favor of ConvnetPredictor) Train an image regression on a webcam feed.
• ConvnetViewer : View the activations of a trained convnet.
• ConvnetPredictor : A combination of ConvnetRegressor and ConvnetClassifier.
• DoodleClassifier : Similar to convnet classifier, but optimized toward classifying images of hand-drawings.
• FaceClassifier : (deprecated in favor of FacePredictor) Train a model and do real time classification of face poses.
• FaceDTW : Perform facial gestures over time and classify them using Dynamic Time Warping.
• FaceRegressor : (deprecated in favor of FacePredictor) Train a model and do real time regression of face poses.
• FacePredictor : A combination of FaceClassifier and FaceRegressor.
• ImageTSNEViewer : View a collection of images organized via t-SNE on extracted features from a convnet, where the feature extraction is done from a python script and the results are imported into the app.
• ImageTSNELive : The same as ImageTSNEViewer (above), but handles the feature extraction and t-SNE assignment internally, and optionally allows for assignment to grid.
• ReverseImageSearch : Search for most similar images in an image collection, given a query image. Analyze it first, then save the results for real-time browsing.
• ReverseImageSearchFast : Fast search and retrieval of similar images in a database to a query image, webcam feed, video, or screengrabber.
• SimpleRegression : Boiled down example showing how to train a model and do real time regression with mouse position as the input.

osc-modules

• AbletonOSC : Receives data to control Ableton Live music parameters.
• AudioUnitOSC : Receives data to control Audio Unit modules.
• ConvnetOSC : Sends convolutional neural network-based feature vectors of a live webcam stream (using ofxCcv).
• FaceTracker2OSC : Sends coordinates of face found in a webcam stream.
• KeyboardOSC : Execute keystrokes via OSC commands.
• KinectOSC : Sends skeleton data of a single tracked skeleton from a Kinect.
• LeapMotionOSC : Sends finger point data detected by Leap Motion hand tracker.

Troubleshooting

If when you open any of the applications which require external files, such as ConvnetOSC, ConvnetPredictor, or FaceOSC, and it throws you an error that it can't find the model file, make sure you've run setup.sh and that the files image-net-2012.sqlite3 and shape_predictor_68_face_landmarks.dat are found inside the data folder. If they still are giving you problems, try individually moving each app you want to use into the data folder and then moving it back out to its original location, then re-running. If you are still having problems, post an issue on the issue tracker.

osc-modules

osc-modules

Standalone openFrameworks apps which send or receive data over open sound control (OSC) between various peripherals and applications.

Includes:

• AbletonOSC : Receives data to control Ableton Live music parameters.
• AudioUnitOSC : Receives data to control Audio Unit modules.
• ConvnetOSC : Sends convolutional neural network-based feature vectors of a live webcam stream (using ofxCcv).
• DarknetOSC : Sends convolutional neural network-based feature vectors of a live webcam stream (using ofxDarknet).
• FaceTracker2OSC : Sends coordinates of face found in a webcam stream.
• KinectOSC : Sends skeleton data of a single tracked skeleton from a Kinect.
• LeapMotionOSC : Sends finger point data detected by Leap Motion hand tracker.

KinectOSC*, FaceTracker2OSC, ConvnetOSC, and DarknetOSC require additional data files in order to work properly. These files can be all be downloaded at once by navigating to the data folder and running sh setup.sh. If you are having any trouble with those, refer to the documentation in each of those pages linked above for more specific usage instructions and links for the individual data files needed.

Additionally, the OSC configuration for each of the applications are also located in the data folder. Editing those files will let you adjust the destination to send OSC to (or in the case of AbletonOSC and AudioUnitOSC where to receive from).

*Note, to use KinectOSC, unzip the zip file openni.zip into the data folder, so that the openni folder is inside data.

Check the releases page for the latest downloadable release.

ml4a-ofx + osc-modules

Binaries for all applications found in ml4a-ofx including the apps folder and osc-modules. Please refer to the documentation in each app for individual instructions.

Installation

In order to complete installation, run the setup.sh script to download necessary large data files.

If you wish to use KinectOSC, you must also download the zip file openni.zip and place the folder inside ('openni') into the data folder.

Known issues

Apps using ofxDarknet (YoloLive, ReverseObjectSearchFast, and GoBot) require an NVIDIA graphics card and CUDA to function. Even with those, they are a bit unstable, and often cause segmentation faults on startup and crash. Proceed with caution with those... your results may vary!

The other apps, including the ones based on ofxCcv and ofxFaceTracker2 are more or less stable.

Apps includes

More documentation can be found in the individual READMEs. This release includes:

ml4a-ofx apps

• AudioClassifier + AudioClassifier2 : Train a classifier on microphone audio recordings. AudioClassifier2 is w.i.p., will merge soon.
• AudioTSNEViewer : View a collection of audio samples organized via t-SNE on extracted audio features. Works both on a collection (folder) of individual files, or a single audio file which can is segmented by onsets or beats.
• ConvnetClassifier : Train an image classifier on a webcam feed.
• ConvnetRegressor : Train an image regression on a webcam feed.
• ConvnetViewer : View the activations of a trained convnet.
• DoodleClassifier : Similar to convnet classifier, but optimized toward classifying images of hand-drawings.
• FaceClassifier : Train a model and do real time classification of face poses.
• FaceDTW : Perform facial gestures over time and classify them using Dynamic Time Warping.
• FaceRegressor : Train a model and do real time regression of face poses.
• Gobot : Automatically scrape Go board position from online-go.com and make next move recommendations.
• ImageTSNEViewer : View a collection of images organized via t-SNE on extracted features from a convnet, where the feature extraction is done from a python script and the results are imported into the app.
• ImageTSNELive : The same as ImageTSNEViewer (above), but handles the feature extraction and t-SNE assignment internally, and optionally allows for assignment to grid.
• ReverseImageSearch : Search for most similar images in an image collection, given a query image. Analyze it first, then save the results for real-time browsing.
• ReverseImageSearchFast : Fast search and retrieval of similar images in a database to a query image, webcam feed, video, or screengrabber.
• ReverseObjectSearchFast : Fast search and retrieval of similar images in a database to multiple objects found in a query image, webcam feed, video, or screengrabber.
• SimpleRegression : Boiled down example showing how to train a model and do real time regression with mouse position as the input.
• YoloLive : Do real-time object detection from 9000 classes on a webcam feed, video, or screengrabber.

osc-modules

• AbletonOSC : Receives data to control Ableton Live music parameters.
• AudioUnitOSC : Receives data to control Audio Unit modules.
• ConvnetOSC : Sends convolutional neural network-based feature vectors of a live webcam stream (using ofxCcv).
• DarknetOSC : Sends convolutional neural network-based feature vectors of a live webcam stream (using ofxDarknet).
• FaceTracker2OSC : Sends coordinates of face found in a webcam stream.
• KeyboardOSC : Execute keystrokes via OSC commands.
• KinectOSC : Sends skeleton data of a single tracked skeleton from a Kinect.
• LeapMotionOSC : Sends finger point data detected by Leap Motion hand tracker.