This is a evolving document and is subject to change (lots of change!)
- IMNY-UT 224 Introduction to Machine Learning for the Arts
- Interactive Media Arts (IMA), Tisch School of the Arts, New York University
- 370 Jay Street, Room 410
- Thursdays 12:20 - 3:20 PM
- Daniel Shiffman
- For contact info and office hours, please refer to internal documents shared via e-mail.
1: Introduction (9/5)
2: Transfer Learning (9/12)
3: Pre-trained models 1: Body (9/19)
4: Pre-trained models 2: Face and Hand (9/26)
5: Training a Neural Network 1 (10/3)
6: Training a Neural Network 2 (10/10)
7: ml5.js project presentations (10/17)
8: Introduction to Transformers.js (10/24)
9: Language Models (10/31)
10: Image Generation Models (11/7)
11: Final Project Proposals + Fine-Tuning (11/14)
12: Final Project Proposals + Fine-Tuning (11/21)
13: Final Project Play Testing (12/05)
14: Final Project Presentations (12/12)
Please read and review the ITP/IMA Community Guidelines. The Guidelines will be discussed as part of the course introduction.
An introductory course designed to provide students with hands-on experience developing creative coding projects with machine learning. The history, theory, and application of machine learning algorithms and related datasets are explored in a laboratory context of experimentation and discussion. Examples and exercises will be demonstrated in JavaScript using the p5.js, ml5.js, and TensorFlow.js libraries. In addition, students will learn to work with open-source generative models including text generation models and image generation models. Principles of data collection and ethics are introduced. Weekly assignments, team and independent projects, and project reports are required.
Your success in this class is important to me. Everyone learns differently and requires different kinds of accommodations. If there are aspects of this course that prevent you from learning or exclude you in any way, please let me know! Together we’ll develop strategies to meet your needs and the requirements of the course.
You all enter this classroom with different sets of skills. My office hours are open to you as an extension of the classroom. If you can’t make it to the scheduled times, please let me know and I'll be very happy to accomondate alternate times. There’s no incorrect way to approach office hours, and they are, by default, informal. I hope to work closely with all of you to cultivate a space of openness and mutual support. I welcome you to contact me outside of class and office hours through email.
At the completion of this course, the student will:
- Develop an intuition for and high level understanding of core machine learning concepts and algorithms, including supervised learning, unsupervised learning, reinforcement learning, transfer learning, classification, and regression.
- Be able to apply machine learning algorithms to real-time interaction in media art projects using pre-trained models and “transfer learning” in JavaScript and related tools.
- Learn how to collect a custom dataset to train a machine learning model and
- Develop a vocabulary for critical discussions around the social impact and ethics of data collection and application of machine learning algorithms.
- Become familiar with the current landscape of new media art generated from machine learning algorithms. Understand how to use a machine learning model to generate media: words, sound, and images.
You will need a modern laptop (4 years old or younger is a good rule of thumb). Limited numbers are available for checkout from the department. Any required software for this course will be freely available.
There is no textbook for the class. Readings and videos will be assigned on the individual session notes pages.
Classes will be a mixture of lecture, discussion, hands-on tutorials, homework review, presentations, and group work. You will need to come to class prepared with a laptop and any other supplies specified for that class.
The course will be once per week for three hours for a total of fourteen weeks.
Grades for the course will follow the standard A through F letter grading system and will be determined by the following breakdown:
- 25% Participation
- 50% Assignments (including reading responses and other written work)
- 25% Final project
There will be weekly assignments that are relevant to the class material. The primary elements of the assignments is documentation (written description, photos, screenshots, screen recording, code, and video all qualify based on the assignment) of your process. Each assignment is due by class time one week after they are assigned unless stated otherwise.
It is expected that you will spend 3 to 6 hours a week on the class outside of class itself. This will include reviewing material, reading, watching video, completing assignments and so on. Please budget your time accordingly.
Each assignment will be marked as complete (full credit), partially complete (half credit), or incomplete (no credit). To be complete, an assignment should meet the criteria specified in the syllabus including documentation. If significant portions are not attempted it may be marked partially complete. If an attempt isn’t made to meet the criteria specified it will be marked incomplete.
Assignments may include responses to reading and other written assignments. Responses to readings are generally to be 200 to 500 words in length unless otherwise specified. Grading will follow the same guidelines as above; on time and meeting the criteria specified will be marked as complete. Partially completed work will be given half credit. Work that is not turned in, or fails to meet the criteria specified will be given no credit.
ITP/IMA is committed to facilitating the fullest possible participation of all students. There are many forms of participation. Please communicate what kinds of engagement are best for you so it can be taken into account.
Examples of modes of participation can look like: asking questions, going to office hours, sending and reading emails, class group discussion, arriving on time, going to class, taking notes, listening to peers, submitting responses to a form (anonymous or not), following instructions, active listening, and more.
An assignment extension may be granted upon request. If you request an extension before the due date, your grade will not be affected. However, if you do not request an extension, the grading rules above apply. Please clarify with your instructor and set a deadline together. The recommended timeline is 1 to 5 days.
Note: There may be instances where having an extension may result in not being able to participate fully in activities such as feedback sessions or workshopping ideas/projects, which likely cannot be made up if it could disrupt the overall course schedule. Extensions are distributed at the discretion of the instructor.
After the first two weeks of the add/drop period, effective in week three onward, students are permitted the following number of absences: 3 absences. There are no excused absences and unexcused absences. There are only absences. Any more than 3 absences will affect your grade, please see the makeup work policy below.
This is an option for those who have attended more than 50% of the class (if you have missed more than 50% of class sessions, it will result in an automatic F for the course). While there is no distinction in this course between excused and unexcused absences, you may inquire about makeup work. Makeup work could be reading or viewing materials, a conversation with someone in class, additional office hours, writing a paper or an additional project. Not all course content can be made up. Please clarify with your instructor and set a deadline together. The recommended timeline is 1 to 5 days.
Incomplete grades may only be given to students who have completed more than half of the class assignments. Incomplete grades are given at the discretion of the instructor.
Plagiarism is presenting someone else’s work as though it were your own. More specifically, plagiarism is to present as your own: A sequence of words quoted without quotation marks from another writer or a paraphrased passage from another writer’s work or facts, ideas or images composed by someone else.
Collaboration is highly valued and often necessary to produce great work. Students build their own work on that of other people and giving credit to the creator of the work you are incorporating into your own work is an act of integrity. Plagiarism, on the other hand, is a form of fraud. Proper acknowledgment and correct citation constitute the difference.
- Link to the Tisch Student Handbook
- Link to Suggested Practices for Syllabus Accessibility Statements
(The following is adapted from Golan Levin’s Interactivity and Computation Course (Fall 2018) at Carnegie Mellon University.)
You must cite the source of any code you use. Please note the following expectations and guidelines:
-
Check the License. When using others' code, pay attention to the license under which it has been released, and be certain to fulfill the terms and requirements of those licenses. Descriptions of common licenses, and their requirements, can be found at choosealicense.com. Some licenses may require permission. If you are confused or aren’t sure how to credit code, ask the course instructor and make your best good faith effort.
-
Use Libraries. The use of general, repurposable libraries is strongly encouraged. The people who developed and contributed these components to the community worked hard, often for no pay; acknowledge them by citing their name and linking to their repository.
-
Be Careful. It sometimes happens that an artist places the entire source code for their sketch or artwork online, as a resource from which others can learn. Assignments professors give in new-media arts courses are often similar (e.g. "Clock"); you may also discover the work of a student in some other class or school, who has posted code for a project which responds to a similar assignment. You should probably avoid this code. At the very least, you should be careful about approaching such code for possible re-use. If it is necessary to do so, it is best to extract components that solve a specific technical problem, rather than those parts which operate to create a poetic experience. Your challenge, if and/or when you work with others' code, is to make it your own. It should be clear that downloading an artwork from someone's GitHub and simply changing the colors would be disgracefully lazy. And doing so without proper citation would be outright plagiarism.
You should treat AI tools just as you would any other source: cite the source and note how it was used (Harvard has a useful guide to citation of AI systems). You should be prepared to explain how your use is an appropriate tool to fit your goal or concept and does not detract from your experience meeting the learning objectives of the assignment or course. There are some cases where the use of generative AI systems may fall under a form of plagiarism. Document your process as part of your work for the class.
It’s crucial for our community to create and uphold learning environments that empower students of all abilities. We are committed to create an environment that enables open dialogue about the various temporary and long term needs of students and participants for their academic success. We encourage all students and participants to discuss with faculty and staff possible accommodations that would best support their learning. Students may also contact the Moses Center for Student Accessibility (212-998-4980) for resources and support. Link to the Moses Center for Student Accessibility.
Your health and safety are a priority at NYU. Emphasizing the importance of the wellness of each individual within our community, students are encouraged to utilize the resources and support services available to them 24 hours a day, 7 days a week via the NYU Wellness Exchange Hotline at 212-443-9999. Additional support is available over email at [email protected] and within the NYU Wellness Exchange app. Link to the NYU Counseling and Wellness Center.
Laptops and other electronic devices are essential tools for learning and interaction in classrooms. However, they can create distractions that hinder students' ability to actively participate and engage. Please be mindful of the ways in which these devices can affect the learning environment, please refrain from doing non-class oriented activities during class.
Tisch School of the Arts is dedicated to providing its students with a learning environment that is rigorous, respectful, supportive and nurturing so that they can engage in the free exchange of ideas and commit themselves fully to the study of their discipline. To that end, Tisch is committed to enforcing University policies prohibiting all forms of sexual misconduct as well as discrimination on the basis of sex and gender. Detailed information regarding these policies and the resources that are available to students through the Title IX office can be found by using the following link: Link to the NYU Title IX Office.
Teachers and students work together to create a supportive learning environment. The educational experience in the classroom is one that is enhanced by integrating varying perspectives and learning modes brought by students.
This syllabus is adapted from the work of previous instructors—Yining Shi (Fall 2023) and Jack Du (Summer 2024)—who have generously shared their materials. Thank you also to Gottfried Haider, his course materials for Machine Learning for Artists and Designers also served as a foundation for the materials presented here.