DS-GA.3001 Advanced Topics in Embodied Learning and Vision (Spring 2025)

Table of contents

  1. DS-GA.3001 Advanced Topics in Embodied Learning and Vision (Spring 2025)
    1. Course Syllabus
    2. About
    3. Recommended Prerequisites
    4. Logistics
    5. Grading
    6. Course Work
      1. Paper Reviews
      2. Topic Presentation
      3. Project
      4. Late Policy
    7. Academic Integrity
    8. GenAI Policy

Course Syllabus

The syllabus document can be accessed here (requires NYU login).

About

How do we build end-to-end learning agents for embodied AI? This graduate level course covers advanced topics in embodied visual learning, perception and planning, with applications to robotics and self-driving. Topics include deep learning, computer vision, 3D perception, unsupervised 3D learning, self-supervised representation learning, continual learning, foundation model agents, etc. The goals of the course are:

  • Understand and apply computer vision and deep learning in the context of embodied agent learning by applying spatial geometric priors, model inductive biases, design multi-task network architectures, and embodied foundation models.
  • Learn to formulate a variety of computer vision and robotics problems using deep learning tools.
  • Develop a deep understanding of supervised and self-supervised representation learning for downstream perception and planning tasks.
  • Develop hands-on skill of implementing embodied learning systems in simulator environments.
  • Machine Learning (DS-GA 1003 or CSCI-GA 2565)
  • Computer Vision (CSCI-GA 2271)
  • Deep Learning (DS-GA 1008)
  • Proficiency in Python Programming

Logistics

  • Lectures: Thursday 4:55pm - 7:35pm (including 1hr recitation)
  • Location 19 West 4th St. Room 102
  • Office Hours: Thursday 1-2pm
  • Communication: We will use Campuswire as our main communication tool for announcements and answering questions related to the lectures, assignments, and projects. The registration link is available on Brightspace.

Grading

  • In-Class Participation (10%): Marks will be given on the amount of in-class participation during the discussion and Q&A period.
  • Paper Review (15%): Students need to submit a one-page paper review on one of the selected readings every week.
  • Topic Presentation (30%): Marks will be given on the prepared content, depth, analysis, and presentation quality.
  • Project (45%):
    • Project Proposal (10%): Project proposal is due in the 6th week. Student groups will need to schedule a mandatory office hour to discuss the project with the instructor.
    • Report (25%): Project report is due in the final week.
    • Presentation (10%): Project presentations are conducted in the final two weeks.

Course Work

Paper Reviews

  • Submission: You need to submit a paper review every week on Gradescope.

  • Paper Selection: You may choose a recent paper (<3 years) on the designated topic. You may choose one paper from the suggested reading list. If you choose your own paper, you should choose from reputable venues such as NeurIPS, ICML, ICLR, AISTATS, CoLLAs for ML papers, CVPR, ICCV, ECCV for CV papers, and ICRA, RSS, CoRL, IROS for robotics papers. You should choose papers that have a potential high impact.

  • Grading: We will release the grades within a week of the submission date.

Topic Presentation

  • Format: You will form a group with other students on the topic. Each of you will be asked to present on a topic for 30 minutes. It should contain content on general background but more importantly on recent research papers.
  • Panel Discussion: After the presentation, the presenter students will form an expert panel, and lead a 30-minute discussion Q&A with the audience. The audience students are required to ask questions related to the general topic, and the presenter students will give their opinions. Participation marks will be awarded based on high quality audience questions. The panel performance will also be part of the presentation score.

Project

  • Goal: The goal of the final project is to let students develop hands-on skills of implementing embodied learning systems for concrete real-world tasks such as toy embodied environments, long-form egocentric video understanding, self-driving, robotic navigation and manipulation.

  • Group: You are asked to form a group of two upon the beginning of the semester.

  • Project Consultation: To guide the project throughout the semester, the students are required to complete project consultations. They need to meet with the instructor and the TAs at least once each. See key dates below on when to meet with the instruction team the latest in the semester.

  • Project Presentation: You can choose to present on Week 14 or Week 15. The groups who present on Week 14 will get a bonus of 2% added to their final grade. Time duration is 25 minutes + 5 min Q&A.

Late Policy

You have 4 late days in total for any deliverables. If your deliverable is a team effort then it would cost late days from each team member. You can use a maximum of 2 late days on each deliverable. If it is beyond 2 late days or you have used up your late days, then there will be a 20% penalty on the marks each day.

Academic Integrity

Work you submit should be your own. Please consult the CAS academic integrity policy for more information: http://cas.nyu.edu/page/academicintegrity. Penalties for violations of academic integrity may include failure of the course, suspension from the University, or even expulsion.

GenAI Policy

AI may not be used in weekly paper reviews and paper presentations. AI may be used towards coding assistance and report writing assistance in the course project. However, if the use of AI can still impact the grade if the report contains poor writings and non-factual statements.