MIT Open Course "How to Do Anything with AI" (PDF file)

"How to Do Anything with AI" is a cutting-edge course offered by the Massachusetts Institute of Technology, deeply exploring artificial intelligence technology and its applications across a variety of real-world data modalities. The course focuses on modern deep learning and foundation models, covering multimodal content from language to vision, audio, sensors, medical data, and more. Through lectures, readings, discussions, and research projects, students will develop critical thinking, understand the latest technological achievements of AI, and master the principles of multimodal AI. The course emphasizes theoretical learning while stressing practical application, encouraging students to innovate and explore the infinite possibilities of AI across multiple domains.

Get the "How to Do Anything with AI" report PDF original file. Scan the QR code to follow and reply: 20250813

Course Content

Course Introduction

How to Conduct AI Research

Guides students on how to generate research ideas, read papers, execute research plans, and write research papers. The course provides a detailed introduction to all stages of the research process, including bottom-up discovery and top-down design methods, emphasizes the importance of scientific questions and hypotheses, and provides examples of multiple research directions, such as multimodal AI, sensor data processing, AI reasoning, interactive agents, embodied AI, socially intelligent AI, human-computer interaction, and ethics and safety. The course introduces methods for literature review, steps for testing research ideas, and the structure and key points of writing papers.

The course provides a wealth of resource links and tool suggestions to help students obtain support during the research process. The file also lists this week's assignment arrangements, including the submission of the project preference form and a preview of next week's course topics. The course provides students with a comprehensive introductory guide to AI research, covering all aspects from theory to practice.

Data, Structure, and Learning

The course mainly introduces the characteristics, structures, and common learning objectives of different data modalities (such as vision, language, audio, sensor data, tabular data, graph data, and set data). The course explores in detail the data representation, distribution, granularity, structure, information content, noise, and correlation of each modality, and discusses multiple learning paradigms such as supervised learning, unsupervised learning, and reinforcement learning, as well as their applications in multimodal and multitask learning. The course emphasizes the importance of data preprocessing, visualization, and model selection, and how to evaluate a model's generalization ability through training, validation, and test data.

At the end of the course, the assignment arrangements for this week were summarized, including the submission of the project preference form, the preparation and presentation of the project proposal, and the optional tutorial on machine learning tools next week. It provides students with foundational knowledge of data processing and machine learning, helping them better understand and apply different modalities of data in AI research.

Practical AI Tools

The course mainly revolves around PyTorch and Hugging Face tools, providing students with a practical guide for using these tools for AI development and debugging. The course introduces the main features of Hugging Face, including the transformers and datasets libraries. The tools integrate seamlessly with PyTorch, providing powerful pretrained models and data loading capabilities. The course mentions the bitsandbytes and flash-attn libraries commonly used with PyTorch, which can optimize model performance and memory usage. The course explains debugging techniques during AI development, including how to establish a close connection with the data, set up an end-to-end skeleton model, diagnose errors through overfitting, improve generalization ability through regularization, tune hyperparameters, and squeeze out the final performance improvements.

The course provides advice on how to design machine learning models for new data, emphasizing the approach of starting with simple models and gradually increasing complexity. It summarizes the key steps when debugging models, such as checking the correctness of input data and labels, observing training loss, and handling issues such as overfitting and underfitting. It can help students avoid common pitfalls during AI development and improve the reliability and performance of models.

Model Architectures

The course introduces the design principles and methods of model architectures for different data modalities (such as sequential data, spatial data, set data, and graph data). The course content includes how to choose an appropriate model architecture based on the characteristics of the data, for example, recurrent neural networks (RNN) for time series data, convolutional neural networks (CNN) for spatial data processing, and graph neural networks (GNN) for graph data processing. It discusses how to achieve invariance and equivariance of data through parameter sharing and information aggregation, and how to design models based on the semantic information, granularity, structure, information content, noise, and correlations of the data.

At the end of the course, the assignment arrangements for this week were summarized, including the submission of the project proposal and preparation for the reading discussion. The course provides students with systematic guidance on model architecture design, helping them better understand and apply different types of model architectures in AI research.

Multimodal AI and Alignment

The course deeply explores the core concepts of multimodal AI, including the heterogeneity of data and the connectivity and interactivity between modalities. The course content covers the historical stages of multimodal research, from the behavioral era to the deep learning era, and then to today's foundation model era. It introduces the diversity of multimodal tasks, such as the combination of language and vision, sentiment analysis, and video event recognition, and discusses the challenges of multimodal alignment, including discrete alignment and continuous alignment methods, and how to achieve alignment between modalities through contrastive learning.

The course introduces in detail the six core challenges of multimodal AI: representation learning, alignment, reasoning, generation, transfer, and quantification. It particularly mentions the application of the CLIP model in language and vision tasks, and how to learn shared and unique modal information through contrastive learning. Finally, it summarizes the assignment arrangements for this week, including reading discussion and feedback on project progress. The course provides students with a comprehensive overview of the field of multimodal AI, helping them understand the complexity of multimodal data and how to address challenges through model design and learning methods.

Multimodal Fusion

The course delves into the core concepts and techniques of multimodal fusion, including early fusion, intermediate fusion, and late fusion, as well as various methods such as additive fusion, multiplicative fusion, tensor fusion, low-rank fusion, and gated fusion. The course content covers techniques ranging from simple linear fusion to complex nonlinear fusion, and how to optimize the representation and learning process of multimodal data through dynamic fusion strategies. It discusses the optimization challenges in multimodal fusion, such as how to balance the overfitting and generalization capabilities of different modalities, and how to solve problems in real-world tasks through contrastive learning and multimodal representation learning.

The course introduces the challenges of multimodal fusion in practical applications, such as how to handle heterogeneity between modalities, how to avoid the impact of bias from a single modality on fusion results, and how to improve model performance through dynamic fusion and architecture search. At the end, the course summarizes the assignment arrangements for the week, including reading discussions and feedback on project progress, helping students better understand and apply multimodal fusion techniques.

Cross-modal Learning

The course delves into the basic concepts and techniques of cross-modal learning, including achieving knowledge transfer between modalities through fusion, alignment, and translation. The course content covers various transfer strategies in multimodal learning, such as transfer of pretrained models, collaborative learning, and model induction. It particularly mentions the High-Modality Multimodal Transformer (HighMMT) model, which can perform knowledge transfer between partially observable modalities and is suitable for multitask learning and transfer learning across various modalities and tasks.

The course discusses the open challenges in cross-modal learning, such as learning with low-resource modalities, applications beyond language and vision modalities, challenges in complex data and model training, and model interpretability. At the end, the document summarizes the assignment arrangements for the week, including reading discussions and feedback on project progress, helping students better understand and apply cross-modal learning techniques.

Large Foundation Models

The course provides a detailed introduction to the development history, pretraining methods, architecture types, instruction fine-tuning, and preference tuning of large language models (LLMs), as well as efficient training and inference techniques. The course content covers the evolution from recurrent neural networks (RNNs) to the Transformer architecture, and how to perform pretraining on large-scale text data through unsupervised learning. It discusses how to optimize model performance through instruction fine-tuning and preference tuning, and how to improve training efficiency and inference speed through techniques such as LoRA (Low-Rank Adaptation) and quantization.

The course discusses the future development directions of large language models, including how to teach models to perform reasoning, how to extend multimodal LLMs to cover more modalities, and how to evaluate and deploy these models in practical applications. At the end, the course provides the assignment arrangements for the week, including the submission of midterm reports and the completion of project resource application forms, helping students better understand and apply the techniques of large foundation models.

Large Multimodal Models

The course provides a detailed introduction to the fundamentals of large multimodal models, pretraining methods, how to adapt large language models (LLMs) into multimodal LLMs, and the latest advances in text-to-multimodal generation. The course content covers representation methods for multimodal foundation models, multimodal Transformer architectures, cross-modal attention mechanisms, and how to optimize the performance of multimodal models through instruction fine-tuning and preference tuning. The course discusses how to achieve multimodal conditional generation through prefix tuning and adapter layers, and how to enhance the model's generalization ability through large-scale pretraining datasets and multimodal instruction fine-tuning datasets.

The course discussed the future development directions of multimodal models, including the design of native multimodal models, the application of multimodal Mixture of Experts (MoE), and how to apply multimodal models to practical scenarios such as time-series data. At the end of the course, the assignments for the week were arranged, including reading discussions and feedback on project progress, to help students better understand and apply the technology of large multimodal models.

Modern Generative AI

The course provided a detailed introduction to the core concepts of generative AI, the current state of technology development, methods of conditional generation, model architectures, and techniques for training these models. The course content covered a variety of generative models from Variational Autoencoders (VAE) to diffusion models and flow matching models, and explored the training objectives, noise handling methods, sampling speed, as well as advantages and disadvantages of the models. The course introduced how to optimize the generation process through conditional vector fields and loss functions, and how to improve model performance through specific architectural designs.

The course also discussed evaluation metrics for generative AI, such as Fréchet Inception Distance (FID), CLIP Score, precision/recall, and aesthetic scores, to help evaluate the quality and diversity of generated content. At the end of the course, the assignments for the week were arranged, including reading tasks and feedback on project progress, to help students better understand and apply the technology of generative AI.

Reinforcement Learning and Interaction

The course provided a detailed introduction to the fundamentals of reinforcement learning, the application of modern reinforcement learning in the alignment and reasoning of large language models (LLMs), and the design and implementation of interactive LLM agents. The course content covered the basic concepts of reinforcement learning, such as Markov Decision Processes (MDP), policy learning, the comparison between model-based and policy-based methods, and how to optimize policies through policy gradient methods such as REINFORCE and PPO. The course explored how to perform reinforcement learning through human feedback, including the training of reward models and preference optimization techniques.

The course deeply explored the challenges of reinforcement learning in practical applications, such as the design of reward functions, the balance between exploration and exploitation of models, and how to improve model performance and efficiency through methods such as Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO). At the end of the course, the assignments for the week were arranged, including the submission of the final project report and preparation for the project presentation, to help students better understand and apply the technology of reinforcement learning.

Latest Development Directions

The course explored frontier fields such as multimodal reasoning, AI agents, human-computer interaction, and ethics and safety. The course content included representation methods for multimodal foundation models, how to adapt large language models into multimodal text generation models, how to implement text and image generation, and how to improve model trustworthiness and safety through human-computer interaction. It specifically mentioned the "Interactive Sketchpad" system, a multimodal tutoring system for collaboration and visual problem solving, which can enhance the effectiveness of problem solving through visual reasoning and code execution.

The course discussed the application of multimodal models in education, for example, helping students understand abstract geometric concepts through visual reasoning. It introduced the challenges of multimodal models in safety and ethics, including how to quantify model shortcomings, predict and control model behavior, and how to address bias and unfairness in models. At the end of the course, the assignments for the week were summarized, including the submission of the final project report and the collection of course feedback.

Course project address

• Project website：https://mit-mi.github.io/how2ai-course/spring2025/
• GitHub repository：https://github.com/MIT-MI/how2ai-course

Course application scenarios

• Healthcare: Based on AI processing of medical imaging and medical record data, develop auxiliary diagnostic systems to improve the accuracy and efficiency of disease detection and treatment.
• Intelligent Transportation: Leveraging AI technology to analyze traffic camera videos and sensor data, enabling autonomous driving assistance systems and traffic flow optimization.
• Art and Creative Design: Using AI to generate creative content such as music and paintings, transforming language descriptions into visual or audio works, providing creative tools for artists and designers.
• Intelligent Education: Developing AI-driven personalized learning systems that provide customized learning paths and teaching content by analyzing student behavior and feedback.
• Environmental Protection: Using AI to process satellite imagery and environmental sensor data, monitor environmental changes, predict natural disasters, and support ecological conservation and sustainable development.

Get the "How to Use AI to Do Anything" report PDF original file. Scan the QR code to follow and reply: 20250813