Purpose Book-of-MLM is the supporting teaching resource repository for the book "Multimodal Large Language Models: A New Paradigm of Artificial Intelligence Technology," maintained by the Human-Cyber-Physical Intelligence Integration Laboratory (HCP-Lab) at Sun Yat-sen University on GitHub. Its goal is to provide accessible learning materials on multimodal large models for junior undergraduates and practitioners who have just entered the field of artificial intelligence, systematically introducing the key technologies, foundation models, and typical applications of multimodal large models, and exploring feasible approaches to achieving artificial general intelligence.
Content Structure The book has 5 chapters in total: Chapter 1 deeply explores the most representative large model architectures; Chapter 2 provides an in-depth analysis of the core technologies of multimodal large models; Chapter 3 introduces several representative multimodal large models; Chapter 4 deeply analyzes 3 typical applications: visual question answering, AIGC, and embodied intelligence; Chapter 5 explores feasible approaches to achieving artificial general intelligence, involving frontier technical fields such as model robustness, interpretability, trustworthiness, causal reasoning, world models, embodied intelligence, and multi-agent systems.
Teaching Resources The repository provides undergraduate teaching courseware from the School of Computer Science and Engineering at Sun Yat-sen University, with 12 lectures in total: Lecture 1 What Are Multimodal Large Models (Chapter 1); Lecture 2 Development and Evolution of Multimodal Large Models (Chapter 1); Lecture 3 From Pre-training to Self-supervision (Chapter 2); Lecture 4 Self-supervision and Contrastive Learning (Chapter 2); Lecture 5 Prompt Learning and In-context Learning (Chapter 2); Lecture 6 Chain of Thought and Value Alignment (Chapter 2); Lecture 7 Multimodal Foundation Models (Chapter 3); Lecture 8 Multimodal Models Moving Toward the Real World (Chapter 3, Chapter 4); Lecture 9 Principles and Applications of AIGC (Chapter 4); Lecture 10 Foundational Theories and Applications of Embodied Intelligence (Chapter 4); Lecture 11 Spatial Reasoning of Multimodal Large Models (Chapter 5); Lecture 12 Embodied World Models (Chapter 5). If a PPT version is needed, please contact liuy856@mail.sysu.edu.cn by email.
Other Resources The repository also lists related open-source resources, including the open-source operating system for embodied agents PhyAgentOS, the open-source visual-language causal reasoning framework CausalVLR, the HCP-Diffusion unified code framework, and the embodied intelligence paper list (Embodied_AI_Paper_List).
Applicable Scenarios This book is suitable for use as a textbook by senior undergraduates and graduate students in related majors at universities.It is also a reference work for IT practitioners of various kinds. In terms of teaching hour arrangements, it can accommodate teaching arrangements of 32 to 56 class hours. These courseware slides have become the teaching PPTs of more than ten universities, including Tongji University, Chongqing University, Northwestern Polytechnical University, Shandong University, Xidian University, and South China Normal University. This book has been selected into the undergraduate textbook series of Sun Yat-sen University.
Authors and Feedback The authors are Liu Yang (Associate Professor at the School of Computer Science and Engineering, Sun Yat-sen University, and a core member of the HCP-Lab, the Human-Cyber-Physical Intelligence Integration Laboratory of Sun Yat-sen University) and Lin Jing (IEEE Fellow, IAPR Fellow, IET Fellow, Director of the Institute of Multi-Agent and Embodied Intelligence at Peng Cheng Laboratory, Second-Class Professor at Sun Yat-sen University, recipient of the National Science Fund for Distinguished Young Scholars, and Chief Scientist of the National Major Artificial Intelligence Project). If you have opinions or suggestions regarding the content of this book, you can provide feedback through the GitHub Issues page.
Limitations This material consists of teaching companion resources and a textbook. Detailed information on specific installation commands, code execution methods, or software tools is not currently provided; the PPT version of the courseware must be obtained by contacting us via email.
