AI Engineering Hub (PDF File) - AI Tutorial Materials

AI Engineering Hub is an open-source project that provides in-depth tutorials and practical case studies on building applications with large language models (LLMs), retrieval-augmented generation (RAG), and AI agents. AI Engineering Hub includes theoretical explanations and provides a large number of actionable code examples to help users get started quickly. The project has earned 12.6K stars on GitHub, and its core tutorials have been compiled into a PDF of more than 500 pages for convenient learning. AI Engineering Hub is suitable for beginners, practitioners, and researchers, and encourages community contributions to jointly advance the development of AI technology.

AI Engineering Hub-website

To obtain the original PDF file of "AI Engineering Hub," scan the QR code, follow, and reply: 20250713

Introduction to AI Engineering Hub Course Content

The AI Engineering Hub course materials are a comprehensive and practical guide to data science and machine learning, covering core topics from beginner to advanced levels. The full text is divided into two major sections:

Deep Learning: Includes cutting-edge paradigms such as transfer learning, federated learning, and multi-task learning, as well as optimization techniques such as mixed-precision training and gradient checkpointing, and deeply explores fine-tuning and deployment strategies for large language models (LLMs).
Classical Machine Learning: Covers traditional methods such as feature engineering, regression analysis, decision trees, and clustering, combines statistical theory with practical cases (such as handling data drift and missing values), and reveals common pitfalls (such as the trap of random splitting).

The materials provide data analysis tools (such as Pandas and SQL), visualization techniques (such as Sankey diagrams and QQ plots), and advanced usage of Python object-oriented programming. Through code examples, visualization charts, and experimental comparisons, the full text balances theory and practice, making it suitable for learners at different levels. The accompanying assessment tool can help readers locate the most relevant chapters and efficiently improve their skills.

Course Content of AI Engineering Hub

Deep Learning

Learning Paradigms

Transfer learning, fine-tuning, multi-task learning, and federated learning：
- Transfer Learning: After training a pretrained model (foundation model) on a related task, replace the last few layers and freeze the weights of the other layers to adapt to a new task with small data.
- Fine-Tuning: Adjust some or all weights of a pretrained model to directly adapt to new data.
- Multi-Task Learning (MTL): Shared layers handle multiple tasks, while task-specific branches are trained independently, improving generalization and saving computational resources.
- Federated Learning: Train models on decentralized devices and aggregate parameters rather than data, protecting privacy (such as mobile phone user data).
Introduction to Federated Learning: Models are distributed to devices for local training, and only parameters are uploaded to a central server for aggregation, avoiding centralized data storage.
Building a Multi-Task Learning Model：
- PyTorch implementation: shared layers + task branches, gradient accumulation updates, and dynamic task weight allocation (such as adjustment based on validation accuracy).
Active Learning: Starting with a small amount of labeled data, the model selects low-confidence samples for manual labeling and iteratively optimizes (suitable for scenarios with high labeling costs).

Runtime and Memory Optimization

Momentum: Reduces oscillations through a moving average of gradients and accelerates convergence (hyperparameters need tuning).
Mixed-Precision Training: Mixed float16 and float32: use float16 for forward/gradient computation to accelerate, use float32 for weight updates to preserve precision, need loss scaling to avoid vanishing gradients.
Gradient Checkpointing: Store activations in segments, recompute during backpropagation, saving 50-60% memory (at the cost of 15-25% time).
Gradient Accumulation：
- During small-batch training, accumulate gradients over multiple batches before updating, simulating the effect of large batches (no time reduction, but lowers memory pressure).
- 4 Strategies for Multi-GPU Training (4 Strategies for Multi-GPU Training)
  - Model Parallelism: Split layers across different GPUs, requiring frequent communication.
  - Tensor Parallelism: Split a single tensor operation (e.g., block-wise computation for matrix multiplication).
  - Data Parallelism: Replicate the model, shard the data for training, then aggregate gradients.
  - Pipeline Parallelism: Micro-batch pipeline processing, improving GPU utilization.

Miscellaneous

Label Smoothing: Adjust the label distribution, lower the probability of the true class, and distribute the remaining probability uniformly, preventing the model from being overconfident (improves generalization, but lowers prediction confidence).
Focal Loss: For class imbalance, reduce the loss contribution of easy-to-classify samples through a weighting factor (γ) and inverse class frequency (α).
How Dropout Actually Works? (How Dropout Works): Randomly drop neurons during training, and scale the retained activations by 1/(1-p) to maintain inference consistency (p is the dropout rate).
Problems with Dropout in CNNs: It disrupts the spatial correlations of convolutional layers; it is recommended to use it in fully connected layers.
The Real Role of Hidden Layers and Activation Functions: Explain how hidden layers extract feature hierarchies and how activation functions introduce nonlinearity.
Shuffle Data Before Training: Avoid the data order between batches affecting model convergence.

Model Compression

Knowledge Distillation): The teacher model guides the student model, compressing model size.
Activation Pruning: Prune low-importance neurons to reduce computation, requiring dynamic importance evaluation.

Deployment

Deploying Models from Jupyter Notebook: Simplify the model migration process from development to production.
Model Testing Methods in Production Environments: Including production environment validation methods such as A/B testing and shadow deployment.
Version Control and Model Registry: Track model iterations and manage production releases.

Large Language Models

Where Did the GPU Memory Go? (GPU Memory Management): Analyze the GPU memory bottlenecks and optimization strategies when training LLMs.
Full Model Fine-Tuning vs. LoRA vs. RAG: Compare applicable scenarios: full-parameter fine-tuning consumes a lot of resources, LoRA (Low-Rank Adaptation) is efficient, and RAG (Retrieval-Augmented Generation) dynamically expands knowledge.
5 LLM Fine-Tuning Techniques: A detailed explanation of lightweight fine-tuning techniques including adapter fine-tuning and prefix tuning.

Classical ML (Classical Machine Learning)

ML Fundamentals

Time Complexity of 10 Algorithms: Compare the training/inference time complexity of 10 algorithms such as SVM and Random Forest.
25 key mathematical definitions: Covers core concepts such as probability and linear algebra.
How to reliably improve multiclass probability models: Calibrate multiclass probability outputs through methods such as Platt Scaling.
Reasons model improvements may be ineffective:Pitfalls such as data quality and choice of evaluation metrics.
Loss functions of 16 algorithms: Includes loss function formulas for logistic regression, SVM, and more.
10 common loss functions: Applicable scenarios for cross-entropy, Huber loss, and others.
How to correctly use training/validation/test sets: Data splitting strategies and avoiding information leakage.
5 cross-validation techniques: Implementation and selection of k-fold, leave-one-out, and others.
Steps after cross-validation: Model selection and final evaluation process.
Double descent and the bias-variance tradeoff: Explains the nonlinear relationship between model complexity and generalization.

Statistical foundations

Differences between MLE and EM: MLE is suitable for complete data, while EM handles latent variables (such as clustering).
Confidence intervals and prediction intervals: Explains the differences between the two and their calculation methods.
Why OLS is an unbiased estimator: Mathematical derivation and the Gauss-Markov theorem.
Bhattacharyya distance: A metric for measuring the similarity of probability distributions.
Why choose Mahalanobis distance):The advantages of Mahalanobis distance taking the covariance structure into account.
11 methods for testing data normality: Includes Shapiro-Wilk, QQ plots, and others.
Probability and likelihood: Conceptual distinction and the connection to maximum likelihood estimation.
11 key probability distributions: Application scenarios such as the Poisson distribution and exponential distribution.
Common misconceptions about continuous probability distributions: Explains the non-probabilistic meaning of probability density functions.

Feature definition, engineering, and selection

11 variable types in datasets: Covers nominal variables (unordered categories), ordinal variables (ordered ranks), continuous variables, and more, guiding the choice of encoding and analysis methods during data preprocessing.
Cyclical feature encoding: Use sin/cos encoding for cyclical features in time series (such as hour, month) to preserve cyclic continuity (such as the closeness of 23:59 and 00:01).
Feature discretization: Bin continuous features (such as age groups) to balance information loss and model robustness, suitable for linear models such as logistic regression.
7 categorical data encoding methods: Includes one-hot encoding (sparse categories), target encoding (high-cardinality categories), hashing encoding (memory optimization), and more, solving the problem of converting categorical variables into numerical values.
Permutation feature importance evaluation: Randomly shuffle feature values and observe changes in model performance to quantify feature importance and avoid overfitting to irrelevant features.
Probe feature selection method: Generate random noise features, screen out and remove features whose importance is lower than the noise, improving model generalization ability.

Regression analysis

The mathematical principles of MSE: The convexity of mean squared error (MSE) guarantees that gradient descent converges to the global optimum, and it is sensitive to outliers (squaring amplifies errors).
Sklearn linear regression has no hyperparameters: Because it uses a closed-form solution (analytical solution) to directly compute the weights, no iterative optimization is needed, but it cannot handle large-scale data (gradient descent is required).
Poisson regression vs. linear regression: Poisson regression handles count data (such as the number of events) through a log link function, while linear regression fits a continuous target variable.
Dummy variable trap: When encoding categorical variables, one category must be omitted (such as "male" in gender) to avoid rank deficiency in the design matrix, which leads to multicollinearity.
Generalized linear models (GLMs): Extends linear regression to exponential family distributions (such as the binomial distribution and Poisson distribution), linking the linear predictor to the target variable through a link function.
Zero-inflated regression: For count data with excessive zeros (such as insurance claims), it combines logistic regression (whether it is zero) with Poisson regression (the count part).

Decision trees and ensemble methods

Compressing a random forest into a single tree: By extracting important rule paths, the forest is simplified into a single decision tree, sacrificing some accuracy to improve interpretability.
Decision trees inevitably overfit: Because of their low-bias characteristic (perfectly fitting the training data), variance must be controlled through pruning, limiting depth, or ensemble methods (such as random forests).
AdaBoost principle: Iteratively train weak classifiers (such as shallow trees), adjust sample weights to focus on misclassified samples, and improve overall performance through weighted voting.
Out-of-bag validation (OOB): Use the unsampled data in a random forest (about 37%) as a validation set, allowing model evaluation without an additional data split.

Dimensionality reduction

PCA variance explained: When selecting principal components, retain 95% cumulative variance, balancing dimensionality reduction effectiveness and information loss, suitable for scenarios with high feature correlation.
t-SNE vs. PCA: t-SNE preserves local similarity through the t distribution (suitable for visualizing clusters), while PCA preserves global variance (suitable for preprocessing).
Kernel PCA: Maps data to a high-dimensional space through a kernel function (such as RBF) to handle nonlinear structures, with higher computational complexity than linear PCA.

Section 2.7: Clustering

KMeans vs. Gaussian mixture model (GMM): KMeans hard-assigns samples to the nearest cluster, while GMM soft-assigns them (probabilistic membership); the latter is more flexible but requires more computational resources.
DBSCAN++ optimization: Reduces the number of neighborhood queries and accelerates density clustering by sampling core points, suitable for large-scale data.
Hierarchical DBSCAN (HDBSCAN): Automatically determines the number of clusters and identifies clusters of varying density, outperforming the fixed-parameter limitation of traditional DBSCAN.

Correlation analysis

Limitations of Pearson correlation: It detects only linear relationships, while Spearman rank correlation can capture monotonic nonlinear relationships (such as exponential relationships).
Lessons from Anscombe's quartet: Identical statistics (mean, variance, correlation coefficient) may correspond to completely different data distributions, requiring visual verification.

Missing data handling

Types of missing values (MCAR/MAR/MNAR): MCAR (Missing Completely at Random) can be deleted; MNAR (Missing Not at Random) requires modeling the missingness mechanism (e.g., survival analysis).
MissForest imputation: Iteratively predicts missing values based on random forests, suitable for mixed-type data (numeric + categorical), and superior to mean/median imputation.

Data analysis tools

Pandas/Polars/SQL/PySpark syntax comparison: For example, grouped aggregation: Pandas'sgroupby vs. PySpark'sgroupBy, helping migrate code across platforms.
GPU-accelerated Pandas (RAPIDS cuDF): Uses GPUs to parallelize data operations (e.g., sorting, JOIN), accelerating large-scale data processing, and is compatible with the Pandas API.

Advanced SQL operations

Semi Join: Returns only the rows in the left table that match the right table (e.g.,WHERE EXISTS); when filtering without needing fields from the right table, it is more efficient than INNER JOIN.
Use NOT IN with caution: If the right table contains NULL values, the result may be empty; useNOT EXISTSorLEFT JOIN + IS NULLinstead, which is safer.

Python object-oriented programming

Descriptors: Through the__get__/__set__methods to control attribute access, enabling lazy computation and type validation (e.g.,@propertyunderlying mechanism).
PyTorch does not directly call forward()：model()automatically triggers forward, integrating hooks (such as pre/post-processing) and automatic differentiation, while maintaining the integrity of the computation graph.

Project address of AI Engineering Hub

GitHub repository：https://github.com/patchy631/ai-engineering-hub

Target audience of AI Engineering Hub

Beginners: Provides basic concept explanations and practical guides for learners who are interested in AI but lack experience, helping them get started quickly.
Developers and practitioners: Provides concrete code examples and practical cases for engineers and developers, helping them apply AI technology in real projects.
Researchers: Provides cutting-edge technology trends and research cases for academic researchers, making it easy to share and exchange research results.
Data scientists: Provides data professionals with data processing and model optimization guides combined with AI technology, improving work efficiency.
Technology enthusiasts: Provides technology enthusiasts with practical cases of the latest AI tools and frameworks, meeting the need to explore new technologies.

Get the original PDF file of "AI Engineering Hub"; scan the code, follow, and reply: 20250713