09. Training vs Inference

Training is how a model learns. Inference is how a model is used. They are completely different operations with different costs, infrastructure, and requirements.

Understanding this distinction is critical for both building and deploying ML systems.

Once you’ve studied (trained), answering a question is fast. You don’t re-read all your notes for every question — you use what you’ve already learned.

What happens: The model processes training data repeatedly, adjusting its internal weights to minimize prediction error.

flowchart TD
    A[Training Data\nMillions of examples] --> B[Forward Pass\nModel makes prediction]
    B --> C[Loss Calculation\nHow wrong was it?]
    C --> D[Backpropagation\nCompute gradients]
    D --> E[Weight Update\nGradient descent step]
    E --> B
    E --> F{Converged?}
    F -->|No| B
    F -->|Yes| G[Trained Model\nFrozen weights]

What happens: The trained model (frozen weights) takes a new input and produces a prediction in real time.

flowchart LR
    A[New Input] --> B[Trained Model\nFrozen weights]
    B --> C[Forward Pass only\nno gradients]
    C --> D[Prediction / Output]

flowchart LR
    A[Training] --> B[Offline\nperiodic]
    A --> C[Expensive\ngradients + backprop]
    A --> D[Entire dataset]
    A --> E[Modifies weights]

    F[Inference] --> G[Online\ncontinuous]
    F --> H[Cheap\nforward pass only]
    F --> I[Single input]
    F --> J[Weights frozen]

Process many predictions at once, asynchronously.

# Batch inference: predict churn for all customers tonight
import pandas as pd
import pickle

model = pickle.load(open("churn_model.pkl", "rb"))
customers = pd.read_csv("all_customers.csv")

# Predict for entire customer base
customers["churn_probability"] = model.predict_proba(
    customers[["age", "tenure", "monthly_charges"]]
)[:, 1]

customers.to_csv("churn_predictions.csv", index=False)
# Results used by marketing team tomorrow

Use when: Recommendations pre-generated nightly, risk scores updated daily, report generation.

Predict immediately when a request arrives.

# FastAPI real-time inference
from fastapi import FastAPI
import pickle
import numpy as np

app = FastAPI()
model = pickle.load(open("fraud_model.pkl", "rb"))

@app.post("/score-transaction")
def score(transaction: dict):
    features = np.array([[
        transaction["amount"],
        transaction["hour_of_day"],
        transaction["merchant_category"],
        transaction["distance_from_home"]
    ]])
    prob = model.predict_proba(features)[0][1]
    return {
        "fraud_probability": prob,
        "decision": "block" if prob > 0.8 else "allow"
    }

Use when: Fraud detection (must be instant), search ranking, real-time recommendations.

For production, raw model inference is often too slow or expensive. Common optimizations:

# Example: quantize a PyTorch model for faster inference
import torch

model = torch.load("model.pt")
model.eval()

# Dynamic quantization: weights stored as int8 at inference time
quantized_model = torch.quantization.quantize_dynamic(
    model,
    {torch.nn.Linear},
    dtype=torch.qint8
)
# Typically 2-4x faster on CPU with minimal accuracy loss

Q: Why does training use so much more compute than inference?

A: Training requires forward pass + loss calculation + backpropagation + weight updates for every batch. Backpropagation stores intermediate activations to compute gradients — consuming significant memory and compute. Inference only needs a single forward pass with frozen weights — no gradient computation, no optimizer state, just matrix multiplications. A training cluster might cost $10,000/hour; the same model at inference might cost $0.01 per 1,000 requests.

Q: When would you choose batch inference over real-time inference?

A: Batch (offline) inference when: results don’t need to be immediate (nightly churn scores, weekly recommendations), high throughput is more important than low latency, or the cost of real-time infrastructure isn’t justified. Real-time (online) inference when: decisions must happen instantly (fraud detection, search ranking), user experience requires immediate response, or actions depend on current context (live recommendation).

• Running gradient computation during inference (wastes memory and compute)
• Loading model weights from disk on every request (use server startup loading)
• Not batching inference requests (leaves GPU underutilized)
• Using training infrastructure for inference (overprovisioned and expensive)

← Previous: 08. Reinforcement Learning Next →: 10. Models