Audio Companion Script: Model Size Is Not One Number
When people talk about running a model locally, they usually start with the model name. Seven billion. Thirty five billion A three billion. One hundred twenty billion.
That is useful, but it is not enough.
The first thing to separate is weight memory from runtime memory. Weight memory is the parameter count times the bytes per parameter. If the model is seven billion parameters at sixteen bits, the weights alone are roughly fourteen gigabytes before overhead.
But inference is not just weights sitting on disk. The runtime also needs a KV cache. That cache stores keys and values for previous tokens so the model does not recompute the full prompt at every generation step. The cache grows with context length, layer count, KV head count, head dimension, batch size, and cache precision.
So a model that fits at four thousand tokens may not fit at thirty two thousand tokens. The weights did not change. The cache did.
Then there is runtime overhead: allocator behavior, temporary buffers, tokenizer state, framework state, and sometimes an extra draft model or adapter loaded beside the target model.
Mixture-of-experts models add another source of confusion. Total parameters and active parameters answer different questions. Total parameters usually describe what has to be stored. Active parameters are closer to the per-token expert compute path. A model can advertise a small active parameter count and still require a large memory footprint if all experts need to be resident.
The practical move is simple: do not ask, "How big is the model?" Ask, "What are the weights, what is the KV cache at my target context, what overhead do I expect, and do I need any auxiliary models loaded?"
That question gives you a real local inference budget.
After this segment, you should be able to explain why parameter count is only the first term in the memory estimate, and why MoE active parameter count is not the same thing as local memory footprint.
Walking Questions
Why is parameter count not enough to know whether a model fits locally?
Expected concepts:
weights are only one memory term
KV cache grows with context
runtime overhead and extra models matter
Hint: Name at least two memory terms besides model weights.
For an MoE model, what is the difference between total parameters and active parameters?
Expected concepts:
total parameters usually drive stored weight memory
active parameters describe the per-token compute path
all experts may still need to be resident
Hint: Separate storage from per-token compute.
Modules
Model Anatomy and Memory Budgets
Build the parameter, quantization, KV-cache, and MoE mental models needed for local inference planning.
1.1 Model Size Is Not One Number
Objective
Build a usable mental model for the first local inference question: what actually has to fit in memory?
Why This Matters
Model names compress too many different numbers into one label. "7B", "35B-A3B", and "120B" do not tell you the same thing. A local runtime has to hold weights, runtime caches, temporary activations, tokenizer state, framework overhead, and sometimes both a target model and a draft model.
The root cause of confusion is usually treating parameter count as the whole memory budget. It is only the largest static term.
The Working Budget
Start with this rough decomposition:
weights: total parameters times bytes per parameter
KV cache: sequence length times layer count times key/value width times bytes per element
activations and temporary buffers: runtime-dependent scratch space
extra models: draft models, rerankers, adapters, or classifiers
For a first estimate, write:
memory ~= weights + KV cache + runtime overhead
Then add margin. Local inference fails in the margin, not in the clean formula.
Dense vs MoE
A dense model usually uses nearly all of its parameters for each token. A mixture-of-experts model has total parameters and active parameters. Total parameters are what you generally need to store. Active parameters are closer to the per-token compute path.
This distinction matters because a 35B-A3B style model can look small from a compute perspective and still require a much larger storage footprint than a dense 3B model.
Exercise
Pick three models you are curious about. For each one, write down:
Total parameters.
Active parameters, if it is an MoE model.
Weight precision or quantization.
Maximum context you want to use.
Whether you need a second model loaded at the same time.
Do not decide whether the model fits yet. The goal is to stop using a single parameter number as the answer.
Checkpoint
You should now be able to explain why a quantized 30B model, a dense 8B model, and a 35B-A3B MoE model can stress a machine in different ways.
1.2 Parameters, Activations, and KV Cache
Objective
Estimate the memory terms that change with architecture and sequence length.
int8: about 1 byte per parameter, plus small metadata overhead
int4: about 0.5 bytes per parameter, plus scale and packing overhead
Quantization reduces the stored weights. It does not erase the KV cache, temporary buffers, or the cost of long context.
KV Cache Memory
The KV cache stores attention keys and values for prior tokens so the model does not recompute the whole prefix every step. It grows with context length and batch size.
You should be able to answer: if a model fits at 4K context, which term grows when you ask for 32K context?
Quiz
For a mixture-of-experts model, which statement is the most useful first-pass memory rule?
Correct: Total parameters usually drive weight storage, while active parameters are closer to per-token compute.
Which change directly increases KV cache memory during generation?
Correct: Increasing context length while keeping the same model architecture.
1.3 Memory Estimation Lab
Objective
Turn the memory budget into a small calculator you can use before downloading a model.
What To Build
Implement a function that estimates inference memory in GB from architecture numbers. The result should be approximate, not authoritative.
The useful behavior is not perfect precision. The useful behavior is catching impossible plans early.
Inputs
The calculator needs:
total parameters in billions
weight precision in bits
context length
number of layers
hidden size
attention head count
KV head count
batch size
KV cache precision in bits
fixed runtime overhead
Interpretation Rules
Use decimal GB for the lab so the math stays readable. Real systems may report GiB instead. Treat the output as a budget estimate with margin, not a guarantee.
If the estimate says 28 GB, do not expect a 32 GB machine to be comfortable once the runtime, OS, UI, tokenizer, draft model, and allocator fragmentation are included.
Manual Test
Estimate a 7B dense model with:
16-bit weights
4096 token context
32 layers
4096 hidden size
32 attention heads
32 KV heads
16-bit KV cache
2 GB overhead
Then repeat with 4-bit weights. The KV cache should not change.
Reflection
What term dominated the 16-bit estimate? What term became more visible after quantization?
Learn the MLX execution model through unified memory, lazy evaluation, model loading, and small tensor experiments.
2.1 MLX Runtime Mental Model
Objective
Understand MLX as a runtime for Apple Silicon local inference, not just another tensor library import.
The Core Idea
MLX is designed around Apple Silicon constraints and strengths. The important mental model is:
arrays can live in unified memory
operations are lazily evaluated
explicit evaluation boundaries matter
compilation can reduce repeated dispatch overhead
peak memory still matters even when CPU and GPU share memory
Unified memory removes a large class of explicit device-copy decisions. It does not make memory infinite. A local model can still fail because weights, KV cache, scratch buffers, and other applications compete for the same system memory pool.
Lazy Evaluation
MLX operations build work until evaluation is forced. Evaluation can happen when you need a value, call mx.eval, materialize output, or cross a boundary the runtime must resolve.
For inference work, lazy evaluation affects:
measured latency
where memory peaks occur
whether a benchmark accidentally times graph construction instead of execution
when generated tokens become available
Why This Matters For DFlash
DFlash's MLX implementation runs a target model and a draft model together. It captures selected hidden states from the target, runs a block draft, verifies the block with the target, accepts matching tokens, and rolls back cache state when the draft overreaches.
That makes MLX behavior visible. Cache shape, evaluation timing, and memory peaks become part of the algorithm, not just implementation trivia.
You should be able to explain why timing a lazy runtime requires forcing evaluation at deliberate points.
2.2 Read an MLX Model Loader
Objective
Learn to read an MLX model loader as a systems document.
What To Inspect
In the DFlash MLX implementation, focus on the loader and config path before reading the generation loop.
Look for:
which Hugging Face files are downloaded
which fields are read from config.json
how layer counts, head counts, block size, and target-layer IDs enter the draft model
where weights are loaded
where the draft model is bound to the target model
Why Config Fields Matter
Architecture fields are not metadata trivia. They decide:
tensor shapes
KV cache width
rotary embedding configuration
how many layers run
which hidden states the draft model consumes
whether a user-facing block_size is legal or sensible
When a model does not load, the root cause is often a mismatch between assumed architecture and actual config.
Exercise
Open dflash/model_mlx.py in the DFlash repo and trace load_draft.
Write down:
The fields used to construct DFlashConfig.
Which fields affect memory.
Which fields affect generation behavior.
Which fields connect the draft model to the target model.
Checkpoint
You should be able to point at a model config and say which numbers will affect local memory before you run inference.
2.3 Minimal MLX Tensor Lab
Objective
Practice the MLX execution model with a tiny tensor workload before using a full model.
Setup
Use this lab when you are on Apple Silicon with MLX installed:
python -m pip install mlx
Then create a scratch file outside this course repo and run:
import mlx.core as mx
x = mx.random.normal((1024, 1024))
w = mx.random.normal((1024, 1024))
y = x @ w
mx.eval(y)
print(y.shape)
print(mx.get_peak_memory() / 1e9)
What To Observe
Change the matrix size and note:
when memory is allocated
how peak memory changes
whether timing includes mx.eval
whether repeated calls behave differently after warmup
Extension
Wrap the matrix multiply in a function and compile it with mx.compile. Compare warm and cold runs. The goal is not to make a universal benchmark. The goal is to feel the difference between Python dispatch, lazy graph construction, and evaluated work.
Checkpoint
Before using MLX for model inference, you should be comfortable answering: did my benchmark time the actual computation, or did it mostly time setup?
Decoding, Speculation, and DFlash
Use DFlash as a concrete case study in speculative decoding, block drafts, verification, and measurement.
3.1 Autoregressive Decoding and Speculation
Objective
Understand why speculative decoding exists and what problem it tries to solve.
Autoregressive Baseline
Standard LLM generation is sequential. The model predicts the next token, appends it to the sequence, updates cache state, and repeats. The KV cache avoids recomputing the entire prefix, but the next-token dependency still limits parallelism.
This creates a bottleneck:
one target-model step usually commits one token
large models are expensive per step
long context increases cache pressure
batching helps serving workloads but does not remove the dependency chain for one response
Speculative Decoding
Speculative decoding adds a cheaper drafter. The drafter proposes multiple tokens. The target model verifies them. Accepted tokens become real output. Rejected tokens are replaced by the target's token and cache state is corrected.
The speedup depends on:
draft cost
verification cost
acceptance length
block size
cache rollback overhead
quality match between drafter and target
Speculation is useful only when the combined draft plus verify path is cheaper than making the target produce each token alone.
DFlash Frame
DFlash uses a block-diffusion draft model for speculative decoding. The interesting part is not "diffusion" as a vague buzzword. The interesting part is that a block proposal can expose more parallel work, while the target model still verifies correctness.
Exercise
For a model you care about, write a speculative decoding budget:
target model:
draft model:
block size:
expected acceptance length:
extra memory for draft:
rollback risk:
metric I would trust:
Checkpoint
You should be able to explain why a draft model can make generation faster without being allowed to change the final target-model distribution unchecked.
3.2 DFlash Reading Lab
Objective
Read DFlash as an inference-systems design, not just as a package to install.
Transformers for direct model hacking and reference behavior
vLLM or SGLang for serving-oriented throughput and speculative decoding support
Do not choose a runtime because it is fashionable. Choose it because it fits the machine, model, and workload.
What Counts As Evidence
Good evidence:
a config-derived memory estimate
a measured prompt-processing speed
a measured generation speed
a peak-memory reading
acceptance-length distribution for speculative decoding
the exact model and draft model IDs
Weak evidence:
"it should fit"
one short generation with no memory measurement
comparing different prompts across runtimes
reporting only tokens per second without context length
Capstone
Produce the memo, then run the smallest possible experiment that could falsify it. If the run fails, update the root-cause section before changing tools.
Checkpoint
You should now have a reusable method for answering: what model can I run, on what machine, with what runtime, and why?