Mixture-of-Experts: Active Parameters Do Not Set the Model Storage

Read what the router selects

In the sparse mixture-of-experts arrangement discussed here, a router chooses some expert networks to process a token's intermediate representation. Their outputs are combined before processing continues. Other parts of the model are shared. Sparse selection reduces the expert population participating in that token's computation; it does not erase the unselected learned weights.

An expert can be a feedforward network inside a transformer layer. It need not be a complete language model with its own independent copy of every shared component. Consequently, an 8x7B name is insufficient grounds for multiplying eight by seven billion to obtain the model's distinct weight inventory.

Treat an advertised active count as an architectural count with a stated boundary. It does not directly measure bytes transferred, elapsed time or useful work completed. Ask which shared components and selected experts the count includes.

References: [M1][H1]

Use the January 2024 Mixtral example at its stated boundary

Mixtral of Experts v1, submitted January 8, 2024, describes eight feedforward experts per layer and two selections per token at each layer. Selections can change over timesteps. Its abstract rounds the counts to 47B available parameters and 13B active during inference.

These historical, model-specific figures illustrate participation versus inventory. They establish neither a universal MoE topology nor a current performance recommendation.

References: [P1]

ME-213: count 10 billion total and 4 billion per token

Fictional example ME-213 stipulates one simplified counting boundary. It contains two billion shared parameters and eight disjoint expert populations of one billion parameters each. All common components, including routing parameters, are included in the stipulated shared population. For this example, that whole shared population participates for each token and exactly two experts are selected.

The distinct model inventory is 2 + 8 × 1 = 10 billion parameters. One token's participating population is 2 + 2 × 1 = 4 billion. The remaining six expert populations still belong to the model. These inputs describe a toy architecture, not Mixtral's layer accounting.

Now stipulate that every stored weight value occupies exactly two bytes. The complete inventory contains 10,000,000,000 × 2 = 20,000,000,000 bytes, or 20 decimal GB, where one GB is one billion bytes. Applying two bytes to the active count gives 8 GB of raw values for that token's participating population. It does not give the storage of the complete inventory.

This arithmetic excludes metadata, file-format overhead and all runtime state, including attention KV cache, activations and temporary buffers. The 20 GB result is a raw-value total, not a measured checkpoint size or GPU allocation requirement.

Two tokens can touch 6 billion distinct parameters

Keep ME-213's assumptions and suppose token A selects experts 1 and 2, while token B selects experts 3 and 4. Each token still has four billion participating parameters. Across those two tokens, the distinct touched inventory contains the two-billion shared population plus four different one-billion expert populations: 2 + 4 × 1 = 6 billion.

Keeping the count at four billion misses the different expert selections. Adding four plus four to obtain eight billion counts the same shared population twice. If both tokens instead selected experts 1 and 2, their distinct union would remain four billion.

This union describes which weights the stipulated tokens touch. It does not specify when those weights are accessed or how many are simultaneously resident on a device. A batch may reach more experts than one token does, and later tokens may reach others. Neither observation changes the complete ten-billion-parameter inventory.

Locate the complete inventory and the path to each selected expert

The complete weights need an account of their locations in the offered arrangement. Their total does not require every expert to reside simultaneously on one GPU.

Mixtral's paper describes expert parallelism: tokens travel to the GPU holding their selected expert, and outputs return. It also identifies load-balancing challenges. Here the communication carries the work to the expert.

Other supported implementations can move weights to the compute device. Hugging Face Accelerate v1.11.0's Big Model Inference guide describes distributing weights across devices and using CPU or disk offload, with transfer overhead during inference. This is an implementation example; an offer must establish support for its own model, backend and versions.

A local resident subset can include experts that the current token does not select. Conversely, a selected expert need not already be resident on the token's starting GPU. Ask what remains on each device, what is held elsewhere and what must move when routing reaches another expert. Include any duplicated shared weights in the physical memory account.

Adding device capacities does not describe one allocation. A placement proposal must explain the local requirements and actual access paths. Parameter arithmetic alone cannot predict latency, communication cost, load balance or task quality.

References: [P1][A1]

Compare an active-count quote with a supported arrangement

Consider two fictional offers for ME-213. Offer A says four billion active parameters and concludes that 8 GB stores the model at two bytes per value. It gives no location for the other weights. The conclusion omits twelve decimal GB of the stipulated raw inventory. A smaller local allocation might still be part of an arrangement, but the offer has not explained that arrangement.

Offer B identifies the same ten-billion total, four-billion per-token count and two-byte representation. It supplies a placement map for the complete weights, states which shared or expert populations have additional copies, and identifies the supported path to experts outside each GPU's resident subset. It names the software versions, devices and any CPU, storage or inter-device resources required. These are fictional proposal contents, not test results.

B makes the capacity claim possible to assess. The next question is whether the offered arrangement meets the intended use. Request relevant per-device peak-memory evidence and complete response-time, throughput and task-quality evidence for the actual model revision, representation, input/output lengths and concurrency. Have the software provider disclose the routing and placement conditions of that evaluation. A quiet run reaching only a limited expert population leaves a broader claim unresolved.

Compare the complete resources and support being purchased. If offload or expert distribution depends on equipment or connections omitted from the price, obtain their scope and cost or confirm that the required resources already exist. No active-parameter ratio supplies a corresponding speed or price ratio.

Use the related quantization guide for representation and runtime-memory questions. Use the training guide for replicated or sharded training state, and the serving-layout guide for aggregated versus separate prefill/decode pools. Those decisions can coexist with MoE routing; none supplies the missing account of its expert weights.

Bring the selected equipment to a physical deployment inquiry

Once the appointed platform team selects the supported arrangement, identify the actual servers, rack locations and physical connections it requires. A parameter count alone cannot produce that installation package.

Leviathan's Complete Physical Deployment service is relevant when the selected equipment needs agreed receiving, rack integration, onsite installation or IT cabling. Use the related service and project-inquiry links to discuss a nonconfidential equipment summary, quantities, locations, connection requirements, receiving team and availability.

Model operation, expert-routing implementation, tuning and training remain with separately appointed software providers. The physical engagement has its own defined work and handoff.

References: [L1]