Skip to the yard
Waxsealai

Mistral Large 4 Combines a 1T-Parameter MoE with Sparse Attention and Vision

Shipped
Shipper
John Jones
Readers
611

Mistral Large 4 pairs a 1M-token context window with sparse expert routing, image input and strong reported coding and cybersecurity results.

Mistral Large 4 Combines a 1T-Parameter MoE with Sparse Attention and Vision

Mistral Large 4, also known as Le Chonk, is a multimodal model built around a large but sparsely activated expert network. It accepts text and images and produces text, with about 1 trillion total parameters, 49 billion active parameters and a reported context window of 1 million tokens. For teams considering deployment, the central trade-off is clear: the model offers broad capacity and local control, while its scale makes hardware choice and inference setup consequential.

The model is available through the Mistral Large 4 collection and under the Mistral Ecosystem Open-Weight License, described as an MIT derivative that sets limits on model-as-a-service offerings. GGUF conversions from Unsloth and the llama.cpp team provide additional ways to run it. Its architecture combines mixture-of-experts routing, sparse attention and a Pixtral vision encoder.

Two kinds of sparsity

Mistral Large 4’s decoder has 72 layers, a hidden dimension of 6,144 and 64 attention heads. Its attention design joins Multi-head Latent Attention (MLA), which stores a compressed key-value representation, with a DeepSeek Sparse Attention indexer. For each query, the indexer selects up to 2,048 earlier key positions. That reduces the context positions considered by attention; it’s separate from expert routing, which chooses feed-forward networks for each token.

The model uses RoPE in three decoder layers out of every four, with NoPE in the fourth, and shares key-value state across layers. Its RoPE variant, LeRope, applies YaRN scaling with a learned multiplier for each frequency.

The first decoder layer uses a dense feed-forward network. Each of the remaining 71 layers has a mixture-of-experts block with 128 experts. A softmax router selects four experts for each token, and one shared expert runs alongside them, giving five active experts per token. This pattern lets the model draw on a large total pool without activating every expert for every input.

Image processing starts with a 1.6-billion-parameter Pixtral encoder. It has 30 layers, a hidden dimension of 2,048 and 16 attention heads. The encoder processes 14-by-14 patches, then uses a patch-merge projector with a spatial merge size of two. The overall design connects visual input to the text-generating decoder.

Diagram of Mistral Large 4 attention

Security work and reported results

Cybersecurity is a notable focus. Mistral Large 4 scored 82% on Artificial Analysis Cyber Index, a test in which a model reproduces a real vulnerability in open-source software and then patches it. Mistral reports that several closed frontier models scored near zero on the same test because they declined to attempt the task. Reproduction matters to defensive work: verifying a vulnerability can require recreating it before a fix can be checked.

Mistral’s internal tests also found the model useful for malware analysis, vulnerability prioritisation and writing detection rules, tasks it wasn’t explicitly trained for. The company frames these as defensive applications, including vulnerability research and incident response. Open weights let users control deployment and inspect refusal behaviour directly. They can also reduce dependence on an external API during an incident, though running a model of this size brings substantial infrastructure needs.

Open access does not mean the model complies with every request. On cyber prompts from JailbreakBench, StrongREJECT and AgentHarm, Mistral Large 4 had a higher average refusal rate than any other open-weight model in the reported comparison. That result is relevant for teams evaluating its behaviour, but a refusal score alone doesn’t establish how well it will handle a particular organization’s workflows.

The model’s reported coding results include 61.7% on DeepSWE v1.1, 59.4% on SWE-Atlas-QnA and 28.3% on Terminal-Bench 4. Its Coding Agent Index score is 49.8%, ahead of DeepSeek V4 Pro 0813 and Qwen3.8 Max in the stated comparison. For broader work tasks, it reached 59.9% on AutomationBench and 1,393 Elo on AA-Briefcase. It also recorded 93.3% attack resistance on Lakera’s B3 benchmark, which evaluates resistance to prompt injection.

As of October 29, 2026, Mistral Large 4 ranked fifth among open-weight models on Artificial Analysis Intelligence Index. These figures cover different tasks and should be read as separate measurements rather than a single general score. They point to breadth across coding, security, image understanding and agent workflows; practical performance will still depend on the task, serving configuration and hardware.

Deployment choices and limits

Day-one support is listed for Transformers, vLLM and llama.cpp. In Transformers, AutoModelForMultimodalLM loads the model and AutoProcessor prepares conversations and images. Reasoning effort can be set to low, medium or high, with high as the default. Reasoning can’t be turned off: setting reasoning_effort to none still uses the high setting.

vLLM supports an NVFP4 checkpoint aimed at lower memory use, but it requires NVIDIA Blackwell GPUs. A DSpark drafter can speed generation through speculative decoding. For hosted deployment, an Inference Endpoints recipe uses eight NVIDIA H200 GPUs, FP8 weights, tensor and expert parallelism, and the DSpark draft head. The described setup serves up to 131K tokens of context and handles up to 16 simultaneous requests. That is below the model’s stated 1-million-token context window, so deployment limits matter when planning workloads.

Inference Endpoints exposes an OpenAI-compatible API. Its reasoning_effort field accepts the same three settings; answers appear in choices[0].message.content, with reasoning in choices[0].message.reasoning. The API also supports tool calls, structured outputs, image inputs and streaming. Dell AI Servers with H200, B200, GB200 or GB300 hardware have an optimized deployment path through the dell-ai CLI.

For local use, GGUF files are available in IQ2_XXS, Q2_K, Q4_K_M and Q8_0 formats, alongside NVFP4 and BF16 options. The 2-bit versions can run on a single Apple Silicon Ultra computer with 512 GB of unified RAM; the other configurations require distributed inference. Multiple NVIDIA or AMD cards are also supported. In one reported Q2_K run on an M3 Ultra, prefill reached about 90 tokens per second and generation about 15 tokens per second. These figures describe that specific setup, not a general speed guarantee.

Fine-tuning support is available through TRL, with the FP8 model used in an example on a node with eight B300 GPUs. One example trains multilingual reasoning, and the full script with a Slurm launcher is in TRL’s examples. More broadly, teams could use traces from security tasks to adapt smaller models to their own code and logs. Reproduce-then-patch tasks also have verifiable outcomes that can support reinforcement-learning experiments.

Mistral Large 4’s appeal rests on the combination of specialist security results, image input and deployable weights. Its scale, hardware requirements, reasoning controls and license terms all shape where it fits. The useful next step for adopters is to test their own workloads and refusal cases on the intended serving setup, rather than infer production behaviour from benchmark scores alone.

Source: John Jones · huggingface.co

Same vesselRelated cargo