Mistral AI · Mistral Family
12B
128k
Dense
Text
Community
2024-07
Attention Structure
MHA
Attention Heads
32
KV Heads
8
Head Dimension
128
Position Embedding
RoPE (θ=1,000,000)
Hidden Dimension
5,120
Intermediate Dim
14,336
Layers
40
Vocab Size
131,072
Activation
SwiGLU
Normalization
RMSNorm
Native Dtype
BF16
Dense Transformer data flow
Input Tokens
Token IDs
Embeddings
131k vocab → 5Kd
Position Embed
RoPE (θ=1,000,000)
Transformer ×40
MHA + SwiGLU + RMSNorm
Final RMSNorm
Layer Normalization
Output Logits
131k vocab probs
Input Tokens
Token IDs
Embeddings
131k vocab → 5Kd
Position Embed
RoPE (θ=1,000,000)
Transformer ×40
MHA + SwiGLU + RMSNorm
Final RMSNorm
Layer Normalization
Output Logits
131k vocab probs
Transformer Block Detail
VRAM demand scaling at FP16 precision
Total Tokens
~9T
Compute
~1.1 × 10²³ FLOPs
Real-world application integrity mapping
Composite Integrity Score
FP16 vs INT4 output stability
Needle-in-a-Haystack accuracy
Instruction adherence consistency
Factual grounding confidence
Interactive VRAM estimation
1× RTX 6000 Ada or 2× RTX 4090
prosumer