DeepSeek AI · DeepSeek Family
671B
128k
MoE
Text
Community
2024-12
Attention Structure
MLA
Attention Heads
128
KV Heads
128
Head Dimension
128
Position Embedding
RoPE (θ=10,000)
Hidden Dimension
7,168
Intermediate Dim
18,432
Layers
61
Vocab Size
129,280
Activation
SwiGLU
Normalization
RMSNorm
Native Dtype
BF16
MoE Transformer data flow
Input Tokens
Token IDs
Embeddings
129k vocab → 7Kd
Position Embed
RoPE (θ=10,000)
Transformer ×61
MLA + SwiGLU + RMSNorm
Final RMSNorm
Layer Normalization
Output Logits
129k vocab probs
Input Tokens
Token IDs
Embeddings
129k vocab → 7Kd
Position Embed
RoPE (θ=10,000)
Transformer ×61
MLA + SwiGLU + RMSNorm
Final RMSNorm
Layer Normalization
Output Logits
129k vocab probs
Transformer Block Detail
VRAM demand scaling at FP16 precision
Total Tokens
14.8T
Compute
~2.8 × 10²⁴ FLOPs
Real-world application integrity mapping
Composite Integrity Score
FP16 vs INT4 output stability
Needle-in-a-Haystack accuracy
Instruction adherence consistency
Factual grounding confidence
Interactive VRAM estimation
Multi-node: 2+ DGX H100 systems (NVLink + InfiniBand)
supercompute