DeepSeek AI · DeepSeek Family
33B
128k
Dense
Text
Community
2025-01
Attention Structure
GQA
Attention Heads
40
KV Heads
8
Head Dimension
128
Position Embedding
RoPE (θ=1,000,000)
Hidden Dimension
5,120
Intermediate Dim
27,648
Layers
64
Vocab Size
152,064
Activation
SwiGLU
Normalization
RMSNorm
Native Dtype
BF16
Dense Transformer data flow
Input Tokens
Token IDs
Embeddings
152k vocab → 5Kd
Position Embed
RoPE (θ=1,000,000)
Transformer ×64
GQA + SwiGLU + RMSNorm
Final RMSNorm
Layer Normalization
Output Logits
152k vocab probs
Input Tokens
Token IDs
Embeddings
152k vocab → 5Kd
Position Embed
RoPE (θ=1,000,000)
Transformer ×64
GQA + SwiGLU + RMSNorm
Final RMSNorm
Layer Normalization
Output Logits
152k vocab probs
Transformer Block Detail
VRAM demand scaling at FP16 precision
Total Tokens
~800B (distill)
Compute
~2.8 × 10²² FLOPs
Real-world application integrity mapping
Composite Integrity Score
FP16 vs INT4 output stability
Needle-in-a-Haystack accuracy
Instruction adherence consistency
Factual grounding confidence
Interactive VRAM estimation
1× A100 80GB or 1× H100 80GB
datacenter