Google DeepMind · Gemma Family
27B
8k
Dense
Text
Community
2024-07
Attention Structure
MHA
Attention Heads
32
KV Heads
16
Head Dimension
128
Position Embedding
RoPE (θ=10,000)
Hidden Dimension
4,608
Intermediate Dim
36,864
Layers
46
Vocab Size
256,128
Activation
GELU
Normalization
RMSNorm
Native Dtype
BF16
Dense Transformer data flow
Input Tokens
Token IDs
Embeddings
256k vocab → 5Kd
Position Embed
RoPE (θ=10,000)
Transformer ×46
MHA + GELU + RMSNorm
Final RMSNorm
Layer Normalization
Output Logits
256k vocab probs
Input Tokens
Token IDs
Embeddings
256k vocab → 5Kd
Position Embed
RoPE (θ=10,000)
Transformer ×46
MHA + GELU + RMSNorm
Final RMSNorm
Layer Normalization
Output Logits
256k vocab probs
Transformer Block Detail
VRAM demand scaling at FP16 precision
Total Tokens
13T
Compute
~2.4 × 10²³ FLOPs
Real-world application integrity mapping
Composite Integrity Score
FP16 vs INT4 output stability
Needle-in-a-Haystack accuracy
Instruction adherence consistency
Factual grounding confidence
Interactive VRAM estimation
1× A100 80GB or 1× H100 80GB
datacenter