Cohere · Command R Family
104B
128k
Dense
Text
Community
2024-04
Attention Structure
GQA
Attention Heads
64
KV Heads
8
Head Dimension
128
Position Embedding
RoPE (θ=2,097,152)
Hidden Dimension
8,192
Intermediate Dim
22,528
Layers
64
Vocab Size
256,000
Activation
SwiGLU
Normalization
LayerNorm
Native Dtype
BF16
Dense Transformer data flow
Input Tokens
Token IDs
Embeddings
256k vocab → 8Kd
Position Embed
RoPE (θ=2,097,152)
Transformer ×64
GQA + SwiGLU + LayerNorm
Final LayerNorm
Layer Normalization
Output Logits
256k vocab probs
Input Tokens
Token IDs
Embeddings
256k vocab → 8Kd
Position Embed
RoPE (θ=2,097,152)
Transformer ×64
GQA + SwiGLU + LayerNorm
Final LayerNorm
Layer Normalization
Output Logits
256k vocab probs
Transformer Block Detail
VRAM demand scaling at FP16 precision
Total Tokens
~4.5T
Compute
~9.0 × 10²³ FLOPs
Real-world application integrity mapping
Composite Integrity Score
FP16 vs INT4 output stability
Needle-in-a-Haystack accuracy
Instruction adherence consistency
Factual grounding confidence
Interactive VRAM estimation
4× A100 80GB or 4× H100 80GB
datacenter