TII (UAE) · Falcon Family
180B
4k
Dense
Text
Community
2023-09
Attention Structure
MQA
Attention Heads
232
KV Heads
8
Head Dimension
64
Position Embedding
RoPE (θ=10,000)
Hidden Dimension
14,848
Intermediate Dim
59,392
Layers
80
Vocab Size
65,024
Activation
GELU
Normalization
LayerNorm
Native Dtype
BF16
Dense Transformer data flow
Input Tokens
Token IDs
Embeddings
65k vocab → 15Kd
Position Embed
RoPE (θ=10,000)
Transformer ×80
MQA + GELU + LayerNorm
Final LayerNorm
Layer Normalization
Output Logits
65k vocab probs
Input Tokens
Token IDs
Embeddings
65k vocab → 15Kd
Position Embed
RoPE (θ=10,000)
Transformer ×80
MQA + GELU + LayerNorm
Final LayerNorm
Layer Normalization
Output Logits
65k vocab probs
Transformer Block Detail
VRAM demand scaling at FP16 precision
Total Tokens
3.5T
Compute
~7.1 × 10²³ FLOPs
Real-world application integrity mapping
Composite Integrity Score
FP16 vs INT4 output stability
Needle-in-a-Haystack accuracy
Instruction adherence consistency
Factual grounding confidence
Interactive VRAM estimation
8× A100 80GB (DGX A100) or 8× H100 (DGX H100)
supercompute