writer Thomas I By AfterPost | 20 October 26
Meta has revealed Prometheus, its gigawatt-scale AI supercluster in New Albany, Ohio, that trained its new Muse foundational models including Muse Wonder, achieving over 85% goodput for pretraining and 90% for reinforcement learning with ∼47,000 GPU hours mean time between interruptions. The first phase is complete with hundreds of thousands of GPUs and Meta is scaling to ∼1 million GPUs.
This is the machine behind Meta's most capable models. Details published Oct 19, 2026 on http://engineering.fb.com: http://go.meta.me/4yXXApZ
What Prometheus actually is:
Two years ago, New Albany was a conventional data center running web traffic. Today it's a single supercomputer spread across multiple buildings - legacy halls, new AI-optimized building, rapid-deployment structures, and partner capacity - linked by Backend Aggregation (BAG) network over fiber distances from 2km to 100km+. That distance forced Meta to engineer around speed of light.
Power: 1 gigawatt target. Hardware: NVIDIA GB200 NVL72 (Catalina) - 72 GPUs, 13.4TB HBM3e, 120kW per 2-rack pair. GB300 NVL72 (Clemente) - 72 GPUs, 20.7TB HBM3e, 1+ exaFLOP FP4, 135kW per rack. Next: Vera Rubin NVL72 (San Miguel) - 170kW+ fully liquid-cooled, 10-20% better perf/watt.
The resilience problem Meta solved:
At supercluster scale, one bad host kills the whole job. Moving from NVL8 to NVL72 increased fault blast radius 9x.
Meta's fix:
- Run 64 of 72 GPUs per rack, hold 8 as instant in-rack spares- Hold ∼2% reserve racks- To run 32,768 active GPUs, provision 37,656 GPUs = 13% over-provisioning- Checkpointing, automated fault recovery, hardware screening
Result: Interruptions peaked mid-2025 at ∼1,000 per day per 8,000 GPUs, mostly NVLink signal issues across 600k+ connections and later HBM faults. After stability sprint March 2026 with NVIDIA engineers embedded on-site, interruptions fell 99%.
Numbers during Muse Wonder training:
- Pretraining goodput: 85%+ (share of time actually advancing model, not lost to restarts)- RL goodput: 90%+- MTBI: ∼47k GPU hours wall-clock, now >54k GPU hours after fleet firmware fixes- Target: 100k GPU hours (2 interruptions per 8,000 GPUs per day)
Why Ohio, why now:
Prometheus features closed-loop liquid cooling for density impossible in 2017 when site broke ground. It currently runs multiple parallel jobs of tens of thousands of GPUs. It trained Muse Wonder and Muse Wonder Full - frontier models for personal and professional agentic tasks.
Next frontier models from Meta Superintelligence Labs will train on expanded Prometheus. Long-term follow-on: Hyperion, planned 5+ GW.
FAQ
Where is Prometheus located?
New Albany, Ohio, at Meta's data center campus, first phase operational October 2026.
How many GPUs does Prometheus have?
Hundreds of thousands today, scaling toward ∼1 million distributed GPUs acting in lockstep.
What does 85% goodput mean?
Goodput = productive training time / total wall-clock time. 85% means only 15% lost to crashes, idle, restarts - extremely high at this scale.
What is MTBI 47k GPU hours?
Mean time between interruptions: cumulative GPU time before something breaks job. 47,000 GPU hours during Muse Wonder training, now >54k after fixes.
What hardware powers Prometheus?
NVIDIA GB200 NVL72 and GB300 NVL72 now, Vera Rubin NVL72 soon, all on Ethernet fabric with closed-loop cooling.
Also read this: https://afterpostint.com/post/h2-strong-gop-threatens-to-block-350b-pentagon-funds-over-russia-strong-h2