Why this exists
Most benchmarks die the same death: their answers leak into training data and the leaderboard turns into a memory test. PowderBench is contamination-free by construction — every round asks about weather that hasn't happened yet. There is nothing to memorize, nothing to overfit, and no way to peek. You, your ML pipeline, or your AI agent make a real forecast, the atmosphere runs its own compute, and a few days later the mountains grade your work.
It's also just fun. Snowfall is one of the hardest quantities in weather forecasting — orographic effects, rain-snow lines, wind loading — and it comes with a built-in villain: the global weather models submit automatically every round. The whole site is one long dare: nobody is beating the weather models. Yet.
Three leagues, three kinds of truth
Every league is named for its ground truth, because the truth is the game:
- Stations — 45 SNOTEL sites in the western US. Automated snow pillows and depth sensors: the closest thing to objective snowfall that exists at scale. Live October–May.
- ERA5 — 23 points across the Andes, New Zealand, and Australia, scored against ERA5 reanalysis. No public station network covers the southern hemisphere's ski country, so the truth is a model analysis — disclosed loudly, smoothed at ~25 km, and identical for everyone. Live all austral winter.
- Resorts — scored against what the mountain itself publishes in its morning snow report. Resort numbers run generous, which is exactly why they get their own league instead of contaminating the others: everyone forecasts the same published number, and the front page keeps the receipts by plotting resort claims next to ERA5 at the same coordinates.
One set of rules everywhere: daily rounds, 24/48/72-hour horizons, Powder Score against climatology, QC that voids bad data for everyone equally.
Fair by design
- The cutoff is the only wall. Any information source is
allowed — NWP output, satellite, webcams, your knees. A submission counts
only if it landed on
mainbefore the cutoff, and GitHub sets merge timestamps, so deadlines can't be forged. - No accounts, no API keys. Entering is a pull request with a CSV in it. The scoring code, the QC rules, and every result are public and deterministic — the leaderboard can be recomputed by anyone.
- Baselines keep everyone honest. Climatology defines zero. Open-Meteo's global models submit every round. Persistence and zeros run too, so there's always a floor under the table.
- Agents welcome. Automated competitors are first-class citizens — label your team honestly and bring your best pipeline.
How resort scraping works (and its manners)
The Resorts league depends on archiving snow reports that vanish daily, so a scraper visits each resort's own website twice a day. It behaves like a good guest, and the policy is enforced in code, not just promised:
- Aggregator sites are never scraped — only individual resort sites, each onboarded after a robots.txt and terms-of-use review that is recorded, dated, in the public registry.
- The scraper identifies itself honestly
(
PowderBench/0.1 (+https://powderbench.com; …)), re-checks robots.txt on every run, spaces its requests, and touches each site at most twice a day — less traffic than one human visitor. - Reported values are reproduced minimally (a daily snowfall number), always attributed, as factual weather data.
- If a resort would rather not be included, an issue on GitHub is enough: the scraper is disabled for that resort going forward.
Data sources & licensing
- SNOTEL — USDA Natural Resources Conservation Service, public domain.
- ERA5 — Copernicus/ECMWF reanalysis, served via Open-Meteo (CC BY 4.0), which also powers the NWP baselines.
- Resort snow reports — © their resorts, reproduced minimally with attribution; per-resort provenance lives in the public registry.
- Code — MIT, at github.com/andrewnakas/powderbench.
The benchmark runs entirely on GitHub Actions: rounds open, baselines submit, reports get archived, truth gets fetched, scores resolve, and this site redeploys — every day, in public.
Contact
Questions, bugs, a resort that wants in (or out), a station we should add? Open an issue. Bug reporters get eternal glory in the README.