AI SOTA Shift Tracker
Tracks state-of-the-art (SOTA) shifts across public LLM leaderboards, token-utilisation rankings and the AI-SOTA KG knowledge graph.
Live app: https://ai-sota-atlas.lovable.app · Docs in the app: /docs · Architecture: docs/ARCHITECTURE.md · How to use: docs/HOW_TO_USE.md · Ontologies: ontology/
Created by Dr Shyam Sundaram.
1. What the dashboard does
| Section | Route | Purpose |
|---|---|---|
| Leaderboard dashboard | / | Normalized benchmark results per category, filters, charts, details table, batch status, Refresh now / Auto-refresh |
| Token utilisation report | /tokens | KPI band, data-quality panel, charts, country drill-down, granular table, CSV export |
| AI-SOTA KG | /knowledge-graph | Interactive graph of models, developers, countries, benchmarks, categories and token slices |
| API docs | /api-docs | Public REST endpoints (OpenAPI JSON at /api/public/docs) |
| Documentation | /docs | README, How to use, Architecture and Ontology (TTL downloads + metrics) |
| Hardware & Chipsets | /hardware | Accelerator catalogue, server systems, side-by-side comparison, workload memory sizing and measured MLPerf results |
Categories
- Global frontier — LMArena (Text Arena ratings, via its Hugging Face dataset), Stanford HELM Capabilities (mean score), BenchLM, Aider, Humanity's Last Exam, SWE-bench Verified, LiveCodeBench, modelgrep (AA Intelligence & Coding index), Berkeley function-calling, ARC Prize, Vectara
- Hallucination & safety — Suprmind, Vectara (hallucination_rate, factual_consistency, faithfulness, groundedness, citation_accuracy)
- Cybersecurity — Nilgiri cyber range (SPARC Labs), BenchLM cybersecurity, Cybench
- Multimodal — MMMU-Pro and MMMU (Val) overall accuracy (from MMMU's published
leaderboard_data.json); MathVision testmini overall (mathllm.github.io/mathvision); Video-MME overall without subtitles; CharXiv reasoning (RQ) and descriptive (DQ) overall accuracy (publishedval_result.csv). MathVerse, OSWorld, ScreenSpot, MMLU-Pro and OlympiadBench are listed as reference sources only — no machine-readable verified scores - AI infrastructure — MLPerf Inference (MLCommons): best closed-division datacenter offline throughput (tokens/s) per LLM workload. Measures hardware/system speed, not model quality
- India, China, Europe — regional benchmarks and model families. China and Europe also get dedicated slices of the global leaderboards: LMArena's published ratings and modelgrep's Intelligence Index are filtered by developer origin (e.g. Mistral, Aleph Alpha for Europe; DeepSeek, Qwen, GLM, Kimi, MiMo, Ernie, Step for China), so regional leaders are ranked against each other on the same scales as the global boards
- Middle East & Africa — reference-only for now: Falcon (UAE, TII), K2 Think (UAE, MBZUAI), Jais (UAE, Inception/G42), ALLaM (Saudi Arabia, SDAIA) and FANAR (Qatar, QCRI). None appear on the readable global leaderboards yet, so no scores are shown; the category will populate automatically once they do
Sources are defined in pipeline/config.json. Pages that draw their tables in the browser, or publish no table, are kept as reference-only — scores are never invented.
2. Using the dashboard
- Refresh now runs the server-side collector for every source (leaderboards + token data), stores a new batch and reloads charts, tables and the knowledge graph.
- Auto-refresh re-reads the latest stored batch at 1h or 6h.
- Leaderboard filters: category, metric, region, search.
- Import JSON previews a normalized records file client-side.
- Token report: pick a slice (Overall / Programming / Language), a country or developer, drill down, export CSV.
- Knowledge graph: filter node types, country, category, slice, top-N; search; click a node for connections, scores and token KPIs; export PNG/JSON. Auto-updates every ~60s, on tab focus and after every refresh.
Token report and knowledge graph pages each show a Sources panel listing every feed in the current batch with row counts.
3. Data model
Normalized leaderboard record:
{
"source_name": "artificial_analysis",
"source_url": "https://artificialanalysis.ai/leaderboards/models",
"retrieved_at": "2026-08-02T10:30:00Z",
"category": "global_frontier",
"model_name": "Claude Opus",
"model_version": "4.8",
"benchmark_name": "AA Leaderboard",
"metric_name": "intelligence",
"metric_value": 57.9,
"metric_unit": "score",
"rank": 1,
"date_reported": "2026-07-23",
"scope_region": "global",
"task_type": "general",
"raw_hash": "..."
}
Token records carry slice, model, developer, country (developer home country), weekly tokens, share and week-over-week growth. KPIs are defined in pipeline/token_config.json.
4. REST API
| Method | Path | Description |
|---|---|---|
| GET | /api/public/dataset | Latest leaderboard dataset |
| POST | /api/public/scrape | Run collector now (leaderboards + tokens) |
| POST | /api/public/ingest | Push a normalized dataset (from the Python pipeline) |
| GET | /api/public/tokens | Token records; slice, country, developer, model, group_by, format=csv, limit |
| POST | /api/public/tokens/refresh | Re-collect token data only |
| GET | /api/public/graph | Knowledge graph; mini, types |
| GET | /api/public/docs | OpenAPI 3 spec |
5. Offline Python pipeline
python -m pip install -r pipeline/requirements.txt
python pipeline/scraper.py --config pipeline/config.json --out pipeline/out/raw_records.json
python pipeline/process.py --in pipeline/out/raw_records.json --out src/data/records.json
python pipeline/scheduler.py --once --dashboard-url https://ai-sota-atlas.lovable.app
Async, robots.txt-aware, randomized 1–5 s delays, rotating User-Agents, Pydantic extraction schema, exponential backoff; pandas cleaning, snake_case columns, dedup, per-category caps. See pipeline/README.md.
6. Known limitations
- Token numbers cover only OpenRouter traffic; under-represents vendors with strong direct APIs.
- Country values are attributed from the developer's home country.
- Per-language slices come from a top-10-only feed.
- Model names differ across sources, so only part of the models are matched in the knowledge graph (shown on the page).
- Some benchmark sites render tables client-side and yield no scores yet.
7. Ontologies
ontology/leaderboard.ttl, ontology/tokens.ttl, ontology/knowledge-graph.ttl, ontology/hardware.ttl — OWL schemas per section; downloadable at /docs → Ontology. See ontology/README.md.
Hardware & Chipsets
The /hardware section connects model evaluation with deployment constraints. Its source-linked catalogue covers NVIDIA H100/H200/B200 and GB200, AMD MI300X/MI325X/MI350X, Google TPU v5p and AWS Trainium2/Inferentia2. Server configurations include NVIDIA DGX and GB200 NVL72, Dell PowerEdge, Lenovo ThinkSystem and Supermicro; HPE is currently a configuration reference, with unverified values left unknown.
- Search/filter by chip vendor and workload; inspect precision, theoretical compute, memory/bandwidth, power, interconnect and software; compare up to four accelerators and export the filtered catalogue or servers as CSV.
- Workloads are pretraining/continued pretraining, full fine-tuning, LoRA, QLoRA and inference. Memory sizing is a planning estimate, not a promised fit, tokens/second prediction or time-to-train calculator.
- Decimal GB is used consistently; source GiB values are converted and identified. Aggregate GPU memory is not automatically one shared memory pool; CPU RAM is separate.
- The sizing panel separates weights, trainable gradients/Adam states, adapters, architecture-specific KV cache, explicit activation/runtime budget and reserved memory. Unknown KV configuration produces a partial estimate.
- Live MLPerf results preserve workload, system, submitter, accelerator description, weight precision, scenario, division, version, source URL and retrieval time. Compare like-for-like workloads; Offline throughput is not serving latency.
- Refresh now runs the primary collection and checks machine-readable MLPerf results. Curated product specifications retain their last verification date until explicitly reverified. Cloud capacity and prices are not guaranteed; prices remain unknown.
- The graph adds accelerator, server-system and hardware-vendor nodes. Reported training is supported by model-specific evidence (initially Meta Llama 3.1 405B on H100); benchmarked-on is a separate relation, never proof of pretraining or production deployment. Sizing estimates are not stored as historical graph facts.
- MLPerf Training is reference-only: prescribed task time-to-target results are not interchangeable with arbitrary LLM training or fine-tuning.
8. Development
git clone <this-repository-url>
cd <repository-name>
npm i
npm run dev
Built with Lovable — changes sync two-way with GitHub.