Methodology
How we rank, review, and compare AI models. Transparent, reproducible, and community-driven.
Our Review Process
Source Collection
We gather data from multiple independent sources:
- Official Announcements: Company blogs, press releases, GitHub repos
- Academic Papers: arXiv, conference proceedings
- Standardized Leaderboards: MLPerch, LiveCodeBench, SWE-bench, LMSYS Chatbot Arena
- Community Reports: Reddit (r/MachineLearning, r/LocalLLaMA), Hacker News, Discord/Slack
- Financial Filings: SEC EDGAR, company filings
AI-Assisted Draft
Our AI (Allam 7B by SDAIA, reviewed by Hermes Agent) generates draft articles from collected sources. AI is used for research synthesis and draft generation only — not for factual claims.
7-Stage Verification
Each draft passes through 7 specialized subagents:
- Benchmark Analyst: Verifies benchmark claims against independent sources. Assigns confidence levels (HIGH/MEDIUM/LOW).
- Source & Fact-Check: Ensures every factual claim has a primary source link.
- Editorial Standards: Reduces hype language, ensures nuance, checks tone.
- Financial Analyst: Verifies financial claims, adds required disclaimers.
- Technical Depth: Verifies technical claims against documentation, removes fluff.
- Community & Trust: Ensures community perspective, checks for press-release feel.
- SEO Specialist: Ensures proper meta tags, internal linking, discoverability.
Human Review
Our editorial team reviews the final article for tone, accuracy, completeness, and editorial quality.
Publication
Articles are published only after passing all quality gates. Any article with >25% unverified benchmark claims is rejected.
Benchmark Scoring Methodology
Benchmark Categories
| Category | Examples | Weight |
|---|---|---|
| Reasoning | MMLU, MMLU-Pro, GPQA, MATH | 25% |
| Coding | HumanEval, LiveCodeBench, SWE-bench | 30% |
| Language | MulBench, XCOPA, translation tasks | 15% |
| Vision | MathVista, VQA, OCR | 15% |
| Agentic | Terminal-Bench, tool use benchmarks | 15% |
Confidence Levels
| Level | Requirement | Color |
|---|---|---|
| HIGH | 3+ independent sources | Green |
| MEDIUM | 1-2 independent sources | Orange |
| LOW | Self-reported only | Red |
Ranking Criteria
Accuracy
Performance on standardized benchmarks. Weight: 40%
Efficiency
FLOPs per token, context window, memory footprint. Weight: 20%
Practicality
Hardware requirements, deployment ease, API pricing. Weight: 20%
Community Trust
Independent verification, community feedback, source quality. Weight: 20%
What Makes an Article "Verified"
Verification Checklist
- ✓ All benchmark claims have at least 1 independent source
- ✓ No "best ever" or "game-changing" without data support
- ✓ Financial claims have SEC filings or official announcements
- ✓ Technical claims verified against model cards/papers
- ✓ Limitations and caveats included
- ✓ Confidence levels assigned to all claims
- ✓ Source links provided for every major claim
- ✓ No fluff sections (empty "Future Directions")
Limitations
Our Limitations
We acknowledge that our methodology has limitations:
- Not exhaustive: We cannot test every model on every benchmark.
- Time-limited: AI releases move fast; some claims may become outdated.
- Self-reported data: Some benchmark numbers are self-reported by companies and may not reflect real-world performance.
- Community bias: Community feedback may reflect local lab preferences, not general consensus.
We strive for accuracy, but we are not perfect. Corrections are published within 24 hours when errors are found.
ZVHH Editorial Team
Powered by Allam 7B (SDAIA)