# Tencent Hy3: Tencent's 295B MoE Open-Source LLM Tencent just open-sourced Hy3 — a 295B-parameter MoE model with 21B active parameters. Benchmarks near Claude Opus level, Apache 2.0 licensed, freely available on HuggingFace. ## Overview Tencent's Hy Team has open-sourced **Hy3**, a 295B-parameter Mixture-of-Experts (MoE) language model that is already generating significant attention in the open-source AI community. The model is available under the **Apache 2.0 license**, making it one of the most permissively licensed frontier-scale models on the market. Following the Hy3 Preview release in late April 2026, Tencent gathered feedback from 50+ products and scaled up post-training with higher-quality data. The full Hy3 model, released in early July 2026, outperforms similar-size models and "rivals flagship open-source models with 2–5× parameters," according to the team. 295B Total Parameters 21B Active Parameters 192 Experts (Top-8) 256K Context Window Apache 2.0 License The key takeaway: **Hy3 activates only 21B parameters per token** while having 295B total. This means inference is comparable to a ~21B dense model in compute cost, but the full 295B capacity gives it knowledge and reasoning that rivals much larger dense models. ## Architecture Deep Dive Hy3 uses a carefully designed MoE architecture that balances scale with efficiency: ### Key Architecture Specifications Hy3 is built on the Hunyuan lineage (code-named `hy_v3`) with several notable design choices: | **Property** | **Value**| --- | --- | Architecture Type | Mixture-of-Experts (MoE)| | Total Parameters | 295B| | Activated Parameters per Token | 21B| | MTP Layer Parameters | 3.8B| | Number of Layers (excl. MTP) | 80| | Number of MTP Layers | 1| | Attention Heads | 64 (GQA, 8 KV heads, head dim 128)| | Hidden Size | 4096| | Intermediate Size | 13,312| | Context Length | 256K tokens| | Vocabulary Size | 120,832| | Number of Experts | 192 (top-8 activated)| | Supported Precision | BF16| ### Multi-Token Prediction (MTP) One of Hy3's most interesting architectural features is its **Multi-Token Prediction (MTP)** layer, which accounts for 3.8B parameters. MTP is a form of speculative decoding built directly into the model's training — the model learns to predict multiple future tokens in parallel during training, which speeds up inference at serving time. Tencent's vLLM configuration recommends `speculative-config.num_speculative_tokens 2` for Hy3 (up from 1 in the preview version), and SGLang uses EAGLE speculative decoding with `speculative-num-steps 2` and `speculative-num-draft-tokens 3`. ### GQA Attention Hy3 uses Grouped Query Attention (GQA) with only **8 KV heads** out of 64 query heads. This reduces the KV cache size by 8× compared to multi-query attention, which is critical for a 256K context window. The head dimension of 128 keeps per-head compute manageable. ## Benchmark Results Hy3's pre-trained base model was evaluated against competitors with similar active parameter counts. Here are the key results from Tencent's evaluation: ### Pre-trained Base Model Benchmarks Compared to other models with ~21–37B activated parameters: | **Benchmark** | **Metric** | **Kimi K2 (32B act.)** | **DeepSeek-V3 (37B act.)** | **GLM-4.5 (32B act.)** | **Hy3 Preview-Base (21B act.)**| --- | --- | --- | --- | --- | --- | MMLU | 5-shot | 88.24 | 87.68 | 87.73 | **87.42**| | MMLU-Pro | 5-shot | 65.98 | 63.98 | 63.67 | **65.76**| | MMLU-Redux | 5-shot | 87.18 | 86.81 | 86.56 | **86.86**| | ARC-Challenge | 0-shot | 96.66 | 94.65 | 96.32 | 95.99| | DROP | 5-shot | 86.40 | **86.50** | 82.90 | 85.50| | PIQA | 4-shot | 84.93 | 84.22 | 84.71 | 84.39| | SuperGPQA | 5-shot | 51.10 | 46.17 | 49.64 | **51.60**| | SimpleQA | 5-shot | **34.37** | 26.15 | 29.26 | 26.47| | MBPP-plus | 3-shot | **81.35** | 75.47 | 78.05 | 78.71| | CRUXEval-I | 3-shot | 68.01 | 67.79 | 68.51 | **71.19**| | LiveCodeBench-v6 | 1-shot | 30.86 | 29.31 | 27.43 | **34.86**| | GSM8K | 4-shot | 93.46 | 88.15 | 90.06 | **95.37**| | MATH | 4-shot | 71.20 | 59.37 | 61.00 | **76.28**| | CMath | 4-shot | 90.83 | 85.50 | 89.33 | **91.17**| | MMMLU | 5-shot | 77.63 | 79.54 | 79.26 | **80.15**| | INCLUDE | 5-shot | 75.66 | 77.86 | 76.27 | **78.64**| **Key takeaways from pre-trained benchmarks:** - **MMLU-Pro (65.76):** Hy3 Preview-Base matches or exceeds Kimi K2 (65.98) despite having 12B fewer active parameters. This is a strong signal that the MoE architecture is being used efficiently. - **Math (GSM8K 95.37, MATH 76.28):** Hy3 Preview-Base leads all competitors by significant margins — 2.9 points on GSM8K and 5.1 points on MATH. This suggests the training data or architecture is particularly well-suited for mathematical reasoning. - **Code (LiveCodeBench 34.86):** Strong lead of 4+ points over Kimi K2, matching the model's emphasis on coding improvements in the full Hy3 release. - **SuperGPQA (51.60):** New high among similarly-sized models, showing strong knowledge and reasoning. - **Multilingual (MMMLU 80.15):** Leads the field on multilingual evaluation. ### Instruct Model: STEM & Reasoning The fine-tuned Hy3 model was evaluated on challenging STEM benchmarks including FrontierScience-Olympiad and IMOAnswerBench. It also achieved "excellent results" in the Tsinghua Qiuzhen College Math PhD qualifying exam (Spring 2026) and the China High School Biology Olympiad (CHSBO 2025), which Tencent cites as evidence of generalizable reasoning capacity. ### Expert Blind Evaluation Tencent ran a blind evaluation with **270 experts** using tasks from their actual work. Results: 2.67/4 Hy3 Score 2.51/4 GLM-5.1 Score +0.16 Advantage The advantage was most substantial in **frontend development**, **data & storage**, and **CI/CD tasks**. This aligns with Hy3's heavy emphasis on agent and coding capabilities in its post-training. ## From Preview to Full: What Changed Tencent released Hy3 Preview in April 2026 and the full Hy3 in July 2026. The gap between them is significant — this is essentially a different quality tier. ### Improvements in Full Hy3 over Preview Tencent identified and fixed several categories of issues based on feedback from 50+ products: 1. **Tool calls and output format stability:** Fixed baseline reliability issues across tool configurations and output constraints. Tool-call error recovery and overall efficiency improved. Accuracy variance across different agent scaffoldings (CodeBuddy, Cline, KiloCode) on SWE-Bench Verified remains within 4%. 1. **Knowledge and anti-hallucination:** Fine-grained data cleaning and training constraints reduced the hallucination rate from **12.5% to 5.4%** and commonsense error rates from **25.4% to 12.7%**. This is a near halving of factual errors. 1. **Complex context retention and multi-turn intent tracking:** Joint SFT+RL optimization reduced the multi-turn issue rate from **17.4% to 7.9%**. Better coreference resolution, ellipsis recovery, and constraint inheritance over long conversations. 1. **RL training scale:** "Scaled up post-training with higher quality data" and "scaled up RL training" — the team rebuilt their RL infrastructure for the preview and expanded it further for full Hy3. The MTP speculative decoding configuration also tightened: speculative tokens increased from 1 (preview) to 2 (full), and EAGLE draft tokens from 2 to 3, suggesting improved token prediction accuracy. ## Agent & Tool-Calling Capabilities Tencent explicitly positioned Hy3 as an agent-first model. The coding and agent benchmarks showed the "biggest gains" of any category: | **Benchmark** | **What it measures** | **Hy3 Result**| --- | --- | --- | SWE-bench Verified | Real-world software engineering tasks | Competitive with larger models| | Terminal-Bench 2.0 | Command-line agent tasks | Competitive scores| | ClawEval | OpenClaw-style agent capabilities | Strong performance| | WildClawBench | Real-world agent scenarios | Strong performance| | BrowseComp | Web search agent tasks | Competitive scores| | WideSearch | Information retrieval agents | Competitive scores| The model supports a structured tool-call format using custom XML-like tokens: ``, ``, ``, ``, ``, and ``. It also supports reasoning mode with ``/`` tags and a `reasoning_effort` parameter that accepts `"no_think"`, `"low"`, or `"high"`. "We don't think public benchmark scores tell the full story. So we ran a blind evaluation with 270 experts using tasks from their work." — Tencent Hy Team ## Deployment & Serving Hy3 is a large model requiring significant infrastructure. Tencent recommends **8× H20-3e GPUs** or equivalent for serving on 8 tensor-parallel ranks. ### vLLM Deployment ``` vllm serve tencent/Hy3 \ --tensor-parallel-size 8 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 2 \ --tool-call-parser hy_v3 \ --reasoning-parser hy_v3 \ --enable-auto-tool-choice \ --port 8000 \ --served-model-name hy3` ``` Notable: vLLM requires setting `VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm` as a workaround for an MNNVL workspace size issue. ### SGLang Deployment ``` python3 -m sglang.launch_server \ --model tencent/Hy3 \ --tp-size 8 \ --tool-call-parser hunyuan \ --reasoning-parser hunyuan \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 \ --speculative-algorithm EAGLE \ --port 8000 \ --served-model-name hy3` ``` ### Inference Efficiency Despite 295B total parameters, Hy3 activates only **21B per token**. At BF16 precision, this means the active weights are ~42GB, plus KV cache and overhead. The 192 experts with top-8 selection means each token routes through 8 of 192 expert networks, giving the model its vast capacity while keeping per-token compute manageable. ## Licensing & Availability Hy3 carries the **Apache 2.0 license** — one of the most permissive licenses for a model at this scale. This is notably different from Hy3 Preview, which uses the more restrictive "Tencent Hy Community License." The model is available on multiple platforms: ### Model Availability - **HuggingFace:** [tencent/Hy3](https://huggingface.co/tencent/Hy3) (Apache 2.0) · [tencent/Hy3-FP8](https://huggingface.co/tencent/Hy3-FP8) (FP8 quantized) - **ModelScope:** [Tencent-Hunyuan/Hy3](https://modelscope.cn/models/Tencent-Hunyuan/Hy3) - **GitCode:** [hy3-preview](https://ai.gitcode.com/tencent_hunyuan/Hy3) - **GitHub:** [Tencent-Hunyuan/Hy3](https://github.com/Tencent-Hunyuan/Hy3) A **FP8 quantized version** (Hy3-FP8) is also available, which reduces memory requirements significantly. The model has 5,500+ downloads on HuggingFace and 597 likes as of July 2026. ### Finetuning Tencent provides a complete finetuning pipeline with DeepSpeed ZeRO configurations (ZeRO-2 and ZeRO-3 with/without offload) and LLaMA-Factory integration. Full fine-tuning and LoRA are both supported. ## Conclusion Tencent Hy3 represents a significant milestone in the open-source LLM space. At 295B parameters with Apache 2.0 licensing, it's one of the largest fully open models available — and the MoE architecture means it's actually *usable* with the right hardware, since only 21B parameters activate per token. The benchmark results are impressive: top-tier performance on math (GSM8K 95.37, MATH 76.28), strong coding (LiveCodeBench 34.86), competitive agent capabilities, and a 2.67/4 score in blind expert evaluation that edges out GLM-5.1. The hallucination reduction (12.5% → 5.4%) and multi-turn reliability improvements (17.4% → 7.9%) show that the full Hy3 has matured well beyond the Preview version. For teams looking for a frontier-capable open-source model that can run agent workflows, handle long contexts (256K), and be fine-tuned without license restrictions, Hy3 is now a strong contender alongside models like Kimi K2 and DeepSeek-V3 — but with a more permissive license and competitive (often superior) benchmarks. **Where to try it:** Deploy via vLLM or SGLang on 8× H20-3e GPUs, or try the openweights on HuggingFace. FP8 quantization is available for reduced memory requirements.