Tencent Hy3: Tencent's 295B MoE Open-Source LLM

Overview

Tencent's Hy Team has open-sourced Hy3, a 295B-parameter Mixture-of-Experts (MoE) language model that is already generating significant attention in the open-source AI community. The model is available under the Apache 2.0 license, making it one of the most permissively licensed frontier-scale models on the market.

Following the Hy3 Preview release in late April 2026, Tencent gathered feedback from 50+ products and scaled up post-training with higher-quality data. The full Hy3 model, released in early July 2026, outperforms similar-size models and "rivals flagship open-source models with 2–5× parameters," according to the team.

The key takeaway: Hy3 activates only 21B parameters per token while having 295B total. This means inference is comparable to a ~21B dense model in compute cost, but the full 295B capacity gives it knowledge and reasoning that rivals much larger dense models.

Architecture Deep Dive

Hy3 uses a carefully designed MoE architecture that balances scale with efficiency:

Multi-Token Prediction (MTP)

One of Hy3's most interesting architectural features is its Multi-Token Prediction (MTP) layer, which accounts for 3.8B parameters. MTP is a form of speculative decoding built directly into the model's training — the model learns to predict multiple future tokens in parallel during training, which speeds up inference at serving time.

Tencent's vLLM configuration recommends speculative-config.num_speculative_tokens 2 for Hy3 (up from 1 in the preview version), and SGLang uses EAGLE speculative decoding with speculative-num-steps 2 and speculative-num-draft-tokens 3.

GQA Attention

Hy3 uses Grouped Query Attention (GQA) with only 8 KV heads out of 64 query heads. This reduces the KV cache size by 8× compared to multi-query attention, which is critical for a 256K context window. The head dimension of 128 keeps per-head compute manageable.

Benchmark Results

Hy3's pre-trained base model was evaluated against competitors with similar active parameter counts. Here are the key results from Tencent's evaluation:

Pre-trained Base Model Benchmarks

Compared to other models with ~21–37B activated parameters:

Key takeaways from pre-trained benchmarks:

• MMLU-Pro (65.76): Hy3 Preview-Base matches or exceeds Kimi K2 (65.98) despite having 12B fewer active parameters. This is a strong signal that the MoE architecture is being used efficiently.
• Math (GSM8K 95.37, MATH 76.28): Hy3 Preview-Base leads all competitors by significant margins — 2.9 points on GSM8K and 5.1 points on MATH. This suggests the training data or architecture is particularly well-suited for mathematical reasoning.
• Code (LiveCodeBench 34.86): Strong lead of 4+ points over Kimi K2, matching the model's emphasis on coding improvements in the full Hy3 release.
• SuperGPQA (51.60): New high among similarly-sized models, showing strong knowledge and reasoning.
• Multilingual (MMMLU 80.15): Leads the field on multilingual evaluation.

Instruct Model: STEM & Reasoning

The fine-tuned Hy3 model was evaluated on challenging STEM benchmarks including FrontierScience-Olympiad and IMOAnswerBench. It also achieved "excellent results" in the Tsinghua Qiuzhen College Math PhD qualifying exam (Spring 2026) and the China High School Biology Olympiad (CHSBO 2025), which Tencent cites as evidence of generalizable reasoning capacity.

Expert Blind Evaluation

Tencent ran a blind evaluation with 270 experts using tasks from their actual work. Results:

The advantage was most substantial in frontend development, data & storage, and CI/CD tasks. This aligns with Hy3's heavy emphasis on agent and coding capabilities in its post-training.

From Preview to Full: What Changed

Tencent released Hy3 Preview in April 2026 and the full Hy3 in July 2026. The gap between them is significant — this is essentially a different quality tier.

1. Tool calls and output format stability: Fixed baseline reliability issues across tool configurations and output constraints. Tool-call error recovery and overall efficiency improved. Accuracy variance across different agent scaffoldings (CodeBuddy, Cline, KiloCode) on SWE-Bench Verified remains within 4%.
2. Knowledge and anti-hallucination: Fine-grained data cleaning and training constraints reduced the hallucination rate from 12.5% to 5.4% and commonsense error rates from 25.4% to 12.7%. This is a near halving of factual errors.
3. Complex context retention and multi-turn intent tracking: Joint SFT+RL optimization reduced the multi-turn issue rate from 17.4% to 7.9%. Better coreference resolution, ellipsis recovery, and constraint inheritance over long conversations.
4. RL training scale: "Scaled up post-training with higher quality data" and "scaled up RL training" — the team rebuilt their RL infrastructure for the preview and expanded it further for full Hy3.

The MTP speculative decoding configuration also tightened: speculative tokens increased from 1 (preview) to 2 (full), and EAGLE draft tokens from 2 to 3, suggesting improved token prediction accuracy.

Agent & Tool-Calling Capabilities

Tencent explicitly positioned Hy3 as an agent-first model. The coding and agent benchmarks showed the "biggest gains" of any category:

The model supports a structured tool-call format using custom XML-like tokens: <tool_calls>, <tool_call>, <arg_key>, <arg_value>, <tool_response>, and <tool_responses>. It also supports reasoning mode with <think>/</think> tags and a reasoning_effort parameter that accepts "no_think", "low", or "high".

Deployment & Serving

Hy3 is a large model requiring significant infrastructure. Tencent recommends 8× H20-3e GPUs or equivalent for serving on 8 tensor-parallel ranks.

vLLM Deployment

vllm serve tencent/Hy3 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 2 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --port 8000 \
  --served-model-name hy3

Notable: vLLM requires setting VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm as a workaround for an MNNVL workspace size issue.

SGLang Deployment

python3 -m sglang.launch_server \
  --model tencent/Hy3 \
  --tp-size 8 \
  --tool-call-parser hunyuan \
  --reasoning-parser hunyuan \
  --speculative-num-steps 2 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 3 \
  --speculative-algorithm EAGLE \
  --port 8000 \
  --served-model-name hy3

Inference Efficiency

Despite 295B total parameters, Hy3 activates only 21B per token. At BF16 precision, this means the active weights are ~42GB, plus KV cache and overhead. The 192 experts with top-8 selection means each token routes through 8 of 192 expert networks, giving the model its vast capacity while keeping per-token compute manageable.

Licensing & Availability

Hy3 carries the Apache 2.0 license — one of the most permissive licenses for a model at this scale. This is notably different from Hy3 Preview, which uses the more restrictive "Tencent Hy Community License."

The model is available on multiple platforms:

A FP8 quantized version (Hy3-FP8) is also available, which reduces memory requirements significantly. The model has 5,500+ downloads on HuggingFace and 597 likes as of July 2026.

Finetuning

Tencent provides a complete finetuning pipeline with DeepSpeed ZeRO configurations (ZeRO-2 and ZeRO-3 with/without offload) and LLaMA-Factory integration. Full fine-tuning and LoRA are both supported.

Conclusion

Tencent Hy3 represents a significant milestone in the open-source LLM space. At 295B parameters with Apache 2.0 licensing, it's one of the largest fully open models available — and the MoE architecture means it's actually usable with the right hardware, since only 21B parameters activate per token.

The benchmark results are impressive: top-tier performance on math (GSM8K 95.37, MATH 76.28), strong coding (LiveCodeBench 34.86), competitive agent capabilities, and a 2.67/4 score in blind expert evaluation that edges out GLM-5.1. The hallucination reduction (12.5% → 5.4%) and multi-turn reliability improvements (17.4% → 7.9%) show that the full Hy3 has matured well beyond the Preview version.

For teams looking for a frontier-capable open-source model that can run agent workflows, handle long contexts (256K), and be fine-tuned without license restrictions, Hy3 is now a strong contender alongside models like Kimi K2 and DeepSeek-V3 — but with a more permissive license and competitive (often superior) benchmarks.

Where to try it: Deploy via vLLM or SGLang on 8× H20-3e GPUs, or try the openweights on HuggingFace. FP8 quantization is available for reduced memory requirements.