# Ternary Bonsai 27B: Running 27B-Class AI on Your Laptop and Phone

PrismML releases Ternary Bonsai 27B: a 7.2GB multimodal model with 95% FP16 quality, running at ~26 tok/s on laptops and enabling 27B-class AI on consumer devices for the first time.

# Ternary Bonsai 27B: 27B-Class AI That Fits on Your Laptop

PrismML's latest release compresses a full 27-billion-parameter multimodal model into 7.2GB with only a 5-point quality drop — enabling local agentic AI at consumer scale.

📅 July 17, 2026

By ZVHH Research

Source: PrismML

⚡ TL;DR

🚀**PrismML released Bonsai 27B** on July 14, 2026 — two variants: Ternary (7.2GB, quality-focused) and 1-bit (3.9GB, phone-class)

🧠**95% of FP16 quality retained** in the ternary variant across 15 benchmarks — the highest retention at this compression ratio

🔬**Every weight is {−1, 0, +1}** with FP16 group-wise scaling — a true 1.71 bits per weight, no hidden high-precision parameters

📊**~26 tok/s on M5 Pro laptop**, ~98 tok/s on H100 — with 262K context and multimodal vision capability

💎**2x the intelligence density** of the best conventional quantization — every stored GB translates into far more usable intelligence

⚠️**Apple is exploring PrismML** for on-device iPhone AI; Apache 2.0 license — fully open

7.2 GB

Ternary deployed size

95%

FP16 retention

1.71

Bits per weight

262K

Context length

For the first time, a 27-billion-parameter language model with full reasoning, tool-calling, and multimodal capabilities runs interactively on a standard laptop. That sounds impossible by any conventional measure — until you understand what PrismML has done to the model's weights.

Released July 14, 2026, **Bonsai 27B** is the latest (and largest) member of PrismML's Bonsai family, built on Qwen3.6-27B and compressed end-to-end using ternary weights. The result: a 27B-class model that occupies less storage than a full-precision 2B model, while retaining 95% of the original's intelligence.

This is not a pruning experiment or a knowledge-distillation hack. Every single weight in the language network — embeddings, attention projections, MLPs, and the LM head — is quantized to one of three values: **{−1, 0, +1}**. No high-precision escape hatches. No mixed-precision tricks hiding behind a low-bit label.

## The Bonsai Family: A Progressive Compression Revolution

PrismML has been quietly building toward this release for over a year. The Bonsai family represents a deliberate, step-by-step expansion of what's possible at extreme compression ratios:

### Phase 1: 1-bit Bonsai (March 31, 2026)

PrismML emerged from stealth on March 31, 2026, with 1-bit Bonsai — a family of binary-weight models in 8B, 4B, and 1.7B sizes. The flagship **1-bit Bonsai 8B** fit in just 1.15 GB while matching its full-precision baseline's benchmark performance. Backed by $16.25M from Khosla Ventures, Cerberus, and Google, the Caltech-born team had demonstrated that binary weights (**{−1, +1}**) could produce commercially useful language models.

### Phase 2: Ternary Bonsai (April 2026)

Just weeks later, PrismML introduced the Ternary Bonsai family (8B, 4B, 1.7B), adding a zero state to create three weight values **{−1, 0, +1}**. The extra state — seemingly trivial — delivered a ~5-point average improvement across benchmarks for only ~600MB of additional memory. The zero state proved critical: it gives the model a "neutral" weight option, preserving more of the original weight distribution than binary's forced bipolar quantization.

### Phase 3: Bonsai 27B (July 14, 2026)

Now, the family reaches its largest member. Bonsai 27B scales the ternary technique to a 27B-parameter model — a capability tier that previously required 54GB in FP16 or at minimum 17.6GB in the best conventional 4-bit build. Bonsai 27B delivers the same tier in 7.2GB (ternary) or 3.9GB (1-bit).

🏢 About PrismML

PrismML was founded by Caltech researchers with support from Khosla Ventures, Cerberus, and Google. Their core research focus is neural network compression without sacrificing reasoning ability — a problem that has been challenging the field for years. Apple is reportedly exploring PrismML's technology for on-device iPhone AI.

## How Ternary Quantization Works

The core innovation of Bonsai 27B isn't the model architecture — it's what PrismML does to the weights. To understand why this matters, we need to look at the mechanics.

### The Ternary Weight Alphabet

Traditional quantization methods approximate continuous FP16 weights with values from a larger set (e.g., 4-bit quantization uses 16 discrete levels per weight). Bonsai's ternary approach reduces every weight to one of **three values**:

```

{−1, 0, +1}`
```

But simply snapping weights to three values would destroy the model. PrismML's solution: **group-wise FP16 scaling**. Each group of 128 ternary weights shares one FP16 scale factor. The actual mathematical operation becomes:

```

actual_weight = ternary_value × group_scale_factor`
```

This means the *effective* weight range is not limited to [−1, +1] — the scale factor adapts per group, allowing the model to preserve important magnitude information across different weight channels. The ternary values act as a directional sign (+1 pushes, −1 pulls back, 0 stays neutral), while the FP16 scale captures the intensity.

### Why 1.71 Bits Per Weight?

A binary value (0 or 1) carries exactly 1 bit. A ternary value (−1, 0, or +1) carries *log₂(3)* ≈ 1.585 bits. Since each group of 128 ternary weights also stores one 16-bit FP16 scale factor, the total storage per weight is:

```

(1.585 bits × 128 weights + 16 bits scale) / 128 weights = ~1.71 bits/weight`
```

This is a **true average** — unlike conventional "2-bit" builds that advertise lower bits per weight while actually storing 2.8 bits/weight when you count the full overhead.

### End-to-End: No Escape Hatches

What makes Bonsai different from other quantization approaches is the **end-to-end** coverage. Every component of the language network is ternary:

- Embedding layer

- Attention projections (all 64 layers)

- MLP projections (SwiGLU gates)

- LM head (vocabulary projection)

No layer is left in FP16 or higher precision. The vision tower ships separately in compact 4-bit HQQ format (0.63 GB), loaded only when an image is provided — text-only serving never pays for it.

⚠️ Ternary vs. Conventional "2-bit"

A widely-used "2-bit" build of Qwen3.6-27B (IQ2_XXS) is actually 2.8 bits/weight at 9.4GB. Bonsai's ternary format is genuinely 1.71 bits/weight at 5.9GB ideal — nearly half the size and with a lower true bit-width. The deployed footprint sits at ~7.2GB because current kernels store each ternary value in a 2-bit slot; native ternary kernels (an active engineering target) would close the remaining gap.

## Architecture & Design

Bonsai 27B is derived from **Qwen3.6-27B**, a hybrid-attention causal language model. The architecture itself is unchanged — what's different is every weight inside it.

### Hybrid Attention Backbone

Qwen3.6-27B uses a hybrid attention pattern: approximately **75% linear attention** and **25% full attention**. This design choice has a profound impact on Bonsai's practical usability:

- **Linear attention** layers grow context linearly in memory, making the 262K-token context window practical on consumer hardware

- Only 16 of 64 layers use full attention, keeping the KV cache manageable (~4.3GB at full context in FP16, ~1.1GB at 4-bit compression)

- The hybrid design enables long-document analysis, full-repository code work, and other context-heavy tasks on-device

### Model Composition

| **Component** | **Parameters** | **Format**|
--- | --- | ---
| 🌳 Language backbone | ~24.8B | Ternary {−1,0,+1} @ 1.71 bpw|
| Embeddings + LM head | ~2.5B | Ternary {−1,0,+1} @ 1.71 bpw|
| Vision tower | ~0.46B | HQQ 4-bit (optional)|
| **Total** | **~27.8B** | —|

### Additional Design Elements

- **SwiGLU MLP** — gated activation function for improved expressiveness

- **RoPE** — Rotary Positional Embeddings for positional encoding

- **RMSNorm** — Root Mean Square Layer Normalization

- **Thinking mode** — enabled by default, toggleable per request via `thinking_budget_tokens`

## Benchmark Performance

Bonsai 27B was evaluated across **15 benchmarks** in six skill categories, all in thinking mode where the model's full reasoning capability is exercised. Here's how it compares to the FP16 baseline and conventional quantization methods:

### By Capability

| **Category** | **Benchmarks** | **Qwen3.6-27B FP16** | **Ternary Bonsai 27B** | **1-bit Bonsai 27B**|
--- | --- | --- | --- | ---
| 🧮 Math | GSM8K, MATH-500, AIME25, AIME26 | 95.3 | 93.4 | 91.7|
| 💻 Coding | HumanEval+, MBPP+, LiveCodeBench | 88.7 | 86.0 | 81.9|
| 🤖 Agentic / Tool-calling | BFCL v3, TauBench | 80.0 | 74.0 | 66.0|
| 📋 Instruction Following | IFEval, IFBench | 78.4 | 71.8 | 65.8|
| 📚 Knowledge / STEM | MMLU-Redux, MuSR | 83.1 | 77.0 | 73.4|
| 👁️ Vision | MMMU Pro, OCRBench | 72.6 | 65.2 | 59.6|
| **Overall (15)** | — | **85.0** | **80.5** | **76.1**|

### The Story in the Numbers

The averages tell a clean story, but the breakdown reveals something more important: **math and coding are nearly untouched**. Math scores are within two points of full precision (93.4 vs 95.3), and coding at 86.0 is a small drop that's barely noticeable in practice. These are exactly the capabilities that matter most for agentic workloads — multi-step reasoning, structured tool use, and code generation.

The gap concentrates in the most demanding categories: agentic tool calling (74.0 vs 80.0) and vision (65.2 vs 72.6). This is not a failure mode — it's the expected shape of compression loss, and it's dramatically better than what conventional methods do.

**Key insight:** Conventional sub-4-bit methods don't just score lower overall — they collapse selectively on reasoning tasks while maintaining superficial competence on knowledge recall. IQ2_XXS scores 88.9 on MMLU-Redux but drops to 57.5 on AIME26. Bonsai holds both, keeping AIME at 87.5–90.8 and LiveCodeBench at 82.8.

### Benchmark Comparison: Same Footprint, Different Methods

For context, here's how Bonsai 27B stacks against conventional quantization methods at similar sizes:

| **Variant** | **True bpw** | **Footprint** | **Thinking avg** | **vs FP16**|
--- | --- | --- | --- | ---
| Qwen3.6-27B FP16 | 16.0 | 54 GB | 85.07 | 100%|
| Qwen3.6-27B Q4_K_XL ("4-bit") | 5.2 | 17.6 GB | 84.99 | 99.9%|
| Qwen3.6-27B IQ2_XXS ("2-bit") | 2.8 | 9.4 GB | 72.73 | 85.5%|
| 🌳 **Ternary Bonsai 27B** | **1.71** | **5.9 GB** | **80.49** | **94.6%**|
| 1-bit Bonsai 27B | 1.125 | 3.9 GB | 76.11 | 89.5%|

At 5.9GB, Ternary Bonsai 27B **outscores both sub-4-bit conventional builds by more than seven points at one-half to two-thirds of their size**. The conventional builds may score higher in aggregate, but their failure mode is qualitatively worse: they maintain surface-level competence while losing reasoning depth.

## Intelligence Density: The New Metric

PrismML introduced a new metric with Bonsai 8B — **intelligence density** — which captures the ratio of capability to deployed size:

```

D = −log₂(1 − score/100) / size_GB`
```

| **Variant** | **Size (GB)** | **Benchmark avg** | **Density (1/GB)**|
--- | --- | --- | ---
| 1-bit Bonsai 27B | 3.9 | 76.11 | **0.530**|
| 🌳 **Ternary Bonsai 27B** | 5.9 | 80.49 | **0.400**|
| Qwen3.6-27B IQ2_XXS | 9.4 | 72.73 | 0.199|
| Qwen3.6-27B Q4_K_XL | 17.6 | 84.99 | 0.155|
| Qwen3.6-27B FP16 | 54 | 85.07 | 0.051|

Ternary Bonsai 27B delivers **2× the density of the densest conventional build** (IQ2_XXS at 0.199) and nearly **8× the density of FP16**. No conventional build of Qwen3.6-27B or Gemma-4-31B exceeds 0.2. Every stored gigabyte translates into far more usable intelligence.

## Real-World Performance

### Throughput

Token generation throughput (tg128) was measured over 128 generated tokens, capturing the memory-bandwidth-bound interactive phase:

| **Platform** | **Footprint** | **Generate (tok/s)** | **Process 512 (tok/s)**|
--- | --- | --- | ---
| M5 Max laptop (Metal) | 7.2 GB | 44.0 | 830|
| M5 Pro laptop (Metal) ⚡ | 7.2 GB | **26.2** | **393**|
| M4 Pro laptop (Metal) | 7.2 GB | 18.0 | 125|
| H100 GPU (CUDA) | 7.2 GB | 98.0 | 2,596|
| 100K context (M5 Pro, 4-bit KV) | 10.1 GB | ~22 | ~350|

💡 Memory Bandwidth Bound

On laptops, the M5 Pro streams ~186 GB/s of weights, confirming the memory-bandwidth-dominated profile that the low-bit representation is built to exploit. The meaningful statement isn't a speedup ratio over FP16 — it's that a 27B model runs interactively on an everyday laptop where the FP16 baseline (54 GB) and even "4-bit" builds (17.6 GB) don't fit at all.

### Peak Memory at Context

What a device must actually accommodate is *peak* memory — weights plus KV cache plus activations (~1.3GB overhead):

| **Build** | **Weights** | **4K ctx** | **10K ctx** | **100K ctx**|
--- | --- | --- | --- | ---
| 🌳 Ternary Bonsai | 7.15 GB | 8.4 GB | 8.7 GB | 14.7 GB|
| Qwen3.6-27B Q4_K_XL | 17.6 GB | 19.2 GB | 19.6 GB | 25.6 GB|
| Qwen3.6-27B FP16 | 51.25 GB | 52.6 GB | 53.3 GB | 59.3 GB|

The ternary build holds a **100K-token context at 14.7GB** without KV-cache compression — fitting mainstream laptops outright. With 4-bit KV cache, the 100K peak drops to ~10.1GB, and the full 262K window fits in ~12.8GB.

### DSpark Speculative Decoding

Bonsai 27B ships with a **DSpark** drafter layer — a semi-autoregressive 6-layer transformer trained against the low-bit target. It uses hidden states tapped from five evenly spaced layers of the target model, with a diffusion-flavored block-denoising objective.

On CUDA (H100), the drafter achieves a **1.34× decode speedup**: 98 → 131.8 tok/s at an accepted draft length of τ ≈ 3.7. On Apple Silicon, the batch-1 verification pass does not yet amortize, so the drafter is not enabled by default on-device.

## The Bigger Picture: Why This Matters

Bonsai 27B isn't just a smaller model — it enables entirely new deployment patterns that were impossible before.

### On-Device Agentic AI

When a model capable of sustained agentic work fits on the device, the marginal cost of a hundred-step loop is zero, and user data never leaves the machine. This unlocks persistent on-device agents, offline assistants, and assistants that reason over private local data by construction.

### Hybrid Deployments

The cost-quality spectrum now has a practical local operating point: non-frontier and privacy-sensitive tasks route to Bonsai locally, while frontier cloud models handle the hardest steps. This collapses the cost-per-task of agentic systems dramatically.

### Phone-Class AI

The 1-bit variant (3.9GB) fits within the iPhone 17 Pro's memory budget — about 6GB available for on-device apps. It's the **first 27B-class model to run on a phone**, bringing multi-step reasoning and tool use to mobile devices.

🍎 Industry Interest

Apple is reportedly exploring PrismML's technology to run large AI models directly on iPhone. This follows years of Apple investing in on-device ML — the MLX framework, on-device Siri improvements, and the broader push toward private, on-device AI. Bonsai 27B's 1-bit variant is a natural fit for this strategy.

### Apache 2.0: Fully Open

Everything is available under the Apache 2.0 license — weights, kernels, demo code. Bonsai 27B runs natively on Apple devices via MLX and NVIDIA GPUs via CUDA, through custom low-bit kernels built for its hybrid-attention architecture. It's also landing in mainline llama.cpp (Q1_0 fully merged, Q2_0 on CPU and Metal, Vulkan in review).

## Use Cases

🎯 Primary Use Cases

💻

Laptop agents

Full 27B reasoning on everyday hardware

🔒

Privacy-first

On-device execution, data never leaves

📱

Phone-class

1-bit variant fits iPhone 17 Pro

🖥️

Single-GPU serving

27B quality from consumer GPU

## Limitations & Trade-offs

An honest article needs to be clear about what Bonsai 27B *doesn't* do:

⚠️ Quality–Footprint Trade-off

The ternary model retains 94.6% of the full-precision average. The gap is modest and predictable — reasoning core (math, coding) stays within a few points of baseline — but it exists. For workloads where every point matters, the full model is still superior.

- **Doesn't fit a phone (ternary):** At ~7.2GB, the ternary build exceeds the ~6GB per-app iOS memory budget. Use the 1-bit companion via MLX Swift for phone deployment.

- **Served in 2-bit slots today:** The deployed footprint (~7.2GB) sits above the representation's ~5.9GB native target. Native ternary kernels are an active engineering target.

- **Agentic coding not yet tuned:** Long-horizon, multi-file, run-test-and-repair workflows are not a strong target of this release. A Bonsai 27B variant tuned for agentic coding is next on the roadmap.

- **Vision at 4-bit:** The vision tower is quantized separately at 4-bit HQQ — it works well but doesn't benefit from the ternary compression that defines the language network.

## Getting Started

Bonsai 27B runs through llama.cpp (CUDA, Metal, CPU) and MLX (Apple Silicon). Here's a quick start for CUDA:

```

# Clone the PrismML fork (includes Q2_0_g128 hybrid-attention kernels)
git clone https://github.com/PrismML-Eng/llama.cpp
cd llama.cpp

# Build with CUDA support
cmake -B build -DGGML_CUDA=ON && cmake --build build -j

# Download the 2-bit GGUF weights
hf download prism-ml/Ternary-Bonsai-27B-gguf \
Ternary-Bonsai-27B-Q2_0.gguf --local-dir .

# Run inference
./build/bin/llama-cli \
-m Ternary-Bonsai-27B-Q2_0.gguf \
-p "Explain quantum computing in simple terms." \
-n 256 \
--temp 0.7 --top-p 0.95 --top-k 20 \
-ngl 99`
```

For the OpenAI-compatible server:

```

./build/bin/llama-server \
-m Ternary-Bonsai-27B-Q2_0.gguf \
--host 0.0.0.0 --port 8080 -ngl 99`
```

**Generation parameters** (used for all reported benchmarks): temperature 0.7, top-p 0.95, top-k 20.

Available formats:

| **Format** | **HF Repository** | **Size**|
--- | --- | ---
| 🌳 Ternary GGUF (Q2_0) | [Ternary-Bonsai-27B-gguf](https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf) | 7.17 GB|
| Ternary MLX (2-bit) | [Ternary-Bonsai-27B-mlx-2bit](https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit) | 7.05 GB|
| 1-bit GGUF (Q1_0) | [Bonsai-27B-gguf](https://huggingface.co/prism-ml/Bonsai-27B-gguf) | 3.53 GB|
| 1-bit MLX (1-bit) | [Bonsai-27B-mlx-1bit](https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit) | 3.92 GB|

Each repo ships an `mmproj` file for the vision tower (+0.63 GB), needed for image input.

## ZVHH Scorecard

📊 Overall Assessment

🧠

95%

FP16 quality retained

💾

7.2 GB

Deployed size

⚡

26 tok/s

M5 Pro laptop

📜

✓

Apache 2.0 License

| **Criteria** | **Rating**|
--- | ---
| 🧮 Reasoning / Math | ★★★★★ 93.4|
| 💻 Coding | ★★★★★ 86.0|
| 🤖 Agentic / Tool Calling | ★★★★★ 74.0|
| 📚 Knowledge | ★★★★★ 77.0|
| 👁️ Vision | ★★★★★ 65.2|
| 💎 Intelligence Density | ★★★★★ 0.400/GB|
| 📜 Openness | ★★★★★ Apache 2.0|

**Final verdict:** Bonsai 27B is a genuine paradigm shift in intelligence density. At 7.2GB with 95% quality retention, it makes 27B-class reasoning practical on hardware people already own. The 1-bit variant (3.9GB) extends this to phones. For anyone running local AI agents, this is the model to watch.

**Primary Sources:**

- [PrismML — Announcing Bonsai 27B](https://prismml.com/news/bonsai-27b)

- [PrismML Docs — Bonsai 27B Specifications](https://docs.prismml.com/models/bonsai-27b)

- [HuggingFace — Ternary-Bonsai-27B-gguf](https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf)

- [PrismML-Eng/Bonsai-demo (GitHub)](https://github.com/PrismML-Eng/Bonsai-demo)

- [Bonsai 27B Whitepaper (PDF)](https://github.com/PrismML-Eng/Bonsai-demo/blob/main/bonsai-27b-whitepaper.pdf)

Article written by ZVHH Research. All benchmark data sourced from official PrismML documentation and HuggingFace model cards.

Powered by Allam 7B (SDAIA)

### Related Coverage

🌳 [Tencent Hy3: Open-Source MoE LLM That Beats Flagship](/tencent-hy3-open-source-moe-llm-beats-flagship.html) — Another open-weight model challenging the frontier.

🧠 [VibeThinker 3B: Beats Frontier Reasoning Models](/vibethinker-3b-beats-frontier-reasoning-models.html) — Small models with exceptional reasoning capabilities.

📊 [Top 10 AI Models of June 2026](/top-10-intelligence-ai-models-of-june-2026.html) — Our monthly intelligence ranking.