Budget Server Titan Hetzner Takes a Swing at the LLM Inference Market! Experimental Qwen3.6 API Goes Live!
What’s the Scoop? News Overview
- Hetzner Launches Experimental LLM Inference API: Hetzner has kicked off an “experimental project” offering an OpenAI-compatible API on its proprietary infrastructure. Currently in a free, no-strings-attached phase.
- Adopting the Qwen3.6-35B Model: Featuring the latest FP8-quantized MoE model, “Qwen/Qwen3.6-35B-A3B-FP8”, it supports both text and image inputs with a staggering 262K context window.
- Mind-Blowing Performance: In tests conducted on July 23, 2026, the first token generation (TTFT) clocked in at 153ms, with an output speed of 224 tokens per second—talk about turbo-charged!
Why Does This Matter? Key Takeaways
- Cost-Competitive Infrastructure King: Hetzner’s strength lies in its “in-house hardware management and drastically low-cost structure,” which could lead to a disruptive drop in inference costs.
- Maximizing GPU Resources: By selling off excess GPU capacity as an API, Hetzner is pushing its infrastructure’s utilization to the max—smart move!
- Easy Migration from Existing Environments: With a fully OpenAI-compatible API, users can experiment with minimal changes to existing client code.
🦈 Shark’s Eye (Curator’s Perspective)
The budget server king has finally bared its teeth! Hetzner’s move to “commoditize inference” is undeniably an exciting development. What’s particularly intriguing is that they haven’t just released an API; they’ve also hidden advanced functionalities like enable_thinking, which haven’t been documented yet. This shows that Hetzner is not only running models but also attempting some serious optimizations at the internal implementation level!
Currently, it seems they’re leveraging workstation GPUs like the RTX PRO 6000 Blackwell Max-Q (with 96GB VRAM), but if this efficient infrastructure know-how translates into large-scale inference clusters, existing AI providers will definitely have a formidable competitor on their hands!
What’s Next?
- Commercialization and Pricing: After this experimental phase, all eyes will be on whether they’ll release a “Hetzner price” that could shake up the market when they announce paid plans with SLA (Service Level Agreement) included.
- Expanding GPU Lineup: The introduction of denser multi-GPU systems from the Blackwell generation could enhance support for even larger models (like GLM-5).
A Word from HaruShark
I have a feeling that Hetzner is ready to devour the waves of inference costs! Let’s dive in! 🦈🔥
Terminology Explained
-
Qwen3.6-35B-A3B: An AI model with 35 billion parameters that uses Mixture-of-Experts (MoE) technology to keep active parameters to a mere 3 billion, ensuring speed without sacrificing performance.
-
TTFT (Time to First Token): The time it takes from when a user sends a request until the first character is returned—an essential metric for user experience.
-
FP8 Quantization: A technique that compresses data precision to 8 bits (1 byte), effectively cutting memory usage in half while dramatically boosting computation speed without losing accuracy.