v0.6.0

What happened

The article reports the release of llama.cpp v0.6.0, which includes a new llama_batch_ext API for handling mixed token and embedding inputs, support for new models such as GLM-5.3-Flash and Clef, performance optimizations for Apple GPUs, and updates to the Web UI with Hugging Face Hub integration.

Why it matters

This update is significant for AI developers and machine learning engineers as it enhances the performance and flexibility of LLM inference in C/C++, supports new models, and improves the user experience with the Web UI.

• llama.cpp v0.6.0 introduces a new API for more flexible LLM inference.
• Support for new models like GLM-5.3-Flash and Clef expands the library's capabilities.
• Performance optimizations on Apple GPUs enhance inference speed and efficiency.

The bigger picture

The release improves the efficiency and capabilities of LLM inference, enabling developers to work with larger models and more complex tasks, while also enhancing the user interface for easier model management and deployment.

The v0.6.0 release of llama.cpp brings significant improvements to LLM inference in C/C++, including new APIs, model support, and performance optimizations, making it a valuable update for AI developers and researchers.

Technical details

• llama_batch_ext API with llama_process() for mixed token/embedding batches and state embeddings.
• Metal tensor API flash attention kernel for F16 KV.
• MMA mat-mul kernels for speculative and batched decoding on Apple GPUs.
• Hugging Face Hub data layer and model download pipeline for the Web UI.
• Support for GLM-5.3-Flash (GLM5-Next) and Clef decision models.

FAQ