b11572
LLM inference in C/C++. Contribute to ggml-org/llama.cpp development by creating an account on GitHub.
llama.cpp1 min read
vLLM
239 stories
b11571
LLM inference in C/C++. Contribute to ggml-org/llama.cpp development by creating an account on GitHub.
llama.cpp1 min read
Performing your first LLM fine-tune on Runpod with Axolotl, and how to host it after
You can teach an open-weight LLM to text like a specific persona: their slang, their reply length, their humor, even how they write in a roleplay scene, creative writing exercise, or chatbot customer interactions.
RunPod28 min read
LLM Batching Explained: Static vs Dynamic vs Continuous Batching
Imagine four people booking a ride to the same destination. If each person takes a separate taxi, four vehicles make the same trip. But if they travel together in a bus, everyone reaches the destination in a single trip. The bus carries more people at once and makes better use of the resources.
Analytics Vidhya12 min readHow Did I Write My First FlashAttention Kernel in CUDA ?
From demystifying the online softmax trick to hitting a 41.6x speedup on an NVIDIA RTX 4050 GPU — a beginner’s hands-on journey into hardware-aware deep learning.
Towards AI11 min read
How distributed inference keeps large AI models moving
Distributed inference serving is an air traffic control tower for AI traffic, and agentic workflows make it more necessary than ever.
IBM Research Blog12 min readWhat hackers built on Runpod at TreeHacks 2026
We sponsored TreeHacks 2026 at Stanford, where teams built on Runpod across 36 hours, shipping projects ranging from GPU-accelerated cancer drug discovery.
RunPod5 min read
What is Decision 3.0? vLLM Semantic Router’s New Open Decision Models
vLLM Semantic Router released Decision 3.0, 5 Apache-2.0 multimodal decision models from 0.8B to 27B. They answer choice, yes/no and score questions about text and images in one call, without generating text. The post What is Decision 3.0?
MarkTechPost5 min readIs a second MSc worth it? [D]
I’m an AI research engineer based in Africa with about 3 YoEs in RL, LLMs, post-training, and systems efficiency (CUDA/vLLM). However, on a smaller scale because of compute limits. My undergrad and Master’s are both in communications engineering.
Reddit - r/ML2 min readFrom OpenAI API to Self-Hosted Model: A Migration Guide
How to switch from OpenAI's API to a self-hosted LLM built on open-source models on Runpod, for teams tired of usage limits and API costs.
RunPod5 min readDeploy Llama 3.1 With vLLM on Runpod Serverless
Learn how to deploy Meta's Llama 3.1 8B Instruct model using the vLLM inference engine on Runpod Serverless for blazing-fast performance and scalable AI.
RunPod7 min readBoost vLLM Performance on Runpod with GuideLLM
Learn how to use GuideLLM to simulate real-world inference loads, fine-tune performance, and optimize cost for vLLM deployments on Runpod.
RunPod3 min readHow to Cut LLM Inference Latency and Cost
Learn how to reduce LLM inference costs and latency using quantization, vLLM, SGLang, and speculative decoding without upgrading your hardware.
RunPod13 min readRAG vs. Fine-Tuning: Which Strategy is Best for Customizing LLMs?
RAG and fine-tuning are two powerful strategies for adapting large language models (LLMs) to domain-specific tasks. This post compares their use cases.
RunPod9 min readSupercharge Your LLMs with SGLang: Boost Performance and Customization
Discover how to boost your LLM inference performance and customize responses using SGLang, an innovative framework for structured LLM workflows.
RunPod8 min readQuantization Methods Compared: Speed vs. Accuracy in Model Deployment
Explore the trade-offs between post-training, quantization-aware training, mixed precision, and dynamic quantization. Learn how each method impacts model.
RunPod8 min readRAG vs. Fine-Tuning: Which Is Best for Your LLM?
Retrieval-Augmented Generation (RAG) and fine-tuning are powerful ways to adapt large language models. Learn the key differences, trade-offs, and when to.
RunPod9 min read
Why Temperature 0 Isn't Deterministic
A one-line formula for how often an LLM's top token flips, and why identical completions fall apart after about a hundred tokens The post Why Temperature 0 Isn't Deterministic appeared first on Towards Data Science.
Towards Data Science11 min read
Showing 54 of 239 stories
Load more storiesAiByts Weekly Brief
Get the top AI news and tools straight to your inbox every week.
Weekly · AI news
Join 15,000+ AI enthusiasts