Know a great AI tool or AI news source? Submit it to AiByts

Inference from Transformer Models

4 stories

Stories

  • Weight-Only Quantization

    Overview This post is divided into five parts; they are: • Asymmetric Affine Quantization • Post-Training Quantization with PyTorch • GPTQ and AWQ • Quantizing a Model to GGUF • Evaluating Weight-Only Quantization Note: The code examples in this post use TorchAO and require a CUDA-enabled PyTorch…

    Machine Learning Mastery Blog15 min read
  • Fast Attention Kernels and Operator Fusion

    Overview This post is divided into four parts; they are: • Operator Fusion • FlashAttention-Style Kernels • Paged Attention Kernels • Fusion and CUDA Graphs with torch.

    Machine Learning Mastery Blog11 min read
  • Attention Variants and Memory Efficiency

    Overview This post is divided into eight parts; they are: • The Checkpoint Fixes the Attention Architecture • A Closer Look at Attention Memory • Multi-Head Attention • Multi-Query and Grouped-Query Attention • Sliding-Window and Mixed Attention • Multi-Head Latent Attention • What the Inference…

    Machine Learning Mastery Blog12 min read
  • Floating-Point Precision and Its Effects to Model Inference

    Overview This post is divided into five parts; they are: • How Floating-Point Numbers Work • Common Precision Formats • Loading a Model at a Different Precision • Mixed Precision and Sensitive Operations • Measuring the Result How Floating-Point Numbers Work A floating-point number contains a sign,…

    Machine Learning Mastery Blog8 min read

Showing 4 of 4 stories

You're all caught up.

AiByts Weekly Brief

Get the top AI news and tools straight to your inbox every week.

Weekly · AI news

Join 15,000+ AI enthusiasts