STAPLE: Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modules
Sparse computation can scale the parametric capacity of LLMs without proportionally increasing active computation, but pretraining sparse architectures remains expensive and data-hungry.
This article introduces STAPLE, a method for enhancing pretrained language models through static sparse memory integration, addressing challenges in scaling model capacity while maintaining efficiency.
What happened
Researchers developed STAPLE, a technique that integrates static sparse memory into pretrained models by adding token-specific embeddings as a residual pathway. This approach enables scalable model expansion without full sparse pretraining, demonstrating significant performance improvements across multiple benchmarks while reducing computational resources.
Why it matters
The method provides a sample-efficient, scalable solution for expanding pretrained models, addressing the high cost and data requirements of traditional sparse pretraining. It offers practical benefits for deployment through reduced memory usage and energy consumption.
- STAPLE offers a novel way to enhance pretrained models with sparse memory
- The method achieves significant performance gains with minimal resource overhead
- It provides a sustainable alternative to full sparse pretraining
- Residual pathway integration preserves pretrained knowledge while adding new capabilities
- Practical benefits include 2x faster decoding and 50% lower memory usage
The bigger picture
The approach enables more efficient model scaling for LLMs, reducing computational costs while maintaining performance. It demonstrates practical benefits for real-world deployment through improved resource efficiency and faster inference speeds.
STAPLE presents an effective approach for expanding pretrained language models through static sparse memory integration, offering improved performance and efficiency without the need for full sparse pretraining.
Technical details
- Static sparse memory with predictable access patterns and token-specific updates
- Residual pathway integration of embeddings with existing FFN up-projections
- Comparison metrics: HumanEval, MBPP, GSM8K, DROP benchmarks
- Throughput-oriented decoding evaluation metrics (GPU memory, energy per token)
- Host-offloaded implementation with 5% GPU embedding caching
FAQ
What problem does STAPLE solve?
STAPLE addresses the challenge of scaling large language models while maintaining computational efficiency, reducing the high costs and data requirements of traditional sparse pretraining methods.
How does STAPLE differ from existing methods?
Unlike previous approaches that replace FFN up-projections, STAPLE adds token-specific embeddings as a residual pathway, preserving pretrained computation while learning complementary information.
What are the key performance benefits?
STAPLE achieves 4.27-point improvements on HumanEval, 2x faster decoding, 50% lower GPU memory usage, and 1.5% higher generation throughput compared to dense baselines.
Is STAPLE compatible with existing models?
Yes, the method works with pretrained checkpoints and maintains high throughput with minimal deviation from dense baseline performance through host-offloaded implementation.
What benchmarks were used to evaluate STAPLE?
The method was tested on HumanEval, MBPP, GSM8K, and DROP benchmarks, showing consistent improvements across multiple tasks.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by arXiv cs.LG on Oct 12, 2026.
Comments
Sign in to join the discussion.