Pick the Mixture, Not the Samples: Combinatorial Data Selection for LLMs
Data selection for LLM fine-tuning faces two facts about real training: data pools draw from many distinct domains, and evaluation spans diverse tasks. As a result, the same sample helps different tasks differently, and samples interact during training so that a subset's value is not the sum of its…
This paper introduces Combinatorial Data Selection (CDS), a framework for optimizing data subset selection in large language model (LLM) fine-tuning by addressing domain diversity and task complexity.
What happened
The article presents a novel framework for data selection in LLM training that addresses the limitations of existing methods by explicitly modeling domain diversity and sample interactions. The proposed Mixture-then-Select algorithm first determines domain sample allocation and then ranks samples within domains.
Why it matters
Efficient data selection is critical for optimizing LLM training costs and performance. The framework provides a systematic approach to leverage domain-specific data while avoiding redundancy, which is particularly valuable for resource-constrained applications.
- Domain-aware data selection improves LLM training efficiency
- Sample interactions require explicit modeling for optimal subset selection
- Two-stage Mixture-then-Select outperforms traditional methods
- Cross-domain complementarity should inform data allocation strategies
- Framework adapts to varying resource constraints
The bigger picture
The approach offers a scalable solution for improving data utilization in LLM training, with potential applications in personalized model development and cost-effective data acquisition strategies.
The research advances data selection methodologies for LLMs by introducing a combinatorial framework that addresses domain diversity and sample interactions, demonstrating significant performance improvements over existing approaches.
Technical details
- Combinatorial Data Selection (CDS) framework
- Mixture-then-Select two-stage algorithm
- Non-additive sample interaction modeling
- Domain-specific attribution scoring
- Budget-constrained data purchasing extension
FAQ
What problem does this research address?
The research addresses the challenge of selecting optimal data subsets for LLM fine-tuning when training data spans multiple domains and evaluation tasks.
How does Mixture-then-Select differ from existing methods?
It uses a two-stage approach: first determining domain sample allocation and then ranking within-domain samples, explicitly modeling non-additive interactions between samples.
What are the key benefits of the proposed framework?
The framework achieves 20-54% performance improvements over baselines, works across different models and data sizes, and adapts to budget constraints in data purchasing scenarios.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by arXiv cs.LG on Oct 14, 2026.
Comments
Sign in to join the discussion.