Unconstrained CTI-REALM with Open-Weights

What happened

The article describes a study comparing open-weight and closed-language models for security tasks using the CTI-REALM benchmark. The research evaluated 26 models, 19 open-weight and 7 closed, under a common framework. The study found that open-weight models can match the performance of closed models at a much lower cost when given sufficient turns, challenging the assumption that closed models are superior. The research also highlights the impact of the benchmark's message limit on results and introduces new metrics for more accurate evaluation.

Why it matters

This study is significant for AI developers and security professionals as it challenges the notion that closed models are inherently better for security tasks. It suggests that open-weight models can be a cost-effective alternative without sacrificing performance, which could influence organizational decisions and resource allocation in AI security applications.

• Open-weight models can match closed models in security tasks at a fraction of the cost.
• The CTI-REALM benchmark's message limit affects performance evaluation, and relaxing it benefits open-weight models.
• Stage-level metrics offer a more accurate way to assess model performance in security workflows.
• Price is not a reliable indicator of model capability, as similar rewards can come from models with vastly different costs.
• The study highlights the importance of considering both cost and performance when selecting AI models for security applications.

The bigger picture

The findings suggest that organizations can achieve comparable security performance with open-weight models at a significantly lower cost, which could lead to more widespread adoption of open models in security applications. The introduction of stage-level metrics provides a more nuanced approach to evaluating model performance, potentially influencing future benchmarking practices.

The study demonstrates that open-weight models can effectively compete with closed models in security tasks, offering a cost-effective solution without compromising performance. The research also introduces new evaluation methods that could enhance the accuracy of model assessments in AI security applications.

Technical details

• CTI-REALM benchmark: A security task where an agent converts threat intelligence reports into working detections by querying a live telemetry store.
• Open-weight models: Models with accessible weights that can be audited and run internally by organizations.
• Closed models: High-cost models with proprietary weights, often considered more advanced but less transparent.
• Stage-level metrics: Metrics that evaluate the performance of models based on the execution of their detection queries and comparison against an answer key.
• Token generation: The study also tracks the number of tokens each model generates, noting that the proportion of output dedicated to thinking does not correlate with reward.

FAQ