Results / CRM actions
Better CRM actions, from a better task contract.
A targeted prompt adapter improved Qwen's mean score on seven write-heavy CRM tasks. The improvement came from reading failures and specifying the missing behavior.
Runs recorded 23 April 2026 · Historical Operant publication
Qwen 3.6 Plus · raw
0.400
± 0.067 · 3 replicates
Sonnet 4.6
0.557
± 0.066 · 3 replicates
Qwen · targeted v5 adapter
0.630
± 0.087 · 10 replicates
Mean partial-credit score · higher is better
Seven tasks that change the CRM.
The action slice covers updating a contact's phone, adding a contact to a campaign, creating a note, creating a contact for an account, creating an opportunity, advancing its stage, and qualifying a lead. The limited_zapier tool surface gives the agent the tools needed to complete each task.
A generic adapter tuned on reasoning-heavy API tasks did not transfer: its 0.369 mean score was below the raw open model's 0.400. The targeted v5 adapter added proof-artifact requirements, latest-pricing rules and account-health checks after inspecting failed trajectories.
This result is prompt and adapter optimization. The publication does not report weight training for the v5 CRM result.
| Route | Mean score ± SD | Strict pass | Replicates | Cost / full slice |
|---|---|---|---|---|
| Sonnet 4.6 · frontier ceiling | 0.557 ± 0.066 | 19.0% | 3 | $1.1204 |
| Qwen 3.6 Plus · open baseline | 0.400 ± 0.067 | 14.3% | 3 | $0.2297 |
| Qwen 3.6 Plus · + hand adapter | 0.369 ± 0.082 | 14.3% | 3 | $0.2323 |
| Qwen 3.6 Plus · + targeted v5 adapter | 0.630 ± 0.087 | 54.3% | 10 | $0.2749 |
Seven tasks per replicate. ± is standard deviation, not a confidence interval. Strict pass requires every assertion to pass. Cost is for one full seven-task slice, not one request.
The reasoning slice is a separate experiment.
Four reasoning-heavy sales API tasks test negative selection, priority selection, implicit rules and cross-reference validation. On this surface, GEPA v3 at temperature zero scored 0.313 ± 0.110 across ten replicates, versus Sonnet's 0.160 ± 0.009 across three.
The two slices use different task and tool surfaces. Their absolute scores and costs should not be pooled. The v3 adapter exceeded the reference mean with substantial remaining rollout variance.
| Route | Mean score ± SD | Strict pass | Replicates | Cost / full slice |
|---|---|---|---|---|
| Sonnet 4.6 · frontier ceiling | 0.160 ± 0.009 | 0.0% | 3 | $0.7070 |
| Qwen 3.6 Plus · open baseline | 0.084 ± 0.073 | 0.0% | 3 | $0.0946 |
| Qwen 3.6 Plus · + GEPA (default temp) | 0.157 ± 0.124 | 8.3% | 3 | $0.1015 |
| Qwen 3.6 Plus · + GEPA v3 @ temp=0 | 0.313 ± 0.110 | 22.5% | 10 | $0.1276 |
Four tasks per replicate. The GEPA v3 result uses n=10; the other rows use n=3. Zero strict passes does not mean zero partial-credit progress.
Method and cost basis.
Benchmark: the sales domain of Zapier AutomationBench. Models: Claude Sonnet 4.6 through Anthropic and Qwen 3.6 Plus through Fireworks. All runs use max_steps=10 and max_tokens=4096.
GEPA used the standalone gepa package with Claude Opus 4.7 as the reflection model, max_metric_calls=20 and reflection_minibatch_size=2. The reasoning-heavy v3 run was replicated to n=10. CRM v5 was a manual failure-mode hill climb, also replicated to n=10.
CRM v5 cost $0.2749 per full slice versus $1.1204 for Sonnet: 24.5% of the reported reference cost. These are evaluation-slice costs. They are not a bill for end-to-end optimization, training or operating the product.
The publication identifies the selected benchmark tasks and run settings. An independent, untouched confirmation set, exact runtime/package pins, confidence intervals and end-to-end optimization cost are not reported in this publication.
Limitations and source record.
- These are narrow benchmark slices, not a claim of general model superiority.
- The optimized and reference rows have different replicate counts; the observed means retain visible variance.
- The same task surfaces informed adapter development. Treat the results as optimization evidence, not proof of unseen-task generalization.
- A 0.630 mean is partial credit. The optimized CRM route's strict-pass rate is 54.3%.
Source: the original Operant publication and its checked-in benchmark snapshot. Adapted on 5 September 2026. This page preserves historical evidence; it is not a new run, live telemetry or a forecast.
Benchmark references: Zapier AutomationBench · Prime Intellect introduction