Results / Open models, cold start
Open models on the cold-start task.
We ran three open models from Hugging Face on one Mac, on the same 92 cases as the first cold-start study. Two open embedding models that compare the first message with the label and description of each pattern got 92.4% and 90.2% right, too close to rank. An open 4B instruct model got 80.4%.
Run on 8 October 2026 on an Apple M4 Max, follow-ups on 9 October · Every number comes from the run files · Research, not a product feature
all-MiniLM-L6-v2
92.4%
of 92 cases · cross-validated threshold
bge-small-en-v1.5
90.2%
of 92 cases · cross-validated threshold
Qwen3-4B-Instruct-2507
80.4%
of 92 cases · top option
A case is right when the matcher picks the expected cluster, or none when no cluster fits
The task.
The cases are the 92 cases of the first cold-start study: 38 family members, 16 conversations that expect none, and 38 leave-one-out cases. We rebuilt each first message from the same demo recordings, and checked that the ids, the passes and the expected answers equal the rows of that study.
The options are the four clusters that Operant made, each as its label and description, and "None of these tasks." The question is the same: "Which of these known tasks is the user asking the assistant to do?"
| Matcher | Family members right | None cases right | Leave-one-out: said none | All cases right | Sanctions sent to SAR |
|---|---|---|---|---|---|
| Qwen3-4B-Instruct-2507, top option | 86.8% | 25.0% | 97.4% | 80.4% | 7 of 7 |
| Qwen3-4B-Instruct-2507, assign only at p ≥ 0.9 | 86.8% | 25.0% | 97.4% | 80.4% | 7 of 7 |
| all-MiniLM-L6-v2, nearest option (no threshold) | 97.4% | 0.0% | 0.0% | 40.2% | 6 of 7 |
| all-MiniLM-L6-v2, cosine ≥ 0.3298 (tuned on this set) | 94.7% | 81.2% | 97.4% | 93.5% | 0 of 7 |
| all-MiniLM-L6-v2, cross-validated threshold | 92.1% | 81.2% | 97.4% | 92.4% | 0 of 7 |
| bge-small-en-v1.5, nearest option (no threshold) | 94.7% | 0.0% | 0.0% | 39.1% | 6 of 7 |
| bge-small-en-v1.5, cosine ≥ 0.5982 (tuned on this set) | 89.5% | 87.5% | 97.4% | 92.4% | 0 of 7 |
| bge-small-en-v1.5, cross-validated threshold | 86.8% | 87.5% | 94.7% | 90.2% | 0 of 7 |
| bge-small-en-v1.5 without its query instruction, cross-validated threshold | 73.7% | 81.2% | 92.1% | 82.6% | 0 of 7 |
| First study: Jev, top option | 92.1% | 37.5% | 100.0% | 85.9% | 7 of 7 |
| First study: Kev-9B, top option | 71.1% | 93.8% | 100.0% | 87.0% | 0 of 7 |
| First study: text-embedding-3-small to the centroids, nearest option (no threshold) | 21.1% | 0.0% | 0.0% | 8.7% | 7 of 7 |
| First study: text-embedding-3-small to the centroids, cosine ≥ 0.5 | 21.1% | 31.2% | 100.0% | 55.4% | 6 of 7 |
| First study: text-embedding-3-small to the centroids, cross-validated threshold | 18.4% | 93.8% | 100.0% | 65.2% | 0 of 7 |
A cross-validated threshold is tuned on the other 53 conversations only, for each conversation. The first-study rows come from its recalculated row files. After a reader's questions, we added on 9 October: bge-small-en-v1.5 without its query instruction, and text-embedding-3-small with no threshold and with the same cross-validated threshold, from the first study's own cosines.
What we found.
The first study's embedding found few family members, and its threshold is not the reason. It compared the first message with the centroid of the goal summaries in each cluster, using text-embedding-3-small, and found 8 of the 38 family members at its fixed floor of 0.5. It found 8 with no threshold at all, and 7 with a threshold cross-validated the same way as here. Here, two small open embedding models compared the message with the label and description, the same text that the decision models read. With a cross-validated threshold, they found 35 and 33. Two things changed at once, the model and the text, so this run does not show which change matters more.
Both open embedding models rank the clusters well. With no threshold, the nearest cluster was the right one for 37 and 36 of the 38 family members (97.4% and 94.7%). The threshold does a different job: it decides when to say none.
With no threshold, the nearest option is never none, so the none cases score 0.0%. With a threshold, the two embedding models got 81.2% and 87.5% of the none cases right, and sent no sanctions-screening conversation to SAR triage.
The two embedding models got 85 and 83 of the 92 cases right with a cross-validated threshold. A gap of 2 cases is too small to say which model is better.
bge-small-en-v1.5 did better with its query instruction. Its model card says to choose the setting that does better on the task, so we ran it both ways: 83 of 92 right with the instruction, and 76 without it. With no threshold, both ways found 36 of the 38 family members.
The 4B model was sure of itself, and sometimes wrong. Qwen3-4B gave its top option a probability of 0.9 or more on 91 of the 92 cases, so the 0.9 floor changed nothing. Like Jev, it sent all 7 sanctions-screening conversations to SAR triage.
Method.
Qwen3-4B-Instruct-2507 ran in 4-bit MLX format, with one forward pass for each case. The options were letters. The probability of an option is the probability of its letter as the next token, normalised over the letters. The embedding models ran with sentence-transformers and compared each first message with each "label: description" by cosine. bge-small-en-v1.5 got its query instruction, and a second run on 9 October went without it. The text-embedding-3-small rows use the cosines stored in the first study's rows, scored with the same rules as the open models.
Every number on this page comes from the run files: one row for each case and model, the metrics, and a run log with the command, the model revisions and the timestamps.
Limitations and source record.
- Operant wrote the cluster descriptions from the goal summaries of these demo conversations, and the same conversations are the test cases. So the descriptions are close to the messages, and every absolute number here is optimistic. A fair test writes the descriptions from one half of the conversations and tests on the other half, or writes them again for each fold. We have not run that.
- 54 conversations from one demo workload. The cases are few, and the families are synthetic.
- Qwen3-4B saw one prompt wording and one option order. Another order can give another result.
- Qwen3-4B was not fine-tuned on these cases. For that, a reader suggested a few hundred labelled cases, with some held back for evaluation. This set has 54 conversations.
- The thresholds tuned on this set are optimistic. The cross-validated numbers are the fair ones, but the set is small.
- A fair split of model and text is a 2 × 2: each model on each text, all with the same cross-validated threshold. This page has 2 of the 4 cells: text-embedding-3-small on the centroids, and the open models on the label and description. Not run: text-embedding-3-small on the label and description, and the open models on the centroids of the goal summaries. So this run mixes the effect of the text with the effect of the model.
Source: our own run on 8 October 2026 (scripts/g3_openmodels.py) and follow-ups on 9 October 2026 (scripts/g3_r21_followups.py), on the cases of the upstream Strata study at commit ada801f. Models from Hugging Face: mlx-community/Qwen3-4B-Instruct-2507-4bit, sentence-transformers/all-MiniLM-L6-v2 and BAAI/bge-small-en-v1.5. This is research evidence, not live telemetry and not a product feature.