QDevHow well does each model write jq?
Every provider runs the real NL→jq pipeline over a fixed corpus of 24 questions. Each generated query is scored by running itand comparing the output to a hand-verified answer — so it's graded on the result, not on how the jq is written.
Same pipeline as the extension: schema-only grounding, a bounded self-correcting agent loop, jq in a Web Worker. See how it works.
Correct-answer rate
Share of the 24 cases whose committed query produced the exact expected result. The tick marks the first try— before the loop's self-correction.
How accuracy has moved
Pass rate (%) per provider across the last 3 recorded runs. Model updates, prompt tuning, and corpus changes all show up here.
Time to a query
Median generation time per question; the label also shows p90. Local models trade the network round-trip for on-device compute.
Where each model is strong
Pass rate sliced by what the query exercises — filters, aggregation, string handling, recursion. Localises a weakness to a class of question.
| TAG | Hosted | Local | Chrome built-in | Ollama |
|---|---|---|---|---|
| aggregation | 100% | 50% | 50% | 83% |
| array | 100% | 0% | 0% | 100% |
| boolean | 100% | 67% | 0% | 100% |
| case-insensitive | 100% | 0% | 0% | 100% |
| count | 100% | 50% | 0% | 100% |
| date | 100% | 0% | 0% | 100% |
| descent | 100% | 0% | 0% | 0% |
| empty | 100% | 100% | 0% | 100% |
| existence | 100% | 100% | 0% | 100% |
| filter | 100% | 63% | 0% | 100% |
| flatten | 100% | 50% | 25% | 50% |
| group | 100% | 0% | 0% | 100% |
| interpolation | 100% | 0% | 0% | 100% |
| keys | 100% | 50% | 0% | 100% |
| max | 100% | 100% | 100% | 0% |
| merge | 100% | 0% | 100% | 0% |
| min | 100% | 100% | 100% | 100% |
| negation | 100% | 100% | 0% | 100% |
| nested | 100% | 25% | 25% | 75% |
| numeric | 100% | 100% | 0% | 100% |
| object | 100% | 100% | 0% | 100% |
| project | 100% | 67% | 0% | 100% |
| range | 100% | 100% | 0% | 100% |
| recursion | 100% | 0% | 0% | 0% |
| slice | 100% | 0% | 0% | 100% |
| sort | 100% | 100% | 0% | 100% |
| string | 100% | 0% | 0% | 100% |
| sum | 100% | 0% | 100% | 100% |
| top-level | 100% | 0% | 0% | 100% |
| unique | 100% | 0% | 0% | 100% |
What it takes to run each one
The trade behind the accuracy: setup, first-run download, and whether anything leaves the device.
| PROVIDER | REQUIRES | FIRST-RUN DOWNLOAD | DATA EGRESS | STATUS |
|---|---|---|---|---|
Hosted (best accuracy) proxy default | Hosted proxy | none | → schema + question | 100% pass |
Local (WebLLM · WebGPU) auto | WebGPU | ~1–2 GB first run | ● nothing | 50% pass |
Chrome built-in (Gemini Nano) gemini-nano | Chrome Prompt API (Gemini Nano) | browser-managed | ● nothing | 17% pass |
Ollama (local) qwen2.5-coder | Ollama @ localhost:11434 | model pull (varies) | ● nothing | 83% pass |
Warm-up (model download + compile), one-time per session: Local 2.2s.