We Ran Three Local Models Through 23 Hidden JavaScript Tests

Qwen 2.5 Coder 14B passed 20 of 23 hidden JavaScript unit tests in our local benchmark. Qwen 3.5 9B passed 11. Hermes 3 8B passed nine.

The fastest model was not the most correct.

Hermes 3 generated 65.2 tokens per second, more than twice the throughput of either Qwen model, but it finished with a 39.1% test pass rate. Qwen 2.5 Coder generated 30.5 tokens per second and passed 87.0%.

All three ran through Ollama on the same Apple M2 Max with 32GB of unified memory. They received the same three prompts, structured-output constraint, context window, temperature, seed, output ceiling, and hidden tests. No hosted model API was called, so measured API fees were $0.

That last number needs a boundary. We did not measure electricity, hardware depreciation, setup time, or maintenance. Local inference moves those costs; it does not erase them.

Figure: Qwen 2.5 Coder 14B passed 20 of 23 tests. A task counted as fully passed only when every hidden test succeeded.

Methodology

The benchmark used three models already installed locally:

The machine was an Apple M2 Max with 32GB of unified memory. Ollama 0.19.0 served the models through its local generate API. Each request used:

• Temperature 0.
• Seed 42.
• Context window 4096.
• Output ceiling 768 tokens.
• Thinking disabled.
• One run per model and task.
• A JSON schema requiring one code string.

The harness unloaded a model before its first task, then kept it available for the remaining two tasks. That captured one model-load event per model without treating every function as a cold start. Operating-system file caches were not cleared.

The three prompts requested:

1. mergeIntervals(intervals), including validation, touching intervals, stable sorting, and a no-mutation rule.
2. parseDuration(input), including decimals, five units, strict rejection cases, whitespace, and case handling.
3. topologicalSort(nodes, edges), including deterministic lexicographic choice, cycles, invalid references, duplicate nodes, and no mutation.

Generated functions ran inside Node’s vm module with no process, filesystem, network, package imports, or tool access. Each invocation had a one-second execution timeout. A model received credit for a test only when its code executed and matched the expected result or error.

The scoring unit was the test, not a human impression of code style. A task was considered completely passed only when every hidden test passed.

This follows the same principle as our coding-agent evaluation framework: a plausible answer is not an accepted outcome until an executable check says it works.

Results

Qwen 2.5 Coder created the best code under these constraints. Its advantage was not marginal: it passed nine more tests than Qwen 3.5 and 11 more than Hermes 3.

It was not perfect. The merge function sorted the caller’s array in place, violating the no-mutation requirement. Its topological sort respected dependencies but failed the stricter rule requiring the lexicographically smallest valid result whenever several nodes were available.

Qwen 3.5 produced the only perfect merge-interval solution. Its duration parser contained a small indexing error: the regular expression stored the unit in capture group three, but the code read group two. That turned otherwise valid inputs into errors. On topological sort, the response reached the 768-token ceiling before the structured JSON object closed, so the harness could not execute it.

Hermes 3 was fastest and smallest, but its generated code left more basic defects. The interval function mutated its input, returned an invalid shape for an empty array, and did not enforce pair validation. The duration parser converted both the numeric value and unit string with Number, turning the unit into NaN before calling toLowerCase.

Figure: Hermes 3 generated more than twice as fast as the other models, but speed did not track correctness. Circle size represents the local model file.

Failure patterns mattered more than one score

The task matrix explains more than the aggregate ranking.

Figure: Each prompt exposed a different failure surface. Qwen 3.5 solved merge intervals completely but hit the output ceiling on topological sort.

All three models produced code that looked reasonable at a glance. The failed tests found different weaknesses:

• Input ownership: Hermes 3 and Qwen 2.5 Coder sorted the supplied interval array in place.
• Capture-group bookkeeping: Qwen 3.5 and Hermes 3 both mishandled the duration unit, through different bugs.
• Determinism: Qwen 2.5 Coder found a valid graph order but not the required lexicographically smallest one.
• Output budgeting: Qwen 3.5 spent the entire 768-token allowance on a hand-written heap and returned incomplete JSON.
• Edge cases: Hermes 3 treated a graph with no edges as a cycle and mishandled an empty interval list.

These are not exotic adversarial cases. Mutation, strict parsing, deterministic output, bounded generation, and empty inputs are common production requirements.

A benchmark that graded only the happy-path example would have hidden most of the difference. So would a reviewer who accepted code because the algorithm name and loop structure looked familiar.

Our guide to verifying model and benchmark claims recommends making the acceptance boundary explicit. Here, the boundary was executable: 23 tests, published prompts, a fixed token ceiling, and preserved outputs.

The cost result has two ledgers

Ollama served every request from 127.0.0.1. No hosted API key or paid endpoint was used, so per-call API charges were exactly $0 for this run.

That is the first ledger.

The second ledger includes the computer, electricity, storage, installation, model downloads, updates, and operator time. We did not attach a dollar value to those inputs because this run did not measure power at the wall or allocate the hardware’s purchase price.

Figure: The benchmark measured API fees, file sizes, wall time, tokens, throughput, and correctness. It did not measure the full cost of owning and operating the machine.

This distinction changes the deployment question. Local inference can be attractive when hardware already exists, utilization is high, data must stay on-device, or hosted token pricing dominates a stable workload. It can be uneconomic when requests are rare, maintenance is expensive, or a hosted model avoids enough failures to reduce review and retry cost.

The right comparison is cost per accepted outcome. In this run, Hermes 3 completed quickly but did not fully pass any task. Qwen 2.5 Coder took the longest overall and still failed two task-level acceptance gates, although it came much closer on individual tests.

For a production route, we would add retry cost, reviewer time, power draw, and a hosted-model baseline. The GPT-5.6 configuration analysis explains why raw token price and outcome cost often diverge.

Limitations

This is a small, local engineering test, not a universal model leaderboard.

Only three JavaScript functions were evaluated. The prompts favored strict, self-contained code and did not test repository navigation, tool use, long context, debugging over multiple turns, framework knowledge, or prose quality.

Each model-task pair ran once at temperature zero with a fixed seed. Repeated runs would show stability and variance. A different Ollama version, quantization, prompt template, output limit, context size, or machine can change both speed and correctness.

The total-time measurement included prompt evaluation, generation, and one model load within an already running Ollama server. It was not a clean operating-system cold start because file caches remained available. Generation throughput is therefore more portable than the reported total seconds, but even tokens per second depends on hardware and runtime.

The topological-sort failure for Qwen 3.5 is partly a constraint result. The model produced a long approach that reached the 768-token ceiling. A larger output budget might have completed. The benchmark intentionally kept the ceiling because output length affects latency, memory, and operational predictability.

No electricity meter was used. The $0 cost figure applies only to API charges, not total ownership.

Finally, hidden tests reduce prompt gaming but do not prove semantic coverage. More tests, property-based generation, mutation testing, and independent review would make the code verdict stronger.

Reproduction

The benchmark harness is checked into this site’s repository as scripts/benchmark_local_models.mjs. It uses only Node’s standard library and an Ollama server with the three model names installed.

Run:

ollama serve
pnpm run lab:local-models

To choose different installed models:

pnpm run lab:local-models -- --models=model-a,model-b

The request shape is fixed in the harness:

{
  stream: false,
  think: false,
  format: {
    type: "object",
    properties: { code: { type: "string" } },
    required: ["code"]
  },
  options: {
    temperature: 0,
    seed: 42,
    num_ctx: 4096,
    num_predict: 768
  }
}

The complete measured summary, test failures, timings, and executable model outputs are available in the downloadable benchmark data.

This is the first Yield Signal Lab run. Future model comparisons should keep the same rule: publish the harness, preserve the failure boundary, state what cost was excluded, and let the code earn the conclusion.

Sources

• Ollama API: Generate a response
• Node.js documentation: VM execution contexts
• Qwen: Qwen 2.5 Coder 14B Instruct model card
• NousResearch: Hermes 3 Llama 3.1 8B model card