Skip to content

About

Experimental intermediate representation for coding agents, published with transparent negative blind results.

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Compact LLM IR

A compact-code experiment whose blind result failed.

Status Version CI License

Results · Reproduce · Raw report · Limitations

Research prototype for a model-agnostic, token-efficient programming IR that compiles to JavaScript while using unchanged mainstream LLM tokenizers.

IR means intermediate representation. It is a small code-like format between an AI model and a normal programming language:

task prompt → model writes compact LIR → compiler produces JavaScript → tests run

The idea was that a model might spend fewer tokens writing LIR than Python or JavaScript. The latest blind result did not support that idea: the model was much less accurate in LIR and used more tokens per correct solution. This repository is published as an experimental implementation and transparent negative result, not as a production tool or a successful compression claim.

The result in one table

Frozen HumanEval 60–79 · Qwen2.5-Coder-7B LIR v0.0.8 Python
Pass@1 4/20 17/20
After one repair 5/20 18/20
Session-amortized tokens/correct 1,197.0 238.5

Caution

LIR was 65 percentage points behind Python at Pass@1 and about 5.02× worse in tokens per correct solution. No breakthrough, no-quality-loss, or token-saving claim is supported.

Current proof status — v0.0.8 failed

There is no validated token-saving or no-quality-loss claim yet.

The frozen v0.0.7 blind run on HumanEval 40–59 failed decisively with Qwen2.5-Coder-7B: LIR scored 0/20 at Pass@1 and after one repair, while Python scored 20/20. All LIR attempts failed compilation before tests because the v0.0.7 parser rejected common model formatting. The raw negative result is retained in docs/BLIND_RESULT_QWEN_V007.md.

v0.0.8 is a separate revision with general whitespace, brace, semicolon, declaration, return, and loop normalization. On the now-public development replay, the parser accepted 20/20 of the v0.0.7 first generations, with 15/20 passing official tests. This replay is not blind evidence.

On the untouched HumanEval 60–79 slice, Qwen2.5-Coder-7B scored 4/20 LIR versus 17/20 Python at Pass@1. After one repair it scored 5/20 versus 18/20. LIR required 1,197.0 session-amortized tokens per correct solution versus Python’s 238.5—about 5.02 times more, not less.

v0.0.8 therefore fails both the quality and token-efficiency gates. See docs/BLIND_RESULT_QWEN_V008.md for raw-result links and failure analysis.

Earlier source-density results — not model Pass@1 evidence

Frozen external HumanEval gate

The language was frozen before HumanEval/20–39 were inspected.

o200k metric result
HumanEval/20–39 executable LIR 20/20
LIR source 888 tokens
Python reference 1008 tokens
held-out source reduction 11.9%
tasks shorter / tied / longer 14 / 1 / 5

This is deliberately much more conservative than the internal benchmark. The losses remain public; Python is extremely dense on operations such as swapcase, Counter, and stride-slice assignment.

External 0–39 combined

HumanEval/0–19 are development data; 20–39 are frozen evaluation data.

  • LIR source: 1460 o200k tokens
  • Python source: 1735
  • source reduction: 15.85%
  • compact LIR core + sequence specification: 177 tokens
  • with the spec amortized once over the 40-task session: 5.65% fewer spec+source tokens than Python

Cold one-off requests are a different result: repeatedly paying the whole spec overwhelms savings on tiny functions. Both modes are reported rather than conflated. See docs/EXTERNAL_GATE_V006.md.

Internal adversarial corpus

The synthetic 100-instance / 25-family design corpus remains useful for regression testing, but is not the external headline:

o200k comparison reduction
readable JavaScript 41.3%
minified JavaScript 39.7%
Python 35.7%

All 100 cases behavior-match their baselines.

Compact model specification

The model receives a short core+sequence teaching prompt. Its exact token count is reported with every frozen run rather than assumed from an earlier version.

has_close_elements(xs,t)=xs.pairAny(abs(x-y)<t)
below_zero(xs)=xs.scan(+).any(x<0)
sum_product(xs)=[xs.sum,xs.prod]

Reproduce

npm test
npm run check
npm run bench:100
npm run external:pilot
npm run external:holdout
npm run external:40

Run blind generation against any OpenAI-compatible model server:

LLM_BASE_URL=http://localhost:8000 \
LLM_MODEL=Qwen/Qwen2.5-Coder-7B-Instruct \
npm run external:llm -- --lang lir --tasks benchmarks/external/blind_v008/humaneval60_79.json --spec compact --repairs 1 --require-frozen

Repeat with --lang python, then with --spec verbose for the specification-ablation run.

Primary metric

The target is tokens per correct solution, not shortest source. The evaluator separately reports source, specification, task prompt, diagnostics/repair, Pass@1, and cold-vs-prefix-amortized accounting.

Anti-gaming rules

  • No primitive is added because a scored held-out task needs it.
  • HumanEval/0–79 is now development data. A later revision must use a new untouched range.
  • Future changes must be mined/designed from independent data, then scored on a new unseen range.
  • Source-only and total-context economics are separate metrics.
  • Negative cases stay in the corpus.
  • Internal synthetic results are never presented as external Pass@1 evidence.

Publication gates remaining

  1. Develop a later revision without using a future headline slice, then freeze it before preparing new tasks.
  2. Pass matched LIR/Python generation on one model, then repeat on at least three unrelated model families.
  3. Verify compact-spec Pass@1 against the retained verbose spec.
  4. Add unchanged Qwen and Llama tokenizer measurements.
  5. Publish tokens/correct-solution including repairs and both cold/prefix-cached accounting.
  6. Only then choose a final product name or make a positive efficiency claim. “Compact LLM IR” remains a descriptive research-repository name.

MIT licensed.

About

Experimental intermediate representation for coding agents, published with transparent negative blind results.

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages