Research prototype for a model-agnostic, token-efficient programming IR that compiles to JavaScript while using unchanged mainstream LLM tokenizers.
IR means intermediate representation. It is a small code-like format between an AI model and a normal programming language:
task prompt → model writes compact LIR → compiler produces JavaScript → tests run
The idea was that a model might spend fewer tokens writing LIR than Python or JavaScript. The latest blind result did not support that idea: the model was much less accurate in LIR and used more tokens per correct solution. This repository is published as an experimental implementation and transparent negative result, not as a production tool or a successful compression claim.
| Frozen HumanEval 60–79 · Qwen2.5-Coder-7B | LIR v0.0.8 | Python |
|---|---|---|
| Pass@1 | 4/20 | 17/20 |
| After one repair | 5/20 | 18/20 |
| Session-amortized tokens/correct | 1,197.0 | 238.5 |
Caution
LIR was 65 percentage points behind Python at Pass@1 and about 5.02× worse in tokens per correct solution. No breakthrough, no-quality-loss, or token-saving claim is supported.
There is no validated token-saving or no-quality-loss claim yet.
The frozen v0.0.7 blind run on HumanEval 40–59 failed decisively with Qwen2.5-Coder-7B: LIR scored 0/20 at Pass@1 and after one repair, while Python scored 20/20. All LIR attempts failed compilation before tests because the v0.0.7 parser rejected common model formatting. The raw negative result is retained in docs/BLIND_RESULT_QWEN_V007.md.
v0.0.8 is a separate revision with general whitespace, brace, semicolon, declaration, return, and loop normalization. On the now-public development replay, the parser accepted 20/20 of the v0.0.7 first generations, with 15/20 passing official tests. This replay is not blind evidence.
On the untouched HumanEval 60–79 slice, Qwen2.5-Coder-7B scored 4/20 LIR versus 17/20 Python at Pass@1. After one repair it scored 5/20 versus 18/20. LIR required 1,197.0 session-amortized tokens per correct solution versus Python’s 238.5—about 5.02 times more, not less.
v0.0.8 therefore fails both the quality and token-efficiency gates. See docs/BLIND_RESULT_QWEN_V008.md for raw-result links and failure analysis.
The language was frozen before HumanEval/20–39 were inspected.
| o200k metric | result |
|---|---|
| HumanEval/20–39 executable LIR | 20/20 |
| LIR source | 888 tokens |
| Python reference | 1008 tokens |
| held-out source reduction | 11.9% |
| tasks shorter / tied / longer | 14 / 1 / 5 |
This is deliberately much more conservative than the internal benchmark. The losses remain public; Python is extremely dense on operations such as swapcase, Counter, and stride-slice assignment.
HumanEval/0–19 are development data; 20–39 are frozen evaluation data.
- LIR source: 1460 o200k tokens
- Python source: 1735
- source reduction: 15.85%
- compact LIR core + sequence specification: 177 tokens
- with the spec amortized once over the 40-task session: 5.65% fewer spec+source tokens than Python
Cold one-off requests are a different result: repeatedly paying the whole spec overwhelms savings on tiny functions. Both modes are reported rather than conflated. See docs/EXTERNAL_GATE_V006.md.
The synthetic 100-instance / 25-family design corpus remains useful for regression testing, but is not the external headline:
| o200k comparison | reduction |
|---|---|
| readable JavaScript | 41.3% |
| minified JavaScript | 39.7% |
| Python | 35.7% |
All 100 cases behavior-match their baselines.
The model receives a short core+sequence teaching prompt. Its exact token count is reported with every frozen run rather than assumed from an earlier version.
has_close_elements(xs,t)=xs.pairAny(abs(x-y)<t)
below_zero(xs)=xs.scan(+).any(x<0)
sum_product(xs)=[xs.sum,xs.prod]
npm test
npm run check
npm run bench:100
npm run external:pilot
npm run external:holdout
npm run external:40Run blind generation against any OpenAI-compatible model server:
LLM_BASE_URL=http://localhost:8000 \
LLM_MODEL=Qwen/Qwen2.5-Coder-7B-Instruct \
npm run external:llm -- --lang lir --tasks benchmarks/external/blind_v008/humaneval60_79.json --spec compact --repairs 1 --require-frozenRepeat with --lang python, then with --spec verbose for the specification-ablation run.
The target is tokens per correct solution, not shortest source. The evaluator separately reports source, specification, task prompt, diagnostics/repair, Pass@1, and cold-vs-prefix-amortized accounting.
- No primitive is added because a scored held-out task needs it.
- HumanEval/0–79 is now development data. A later revision must use a new untouched range.
- Future changes must be mined/designed from independent data, then scored on a new unseen range.
- Source-only and total-context economics are separate metrics.
- Negative cases stay in the corpus.
- Internal synthetic results are never presented as external Pass@1 evidence.
- Develop a later revision without using a future headline slice, then freeze it before preparing new tasks.
- Pass matched LIR/Python generation on one model, then repeat on at least three unrelated model families.
- Verify compact-spec Pass@1 against the retained verbose spec.
- Add unchanged Qwen and Llama tokenizer measurements.
- Publish tokens/correct-solution including repairs and both cold/prefix-cached accounting.
- Only then choose a final product name or make a positive efficiency claim. “Compact LLM IR” remains a descriptive research-repository name.
MIT licensed.