Skip to content

DATASET_WORKFLOW.md

Latest commit

 

History

History
121 lines (95 loc) · 5.89 KB

DATASET_WORKFLOW.md

File metadata and controls

121 lines (95 loc) · 5.89 KB
Raw

Cross-project dataset and validation workflow

The broader dataset records useful examples independently of current analyzer coverage. Metadata, acquired source, analysis, generated candidates, and build/test results are separate artifacts with explicit source identities.

Select before running

The September research amendment defines inclusion and grouping. The metadata inventory records source identities, checksums, source-inferred labels, project splits, and missing tool support. An agent's source review does not constitute either required independent human review.

Keep related fixes and copied source lineages in one split. Reserve entire projects before tuning. zlib is a development example; the existing libxml2 holdout remains unopened for analysis or adapter development.

Prepare source archives and ordered patches

source-package accepts a local JSON request with these fields:

  • base: exact path, source_url, and sha256 of the acquired archive.
  • patches: an ordered list with id, path, source_url, and sha256 for each patch. An empty list represents the base without added patches.
  • recipe: optional source_root for an archive with a containing directory.

Paths resolve relative to the request file. The command verifies supplied bytes, extracts a bounded regular-file tree, checks and applies patches in order, and retains original inputs and derived file hashes. Unsupported members, path escapes, failed patches, and limit failures cannot produce a successful package artifact.

uv run tianoshield-propagate source-package /path/package-request.json \
  --output /path/new-source-package

The product includes source-package.json, exact archive and patch bytes, and the derived tree under materialized/. It does not assign derived files a downstream Git commit or blob identity. Loading verifies retained evidence without executing Git. Explicit patch replay uses replay_source_package in the preparation module. Neither operation executes package source code.

Analyze a package-derived file

A case request can use mode: SOURCE_PACKAGE. Its preparation parameters name source_package, exact source_package_id, selected source_path, and the same upstream Git fix/parent parameters used by repository cases. The upstream proof and full package evidence travel with the saved case.

uv run tianoshield-propagate case /path/package-case.json \
  --output /path/new-case
uv run tianoshield-propagate packet /path/new-case \
  --output /path/new-review

The analyzer inspects these files and saves observations. The current sealed classifier/generator contract still requires Git target provenance. Package cases stop with SOURCE_PACKAGE_CLASSIFIER_UNSUPPORTED after analysis, preserving results and the precise missing capability. They are not relabeled as committed fixtures or assigned fabricated Git IDs.

The parser recovers function boundaries from balanced top-level source regions when a file parse error hides a definition. Tree-sitter parses the exact original byte ranges; macros are not removed or expanded. Original errors remain visible, and recovered definitions always have PARTIAL coverage. K&R parameters retain their declared types and signature order. Missing parameter types and ambiguous unnamed types retain incomplete coverage. Callback parameter names do not enter the outer function's parameter scope or normalization.

The three zlib development package states now complete analysis. Against the initial fix, the base matches VULNERABLE, the initial fix matches ALREADY_PATCHED, and the corrected series remains UNCERTAIN. All three have partial function coverage and stop before sealed classifier evidence or generation. The initial fix's known NULL-header regression also demonstrates why a patch match cannot replace build and regression evidence.

Review diagnostics distinguish complete semantic NOT_APPLICABLE observations from partial parsing, even when both have an INCOMPLETE summary. Whole-file coverage remains separate from coverage of the selected function.

Run build and regression checks

validate-build runs an explicit local recipe. It binds each variant to an actual package product and source-tree digest, and records tool identities, fixed command arguments, limits, expectations, and logs. Source-corpus metadata never becomes an executable recipe automatically.

uv run tianoshield-propagate validate-build /path/recipe.json \
  --source pre=/path/pre-package/materialized \
  --source-package pre=/path/pre-package \
  --source post=/path/post-package/materialized \
  --source-package post=/path/post-package \
  --output /path/new-build-result

Support files resolve from the recipe directory unless --auxiliary-root is supplied. Commands run in disposable copies with bounded execution and logs. This is local process isolation, not a security sandbox. Use known development source and a reviewed recipe.

Compile and test outcomes are independent. A failing test on the vulnerable variant can be an expected negative control; its recorded outcome remains FAIL. The result separately states whether expectations were met. A failed prerequisite leaves dependent steps skipped.

Build/test evidence does not rewrite an incomplete analysis result or supply a propagation label. Passing selected tests supports only the tested behavior. Retained build verification does not rerun compilers or tests.

Evidence needed for stronger claims

Retain unknown and unsupported examples, report source acquisition separately from execution, and count related states as dependent cases. Only the subset with a reviewed downstream relationship can measure propagation. Behavioral regression checks and upstream patch analysis have separate denominators.