Skip to content

[Ranking] Gate the synthetic MMoE model behind a demo baseline #28

Description

@theg1239

Problem

The bundled MMoE model is loaded by the recommendation path as the heavy ranker:

However, its documented training generator creates random user vectors, random repository vectors, simulated metadata, and 500,000 labels produced by hand-authored formulas:

A network trained on this data primarily learns to reproduce the simulator's assumptions. It is useful for plumbing and latency tests, but it is not evidence of recommendation quality and should not silently act as a production model.

Proposed change

  • Mark the current artifact explicitly as synthetic/demo-only.
  • Add a configurable ranker mode, for example heuristic and mmoe.
  • Default to a transparent deterministic ranker until a real interaction dataset and evaluated model are available.
  • Store model provenance with every artifact:
    • training-data snapshot/version
    • feature schema
    • training commit
    • evaluation metrics
    • creation timestamp
  • Require offline holdout evaluation before enabling an MMoE artifact.

Acceptance criteria

  • The production path cannot silently load a synthetic model.
  • A documented deterministic fallback ranks semantic relevance plus structured quality/activity features.
  • Model artifacts include machine-readable provenance and feature-schema compatibility checks.
  • Enabling MMoE requires an explicit configuration flag and a non-synthetic evaluated artifact.
  • Evaluation reports include ranking metrics such as NDCG/Recall and calibration for predicted actions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions