[Feature] Improve Job and Group metadata persistence #1306

Feature Category

• SDK & API
• CLI

Problem Statement

Job metadata is currently stored together with OSS artifacts. This makes structured queries, transactional updates, and lifecycle management difficult.

The current metadata model cannot reliably support:

• uniquely identifying Jobs when names are duplicated;
• tracking Group and Job status changes;
• finding unfinished Jobs after an interrupted execution;
• retrying failed or unsuccessful Jobs;
• listing and filtering Jobs within a Group;
• aggregating Job statuses and scores;
• listing Groups under a namespace with pagination.

The local CLI also depends on metadata concepts that should be owned by a trusted server-side service.

Proposed Solution

Provide structured database-backed persistence and query capabilities for Job and Group metadata while keeping execution artifacts in OSS.

Detailed Feature Description

The SDK should support:

• creating, updating, and retrieving Groups and Jobs;
• uniquely retrieving a Job by job_id;
• listing Groups under a namespace;
• listing and filtering Jobs under a Group;
• querying active, completed, unsuccessful, and unfinished Jobs;
• retrieving Group status and score statistics;
• optional cursor pagination.

The local CLI should continue to support task execution and explicit single-task resume without directly accessing persisted metadata.