[feat] Bill sandbox usage as credits with quota gating by junaway · Pull Request #5040 · Agenta-AI/agenta

junaway · 2026-07-02T14:57:57Z

Context

Track B records raw sandbox usage but bills nothing. Track C adds the billing layer: it rolls the raw resource-seconds up into a single credits unit, gates on it, and reports that one unit to Stripe. Base is Track B (feat/metering-track-b).

The reason for a credits unit: not every provider bills per second, and the raw dimensions are heterogeneous (core-seconds vs GiB-seconds vs GPU-seconds), so there is no single number to gate or bill on. Credits is that common number.

Changes

Each raw dimension converts to credits on its own rate, and the per-dimension credits sum into one total. So a usage event now records three layers instead of one:

raw:        SANDBOX_CPU_CORE_SECONDS, SANDBOX_RAM_GIBI_SECONDS, ...   (from Track B)
per-dim:    SANDBOX_CPU_CORE_CREDITS, SANDBOX_RAM_GIBI_CREDITS, ...   (new)
total:      SANDBOX_CREDITS = sum of the per-dimension credits        (new, the billable unit)

Conversion is a per-provider by per-dimension typed rate table (a Dimension enum plus a ProviderRates model), with a pure to_credits() that returns credits only as a Decimal. It never computes money. Stripe owns credit-to-money through the plan price, the same way traces work today. Rates are env-configurable via AGENTA_SANDBOX_CREDIT_RATES. Credits are stored as millicredits (value times 1000) because MeterDTO.delta is an integer.

Gating runs in two places on SANDBOX_CREDITS: a create-time soft pre-check before launching a sandbox, and a post-hoc true-up after usage lands. RBAC (RUN_SESSIONS) stays a separate check. Permission is may-run; entitlement is has-quota.

REPORTS gains exactly one entry, sandbox_credits. The raw seconds and the per-dimension credits are recorded but not reported, so splitting billing per dimension later is a config change, not a rewrite. Migration ee0000000005 appends the five credit values to the meters_type enum (down_revision = ee0000000004).

Tests / notes

42 unit tests pass: 30 for the credit conversion (rate round-trips, non-per-second providers, the reference-scenario cross-check) and 12 for gating. ruff is clean and the composition root imports.
Pricing numbers are marked # TODO(pricing). The quotas are non-blocking until pricing sets real free/limit values.
Base this PR on Track B.

vercel · 2026-07-02T14:58:04Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agenta-documentation	Ready	Preview, Comment	Jul 6, 2026 7:51pm

coderabbitai · 2026-07-02T14:58:06Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0b91cb59-23ef-4299-913b-73201e06bd95

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/metering-track-c

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Copilot

Pull request overview

Implements Track C’s billing layer for sandbox usage in the EE API: converts raw sandbox resource-seconds into a single billable SANDBOX_CREDITS unit (plus per-dimension credit meters), wires reporting to Stripe via REPORTS, and adds quota-based gating (pre-check + post-hoc true-up).

Changes:

Add a provider/dimension rate table and pure to_credits() conversion (Decimal-based) with env-configurable overrides.
Record per-dimension credits and total SANDBOX_CREDITS (millicredits in the meters layer) and expose only the roll-up in REPORTS.
Add create-time quota gating and post-hoc true-up checks, plus EE migration and unit tests.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
docs/designs/sandbox-metering/TRACK_C_FINDINGS.md	Track C design/findings write-up for credits + gating + reporting wiring.
api/oss/src/utils/env.py	Adds `env.sandbox` config (credit-rate overrides + estimate knobs).
api/ee/src/core/sandboxes/credits.py	Implements provider/dimension credit conversion and env override merge.
api/ee/src/core/sandboxes/sink.py	Writes per-dimension credit meters and total `SANDBOX_CREDITS` via entitlements adjust.
api/ee/src/core/sandboxes/service.py	Wires the credits sink after raw-seconds metering.
api/ee/src/core/sandboxes/gating.py	Adds Layer 1 (create-time) quota pre-check and Layer 2 (true-up) quota check.
api/ee/src/core/access/entitlements/types.py	Introduces new credit counters, default quotas, and `REPORTS` entry for `sandbox_credits`.
api/ee/src/core/meters/types.py	Mirrors the new entitlements counters into the `Meters` enum.
api/ee/databases/postgres/migrations/core_ee/versions/ee0000000005_add_sandbox_credit_meters.py	Adds the new sandbox credit enum labels to `meters_type`.
api/ee/tests/pytest/unit/test_sandbox_credits.py	Unit tests for conversion rate table and `to_credits()` behavior.
api/ee/tests/pytest/unit/test_sandbox_gating.py	Unit tests for gating logic and wiring assertions (REPORTS/constraints/quotas).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+| `SANDBOX_CPU_CREDITS`     | `sandbox_cpu_credits`     |
+| `SANDBOX_RAM_CREDITS`     | `sandbox_ram_credits`     |
+| `SANDBOX_SSD_CREDITS`     | `sandbox_ssd_credits`     |
+| `SANDBOX_GPU_CREDITS`     | `sandbox_gpu_credits`     |
+| `SANDBOX_CREDITS`         | `sandbox_credits`         |


+    try:
+        return await _check_sandbox_quota_ee(
+            organization_id=organization_id,
+            provider=provider,
+        )
+    except Exception:  # pylint: disable=broad-exception-caught
+        log.warning(
+            "[sandboxes] check_sandbox_quota failed for org=%s; failing open",
+            organization_id,
+            exc_info=True,
+        )
+        return True, None


+    try:
+        return await _check_sandbox_credits_true_up_ee(organization_id=organization_id)
+    except Exception:  # pylint: disable=broad-exception-caught
+        log.warning(
+            "[sandboxes] check_sandbox_credits_true_up failed for org=%s; failing open",
+            organization_id,
+            exc_info=True,
+        )
+        return True


+        for dim_key, rate_str in dims.items():
+            if dim_key in base_data:
+                try:
+                    base_data[dim_key] = Decimal(str(rate_str))
+                except Exception:  # pylint: disable=broad-exception-caught
+                    pass


+
+

Pure Decimal to_credits() conversion (raw resource-seconds -> credits) per provider x dimension (CPU/RAM/SSD/GPU), env-overridable via AGENTA_SANDBOX_CREDIT_RATES. No money math -- Stripe owns credit->money like traces. Ported from feat/metering-credits-layer's sandbox_metering/credits.py, renamed into core/sandboxes/ and re-keyed to the locked CPU/RAM/SSD/GPU scheme (docs/designs/sandbox-metering/NAMING.md).

- sink.py: record_usage_credits() writes SANDBOX_{CPU,RAM,SSD,GPU}_CREDITS and the sandbox_credits roll-up as millicredits (int-safe, x1000), one call per usage event, org-scoped. Wired into SandboxMeteringService.record_usage() right after the raw *_seconds adjust. - entitlements/types.py: add the 5 credit Counter members, per-plan Quota(period=MONTHLY) TODO(pricing) placeholders on every plan, add Counter.SANDBOX_CREDITS (only) to REPORTS as "sandbox_credits" -- the single billable sandbox meter; raw *_seconds and per-dimension *_credits stay measurement-only. Add the 5 credit counters to CONSTRAINTS[READ_ONLY][COUNTERS]. Add TODO(pricing) sandbox_credits price-slot placeholders to the Pro/Business DEFAULT_CATALOG entries (AGENTA_BILLING_PRICING itself needs no code change -- REPORTS already makes "sandbox_credits" a valid Stripe slot name). - meters/types.py: mirror the 5 credit Counter members into Meters. - env.py: add SandboxConfig (env.sandbox) for AGENTA_SANDBOX_CREDIT_RATES and the Layer-1 in-flight accrual estimate knobs.

gating.py (core/sandboxes/): two-layer entitlement gate mirroring the tracing pattern, ported from feat/sandbox-metering-phase-2's core/sandbox/metering.py and moved into core/sandboxes/. - check_sandbox_quota(): Layer 1 create-time soft pre-check (cached read) against Counter.SANDBOX_CREDITS, adding an estimated in-flight accrual converted through credits.to_credits() (CPU dimension, env-configurable estimated_vcpu/estimated_run_seconds). Fails open on infra errors. - check_sandbox_credits_true_up(): Layer 2 post-hoc true-up, a read-only recheck (delta=0) of SANDBOX_CREDITS after the sink has already written the authoritative adjust(). Returns False when over quota; kill-session wiring (Layer 2b) is a documented stub pending DELETE /sessions/streams/{id}. Deleted phase-2's inline _derive_credits() (ad hoc vcpu+ram sum) -- credits now come exclusively from credits.py's rate-table conversion. RBAC (RUN_SESSIONS) stays strictly separate; this module only checks quota.

ee0000000005, down_revision=ee0000000004 (Track B's sandbox+storage meters migration, current head). Appends the 5 credit enum labels (SANDBOX_{CPU,RAM,SSD,GPU}_CREDITS, SANDBOX_CREDITS) to meters_type via ALTER TYPE ... ADD VALUE IF NOT EXISTS. downgrade() is a no-op (Postgres can't drop enum labels), matching ee0000000003/ee0000000004.

…ds keys Mirror Track B: SANDBOX_<RESOURCE>_<UNIT>_{SECONDS,CREDITS}. Total SANDBOX_CREDITS unchanged.

…eakdown Sum the per-dimension millicredits actually written instead of re-truncating the exact credit sum, so SANDBOX_CREDITS == sum of the *_CREDITS meters.

After the RUN_SESSIONS permission check in set_session_stream, call check_sandbox_quota (EE, is_ee()-guarded local import) and 429 if over quota. Permission (may-run) stays separate from entitlement (has-quota).

junaway · 2026-07-02T16:31:40Z

Post-review fixes applied (from a high-effort code review of this branch):

Gating was unwired — check_sandbox_quota had no production caller, so nothing enforced the credits quota. Now wired into set_session_stream (our runner's launch path) right after the existing RUN_SESSIONS permission check: is_ee()-guarded local import, returns 429 when over quota. Permission (may-run) stays separate from entitlement (has-quota).
Millicredit divergence — SANDBOX_CREDITS (the billed total) was int(sum_of_exact_credits * 1000) while each per-dimension *_CREDITS meter was gated on its own truncated value, so the total could go nonzero while every breakdown meter stayed 0. Now the total is the sum of the per-dimension millicredits actually written, so SANDBOX_CREDITS == Σ *_CREDITS always.

Still open (lower severity, left as notes): rate-override parse silently drops malformed values / unknown providers (add logging); Layer-2 mid-session kill remains a stub pending the runner /kill endpoint. 42 unit tests pass; ruff clean.

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.

+    try:
+        return await _check_sandbox_quota_ee(
+            organization_id=organization_id,
+            provider=provider,
+        )
+    except Exception:  # pylint: disable=broad-exception-caught
+        log.warning(
+            "[sandboxes] check_sandbox_quota failed for org=%s; failing open",
+            organization_id,
+            exc_info=True,
+        )
+        return True, None


+    try:
+        return await _check_sandbox_credits_true_up_ee(organization_id=organization_id)
+    except Exception:  # pylint: disable=broad-exception-caught
+        log.warning(
+            "[sandboxes] check_sandbox_credits_true_up failed for org=%s; failing open",
+            organization_id,
+            exc_info=True,
+        )
+        return True


+        # Billing layer: per-dimension + total sandbox_credits, derived from
+        # the raw seconds just recorded above (see sink.py).
+        await record_usage_credits(
+            provider=usage.provider,
+            organization_id=org_id,
+            cpu_seconds=Decimal(usage.vcpu_seconds),
+            ram_seconds=Decimal(usage.ram_gib_seconds),
+            ssd_seconds=Decimal(usage.disk_gib_seconds),
+            gpu_seconds=Decimal(usage.gpu_seconds) if usage.gpu_seconds else None,
+        )
+


+| Counter key              | value                     |
+|---------------------------|---------------------------|
+| `SANDBOX_CPU_CREDITS`     | `sandbox_cpu_credits`     |
+| `SANDBOX_RAM_CREDITS`     | `sandbox_ram_credits`     |
+| `SANDBOX_SSD_CREDITS`     | `sandbox_ssd_credits`     |
+| `SANDBOX_GPU_CREDITS`     | `sandbox_gpu_credits`     |
+| `SANDBOX_CREDITS`         | `sandbox_credits`         |


Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

mmabrouk · 2026-07-25T14:09:46Z

Linking back to #5505 (usage/cost telemetry, created from #5253), which already references this PR as related open work.

Copilot AI review requested due to automatic review settings July 2, 2026 14:57

Copilot started reviewing on behalf of junaway July 2, 2026 14:58 View session

junaway mentioned this pull request Jul 2, 2026

[integration] big-agents #4791

Merged

12 tasks

Copilot AI reviewed Jul 2, 2026

View reviewed changes

jp-agenta added 9 commits July 2, 2026 18:17

docs(metering): Track C findings — credits unit, REPORTS wiring, gating

69f5ded

refactor(billing): add CORE/GIBI unit token to sandbox credit + secon…

146e133

…ds keys Mirror Track B: SANDBOX_<RESOURCE>_<UNIT>_{SECONDS,CREDITS}. Total SANDBOX_CREDITS unchanged.

chore(metering): ruff-format sandbox credit files

676b0d0

fix(billing): keep sandbox_credits total reconcilable with per-dim br…

4a1dc33

…eakdown Sum the per-dimension millicredits actually written instead of re-truncating the exact credit sum, so SANDBOX_CREDITS == sum of the *_CREDITS meters.

feat(billing): gate session launch on sandbox_credits entitlement

6fbdd23

After the RUN_SESSIONS permission check in set_session_stream, call check_sandbox_quota (EE, is_ee()-guarded local import) and 429 if over quota. Permission (may-run) stays separate from entitlement (has-quota).

junaway force-pushed the feat/metering-track-c branch from a32cf36 to 6fbdd23 Compare July 2, 2026 16:30

vercel Bot deployed to Preview July 2, 2026 16:32 View deployment

STORAGE_BYTES > BYTES

d1fe31f

Copilot AI review requested due to automatic review settings July 2, 2026 20:59

Copilot started reviewing on behalf of junaway July 2, 2026 21:00 View session

vercel Bot deployed to Preview July 2, 2026 21:01 View deployment

Copilot AI reviewed Jul 2, 2026

View reviewed changes

mmabrouk mentioned this pull request Jul 5, 2026

[docs] Sandbox runtime metering — scoped-resource design #4783

Closed

jp-agenta and others added 2 commits July 6, 2026 21:42

docs(billing): track C specs + tasks — debit population, gating, wallet

a9fb0db

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

WIP

c3d6b12

vercel Bot deployed to Preview July 6, 2026 19:51 View deployment

This was referenced Jul 25, 2026

Usage and cost telemetry #5505

Open

docs(agent-workflows): Plan usage and cost telemetry #5253

Closed

Navigation Menu

Uh oh!

[feat] Bill sandbox usage as credits with quota gating - #5040

[feat] Bill sandbox usage as credits with quota gating#5040
junaway wants to merge 12 commits into
feat/metering-track-bfrom
feat/metering-track-c

junaway commented Jul 2, 2026

Uh oh!

vercel Bot commented Jul 2, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 2, 2026 •

edited

Loading

Review skipped

Uh oh!

Copilot AI left a comment

Uh oh!

junaway commented Jul 2, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

mmabrouk commented Jul 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

junaway commented Jul 2, 2026

Context

Changes

Tests / notes

Uh oh!

vercel Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

junaway commented Jul 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

mmabrouk commented Jul 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vercel Bot commented Jul 2, 2026 •

edited

Loading

coderabbitai Bot commented Jul 2, 2026 •

edited

Loading