Skip to content

feat(llmobs): prompt management SDK methods#18186

Draft
PROFeNoM wants to merge 9 commits into
mainfrom
alex/MLOB-7524_prompt-crud-api
Draft

feat(llmobs): prompt management SDK methods#18186
PROFeNoM wants to merge 9 commits into
mainfrom
alex/MLOB-7524_prompt-crud-api

Conversation

@PROFeNoM
Copy link
Copy Markdown
Contributor

@PROFeNoM PROFeNoM commented May 20, 2026

Description

Adds prompt management methods to the LLMObs Python SDK, calling the new public API endpoints from MLOB-7523.

Blocked by: https://github.com/DataDog/dd-source/pull/443753 (public CRUD API routes)

New public API

# Write methods (require DD_API_KEY + DD_APP_KEY)
LLMObs.create_prompt(prompt_id, template, *, title, description, user_version, labels)
LLMObs.create_prompt_version(prompt_id, template, *, description, user_version, labels)
LLMObs.update_prompt(prompt_id, *, title, description)
LLMObs.update_prompt_version(prompt_id, version, *, labels, description)
LLMObs.delete_prompt(prompt_id)

# Read methods (require DD_API_KEY only)
LLMObs.list_prompts(*, ml_app)
LLMObs.list_prompt_versions(prompt_id)

Not covered: GET /prompts/{prompt_id}/versions/{version} is intentionally left out. Prefer to add it once #18127 (hybrid prompt delivery) lands, since it changes the get_prompt signature - at that point we can decide whether to add version= as an optional parameter to get_prompt rather than a separate method.

New types (ddtrace/llmobs/types.py)

  • PromptLabel = Literal["development", "production"] - allowed label values
  • ChatMessage(TypedDict) - template message format (role + content only)
  • PromptResponse(TypedDict) - returned by create/update/list prompt operations
  • PromptVersionResponse(TypedDict) - returned by create/update/list version operations
  • DeletedPromptResponse(TypedDict) - returned by delete

Exception hierarchy

PromptAPIError (base)
  PromptAuthError        - 401/403 (bad API/app key)
  PromptValidationError  - 400 (bad input)
  PromptNotFoundError    - 404
  PromptConflictError    - 409 (duplicate prompt_id)
  PromptServerError      - 5xx

Each method documents which exceptions it can raise.

Cache changes (cache.py)

  • WarmCache now uses per-prompt subdirectories: ~/.cache/datadog/llmobs/prompts/{prompt_id}/{label}.json
  • Added evict_prompt(prompt_id) - uses shutil.rmtree on the prompt directory
  • Previous flat-file layout had prefix collision bugs (e.g., deleting "greeting" could evict "greeting-v2" cache entries)

Files changed

File Change
types.py PromptLabel, ChatMessage, 3 response TypedDicts, 5 exception classes
cache.py Per-prompt subdirectory layout, evict_prompt method
manager.py _request helper, 7 CRUD methods, API key validation on get_prompt
_llmobs.py 7 public classmethods delegating to manager
test_prompts.py Exception mapping test, cache eviction test

Add CRUD operations for LLM Observability prompt registry to the Python SDK.
@datadog-prod-us1-6
Copy link
Copy Markdown

datadog-prod-us1-6 Bot commented May 20, 2026

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 4bbd190 | Docs | Datadog PR Page | Give us feedback!

@PROFeNoM PROFeNoM changed the title feat(llmobs): prompt management write methods [MLOB-7524] feat(llmobs): prompt management write methods May 20, 2026
@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da Bot commented May 20, 2026

Codeowners resolved as

ddtrace/llmobs/_prompts/manager.py                                      @DataDog/ml-observability

PROFeNoM added 3 commits May 20, 2026 15:15
list_prompts and list_prompt_versions only need DD_API_KEY since the
backend uses ValidReportingAPIUser for read endpoints.
@PROFeNoM PROFeNoM changed the title feat(llmobs): prompt management write methods feat(llmobs): prompt management SDK methods [MLOB-7524] May 20, 2026
@PROFeNoM PROFeNoM changed the title feat(llmobs): prompt management SDK methods [MLOB-7524] feat(llmobs): prompt management SDK methods May 21, 2026
PROFeNoM added 2 commits May 21, 2026 09:22
_request() passes body= to conn.request(), mock signature needed to match.
…[MLOB-7524]

Handler expects filter[ml_app], SDK was sending ml_app.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant