feat(llmobs): prompt management SDK methods#18186
Draft
PROFeNoM wants to merge 9 commits into
Draft
Conversation
Add CRUD operations for LLM Observability prompt registry to the Python SDK.
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 4bbd190 | Docs | Datadog PR Page | Give us feedback! |
Codeowners resolved as |
list_prompts and list_prompt_versions only need DD_API_KEY since the backend uses ValidReportingAPIUser for read endpoints.
_request() passes body= to conn.request(), mock signature needed to match.
…[MLOB-7524] Handler expects filter[ml_app], SDK was sending ml_app.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds prompt management methods to the LLMObs Python SDK, calling the new public API endpoints from MLOB-7523.
New public API
Not covered:
GET /prompts/{prompt_id}/versions/{version}is intentionally left out. Prefer to add it once #18127 (hybrid prompt delivery) lands, since it changes theget_promptsignature - at that point we can decide whether to addversion=as an optional parameter toget_promptrather than a separate method.New types (
ddtrace/llmobs/types.py)PromptLabel = Literal["development", "production"]- allowed label valuesChatMessage(TypedDict)- template message format (role+contentonly)PromptResponse(TypedDict)- returned by create/update/list prompt operationsPromptVersionResponse(TypedDict)- returned by create/update/list version operationsDeletedPromptResponse(TypedDict)- returned by deleteException hierarchy
Each method documents which exceptions it can raise.
Cache changes (
cache.py)~/.cache/datadog/llmobs/prompts/{prompt_id}/{label}.jsonevict_prompt(prompt_id)- usesshutil.rmtreeon the prompt directoryFiles changed
types.pyPromptLabel,ChatMessage, 3 response TypedDicts, 5 exception classescache.pyevict_promptmethodmanager.py_requesthelper, 7 CRUD methods, API key validation onget_prompt_llmobs.pytest_prompts.py