-
-
Notifications
You must be signed in to change notification settings - Fork 4.7k
Add OpenAI video generation and content retrieval support #15541
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Sameerlite
wants to merge
46
commits into
litellm_sameer_oct_staging
from
litellm_videos_openai_retrieval
Closed
Changes from all commits
Commits
Show all changes
46 commits
Select commit
Hold shift + click to select a range
f6f6480
Add openai videos generation and retrieval support
Sameerlite 8322c62
add retrieval endpoint
Sameerlite 6e9b69e
Add docs
Sameerlite 16c888c
Add imports
Sameerlite 269e40a
remove orjson
Sameerlite 5c0e9c8
remove double import
Sameerlite 9018a4c
fix openai videos format
Sameerlite 4e3dded
remove mock code
Sameerlite 4ef28a4
remove not required comments
Sameerlite 06b8bb1
Add tests
Sameerlite 87666a1
Add tests
Sameerlite c44298b
Add other video endpoints
Sameerlite 8894bee
Fix cost calculation and transformation
Sameerlite e820162
Implement fix for thinking_blocks and converse API calls
lcfyi 01d672f
Add thinking literal
lcfyi f4557be
Fix mypy issues
lcfyi af6b425
Type fix for redacted thinking
lcfyi c4d2c0a
Add voyage model integration in sagemaker
Sameerlite aeceb19
Add config file logic
Sameerlite aed48af
Use already exiting voyage transformation
Sameerlite 6cff936
refactor code as per comments
Sameerlite 8d6877b
fix merge error
Sameerlite 04fdc48
refactor code as per comments
Sameerlite 349635f
refactor code as per comments
Sameerlite a5ae176
UI new build
ishaan-jaff 8517e87
[Fix] router - regression when adding/removing models (#15451)
AlexsanderHamir 02389ff
fix(prometheus): Fix Prometheus metric collection in a multi-workers …
LoadingZhang 3d4b956
Add tiered pricing and cost calculation for xai
Sameerlite 71fc43d
Use generic cost calculator
Sameerlite d526b41
Resolve conflicts in generated HTML files
Sameerlite 1b3d998
Remove penalty params as supported params for gemini preview model (#…
Sameerlite b64b540
fix conversion of thinking block
Sameerlite 8f36e80
add application level encryption in SQS (#15512)
deepanshululla a5ca5ff
[Feat] Bedrock Knowledgebase - return search_response when using /cha…
ishaan-jaff 825e63d
[Feat] Add dynamic rate limits on LiteLLM Gateway (#15518)
ishaan-jaff 6543cb2
Add google rerank endpoint
Sameerlite 5da242e
Add docs
Sameerlite acb2d2d
fix mypy error
Sameerlite 39e8e40
fix mypy and lint errors
Sameerlite edd05d9
Add haiku 4.5 integration
Sameerlite a7a4bee
Add haiku 4.5 integration for other regions as well
Sameerlite deed425
Handle citation field correctly
Sameerlite 1b173d0
Fix filtering headers for signature calcs
Sameerlite 5a637cc
Add haiku 4.5 integration (#15650)
Sameerlite 53c94f3
Merge branch 'litellm_staging_oct' into litellm_videos_openai_retrieval
Sameerlite cb6683b
Fixed mypy tests
Sameerlite File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
143 changes: 143 additions & 0 deletions
143
docs/my-website/docs/providers/openai/video_generation.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,143 @@ | ||
| import Tabs from '@theme/Tabs'; | ||
| import TabItem from '@theme/TabItem'; | ||
|
|
||
| # OpenAI Video Generation | ||
|
|
||
| LiteLLM supports OpenAI's video generation models including Sora. | ||
|
|
||
| ## Quick Start | ||
|
|
||
| ### Required API Keys | ||
|
|
||
| ```python | ||
| import os | ||
| os.environ["OPENAI_API_KEY"] = "your-api-key" | ||
| ``` | ||
|
|
||
| ### Basic Usage | ||
|
|
||
| ```python | ||
| from litellm import video_generation, video_retrieval | ||
| import os | ||
|
|
||
| os.environ["OPENAI_API_KEY"] = "your-api-key" | ||
|
|
||
| # Generate a video | ||
| response = video_generation( | ||
| prompt="A cat playing with a ball of yarn in a sunny garden", | ||
| model="sora-2", | ||
| seconds="8", | ||
| size="720x1280" | ||
| ) | ||
|
|
||
| print(f"Video ID: {response.id}") | ||
| print(f"Status: {response.status}") | ||
|
|
||
| # Download video content when ready | ||
| video_bytes = video_retrieval( | ||
| video_id=response.id, | ||
| model="sora-2" | ||
| ) | ||
|
|
||
| # Save to file | ||
| with open("generated_video.mp4", "wb") as f: | ||
| f.write(video_bytes) | ||
| ``` | ||
|
|
||
| ## Supported Models | ||
|
|
||
| | Model Name | Description | Max Duration | Supported Sizes | | ||
| |------------|-------------|--------------|-----------------| | ||
| | sora-2 | OpenAI's latest video generation model | 8 seconds | 720x1280, 1280x720 | | ||
|
|
||
| ## Video Generation Parameters | ||
|
|
||
| - `prompt` (required): Text description of the desired video | ||
| - `model` (optional): Model to use, defaults to "sora-2" | ||
| - `seconds` (optional): Video duration in seconds (e.g., "8", "16") | ||
| - `size` (optional): Video dimensions (e.g., "720x1280", "1280x720") | ||
| - `input_reference` (optional): Reference image for video editing | ||
| - `user` (optional): User identifier for tracking | ||
|
|
||
| ## Video Content Retrieval | ||
|
|
||
| ```python | ||
| # Download video content | ||
| video_bytes = video_retrieval( | ||
| video_id="video_1234567890", | ||
| model="sora-2" | ||
| ) | ||
|
|
||
| # Save to file | ||
| with open("video.mp4", "wb") as f: | ||
| f.write(video_bytes) | ||
| ``` | ||
|
|
||
| ## Complete Workflow | ||
|
|
||
| ```python | ||
| import litellm | ||
| import time | ||
|
|
||
| def generate_and_download_video(prompt): | ||
| # Step 1: Generate video | ||
| response = litellm.video_generation( | ||
| prompt=prompt, | ||
| model="sora-2", | ||
| seconds="8", | ||
| size="720x1280" | ||
| ) | ||
|
|
||
| video_id = response.id | ||
| print(f"Video ID: {video_id}") | ||
|
|
||
| # Step 2: Wait for processing (in practice, poll status) | ||
| time.sleep(30) | ||
|
|
||
| # Step 3: Download video | ||
| video_bytes = litellm.video_retrieval( | ||
| video_id=video_id, | ||
| model="sora-2" | ||
| ) | ||
|
|
||
| # Step 4: Save to file | ||
| with open(f"video_{video_id}.mp4", "wb") as f: | ||
| f.write(video_bytes) | ||
|
|
||
| return f"video_{video_id}.mp4" | ||
|
|
||
| # Usage | ||
| video_file = generate_and_download_video( | ||
| "A cat playing with a ball of yarn in a sunny garden" | ||
| ) | ||
| ``` | ||
|
|
||
| ## Video Editing with Reference Images | ||
|
|
||
| ```python | ||
| # Video editing with reference image | ||
| response = litellm.video_generation( | ||
| prompt="Make the cat jump higher", | ||
| input_reference="path/to/image.jpg", # Reference image | ||
| model="sora-2", | ||
| seconds="8" | ||
| ) | ||
|
|
||
| print(f"Video ID: {response.id}") | ||
| ``` | ||
|
|
||
| ## Error Handling | ||
|
|
||
| ```python | ||
| from litellm.exceptions import BadRequestError, AuthenticationError | ||
|
|
||
| try: | ||
| response = video_generation( | ||
| prompt="A cat playing with a ball of yarn", | ||
| model="sora-2" | ||
| ) | ||
| except AuthenticationError as e: | ||
| print(f"Authentication failed: {e}") | ||
| except BadRequestError as e: | ||
| print(f"Bad request: {e}") | ||
| ``` |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you have this be a separate doc - it'll be easier to find
have it be in the same openai folder though
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added