Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade from DALL·E 2 to DALL·E 3 #717

Merged
merged 7 commits into from
Feb 21, 2024
Merged

Upgrade from DALL·E 2 to DALL·E 3 #717

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
aae26dc
Switch to using DALLE 3 as our image generation model. Add in all new…
dkotter Feb 14, 2024
ff85b88
Fix an error on the debug screen
dkotter Feb 14, 2024
debd31e
Fix tests; update readmes
dkotter Feb 14, 2024
6d391cc
One more readme fix
dkotter Feb 14, 2024
6abab98
Merge branch 'develop' into feature/dalle3
dkotter Feb 15, 2024
af0e87c
Add label to each image size that describes the type of image (square…
dkotter Feb 15, 2024
af9c669
Merge branch 'develop' of github.com:10up/classifai into feature/dalle3
iamdharmesh Feb 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Tap into leading cloud-based services like [OpenAI](https://openai.com/), [Micro
* Generate a summary of post content and store it as an excerpt using [OpenAI's ChatGPT API](https://platform.openai.com/docs/guides/chat) and [Google's Gemini API](https://ai.google.dev/docs/gemini_api_overview)
* Generate titles from post content using [OpenAI's ChatGPT API](https://platform.openai.com/docs/guides/chat) and [Google's Gemini API](https://ai.google.dev/docs/gemini_api_overview)
* Expand or condense text content using [OpenAI's ChatGPT API](https://platform.openai.com/docs/guides/chat) and [Google's Gemini API](https://ai.google.dev/docs/gemini_api_overview)
* Generate new images on demand to use in-content or as a featured image using [OpenAI's DALL·E API](https://platform.openai.com/docs/guides/images)
* Generate new images on demand to use in-content or as a featured image using [OpenAI's DALL·E 3 API](https://platform.openai.com/docs/guides/images)
* Generate transcripts of audio files using [OpenAI's Whisper API](https://platform.openai.com/docs/guides/speech-to-text)
* Moderate incoming comments for sensitive content using [OpenAI's Moderation API](https://platform.openai.com/docs/guides/moderation)
* Convert text content into audio and output a "read-to-me" feature on the front-end to play this audio using [Microsoft Azure's Text to Speech API](https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/text-to-speech)
Expand Down Expand Up @@ -397,7 +397,7 @@ Note that [Azure AI Vision](https://docs.microsoft.com/en-us/azure/cognitive-ser

### 2. Configure OpenAI API Keys under Tools > ClassifAI > Image Processing > Image Generation

* Select **OpenAI DALL-E** in the provider dropdown.
* Select **OpenAI DALL·E 3** in the provider dropdown.
* Enter your API Key copied from the above step into the `API Key` field.

### 3. Enable specific Image Processing features
Expand Down
4 changes: 3 additions & 1 deletion includes/Classifai/Features/Feature.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1113,11 +1113,13 @@ function ( $role ) {
__( 'Provider', 'classifai' ) => $feature_settings['provider'],
];

if ( method_exists( $provider, 'get_debug_information' ) ) {
if ( $provider && method_exists( $provider, 'get_debug_information' ) ) {
$all_debug_info = array_merge(
$common_debug_info,
$provider->get_debug_information()
);
} else {
$all_debug_info = $common_debug_info;
}

/**
Expand Down
2 changes: 1 addition & 1 deletion includes/Classifai/Features/ImageGeneration.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ public function __construct() {

// Contains just the providers this feature supports.
$this->supported_providers = [
DallE::ID => __( 'OpenAI Dall-E', 'classifai' ),
DallE::ID => __( 'OpenAI DALL·E 3', 'classifai' ),
];
}

Expand Down
183 changes: 137 additions & 46 deletions includes/Classifai/Providers/OpenAI/DallE.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,18 @@ class DallE extends Provider {
const ID = 'openai_dalle';

/**
* OpenAI DALL·E URL
* OpenAI DALL·E URL.
*
* @var string
*/
protected $dalle_url = 'https://api.openai.com/v1/images/generations';

/**
* Maximum number of characters a prompt can have
* Maximum number of characters a prompt can have.
*
* @var int
*/
public $max_prompt_chars = 1000;
public $max_prompt_chars = 4000;

/**
* OpenAI DALL·E constructor.
Expand Down Expand Up @@ -64,33 +64,53 @@ public function register_endpoints() {
'methods' => WP_REST_Server::READABLE,
'callback' => [ $this->feature_instance, 'rest_endpoint_callback' ],
'args' => [
'prompt' => [
'prompt' => [
'required' => true,
'type' => 'string',
'sanitize_callback' => 'sanitize_text_field',
'validate_callback' => 'rest_validate_request_arg',
'description' => esc_html__( 'Prompt used to generate an image', 'classifai' ),
],
'n' => [
'n' => [
'type' => 'integer',
'minimum' => 1,
'maximum' => 10,
'sanitize_callback' => 'absint',
'validate_callback' => 'rest_validate_request_arg',
'description' => esc_html__( 'Number of images to generate', 'classifai' ),
],
'size' => [
'quality' => [
'type' => 'string',
'enum' => [
'standard',
'hd',
],
'sanitize_callback' => 'sanitize_text_field',
'validate_callback' => 'rest_validate_request_arg',
'description' => esc_html__( 'Quality of generated image', 'classifai' ),
],
'size' => [
'type' => 'string',
'enum' => [
'256x256',
'512x512',
'1024x1024',
'1792x1024',
'1024x1792',
],
'sanitize_callback' => 'sanitize_text_field',
'validate_callback' => 'rest_validate_request_arg',
'description' => esc_html__( 'Size of generated image', 'classifai' ),
],
'format' => [
'style' => [
'type' => 'string',
'enum' => [
'vivid',
'natural',
],
'sanitize_callback' => 'sanitize_text_field',
'validate_callback' => 'rest_validate_request_arg',
'description' => esc_html__( 'Style of generated image', 'classifai' ),
],
'format' => [
'type' => 'string',
'enum' => [
'url',
Expand Down Expand Up @@ -123,7 +143,7 @@ public function render_provider_fields() {
'label_for' => 'api_key',
'input_type' => 'password',
'default_value' => $settings['api_key'],
'class' => 'classifai-provider-field hidden provider-scope-' . static::ID, // Important to add this.
'class' => 'classifai-provider-field hidden provider-scope-' . static::ID,
'description' => sprintf(
wp_kses(
/* translators: %1$s is replaced with the OpenAI sign up URL */
Expand All @@ -141,7 +161,7 @@ public function render_provider_fields() {
);

add_settings_field(
static::ID . 'number_of_images',
static::ID . '_number_of_images',
esc_html__( 'Number of images', 'classifai' ),
[ $this->feature_instance, 'render_select' ],
$this->feature_instance->get_option_name(),
Expand All @@ -152,12 +172,31 @@ public function render_provider_fields() {
'options' => array_combine( range( 1, 10 ), range( 1, 10 ) ),
'default_value' => $settings['number_of_images'],
'description' => __( 'Number of images that will be generated in one request. Note that each image will incur separate costs.', 'classifai' ),
'class' => 'classifai-provider-field hidden provider-scope-' . static::ID, // Important to add this.
'class' => 'classifai-provider-field hidden provider-scope-' . static::ID,
]
);

add_settings_field(
static::ID . 'image_size',
static::ID . '_quality',
esc_html__( 'Image quality', 'classifai' ),
[ $this->feature_instance, 'render_select' ],
$this->feature_instance->get_option_name(),
$this->feature_instance->get_option_name() . '_section',
[
'option_index' => static::ID,
'label_for' => 'quality',
'options' => [
'standard' => __( 'Standard', 'classifai' ),
'hd' => __( 'High Definition', 'classifai' ),
],
'default_value' => $settings['quality'],
'description' => __( 'The quality of the image that will be generated. High Definition creates images with finer details and greater consistency across the image but costs more.', 'classifai' ),
'class' => 'classifai-provider-field hidden provider-scope-' . static::ID,
]
);

add_settings_field(
static::ID . '_image_size',
esc_html__( 'Image size', 'classifai' ),
[ $this->feature_instance, 'render_select' ],
$this->feature_instance->get_option_name(),
Expand All @@ -166,13 +205,32 @@ public function render_provider_fields() {
'option_index' => static::ID,
'label_for' => 'image_size',
'options' => [
'256x256' => '256x256',
'512x512' => '512x512',
'1024x1024' => '1024x1024',
'1792x1024' => '1792x1024',
'1024x1792' => '1024x1792',
],
'default_value' => $settings['image_size'],
'description' => __( 'Size of generated images.', 'classifai' ),
'class' => 'classifai-provider-field hidden provider-scope-' . static::ID, // Important to add this.
'description' => __( 'Size of generated images. Larger sizes cost more.', 'classifai' ),
'class' => 'classifai-provider-field hidden provider-scope-' . static::ID,
]
);

add_settings_field(
static::ID . '_style',
esc_html__( 'Image style', 'classifai' ),
[ $this->feature_instance, 'render_select' ],
$this->feature_instance->get_option_name(),
$this->feature_instance->get_option_name() . '_section',
[
'option_index' => static::ID,
'label_for' => 'style',
'options' => [
'vivid' => __( 'Vivid', 'classifai' ),
'natural' => __( 'Natural', 'classifai' ),
],
'default_value' => $settings['style'],
'description' => __( 'The style of the generated images. Vivid causes more hyper-real and dramatic images. Natural causes more natural, less hyper-real looking images.', 'classifai' ),
'class' => 'classifai-provider-field hidden provider-scope-' . static::ID,
]
);
}
Expand All @@ -194,7 +252,9 @@ public function get_default_provider_settings(): array {
$common_settings,
[
'number_of_images' => 1,
'image_size' => '256x256',
'quality' => 'standard',
'image_size' => '1024x1024',
'style' => 'vivid',
]
);
}
Expand All @@ -217,8 +277,22 @@ public function sanitize_settings( array $new_settings ): array {
if ( $this->feature_instance instanceof ImageGeneration ) {
$new_settings[ static::ID ]['number_of_images'] = absint( $new_settings[ static::ID ]['number_of_images'] ?? $settings[ static::ID ]['number_of_images'] );

if ( in_array( $new_settings[ static::ID ]['image_size'], [ '256x256', '512x512', '1024x1024' ], true ) ) {
$new_settings[ static::ID ]['image_size'] = sanitize_text_field( $new_settings[ static::ID ]['image_size'] ?? $settings[ static::ID ]['image_size'] );
if ( in_array( $new_settings[ static::ID ]['quality'], [ 'standard', 'hd' ], true ) ) {
$new_settings[ static::ID ]['quality'] = sanitize_text_field( $new_settings[ static::ID ]['quality'] );
} else {
$new_settings[ static::ID ]['quality'] = $settings[ static::ID ]['quality'];
}

if ( in_array( $new_settings[ static::ID ]['image_size'], [ '1024x1024', '1792x1024', '1024x1792' ], true ) ) {
$new_settings[ static::ID ]['image_size'] = sanitize_text_field( $new_settings[ static::ID ]['image_size'] );
} else {
$new_settings[ static::ID ]['image_size'] = $settings[ static::ID ]['image_size'];
}

if ( in_array( $new_settings[ static::ID ]['style'], [ 'vivid', 'natural' ], true ) ) {
$new_settings[ static::ID ]['style'] = sanitize_text_field( $new_settings[ static::ID ]['style'] );
} else {
$new_settings[ static::ID ]['style'] = $settings[ static::ID ]['style'];
}
}

Expand Down Expand Up @@ -264,12 +338,19 @@ public function generate_image( string $prompt = '', array $args = [] ) {
$args = wp_parse_args(
array_filter( $args ),
[
'num' => $settings['number_of_images'] ?? 1,
'size' => $settings['image_size'] ?? '1024x1024',
'format' => 'url',
'num' => $settings['number_of_images'] ?? 1,
'quality' => $settings['quality'] ?? 'standard',
'size' => $settings['image_size'] ?? '1024x1024',
'style' => $settings['style'] ?? 'vivid',
'format' => 'url',
]
);

// Force proper image size for those that had been using DALL·E 2 and haven't updated settings.
if ( ! in_array( $args['size'], [ '1024x1024', '1792x1024', '1024x1792' ], true ) ) {
$args['size'] = '1024x1024';
}

if ( ! $image_generation->is_feature_enabled() ) {
return new WP_Error( 'not_enabled', esc_html__( 'Image generation is disabled or OpenAI authentication failed. Please check your settings.', 'classifai' ) );
}
Expand Down Expand Up @@ -307,40 +388,47 @@ public function generate_image( string $prompt = '', array $args = [] ) {
'classifai_dalle_request_body',
[
'prompt' => sanitize_text_field( $prompt ),
'n' => absint( $args['num'] ),
'size' => sanitize_text_field( $args['size'] ),
'model' => 'dall-e-3',
'n' => 1,
'quality' => sanitize_text_field( $args['quality'] ),
'response_format' => sanitize_text_field( $args['format'] ),
'size' => sanitize_text_field( $args['size'] ),
'style' => sanitize_text_field( $args['style'] ),
]
);

// Make our API request.
$response = $request->post(
$this->dalle_url,
[
'body' => wp_json_encode( $body ),
]
);
$responses = [];

set_transient( 'classifai_openai_dalle_latest_response', $response, DAY_IN_SECONDS * 30 );
// DALL·E 3 doesn't support multiple images in a single request so make one request per image.
for ( $i = 0; $i < $args['num']; $i++ ) {
$responses[] = $request->post(
$this->dalle_url,
[
'body' => wp_json_encode( $body ),
]
);
}

set_transient( 'classifai_openai_dalle_latest_response', $responses[ array_key_last( $responses ) ], DAY_IN_SECONDS * 30 );

// Extract out the image response, if it exists.
if ( ! is_wp_error( $response ) && ! empty( $response['data'] ) ) {
$cleaned_response = [];
$cleaned_responses = [];

foreach ( $response['data'] as $data ) {
if ( ! empty( $data[ $args['format'] ] ) ) {
if ( 'url' === $args['format'] ) {
$cleaned_response[] = [ 'url' => esc_url_raw( $data[ $args['format'] ] ) ];
} else {
$cleaned_response[] = [ 'url' => $data[ $args['format'] ] ];
foreach ( $responses as $response ) {
// Extract out the image response, if it exists.
if ( ! is_wp_error( $response ) && ! empty( $response['data'] ) ) {
foreach ( $response['data'] as $data ) {
if ( ! empty( $data[ $args['format'] ] ) ) {
if ( 'url' === $args['format'] ) {
$cleaned_responses[] = [ 'url' => esc_url_raw( $data[ $args['format'] ] ) ];
} else {
$cleaned_responses[] = [ 'url' => $data[ $args['format'] ] ];
}
}
}
}

$response = $cleaned_response;
}

return $response;
return $cleaned_responses;
}

/**
Expand All @@ -354,7 +442,10 @@ public function get_debug_information(): array {
$debug_info = [];

if ( $this->feature_instance instanceof ImageGeneration ) {
$debug_info[ __( 'Number of images', 'classifai' ) ] = $provider_settings['number_of_images'];
$debug_info[ __( 'Number of images', 'classifai' ) ] = $provider_settings['number_of_images'] ?? 1;
$debug_info[ __( 'Quality', 'classifai' ) ] = $provider_settings['quality'] ?? 'standard';
$debug_info[ __( 'Size', 'classifai' ) ] = $provider_settings['image_size'] ?? '1024x1024';
$debug_info[ __( 'Style', 'classifai' ) ] = $provider_settings['style'] ?? 'vivid';
$debug_info[ __( 'Latest response:', 'classifai' ) ] = $this->get_formatted_latest_response( get_transient( 'classifai_openai_dalle_latest_response' ) );
}

Expand Down
2 changes: 1 addition & 1 deletion readme.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Tap into leading cloud-based services like [OpenAI](https://openai.com/), [Micro
* Generate a summary of post content and store it as an excerpt using [OpenAI's ChatGPT API](https://platform.openai.com/docs/guides/chat) and [Google's Gemini API](https://ai.google.dev/docs/gemini_api_overview)
* Generate titles from post content using [OpenAI's ChatGPT API](https://platform.openai.com/docs/guides/chat) and [Google's Gemini API](https://ai.google.dev/docs/gemini_api_overview)
* Expand or condense text content using [OpenAI's ChatGPT API](https://platform.openai.com/docs/guides/chat) and [Google's Gemini API](https://ai.google.dev/docs/gemini_api_overview)
* Generate new images on demand to use in-content or as a featured image using [OpenAI's DALL·E API](https://platform.openai.com/docs/guides/images)
* Generate new images on demand to use in-content or as a featured image using [OpenAI's DALL·E 3 API](https://platform.openai.com/docs/guides/images)
* Generate transcripts of audio files using [OpenAI's Whisper API](https://platform.openai.com/docs/guides/speech-to-text)
* Convert text content into audio and output a "read-to-me" feature on the front-end to play this audio using [Microsoft Azure's Text to Speech API](https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/text-to-speech)
* Classify post content using [IBM Watson's Natural Language Understanding API](https://www.ibm.com/watson/services/natural-language-understanding/) and [OpenAI's Embedding API](https://platform.openai.com/docs/guides/embeddings)
Expand Down
17 changes: 14 additions & 3 deletions tests/cypress/integration/image-processing/image-generation-openai-dalle.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ describe( 'Image Generation (OpenAI DALL·E) Tests', () => {
'/wp-admin/tools.php?page=classifai&tab=image_processing&feature=feature_image_generation'
);
cy.get( '#status' ).check();
cy.get( '#provider' ).select( 'openai_dalle' );
cy.get( '#submit' ).click();
cy.optInAllFeatures();
} );
Expand All @@ -19,13 +20,23 @@ describe( 'Image Generation (OpenAI DALL·E) Tests', () => {
);

cy.get( '#api_key' ).clear().type( 'password' );
cy.get(
'select[name="classifai_feature_image_generation[openai_dalle][number_of_images]"]'
).select( '2' );
cy.get(
'select[name="classifai_feature_image_generation[openai_dalle][quality]"]'
).select( 'hd' );
cy.get(
'select[name="classifai_feature_image_generation[openai_dalle][image_size]"]'
).select( '1024x1792' );
cy.get(
'select[name="classifai_feature_image_generation[openai_dalle][style]"]'
).select( 'natural' );

cy.get( '#status' ).check();
cy.get(
'#classifai_feature_image_generation_roles_administrator'
).check();
cy.get( '#number_of_images' ).select( '2' );
cy.get( '#image_size' ).select( '512x512' );

cy.get( '#submit' ).click();
} );

Expand Down
Loading
Loading