Skip to content

feat: add AWS Bedrock service tier support #9875

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
roomote wants to merge 1 commit into from
Closed

feat: add AWS Bedrock service tier support #9875

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions packages/types/src/provider-settings.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,8 @@ const bedrockSchema = apiModelIdProviderModelSchema.extend({
awsBedrockEndpointEnabled: z.boolean().optional(),
awsBedrockEndpoint: z.string().optional(),
awsBedrock1MContext: z.boolean().optional(), // Enable 'context-1m-2025-08-07' beta for 1M context window.
// AWS Bedrock service tier for models that support it (STANDARD, FLEX, PRIORITY)
awsBedrockServiceTier: serviceTierSchema.optional(),
})

const vertexSchema = apiModelIdProviderModelSchema.extend({
Expand Down
53 changes: 53 additions & 0 deletions packages/types/src/providers/bedrock.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,25 @@ export const bedrockModels = {
minTokensPerCachePoint: 1024,
maxCachePoints: 4,
cachableFields: ["system", "messages", "tools"],
// Service tier pricing (FLEX ~20% discount, PRIORITY ~20% premium)
tiers: [
{
name: "flex" as const,
contextWindow: 200_000,
inputPrice: 2.4, // 20% discount
outputPrice: 12.0, // 20% discount
cacheWritesPrice: 3.0, // 20% discount
cacheReadsPrice: 0.24, // 20% discount
},
{
name: "priority" as const,
contextWindow: 200_000,
inputPrice: 3.6, // 20% premium
outputPrice: 18.0, // 20% premium
cacheWritesPrice: 4.5, // 20% premium
cacheReadsPrice: 0.36, // 20% premium
},
],
},
"amazon.nova-pro-v1:0": {
maxTokens: 5000,
Expand Down Expand Up @@ -184,6 +203,25 @@ export const bedrockModels = {
minTokensPerCachePoint: 2048,
maxCachePoints: 4,
cachableFields: ["system", "messages", "tools"],
// Service tier pricing (FLEX ~20% discount, PRIORITY ~20% premium)
tiers: [
{
name: "flex" as const,
contextWindow: 200_000,
inputPrice: 0.64, // 20% discount
outputPrice: 3.2, // 20% discount
cacheWritesPrice: 0.8, // 20% discount
cacheReadsPrice: 0.064, // 20% discount
},
{
name: "priority" as const,
contextWindow: 200_000,
inputPrice: 0.96, // 20% premium
outputPrice: 4.8, // 20% premium
cacheWritesPrice: 1.2, // 20% premium
cacheReadsPrice: 0.096, // 20% premium
},
],
},
"anthropic.claude-haiku-4-5-20251001-v1:0": {
maxTokens: 8192,
Expand Down Expand Up @@ -235,6 +273,21 @@ export const bedrockModels = {
supportsNativeTools: true,
inputPrice: 0.25,
outputPrice: 1.25,
// Service tier pricing (FLEX ~20% discount, PRIORITY ~20% premium)
tiers: [
{
name: "flex" as const,
contextWindow: 200_000,
inputPrice: 0.2, // 20% discount
outputPrice: 1.0, // 20% discount
},
{
name: "priority" as const,
contextWindow: 200_000,
inputPrice: 0.3, // 20% premium
outputPrice: 1.5, // 20% premium
},
],
},
"anthropic.claude-2-1-v1:0": {
maxTokens: 4096,
Expand Down
66 changes: 61 additions & 5 deletions src/api/providers/bedrock.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,13 +53,14 @@ interface BedrockInferenceConfig {
}

// Define interface for Bedrock additional model request fields
// This includes thinking configuration, 1M context beta, and other model-specific parameters
// This includes thinking configuration, 1M context beta, service tier, and other model-specific parameters
interface BedrockAdditionalModelFields {
thinking?: {
type: "enabled"
budget_tokens: number
}
anthropic_beta?: string[]
service_tier?: "STANDARD" | "FLEX" | "PRIORITY"
[key: string]: any // Add index signature to be compatible with DocumentType
}

Expand Down Expand Up @@ -433,6 +434,28 @@ export class AwsBedrockHandler extends BaseProvider implements SingleCompletionH
additionalModelRequestFields.anthropic_beta = anthropicBetas
}

// Add service tier if specified and model supports it
if (this.options.awsBedrockServiceTier && modelConfig.info.tiers && modelConfig.info.tiers.length > 0) {
if (!additionalModelRequestFields) {
additionalModelRequestFields = {} as BedrockAdditionalModelFields
}
// Convert from lowercase to uppercase for API
const tierMap: Record<string, "STANDARD" | "FLEX" | "PRIORITY"> = {
default: "STANDARD",
flex: "FLEX",
priority: "PRIORITY",
}
const mappedTier = tierMap[this.options.awsBedrockServiceTier as string]
if (mappedTier) {
additionalModelRequestFields.service_tier = mappedTier
logger.info("Service tier specified for Bedrock request", {
ctx: "bedrock",
modelId: modelConfig.id,
serviceTier: mappedTier,
})
}
}
Comment on lines +437 to +457
Copy link
Contributor Author

@roomote roomote bot Dec 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "default" tier should not send service_tier parameter to AWS. When the user selects "default", the service_tier field should be omitted from the API request entirely, allowing AWS Bedrock to use its implicit default behavior. Currently, the code maps "default" to "STANDARD" and includes it in the request, which is inconsistent with AWS best practices and the pricing logic (which correctly skips tier adjustments when tier is "default"). The condition at line 438 should check tier !== "default" before setting the parameter, or the tierMap should not include a "default" entry.

Fix it with Roo Code or mention @roomote and request a fix.


// Build tool configuration if native tools are enabled
let toolConfig: ToolConfiguration | undefined
if (useNativeTools && metadata?.tools) {
Expand Down Expand Up @@ -1027,15 +1050,18 @@ export class AwsBedrockHandler extends BaseProvider implements SingleCompletionH
reasoningBudget?: number
} {
if (this.costModelConfig?.id?.trim().length > 0) {
// Apply service tier pricing if specified
const tierAdjustedInfo = this.applyServiceTierPricing(this.costModelConfig.info)

// Get model params for cost model config
const params = getModelParams({
format: "anthropic",
modelId: this.costModelConfig.id,
model: this.costModelConfig.info,
model: tierAdjustedInfo,
settings: this.options,
defaultTemperature: BEDROCK_DEFAULT_TEMPERATURE,
})
return { ...this.costModelConfig, ...params }
return { ...this.costModelConfig, info: tierAdjustedInfo, ...params }
}

let modelConfig = undefined
Expand Down Expand Up @@ -1080,17 +1106,20 @@ export class AwsBedrockHandler extends BaseProvider implements SingleCompletionH
}
}

// Apply service tier pricing if specified
const tierAdjustedInfo = this.applyServiceTierPricing(modelConfig.info)

// Get model params including reasoning configuration
const params = getModelParams({
format: "anthropic",
modelId: modelConfig.id,
model: modelConfig.info,
model: tierAdjustedInfo,
settings: this.options,
defaultTemperature: BEDROCK_DEFAULT_TEMPERATURE,
})

// Don't override maxTokens/contextWindow here; handled in getModelById (and includes user overrides)
return { ...modelConfig, ...params } as {
return { ...modelConfig, info: tierAdjustedInfo, ...params } as {
id: BedrockModelId | string
info: ModelInfo
maxTokens?: number
Expand Down Expand Up @@ -1230,6 +1259,33 @@ export class AwsBedrockHandler extends BaseProvider implements SingleCompletionH
return false
}

/************************************************************************************
*
* SERVICE TIER PRICING
*
*************************************************************************************/

/**
* Returns a shallow-cloned ModelInfo with pricing overridden for the given tier, if available.
* If no tier or no overrides exist, the original ModelInfo is returned.
*/
private applyServiceTierPricing(info: ModelInfo): ModelInfo {
const tier = this.options.awsBedrockServiceTier
if (!tier || tier === "default") return info

// Find the tier with matching name in the tiers array
const tierInfo = info.tiers?.find((t) => t.name === tier)
if (!tierInfo) return info

return {
...info,
inputPrice: tierInfo.inputPrice ?? info.inputPrice,
outputPrice: tierInfo.outputPrice ?? info.outputPrice,
cacheReadsPrice: tierInfo.cacheReadsPrice ?? info.cacheReadsPrice,
cacheWritesPrice: tierInfo.cacheWritesPrice ?? info.cacheWritesPrice,
}
}

/************************************************************************************
*
* ERROR HANDLING
Expand Down
23 changes: 23 additions & 0 deletions webview-ui/src/components/settings/providers/Bedrock.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ export const Bedrock = ({ apiConfiguration, setApiConfigurationField, selectedMo
!!apiConfiguration?.apiModelId &&
BEDROCK_GLOBAL_INFERENCE_MODEL_IDS.includes(apiConfiguration.apiModelId as any)

// Check if the selected model supports service tiers
const supportsServiceTiers = !!(selectedModelInfo?.tiers && selectedModelInfo.tiers.length > 0)

// Update the endpoint enabled state when the configuration changes
useEffect(() => {
setAwsEndpointSelected(!!apiConfiguration?.awsBedrockEndpointEnabled)
Expand Down Expand Up @@ -150,6 +153,26 @@ export const Bedrock = ({ apiConfiguration, setApiConfigurationField, selectedMo
</SelectContent>
</Select>
</div>
{supportsServiceTiers && (
<div>
<label className="block font-medium mb-1">Service Tier</label>
<Select
value={apiConfiguration?.awsBedrockServiceTier || "default"}
onValueChange={(value) => setApiConfigurationField("awsBedrockServiceTier", value)}>
<SelectTrigger className="w-full">
<SelectValue placeholder="Select service tier" />
</SelectTrigger>
<SelectContent>
<SelectItem value="default">Standard (Default)</SelectItem>
<SelectItem value="flex">Flex (Lower cost, higher latency)</SelectItem>
<SelectItem value="priority">Priority (Lower latency, higher cost)</SelectItem>
</SelectContent>
</Select>
<div className="text-sm text-vscode-descriptionForeground mt-1">
Choose the service tier based on your cost and latency requirements
</div>
</div>
)}
{supportsGlobalInference && (
<Checkbox
checked={apiConfiguration?.awsUseGlobalInference || false}
Expand Down
Loading