Inconsistent error handling in S3: old boto3.exceptions.S3UploadFailedError replaces modern botocore.exceptions.ClientError

[glerb]

Describe the bug

The boto3 S3 client does not process S3 service errors like every other service client. It throws a different error than other service clients do when encountering AWS service errors.

This is a bump of #1986 from 2019(!).

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

A ClientError object should be returned to an except ClientError as error: clause when an S3 service error is returned to botocore, per current docs (which apparently haven't changed for years).

Current Behavior

The boto3 client raises a second error, S3UploadFailedError, from the boto3 exceptions when executing boto3.client('s3').upload_file():

$ python test_copy.py 
Traceback (most recent call last):
  File "<my_path>.venv/lib64/python3.9/site-packages/boto3/s3/transfer.py", line 372, in upload_file
    future.result()
  File "<my_path>.venv/lib64/python3.9/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "<my_path>.venv/lib64/python3.9/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "<my_path>.venv/lib64/python3.9/site-packages/s3transfer/tasks.py", line 139, in __call__
    return self._execute_main(kwargs)
  File "<my_path>.venv/lib64/python3.9/site-packages/s3transfer/tasks.py", line 162, in _execute_main
    return_value = self._main(**kwargs)
  File "<my_path>.venv/lib64/python3.9/site-packages/s3transfer/upload.py", line 764, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "<my_path>.venv/lib64/python3.9/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "<my_path>.venv/lib64/python3.9/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the PutObject operation: Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<my_path>test_copy.py", line 15, in <module>
    response = s3.upload_file(
  File "<my_path>.venv/lib64/python3.9/site-packages/boto3/s3/inject.py", line 145, in upload_file
    return transfer.upload_file(
  File "<my_path>.venv/lib64/python3.9/site-packages/boto3/s3/transfer.py", line 378, in upload_file
    raise S3UploadFailedError(
boto3.exceptions.S3UploadFailedError: Failed to upload ./test.txt to <my_bucket>/./test.txt: An error occurred (InvalidRequest) when calling the PutObject operation: Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters

Reproduction Steps

import boto3
import botocore.exceptions

BUCKET_NAME = '<bucket_name>'
PROFILE_NAME = '<profile_name>'
REGION_NAME = '<region>'
FILE_NAME = './test.txt'

boto3.setup_default_session(profile_name=PROFILE_NAME, region_name=REGION_NAME)
s3 = boto3.client('s3')

try:
    response = s3.upload_file(
        Filename=FILE_NAME,
        Bucket=BUCKET_NAME,
        Key=FILE_NAME
    )
    print(response)
except botocore.exceptions.ClientError as error:
    print(f"exiting: S3 error uploading file to s3://{BUCKET_NAME}: "
          f"{error.response['Error']['Code']}: "
          f"{error.response['Error']['Message']}"
          )

Possible Solution

The reason this is a "bug" is that there is some "backwards compatibility" error handler in boto3/s3/transfer.py that raises the (old?) S3UploadFailedError when the (modern?) botocore/client.py raises a ClientError. Thus, the S3 client does not behave as the current error handling instructions suggest, and there is no additional information in any of the documentation about this. Additionally, this old error handler does not return the full response dictionary in a way that is expected by developers catching a ClientError. Instead, the original error is returned as its __str__/__repr__ by the S3UploadFailedError. Developers have to go spelunking through the library to discover this.

Additional Information/Context

No response

SDK version used

1.35.25

Environment details (OS name and version, etc.)

Linux 5.15.0-202.135.2.el9uek.x86_64 #2 SMP x86_64 x86_64 x86_64 GNU/Linux

[adev-code]

Hi @glerb, thanks for reaching out. I have used the code that you have provided and I was not able to replicate the errors. For a further look, please provide the full debug logs by adding boto3.set_stream_logger('') to your code and redacting any sensitive information. Also, please let me know the type of file you were uploading, content type, and its size. Thanks.

[glerb]

@adev-code the error I used above was just an example intentionally designed to create an error (in this case, trying to upload a file to a bucket with Object Lock enabled, but without specifying a checksum algorithm). The specific error is almost certainly irrelevant. You can probably cause an error any way you want. The issue I'm flagging is the handling of errors generally.

[adev-code]

Hi @glerb, thanks for the update. Boto3 is throwing a ClientError as documented, and it's throwing the S3UploadFailedError to provide additional information. This behavior is expected; please let me know if you have any follow-up questions. Thanks!