InstructionFile vs MetaData

[janSchumacherPayments]

Problem:

In the context of client-side encryption using a customer-provided encryption key (Java, AES, SSE-C), the AWS SDK v1 offers the option to use instruction files to store the encryption metadata. This supports a cost-efficient way of rotation the client-side key (See AWS blog).

Now with the AWS SDK v1 being in maintenance mode, it is suggested to migrate to this library. According to the documentation, the V3 library supports decryption of encrypted files that use instruction files. However, it is mentioned in the 'legacy' section and it seems as if it's not possible to encrypt new files using the instruction files.

Moving forward, what is the recommended way to implement client-side encryption with this library? Are instruction files not recommended anymore? Relying on object metadata instead of instruction files makes it more expensive to rotate the client-side key because the metadata is immutable and a copy of the file is required.

Solution:

Clarification on the usage of instruction files. Are they future-proof? If not, is there a different (cheap) way to rotate client-side keys?

[kessplas]

Hello Jan,

Thanks for reaching out.

It is indeed possible to "re-envelope" an S3EC encrypted object that uses object metadata by calling HeadObject and CopyObject. The first operation (Head) retrieves just the object metadata. The second (Copy) allows the customer to replace the object metadata "in-place" without downloading the entire object using MetadataDirective.REPLACE. I'll follow up with an example.

This should suffice to provide the same functionality as re-enveloping through instruction files, with roughly the same cost profile as Get/Put on the instruction file. HEAD/COPY requests are cheaper than GET/PUT in some cases, depending on storage tier and network topology. However, this approach does have slightly different cost properties when the bucket containing the object(s) has versioning enabled; updating object metadata will create a new version, which is billed as another object. If the object is larger than the instruction file, this would cause your storage costs to increase unless you delete the new versions. Though, if the objects are very small, i.e. smaller than the instruction file, then you would save on costs using this scheme. That said, if this nuance in billing is a blocker for the customer adopting the HEAD/COPY approach, we can add the ability to store crypto parameters in an instruction file on encrypt (PutObject), but this would need to be prioritized amongst other issues/asks/features.

[kessplas]

Example:

public void s3EncryptionClientReEnvelopeInPlace() {
    final String objectKey = appendTestSuffix("re-envelope-in-place");

    KmsClient kmsClient = KmsClient.create();

    S3Client s3Client = S3EncryptionClient.builder()
            .kmsKeyId(KMS_KEY_ID)
            .build();

    // Put a valid encrypted object
    final String input = "SimpleTestOfV3EncryptionClient-ReEnvelope";

    s3Client.putObject(builder1 -> builder1
                    .bucket(BUCKET)
                    .key(objectKey)
                    .build(),
            RequestBody.fromString(input));

    // optional - sanity check decryption
    ResponseBytes<GetObjectResponse> objectResponse = s3Client.getObjectAsBytes(builder1 -> builder1
            .bucket(BUCKET)
            .key(objectKey)
            .build());
    String output1 = objectResponse.asUtf8String();
    assertEquals(input, output1);

    // Call HEAD to get the metadata
    HeadObjectResponse headObjectResponse = s3Client.headObject(builder -> builder
            .key(objectKey)
            .bucket(BUCKET));
    Map<String, String> objectMetadata = headObjectResponse.metadata();

    System.out.println("previous metadata:");
    for (String key : objectMetadata.keySet()) {
        System.out.println("key: " + key + "; value: " + objectMetadata.get(key));
    }

    // just use the default minimal EC for this example
    Map<String, String> encryptionContext = new HashMap<>(1);
    encryptionContext.put("aws:x-amz-cek-alg", "AES/GCM/NoPadding");

    final String EDK_KEY = "x-amz-key-v2";
    // decode b64 encoded EDK
    byte[] edkCiphertext = Base64.getDecoder().decode(objectMetadata.get(EDK_KEY));

    final String ecForMd;
    Map<String, String> encryptionContextNew = new HashMap<>(2);
    encryptionContextNew.put("aws:x-amz-cek-alg", "AES/GCM/NoPadding");
    encryptionContextNew.put("rotated?", "yes");
    try (JsonWriter jsonWriter = JsonWriter.create()) {
        jsonWriter.writeStartObject();
        for (Map.Entry<String, String> entry : encryptionContextNew.entrySet()) {
            jsonWriter.writeFieldName(entry.getKey()).writeValue(entry.getValue());
        }
        jsonWriter.writeEndObject();

        ecForMd = new String(jsonWriter.getBytes(), StandardCharsets.UTF_8);
    }

    ReEncryptRequest reEncryptRequest = ReEncryptRequest.builder()
            .sourceKeyId(KMS_KEY_ID)
            .destinationKeyId(KMS_KEY_ID)
            .sourceEncryptionContext(encryptionContext) // just use the default one for this example
            .destinationEncryptionContext(encryptionContextNew)
            .ciphertextBlob(SdkBytes.fromByteArray(edkCiphertext))
            .build();
    ReEncryptResponse reEncryptResponse = kmsClient.reEncrypt(reEncryptRequest);

    // base64 encode new EDK
    final String encryptedDataKeyNew = Base64.getEncoder().encodeToString(reEncryptResponse.ciphertextBlob().asByteArray());

    System.out.println("New EDK: " + encryptedDataKeyNew);
    Map<String, String> newMetadata = new HashMap<>(objectMetadata);
    newMetadata.replace(EDK_KEY, encryptedDataKeyNew);
    newMetadata.replace("x-amz-matdesc", ecForMd);

    // Copy object in-place
    s3Client.copyObject(builder -> builder
            .sourceKey(objectKey)
            .sourceBucket(BUCKET)
            .destinationKey(objectKey)
            .destinationBucket(BUCKET)
            .metadata(newMetadata)
            .metadataDirective(MetadataDirective.REPLACE)
    );

    HeadObjectResponse headObjectResponseNew = s3Client.headObject(builder -> builder
            .key(objectKey)
            .bucket(BUCKET));
    Map<String, String> objectMetadata2 = headObjectResponseNew.metadata();

    System.out.println("new metadata:");
    for (String key : objectMetadata2.keySet()) {
        System.out.println("key: " + key + "; value: " + objectMetadata2.get(key));
    }

    encryptionContextNew.remove("aws:x-amz-cek-alg");
    ResponseBytes<GetObjectResponse> getObjectResponseNew = s3Client.getObjectAsBytes(builder -> builder
            .key(objectKey)
            .bucket(BUCKET)
            .overrideConfiguration(withAdditionalConfiguration(encryptionContextNew)));

    // assert object decrypts
    String output = getObjectResponseNew.asUtf8String();
    assertEquals(input, output);
    System.out.println("Output: " + output);

    // Cleanup
    deleteObject(BUCKET, objectKey, s3Client);
    s3Client.close();
    kmsClient.close();
}

output:

previous metadata:
key: x-amz-tag-len; value: 128
key: x-amz-iv; value: PXPH/rNBtvNRS9v1
key: x-amz-wrap-alg; value: kms+context
key: x-amz-cek-alg; value: AES/GCM/NoPadding
key: x-amz-key-v2; value: AQIDAHjLUgvQf/mn7MrtEkg/LDE2GU7e9/fK3kCVEjUy4jqAdwEq5HPi8nEV6+jwHPuVQ03oAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMODlWxV7y3kIQ2xcPAgEQgDtQE1hsFbh4OdIld0j7jO7nOCKZQ46M0N02fiistI5XWUqNZwRZ5fVgLjjmoTk/lC2UuTeKXO5RxkVQeg==
key: x-amz-matdesc; value: {"aws:x-amz-cek-alg":"AES/GCM/NoPadding"}
New EDK: AQICAHjLUgvQf/mn7MrtEkg/LDE2GU7e9/fK3kCVEjUy4jqAdwFwSbfE76m8ctJWa8uf7PEVAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMUP0UR7Qs6fc4nGMsAgEQgDvLDVxP/N+oHaL8IDWH+9+9uoRRth2ubuDKiRFKmITOFRnUJpvzpdvJVIRQKGlyRS/FPWVB6hYrckzjng==
new metadata:
key: x-amz-tag-len; value: 128
key: x-amz-iv; value: PXPH/rNBtvNRS9v1
key: x-amz-wrap-alg; value: kms+context
key: x-amz-cek-alg; value: AES/GCM/NoPadding
key: x-amz-key-v2; value: AQICAHjLUgvQf/mn7MrtEkg/LDE2GU7e9/fK3kCVEjUy4jqAdwFwSbfE76m8ctJWa8uf7PEVAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMUP0UR7Qs6fc4nGMsAgEQgDvLDVxP/N+oHaL8IDWH+9+9uoRRth2ubuDKiRFKmITOFRnUJpvzpdvJVIRQKGlyRS/FPWVB6hYrckzjng==
key: x-amz-matdesc; value: {"aws:x-amz-cek-alg":"AES/GCM/NoPadding","rotated?":"yes"}
Output: SimpleTestOfV3EncryptionClient-ReEnvelope

[janSchumacherPayments]

@justplaz Thank you for this example and the explanation. I'm a little bit surprised about the complexity of the example. Compared to the simple putInstructionFile request, rotating the wrapping key seems to be much more complicated with the new client and it seems to be a more manual process involving kms and so on. But to be fair I did not have a look into the V2 API on key rotation using metadata.
For our case where we are talking about 285mil documents with avg 100kb filesize and bucket replication enabled (versioning required) copying each file is no option for us.
How does the feature request work in such cases?
As of now the old API (V2) seem to be at least maintained until December 2025. Is it realistic to get a similar cost efficient feature ready until mid next year?

[kessplas]

I see, indeed there isn't as simple of a solution in the V3 client. Your usecase is valid and we will add this feature in V3. This should be ready well before the V2 client (through the AWS SDK v1) is fully deprecated at the end of 2025.

[janSchumacherPayments]

@justplaz I'm really happy to read that!

[atennapel]

We are blocked on upgrading from v1 to v2 because of the lack of instructionFile support when storing new files. Is there a timeline for the addition of this feature?

[kessplas]

Hey @atennapel,

To clarify, do you need the key rotation functionality through an equivalent to putInstructionFile in v1/v2, or do you only need the ability to store metadata in instruction files on putObject? Thanks.

[janSchumacherPayments]

Hi @kessplas!
How is the timeline on this issue? Did you already made some progress?

[atennapel]

@kessplas Sorry for the late reply.

I don't know the underlying implementation, but we are using the following construct:
new com.amazonaws.services.s3.model.CryptoConfiguration().withStorageMode(CryptoStorageMode.InstructionFile)
Do you know if there's a replacement for this construct planned for v2?

[atennapel]

Hey @kessplas , any updates on this?