Parser: Why does S3SqsEventNotificationRecordSchema not use S3Schema as type for body property?

[cyrildewit]

Hi, I am implementing a Lambda handler that will consume a SQS queue filled with S3 event notifications. I expected that S3SqsEventNotificationSchema would also validate and parse the payload of the S3 Event Notification within the SQS Message, but that is currently not the case.

Here's a copy of the official schemas for easy reference source:

const S3SqsEventNotificationSchema = z.object({
  Records: z.array(S3SqsEventNotificationRecordSchema),
});

const S3SqsEventNotificationRecordSchema = SqsRecordSchema.extend({
  body: z.string(),
});

I wonder if this design decision was made on purpose?

I have solved it by creating a custom version of S3SqsEventNotificationSchema, like so:

const extendedS3SqsEventNotificationSchema = S3SqsEventNotificationSchema.extend({
    Records: z.array(SqsRecordSchema.extend({
        body: JSONStringified(S3Schema),
    })),
});
type ExtendedS3SqsEventNotification = z.infer<typeof extendedS3SqsEventNotificationSchema>;

Full code example for reference:

```ts import type { SQSHandler } from 'aws-lambda'; import middy from '@middy/core'; import { Logger } from '@aws-lambda-powertools/logger'; import { injectLambdaContext } from '@aws-lambda-powertools/logger/middleware'; import { parser } from '@aws-lambda-powertools/parser/middleware'; import { S3Schema, S3SqsEventNotificationSchema, SqsRecordSchema, } from '@aws-lambda-powertools/parser/schemas'; import { JSONStringified } from '@aws-lambda-powertools/parser/helpers'; import { BatchProcessor, EventType, processPartialResponse } from '@aws-lambda-powertools/batch'; import { z } from 'zod';

const processor = new BatchProcessor(EventType.SQS);

const logger = new Logger({
logLevel: 'debug',
});

const extendedS3SqsEventNotificationSchema = S3SqsEventNotificationSchema.extend({
Records: z.array(SqsRecordSchema.extend({
body: JSONStringified(S3Schema),
})),
});

type ExtendedS3SqsEventNotification = z.infer;

const recordHandler = async (record: ExtendedS3SqsEventNotification['Records'][number]): Promise => {
logger.debug('S3 Event Records', {
s3EventRecords: record.body.Records,
});
};

const lambdaSqsHandler: SQSHandler = async (event, context) =>
processPartialResponse(event, recordHandler, processor, {
context,
});

export const handler = middy(lambdaSqsHandler)
.use(injectLambdaContext(logger, { logEvent: true }))
.use(parser({ schema: extendedS3SqsEventNotificationSchema }));

</details>

[dreamorosi]

Hi @cyrildewit - apologies for the late reply.

The body field in a SQS event is always a string. Because of this the correct type is z.string() and not S3Schema.

If we defined the schema like this:

const S3SqsEventNotificationRecordSchema = SqsRecordSchema.extend({
  body: S3Schema,
});

The parsing would always fail since body is a string and S3Schema is a z.object({ ... }).

When it comes to generic SQS events we can't make the assumption of the body being a JSON string, since you could send anything as message to the queue. For the specific S3 -> SQS integration however it should be safe to assume that the body is always a stringified object, and thus we could potentially JSON.parse it for you.

If you'd like us to pre-parse the body for this specific type of events let us know and I'll convert this to a feature request.

[cyrildewit]

Hi @dreamorosi,

First of all, please do not apologize for something that you shouldn't ;). There's no agreement between us and definitely no obligation. I am grateful for your eloborate response and would like to take this moment to show my appreciation for this project in general! It's definitely a game changer! I just started with implementing Lambda's using TypeScript. I did a quite extensive research on the topic and can conclude that these utilities lay a foundation with the best practices that allows me to focus on the things that matter. Well done!

Back to the topic, the reason why I first wanted to reach out within the discussions section is to make sure that I fully understand why this schema was designed this way, before initiation a feature request. I was simply too fresh with this project, but my idea/reasioning was that we could safely pre-parse this body field, since it's a schema for a specific use-case. In contrary to the generic schemas.
It's good to hear that you made the same reasoning.

For the specific S3 -> SQS integration however it should be safe to assume that the body is always a stringified object, and thus we could potentially JSON.parse it for you.

Yes please, if you could could convert this to a feature request that would be great. I think that this little change will make the schema "S3SqsEventNotificationSchema" more praticable for most cases. I can also take the initiative to propose the change and work on the implementation. I'm not sure which steps actually need to be taken, so if could list the actions in shorrt down that would be great. Does it require an RFC for example.

Let me know if I can mean anything in this!
Have a great day!

[dreamorosi]

Hi @cyrildewit, thank you for your kind works - it always makes my day to hear people find my work useful!

For this specific instance, since the change is quite self-contained we don't need an RFC and we can move directly to implementation (learn more about our process here).

I have opened a feature request issue at #3525 and left some details on the implementation in this comment. If you're interested in picking up the issue, please leave a comment so I can assign it to you.

Here you should find details on how to set up the dev environment, if you encounter any issues with that please let me know and I'll do my best to help you get up and running.