|
| 1 | +--- |
| 2 | +title: NServiceBus DataBus feature |
| 3 | +summary: How to handle messages that are too large to be sent by a transport natively by using NServiceBus DataBus |
| 4 | +component: Databus |
| 5 | +reviewed: 2024-08-01 |
| 6 | +redirects: |
| 7 | + - nservicebus/databus |
| 8 | + - samples/pipeline/stream-properties |
| 9 | +related: |
| 10 | + - samples/databus/file-share-databus |
| 11 | + - samples/databus/custom-serializer |
| 12 | + - samples/databus/blob-storage-databus |
| 13 | +--- |
| 14 | + |
| 15 | +Although messaging systems work best with small message sizes, some scenarios require sending binary large objects ([blobs](https://en.wikipedia.org/wiki/Binary_large_object)) data along with a message (also known as a [_Claim Check_](https://learn.microsoft.com/en-us/azure/architecture/patterns/claim-check)). For this purpose, NServiceBus has a `DataBus` feature to overcome the message size limitations imposed by the underlying transport. |
| 16 | + |
| 17 | +## How it works |
| 18 | + |
| 19 | +Instead of serializing the payload along with the rest of the message, the `DataBus` approach involves storing the payload in a separate location that both the sending and receiving parties can access, then putting the reference to that location in the message. |
| 20 | + |
| 21 | +If the location is not available upon sending, the send operation will fail. When a message is received and the payload location is not available, the receive operation will fail too, resulting in the standard NServiceBus retry behavior, possibly resulting in the message being moved to the error queue if the error could not be resolved. |
| 22 | + |
| 23 | +## Transport message size limits |
| 24 | + |
| 25 | +The `DataBus` may be used to send messages which exceed the transport's message size limit, which is determined by the message size limit of the underlying queuing/storage technologies. |
| 26 | + |
| 27 | +> [!NOTE] |
| 28 | +> Not all transports have message size limits and some technologies, such as Azure Service Bus, have increased over time. Current message size limits are stated in the documentation linked in the table below. |
| 29 | +
|
| 30 | +| Transport | Maximum size | |
| 31 | +| --------------------------------- | ------------:| |
| 32 | +| Amazon SQS | [256KB](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/quotas-messages.html) | |
| 33 | +| Amazon SQS (using S3) | [2GB](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/quotas-messages.html) | |
| 34 | +| Azure Storage Queues | [64KB](https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted#capacity-and-quotas) | |
| 35 | +| Azure Service Bus (Standard tier) | [256KB](https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted#capacity-and-quotas) | |
| 36 | +| Azure Service Bus (Premium tier) | [100MB](https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-premium-messaging#large-messages-support) | |
| 37 | +| RabbitMQ | Configured by [`max_message_size`](https://www.rabbitmq.com/configure.html#config-items) | |
| 38 | +| SQL Server | No limit | |
| 39 | +| Learning | No limit | |
| 40 | +| MSMQ | [4MB](https://learn.microsoft.com/en-us/previous-versions/windows/desktop/msmq/ms711436(v=vs.85)#maximum-message-size) | |
| 41 | + |
| 42 | +## Enabling the `DataBus` |
| 43 | + |
| 44 | +See the individual `DataBus` implementations for details on enabling and configuring the `DataBus`. |
| 45 | + |
| 46 | +- [File Share Data Bus](file-share.md) |
| 47 | +- [Azure Blob Storage Data Bus](azure-blob-storage.md) |
| 48 | + |
| 49 | +## Cleanup |
| 50 | + |
| 51 | +By default, blobs are stored with no set expiration. If messages have a [time to be received](/nservicebus/messaging/discard-old-messages.md) set, the `DataBus` will pass this along to the data bus storage implementation. |
| 52 | + |
| 53 | +> [!NOTE] |
| 54 | +> The value used should be aligned with the [ServiceContol audit retention period](/servicecontrol/how-purge-expired-data.md) if it is required that `DataBus` blob keys in messages sent to the audit queue can still be fetched. |
| 55 | +
|
| 56 | +## Specifying `DataBus` properties |
| 57 | + |
| 58 | +There are two ways to specify the message properties to be sent using the `DataBus`: |
| 59 | + |
| 60 | + 1. Using the `DataBusProperty<T>` type |
| 61 | + 1. Message conventions |
| 62 | + |
| 63 | +> [!NOTE] |
| 64 | +> `DataBus` properties must be top-level properties on a message class. |
| 65 | +
|
| 66 | +### Using `DataBusProperty<T>` |
| 67 | + |
| 68 | +Set the type of the property to be sent over as `DataBusProperty<byte[]>`: |
| 69 | + |
| 70 | +snippet: MessageWithLargePayload |
| 71 | + |
| 72 | +### Using message conventions |
| 73 | + |
| 74 | +NServiceBus also supports defining `DataBus` properties by convention. This allows data properties to be sent using the `DataBus` without using `DataBusProperty<T>`, thus removing the need for having a dependency on NServiceBus from the message types. |
| 75 | + |
| 76 | +In the configuration of the endpoint include: |
| 77 | + |
| 78 | +snippet: DefineMessageWithLargePayloadUsingConvention |
| 79 | + |
| 80 | +Set the type of the property as `byte[]`: |
| 81 | + |
| 82 | +snippet: MessageWithLargePayloadUsingConvention |
| 83 | + |
| 84 | +partial: serialization |
| 85 | + |
| 86 | +## `DataBus` attachments cleanup |
| 87 | + |
| 88 | +The `DataBus` implementations each behave differently when cleaning up the physical attachments used to transfer the data properties. |
| 89 | + |
| 90 | +### Why attachments are not removed by default |
| 91 | + |
| 92 | +Automatically removing these attachments can cause problems in many situations. For example: |
| 93 | + |
| 94 | +- The supported implementations do not participate in distributed transactions. If the message handler throws an exception and the transaction rolls back, the delete operation on the attachment cannot be rolled back. Therefore, when the message is retried, the attachment will no longer be present, causing additional problems. |
| 95 | +- The message can be deferred so that the file will be processed later. Removing the file after deferring the message, results in a message without the corresponding file. |
| 96 | +- Functional requirements might dictate the message to be available for a longer duration. |
| 97 | +- If the `DataBus` feature is used when publishing an event to multiple subscribers, neither the publisher nor any specific subscribing endpoint can determine when all subscribers have successfully processed the message, allowing the file to be cleaned up. |
| 98 | +- If message processing fails, it will be handled by the [recoverability feature](/nservicebus/recoverability/). This message can then be retried some period after that failure. The attachment files need to exist for that message to be re-processed correctly. |
| 99 | + |
| 100 | +## Alternatives |
| 101 | + |
| 102 | +> [!NOTE] |
| 103 | +> A combination of these techniques may be used. |
| 104 | +
|
| 105 | +- Use a different transport or different tier (e.g. Azure Service Bus _Premium_ instead of _Standard_). |
| 106 | +- Use message body compression, which works well on text-based payloads like XML and JSON or any payload (text or binary) that contains repetitive data. |
| 107 | + - The [message mutator sample](/samples/messagemutators/) demonstrates message body compression. |
| 108 | +- Use a more efficient serializer, such as a binary serializer. |
| 109 | + - A custom serializer can usually be implemented in only a few lines of code. |
| 110 | + - Some binary [serializers are maintained by the community](/nservicebus/community/#serializers). |
| 111 | +- Use [NServiceBus.Attachments](/nservicebus/community/#nservicebus-attachments) for unbounded binary payloads. The package is similar to the `DataBus`, but has some differences: |
| 112 | + - Read on demand: Attachments are only retrieved when read by a consumer. |
| 113 | + - Async enumeration: The package supports processing all data items using an `IAsyncEnumerable`. |
| 114 | + - No serialization: The serializer is not used, which may result in a significant reduction in memory usage. |
| 115 | + - Direct stream access: This makes the package more suitable for [binary large objects (blobs](https://en.wikipedia.org/wiki/Binary_large_object) since stream contents do not necessarily have to be loaded into memory before storing them or when retrieving them. |
| 116 | + |
| 117 | +## Other considerations |
| 118 | + |
| 119 | +### Monitoring and reliability |
| 120 | + |
| 121 | +The storage location for `DataBus` blobs is critical to the operation of endpoints. As such, it should be as reliable as other infrastructure, such as the transport or persistence. It should also be monitored for errors and be actively maintained. Since messages cannot be sent or received when the storage location is unavailable, it may be necessary to stop endpoints when maintenance tasks occur. |
| 122 | + |
| 123 | +### Auditing |
| 124 | + |
| 125 | +The data stored in `DataBus` blobs may be considered part of an audit record. In these cases, the blobs should be archived alongside messages for as long as the audit record is required. |
0 commit comments