Provide a simple way to provision the topology required by the transport

[SeanFeldman]

As of now, there's no easy way to deploy a function-hosted endpoint. One has to identify the endpoint name (queue name), created the queue, manually subscribe to all the events, and provision the error and the audit queue using the ASB transport CLI tool to manually execute multiple commands to have the topology created.

[JSCProjects]

Yeah this is for a lot of our developers a problem and they still want to use webjobs because they don't have to think about creating this in our development pipeline. We are also facing problems executing the CLI tool on our Azure Dev Ops (Hosted) Agents.

I'm aware that it's a chicken egg problem

[andrekiba]

Yes, this should be a "must have" 👍

[SeanFeldman]

Once Azure/azure-functions-servicebus-extension#82 is resolved, there's potentially a way to have this unblocked.

[SeanFeldman]

While it’s not a full solution, here’s a hint of how to get unstuck if you want the topology to be created along with a function deployment.

[andrekiba]

Thank @SeanFeldman I was trying with something like that!

[SeanFeldman]

Anyone's looking to provision their topology automatically here's an idea of how to get it implemented today. There's also a Functions feature request you could upvote to make this idea supported ootb.

[andrekiba]

@SeanFeldman what about the "other" thing that is for now missing, the database side with the necessary tables like SagaData and OutboxData.
Is there a plan to have the EnableInstallers also for functions?
In the meantime the only way is to create them before?
Thanks!

[SeanFeldman]

@andrekiba, the comment above was just an idea to unblock those that absolutely need it. This is by no means an official solution by Particular.

what about the "other" thing that is for now missing, the database side with the necessary tables like SagaData and OutboxData.

When an endpoint is created, which is what will happen when using the idea I've described, all installers will be executed, assuming endpointConfiguration.EnableInstallers() is applied. Note that I have not tested persistence in my post. You're welcome to verify that.

[andrekiba]

Thank you @SeanFeldman I try today and I let you know...but as you said I suppose yes.
Starting from a "classic" endpoint configuration we are trying to understand what is useful and what is not for a function endpoint.
We need to understand about what is commented now.

Endpoint configuration code

public static ServiceBusTriggeredEndpointConfiguration BuildEndpointConfiguration(IConfiguration configuration)
{
    var endpointConfiguration = new ServiceBusTriggeredEndpointConfiguration(configuration["NServiceBus:EndpointName"]);
    endpointConfiguration.LogDiagnostics();
    var e = endpointConfiguration.AdvancedConfiguration;

    //var transport = e.UseTransport<AzureServiceBusTransport>();
    //transport.ConnectionString(configuration["AzureWebJobsServiceBus"]);

    var persistence = e.UsePersistence<SqlPersistence>();
    persistence.ConnectionBuilder(() => new SqlConnection(configuration.GetConnectionString("Db")));
    persistence.TablePrefix(string.Empty);
    var dialect = persistence.SqlDialect<SqlDialect.MsSqlServer>();
    dialect.Schema("nservicebus");
    var sagaSettings = persistence.SagaSettings();
    sagaSettings.JsonSettings(EventSerialization.Settings);

    //var outboxSettings = e.EnableOutbox();
    //outboxSettings.UseTransactionScope();
    //outboxSettings.KeepDeduplicationDataFor(TimeSpan.FromDays(6));
    //outboxSettings.RunDeduplicationDataCleanupEvery(TimeSpan.FromMinutes(15));

    var serialization = e.UseSerialization<NewtonsoftSerializer>();
    // Newtonsoft serializer doesn't properly deserialize properties with protected setter,
    // the following serializer settings will be applied by NServiceBus during all messages's deserializations
    serialization.Settings(EventSerialization.Settings);

    e.Conventions().DefiningEventsAs(t => t.Namespace != null && t.Namespace.EndsWith("Events"));

    e.AuditProcessedMessagesTo(configuration["NServiceBus:AuditQueue"]);
    e.SendFailedMessagesTo(configuration["NServiceBus:ErrorQueue"]);
    //e.SendHeartbeatTo(serviceControlQueue: configuration["NServiceBus:ServiceControlInstance"],
    //    frequency: TimeSpan.FromSeconds(15),
    //    timeToLive: TimeSpan.FromSeconds(30));

    //var endpointName = configuration["NServiceBus:EndpointName"];
    //var machineName = $"{Dns.GetHostName()}.{IPGlobalProperties.GetIPGlobalProperties().DomainName}";
    //var instanceIdentifier = $"{endpointName}@{machineName}";

    //var metrics = e.EnableMetrics();
    //metrics.SendMetricDataToServiceControl(serviceControlMetricsAddress: configuration["NServiceBus:ServiceControlMonitoringInstance"],
    //    interval: TimeSpan.FromSeconds(10),
    //    instanceId: instanceIdentifier);

    //For Development ONLY you can install NServiceBus.SagaAudit package and activate this monitoring feature
    //https://docs.particular.net/nservicebus/sagas/saga-audit
    //e.AuditSagaStateChanges(serviceControlQueue: configuration["NServiceBus:ServiceControlInstance"]);

    //For Data Annotation validations you can activate the NServiceBus.DataAnnotations unofficial extension
    //https://www.nuget.org/packages/NServiceBus.DataAnnotations/
    //e.UseDataAnnotationsValidation();

    //create topology, sync for now
    CreateTopology(configuration, auditQueue: configuration["NServiceBus:AuditQueue"], errorQueue: configuration["NServiceBus:ErrorQueue"])
        .GetAwaiter().GetResult();
    
    //This instruction checks every time the application starts up in order to create
    //all the necessary NServiceBus objects in the database automatically
    e.EnableInstallers();

    return endpointConfiguration;
}

As you can see we have the method that creates the topology inside the BuildEndpointConfiguration that is called inside the Startup class as usual with builder.UseNServiceBus(() => NServiceBusConfiguration.BuildEndpointConfiguration(configuration));

Does the call to EnableInstallers() must be after the creation of the topology?

[SeanFeldman]

@SeanFeldman, EnableInstallers should be where you have it right now. CreateTopology on the other hand is in the wrong spot. Also, you're using the static approach and the implementation I've covered is using the IFunctionHostBuilder approach. Which I recommend using.

For more questions specific to your issue I suggest going through support (you can reference this thread) and allow this issue to stay focused on the feature request comments. Thank you.

[andrekiba]

@SeanFeldman it's not an issue :-) ...and we use the IFunctionHostBuilder approach and not the static one...I'm not understanding.
We have only created an extension method.

[assembly: FunctionsStartup(typeof(Startup))]
namespace Elfo.NsbFunctions.Endpoint
{
    public class Startup : FunctionsStartup
    {
        public override void Configure(IFunctionsHostBuilder builder)
        {
            var configuration = builder.GetContext().Configuration;
            
            var logger = ConfigureLogger(configuration);

            builder.Services
                .AddLogging(loggingBuilder => loggingBuilder.AddSerilog(logger));
            
            builder.UseNServiceBus(() => NServiceBusConfiguration.BuildEndpointConfiguration(configuration));
        }
    }
}

The implementation of BuildEndpointConfiguration is the one I posted above.
We move back the CreateTopology directly inside the Configure ...but practically seems it works in both cases...

[codewisdom]

Hey @SeanFeldman, thanks for all the hard work on this! There is a debate at work whether or not we need to setup routing in code for our NServiceBus Azure Function apps. I believe we don't need to and it works without it, as long as we have our queues and subscriptions setup properly with the CLI tool. To be clear, do we need to call RouteToEndpoint on the transport RoutingSettings when setting up a FunctionEndpoint or an EndpointInstance with AzureServiceBusTransport?

Of course, I'd love to be able to call EnableInstallers with the rest of the world here. :)

[SeanFeldman]

Thank you, @codewisdom.

do we need to call RouteToEndpoint on the transport RoutingSettings when setting up a FunctionEndpoint or an EndpointInstance with AzureServiceBusTransport?

TLDR: You do.

Topology and routing are two different concerns. While related to each other, they are not the same. Topology, queues/topics/subscriptions/rules is the infrastructure on top of the Service Bus namespace needed for the endpoint to send/publish/receive/subscribe. Topology is provisioned either with the CLI tool, NServiceBus Installers feature, or any other scripting method such as ARM, Bicep, Terraform, etc.

Routing, on the other hand, is a logical concept required by NServiceBus for sending messages marked as commands to the right destination w/o developers specifying that destination each time. No matter how the topology is provisioned, to send a command requires knowing the destination. That destination is either provided once with RouteToEndpoint or each time via sending options. There's an option of extending the routing system, but it's still more or less what the first method is doing, allowing some dynamic options

I hope that helps.

[codewisdom]

Thanks for that fantastic explanation @SeanFeldman. For some reason, I must have missed some of that in the NServiceBus docs.

One follow up question - is the CLI tool subscribe features required for command routing that is not local? I thought so, but perhaps not?

[SeanFeldman]

The CLI tool does not handle routing; it literally builds up the topology based on what you know about the routing. Therefore, topology needs to consider the routing information. So, if you have a destination, you need to create an endpoint using asb-transport endpoint create <name> If you send commands to a destination that is not an endpoint, the destination is likely controlled and provisioned by another party (sub-system in your organization), and requires no CLI work. If that's not the case, you can always use the CLI to create the necessary queue with asb-transport queue create <name>.