Orleans Clustering Fails with Multiple Instances in Azure App Service using a Linux Container

[KobusInversion]

I have the following setup with Orleans to allow for clustering. Using Docker locally, I can spin up two instances with the code below, and both instances join the cluster successfully.

builder.Host.UseOrleans((context, siloBuilder) =>
{
    siloBuilder.Services.AddSerializer(serializerBuilder => { serializerBuilder.ConfigureSerializer(); });  
     
    siloBuilder
        .ConfigureEndpoints(11111, 30000, listenOnAnyHostAddress: true)
        .UseAzureStorageClustering(opt =>
        {
            opt.TableName = "linuxOrleansSiloInstances";
            opt.TableServiceClient =
                new TableServiceClient(settings.TableStorageConfig.ConnectionString);
        })
        .Configure<ClusterOptions>(options =>
        {
            options.ClusterId = settings.ClusterConfiguration.ClusterId;
            options.ServiceId = settings.ClusterConfiguration.ServiceId;
        });
});

I want to host it in an Azure App Service within a Linux container. The first instance spins up correctly, but I start seeing the errors below when the second instance attempts to join the cluster.

Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S169.254.129.15:11111:94653375. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 169.254.129.15:11111. Error: ConnectionRefused
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/Shared/SocketConnectionFactory.cs:line 65

Here is my partial Dockerfile setup attempting to expose ports 11111 and 30000:

FROM mcr.microsoft.com/dotnet/aspnet:9.0 AS base
WORKDIR /app

EXPOSE 5000
EXPOSE 11111
EXPOSE 30000

# Build stage uses the .NET SDK for build
FROM mcr.microsoft.com/dotnet/sdk:9.0 AS build
WORKDIR /src

...

The only documentation I could find was here https://learn.microsoft.com/en-us/dotnet/orleans/deployment/deploy-to-azure-app-service#configure-host-networking, which mentions the following:

If deploying to Linux, ensure that your hosts are listening on all IP addresses as described in the Configure host networking section.
The above article is accurate when it comes to windows-based environment, which I have done before with success but seems when scaling out with a docker linux based environment these instructions don't work.

What configuration am I missing to enable instance communication and ensure proper silo coordination?

[ReubenBond]

Hi @KobusInversion, your silo is listening on 11111 and 30000, but App Service provides ports for you via the WEBSITE_PRIVATE_PORTS environment variable and you should set your silos' AdvertisedIPAddress to the WEBSITE_PRIVATE_IP environment variable, as shown in the docs here:

var endpointAddress = IPAddress.Parse(builder.Configuration["WEBSITE_PRIVATE_IP"]!);
var strPorts = builder.Configuration["WEBSITE_PRIVATE_PORTS"]!.Split(',');
if (strPorts.Length < 2)
{
    throw new Exception("Insufficient private ports configured.");
}

var (siloPort, gatewayPort) = (int.Parse(strPorts[0]), int.Parse(strPorts[1]));

siloBuilder
    .ConfigureEndpoints(endpointAddress, siloPort, gatewayPort, listenOnAnyHostAddress: true)

Note that this requires that you configure App Service to provide private ports in the first place, as demonstrated in the Configure private port count using Azure CLI section.

Please let us know if that works for you.

[KobusInversion]

I was finally able to get it working.

Here is the command I used to create the private IP and ports before:

az webapp config set -g '{resource-group}' -n '{app-name}' --generic-configurations '{"vnetPrivatePortsCount": 2}'

However, it seems that the command has actually changed to:

az webapp config set -g '<resource-group-name>' --subscription '<subscription-id>' -n '<app-service-app-name>' --generic-configurations '{\"vnetPrivatePortsCount\": "2"}'

The key difference is that the previous command did not include the --subscription parameter.