feat: add a federated learning example using nvidia flare

[ferrarimarco]

• Federated Learning use case:

  • Implement an example that deploys NVIDIA FLARE in the cluster.
  • Support getting Terraform output in JSON format
  • Implement a service to provision Cloud Storage buckets
  • Enable Kubernetes network policy logging
  • Don't enforce injecting Cloud Service Mesh sidecars in the kube-system namespace
  • Fix Cloud Service Mesh auto-injection namespace labels
  • Add tolerations to istio-egress so it can be deployed in the cluster
  • Fix mutator configuration to add the right tolerations to federated learning workloads
  • Allow DNS queries to GKE control plane
  • Create missing tenant service account
  • Configure Terraform outputs to get Artifact Registry repository FQDN.
  • Use the truncated Cloud Build BUILD_ID as the project name suffix to avoid conflicts with projects pending deletion.

[arueth]

It seems like you are really intertwining this use case into your "core" federated learning platform. Is it possible to keep them more distinct? Possibly using a separate config file or a different folder structure.

To me this seems like an additional use case on top of your "core" platform, similar to Fine Tuning and RAG on the AI/ML platform.

[ferrarimarco]

Good point, let me see how I can restructure this.

[ferrarimarco]

@arueth I've reworked things a bit, PTAL, thanks.

[arueth]

It looks like we need to find a way to generate a unique project name per build. Right now the job is failing because the project with that SHORT_SHA already exists in a pending delete state. Maybe we can use a truncated version of BUILD_ID.

[ferrarimarco]

It looks like we need to find a way to generate a unique project name per build. Right now the job is failing because the project with that SHORT_SHA already exists in a pending delete state. Maybe we can use a truncated version of BUILD_ID.

Done in the latest commit. I used the BUILD_ID as you suggested, truncating it at 7 characters as a short Git hash.