Skip to content

Commit

Permalink
feat: add nvidia flare demo
Browse files Browse the repository at this point in the history 
chore: fix linting issues

feat: add namespace variable and remove environment variable

feat: change namespace as variable

fix: reviews

chore(deps): bump super-linter/super-linter from 7.0.0 to 7.1.0 (#477)

Bumps [super-linter/super-linter](https://github.com/super-linter/super-linter) from 7.0.0 to 7.1.0.
- [Release notes](https://github.com/super-linter/super-linter/releases)
- [Changelog](https://github.com/super-linter/super-linter/blob/main/CHANGELOG.md)
- [Commits](super-linter/super-linter@v7.0.0...v7.1.0)

---
updated-dependencies:
- dependency-name: super-linter/super-linter
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

fix!: use an external git repository (#475)

- Don't enable the Cloud Source Repositories API.
- Don't provision a Cloud Source Repository.
- Get the Config Sync Git repository with an input variable.
- Let users specify the credentials type to authenticate Config Sync
  with their repositories.

Fix #449

chore: gitignore terraform stuff (#480)

feat: allow config sync to access source repository for synchronization (#479)

fix: minor changes to comply with pr review

Co-authored-by: Laurent Grangeau <[email protected]>

chore: manually build the example container image (#481)

Add instructions to build the distributed TensorFlow Federated example
in the example README, instead of having Terraform orchestrate the build
and push process.

This simplifies the root Terraform module, by removing something that's
needed only for an example, and it's also unlikely for users to have a
build pipeline orchestrated this way.

feat: add nvidia flare demo

chore: fix linting issues

feat: add namespace variable and remove environment variable

feat: change namespace as variable

fix: reviews

chore(deps): bump super-linter/super-linter from 7.0.0 to 7.1.0 (#477)

Bumps [super-linter/super-linter](https://github.com/super-linter/super-linter) from 7.0.0 to 7.1.0.
- [Release notes](https://github.com/super-linter/super-linter/releases)
- [Changelog](https://github.com/super-linter/super-linter/blob/main/CHANGELOG.md)
- [Commits](super-linter/super-linter@v7.0.0...v7.1.0)

---
updated-dependencies:
- dependency-name: super-linter/super-linter
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

feat: add nvflare demo

feat: add nvflare demo

fix: review

fix: peer review
  • Loading branch information
Laurent Grangeau committed Sep 12, 2024
1 parent 5b00f9e commit cb0e642
Show file tree
Hide file tree
Showing 31 changed files with 964 additions and 35 deletions.
10 changes: 10 additions & 0 deletions .github/workflows/build-container-images.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,13 @@ jobs:
contents: read
runs-on: ubuntu-latest
steps:
- name: Free Disk space
shell: bash
run: |
sudo rm -rf /usr/local/lib/android || true
sudo rm -rf /usr/share/dotnet || true
sudo rm -rf /opt/ghc || true
sudo rm -rf /usr/local/.ghcup || true
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- uses: actions/checkout@v4
Expand Down Expand Up @@ -76,3 +83,6 @@ jobs:
context-directory: examples/federated-learning/tff/distributed-fl-simulation-k8s/container-image
# Don't run an instance of this because it needs instrumentation
run: false
- name: nvflare
context-directory: examples/federated-learning/tff/nvflare/container-image
run: true
1 change: 1 addition & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@ To familiarize with the environment that you provisioned, you can also deploy
the following examples in the GKE cluster:

- [Distributed TensorFlow Federated training](./examples/federated-learning/tff/distributed-fl-simulation-k8s/README.md)
- [Nvflare training](./examples/federated-learning/tff/nvflare/README.md)

Federated learning is typically split into Cross-silo and Cross-device federated learning. Cross-silo federated computation is where the participating members are organizations or companies, and the number of members is usually small (e.g., within a hundred).

Expand Down
4 changes: 4 additions & 0 deletions assets/nvflare.svg
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
277 changes: 277 additions & 0 deletions config/lint/.checkov.baseline
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,28 @@
}
]
},
{
"file": "/terraform/nvflare/iam.tf",
"findings": [
{
"resource": "module.nvflare.storage_bucket_iam_bindings",
"check_ids": [
"CKV_TF_1"
]
}
]
},
{
"file": "/terraform/nvflare/main.tf",
"findings": [
{
"resource": "module.nvflare.buckets",
"check_ids": [
"CKV_TF_1"
]
}
]
},
{
"file": "/examples/federated-learning/tff/distributed-fl-simulation-k8s/distributed-fl-workload-pkg/coordinator.yaml",
"findings": [
Expand Down Expand Up @@ -103,6 +125,261 @@
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/base-server/base-server-deployment.yml",
"findings": [
{
"resource": "Deployment.nvflare.nvflare-server1",
"check_ids": [
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_22",
"CKV_K8S_38",
"CKV_K8S_9",
"CKV_K8S_31"
]
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/base-server/kustomization.yaml",
"findings": [
{
"resource": "base:Deployment.nvflare.nvflare-server1",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
},
{
"resource": "base:Pod.nvflare.nvflare-server1.run-nvflare-server1",
"check_ids": [
"CKV2_K8S_6"
]
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/base-deployment/base-deployment.yaml",
"findings": [
{
"resource": "Deployment.nvflare.nvflare",
"check_ids": [
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_22",
"CKV_K8S_38",
"CKV_K8S_9",
"CKV_K8S_31"
]
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/base-deployment/kustomization.yaml",
"findings": [
{
"resource": "base:Deployment.nvflare.nvflare",
"check_ids": [
"CKV_K8S_31",
"CKV_K8S_14",
"CKV_K8S_38",
"CKV_K8S_9",
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_8"
]
},
{
"resource": "overlay:nvflare:Pod.default.nvflare-server1.run-nvflare-server1",
"check_ids": [
"CKV2_K8S_6"
]
},
{
"resource": "overlay:nvflare:Pod.default.nvflare-client1.run-nvflare-client1",
"check_ids": [
"CKV2_K8S_6"
]
},
{
"resource": "overlay:nvflare:Pod.default.nvflare-client2.run-nvflare-client2",
"check_ids": [
"CKV2_K8S_6"
]
},
{
"resource": "base:Pod.default.nvflare.run-nvflare",
"check_ids": [
"CKV2_K8S_6"
]
},
{
"resource": "overlay:client2:Pod.default.nvflare-client2.run-nvflare-client2",
"check_ids": [
"CKV2_K8S_6"
]
},
{
"resource": "overlay:client1:Pod.default.nvflare-client1.run-nvflare-client1",
"check_ids": [
"CKV2_K8S_6"
]
},
{
"resource": "overlay:server1:Pod.default.nvflare-server1.run-nvflare-server1",
"check_ids": [
"CKV2_K8S_6"
]
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/base-client/base-client.yaml",
"findings": [
{
"resource": "Deployment.nvflare.nvflare-client",
"check_ids": [
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_22",
"CKV_K8S_38",
"CKV_K8S_9",
"CKV_K8S_31"
]
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/base-client/kustomization.yaml",
"findings": [
{
"resource": "base:Deployment.nvflare.nvflare-client",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
},
{
"resource": "base:Pod.nvflare.nvflare-client.run-nvflare-client",
"check_ids": [
"CKV2_K8S_6"
]
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/server1/kustomization.yaml",
"findings": [
{
"resource": "base:Deployment.nvflare.nvflare-client1",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
},
{
"resource": "overlay:server1:Deployment.nvflare.nvflare-server1",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
},
{
"resource": "overlay:server1:Service.nvflare.nvflare-server1",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/client1/kustomization.yaml",
"findings": [
{
"resource": "base:Deployment.nvflare.nvflare-client1",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
},
{
"resource": "overlay:client1:Deployment.nvflare.nvflare-client1",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/client2/kustomization.yaml",
"findings": [
{
"resource": "base:Deployment.nvflare.nvflare-client2",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
},
{
"resource": "overlay:client2:Deployment.nvflare.nvflare-client2",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
}
]
},
{
"file": "/examples/federated-learning/tff/distributed-fl-simulation-k8s/distributed-fl-workload-pkg/worker.yaml",
"findings": [
Expand Down
Loading

0 comments on commit cb0e642

Please sign in to comment.