Skip to content

Commit

Permalink
feat: add nvidia flare demo
Browse files Browse the repository at this point in the history 
chore: fix linting issues

feat: add namespace variable and remove environment variable

feat: change namespace as variable

fix: reviews

chore(deps): bump super-linter/super-linter from 7.0.0 to 7.1.0 (#477)

Bumps [super-linter/super-linter](https://github.com/super-linter/super-linter) from 7.0.0 to 7.1.0.
- [Release notes](https://github.com/super-linter/super-linter/releases)
- [Changelog](https://github.com/super-linter/super-linter/blob/main/CHANGELOG.md)
- [Commits](super-linter/super-linter@v7.0.0...v7.1.0)

---
updated-dependencies:
- dependency-name: super-linter/super-linter
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

fix!: use an external git repository (#475)

- Don't enable the Cloud Source Repositories API.
- Don't provision a Cloud Source Repository.
- Get the Config Sync Git repository with an input variable.
- Let users specify the credentials type to authenticate Config Sync
  with their repositories.

Fix #449

chore: gitignore terraform stuff (#480)

feat: allow config sync to access source repository for synchronization (#479)

fix: minor changes to comply with pr review

Co-authored-by: Laurent Grangeau <[email protected]>

chore: manually build the example container image (#481)

Add instructions to build the distributed TensorFlow Federated example
in the example README, instead of having Terraform orchestrate the build
and push process.

This simplifies the root Terraform module, by removing something that's
needed only for an example, and it's also unlikely for users to have a
build pipeline orchestrated this way.

feat: add nvidia flare demo

chore: fix linting issues

feat: add namespace variable and remove environment variable

feat: change namespace as variable

fix: reviews

chore(deps): bump super-linter/super-linter from 7.0.0 to 7.1.0 (#477)

Bumps [super-linter/super-linter](https://github.com/super-linter/super-linter) from 7.0.0 to 7.1.0.
- [Release notes](https://github.com/super-linter/super-linter/releases)
- [Changelog](https://github.com/super-linter/super-linter/blob/main/CHANGELOG.md)
- [Commits](super-linter/super-linter@v7.0.0...v7.1.0)

---
updated-dependencies:
- dependency-name: super-linter/super-linter
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

feat: add nvflare demo

feat: add nvflare demo

fix: review

fix: peer review

fix: peer review

fix: pr review

fix: calculate the network id first

feat: update terraform providers version

fix: revert tenant config

fix: generate the files directly into the acm folder

fix: pr comments

fix: pr review

fix: pr comment

fix: order to generate kustomization.yaml

fix: error on kustomize

fix: failed build

fix: pr comment

fix: random string for bucket

fix: pr comment
  • Loading branch information
Laurent Grangeau committed Sep 27, 2024
1 parent 5b00f9e commit 1e5fdb4
Show file tree
Hide file tree
Showing 33 changed files with 1,130 additions and 128 deletions.
2 changes: 1 addition & 1 deletion .editorconfig
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ trim_trailing_whitespace = true
spaces_around_operators = true
spaces_around_brackets = false

[*.{hcl.tpl,json,lock.hcl,js,md,sh,tf,tfvars,yml,yaml}]
[*.{hcl.tpl,json,lock.hcl,js,md,sh,tf,tfvars,yml,yaml,yml.tpl,yaml.tpl}]
indent_size = 2

[*.md]
Expand Down
10 changes: 10 additions & 0 deletions .github/workflows/build-container-images.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,13 @@ jobs:
contents: read
runs-on: ubuntu-latest
steps:
- name: Free Disk space
shell: bash
run: |
sudo rm -rf /usr/local/lib/android || true
sudo rm -rf /usr/share/dotnet || true
sudo rm -rf /opt/ghc || true
sudo rm -rf /usr/local/.ghcup || true
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- uses: actions/checkout@v4
Expand Down Expand Up @@ -76,3 +83,6 @@ jobs:
context-directory: examples/federated-learning/tff/distributed-fl-simulation-k8s/container-image
# Don't run an instance of this because it needs instrumentation
run: false
- name: nvflare
context-directory: examples/federated-learning/tff/nvflare/container-image
run: true
1 change: 1 addition & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@ To familiarize with the environment that you provisioned, you can also deploy
the following examples in the GKE cluster:

- [Distributed TensorFlow Federated training](./examples/federated-learning/tff/distributed-fl-simulation-k8s/README.md)
- [Nvflare training](./examples/federated-learning/tff/nvflare/README.md)

Federated learning is typically split into Cross-silo and Cross-device federated learning. Cross-silo federated computation is where the participating members are organizations or companies, and the number of members is usually small (e.g., within a hundred).

Expand Down
4 changes: 4 additions & 0 deletions assets/nvflare.svg
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
277 changes: 277 additions & 0 deletions config/lint/.checkov.baseline
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,28 @@
}
]
},
{
"file": "/terraform/nvflare/iam.tf",
"findings": [
{
"resource": "module.nvflare.storage_bucket_iam_bindings",
"check_ids": [
"CKV_TF_1"
]
}
]
},
{
"file": "/terraform/nvflare/main.tf",
"findings": [
{
"resource": "module.nvflare.buckets",
"check_ids": [
"CKV_TF_1"
]
}
]
},
{
"file": "/examples/federated-learning/tff/distributed-fl-simulation-k8s/distributed-fl-workload-pkg/coordinator.yaml",
"findings": [
Expand Down Expand Up @@ -103,6 +125,261 @@
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/base-server/base-server-deployment.yml",
"findings": [
{
"resource": "Deployment.nvflare.nvflare-server1",
"check_ids": [
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_22",
"CKV_K8S_38",
"CKV_K8S_9",
"CKV_K8S_31"
]
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/base-server/kustomization.yaml",
"findings": [
{
"resource": "base:Deployment.nvflare.nvflare-server1",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
},
{
"resource": "base:Pod.nvflare.nvflare-server1.run-nvflare-server1",
"check_ids": [
"CKV2_K8S_6"
]
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/base-deployment/base-deployment.yaml",
"findings": [
{
"resource": "Deployment.nvflare.nvflare",
"check_ids": [
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_22",
"CKV_K8S_38",
"CKV_K8S_9",
"CKV_K8S_31"
]
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/base-deployment/kustomization.yaml",
"findings": [
{
"resource": "base:Deployment.nvflare.nvflare",
"check_ids": [
"CKV_K8S_31",
"CKV_K8S_14",
"CKV_K8S_38",
"CKV_K8S_9",
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_8"
]
},
{
"resource": "overlay:nvflare:Pod.default.nvflare-server1.run-nvflare-server1",
"check_ids": [
"CKV2_K8S_6"
]
},
{
"resource": "overlay:nvflare:Pod.default.nvflare-client1.run-nvflare-client1",
"check_ids": [
"CKV2_K8S_6"
]
},
{
"resource": "overlay:nvflare:Pod.default.nvflare-client2.run-nvflare-client2",
"check_ids": [
"CKV2_K8S_6"
]
},
{
"resource": "base:Pod.default.nvflare.run-nvflare",
"check_ids": [
"CKV2_K8S_6"
]
},
{
"resource": "overlay:client2:Pod.default.nvflare-client2.run-nvflare-client2",
"check_ids": [
"CKV2_K8S_6"
]
},
{
"resource": "overlay:client1:Pod.default.nvflare-client1.run-nvflare-client1",
"check_ids": [
"CKV2_K8S_6"
]
},
{
"resource": "overlay:server1:Pod.default.nvflare-server1.run-nvflare-server1",
"check_ids": [
"CKV2_K8S_6"
]
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/base-client/base-client.yaml",
"findings": [
{
"resource": "Deployment.nvflare.nvflare-client",
"check_ids": [
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_22",
"CKV_K8S_38",
"CKV_K8S_9",
"CKV_K8S_31"
]
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/base-client/kustomization.yaml",
"findings": [
{
"resource": "base:Deployment.nvflare.nvflare-client",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
},
{
"resource": "base:Pod.nvflare.nvflare-client.run-nvflare-client",
"check_ids": [
"CKV2_K8S_6"
]
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/server1/kustomization.yaml",
"findings": [
{
"resource": "base:Deployment.nvflare.nvflare-client1",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
},
{
"resource": "overlay:server1:Deployment.nvflare.nvflare-server1",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
},
{
"resource": "overlay:server1:Service.nvflare.nvflare-server1",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/client1/kustomization.yaml",
"findings": [
{
"resource": "base:Deployment.nvflare.nvflare-client1",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
},
{
"resource": "overlay:client1:Deployment.nvflare.nvflare-client1",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
}
]
},
{
"file": "/examples/federated-learning/tff/nvflare/client2/kustomization.yaml",
"findings": [
{
"resource": "base:Deployment.nvflare.nvflare-client2",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
},
{
"resource": "overlay:client2:Deployment.nvflare.nvflare-client2",
"check_ids": [
"CKV_K8S_22",
"CKV_K8S_43",
"CKV_K8S_14",
"CKV_K8S_8",
"CKV_K8S_9",
"CKV_K8S_38",
"CKV_K8S_31"
]
}
]
},
{
"file": "/examples/federated-learning/tff/distributed-fl-simulation-k8s/distributed-fl-workload-pkg/worker.yaml",
"findings": [
Expand Down
Loading

0 comments on commit 1e5fdb4

Please sign in to comment.