Skip to content

Commit 6e3e912

Browse files
yessawabyessawab
authored
Feat: Adding two new modules for exfiltration protection with and without Private link. And examples using this modules (databricks#33)
* docs:Updating main Readme * docs:Updating main Readme * docs:Updating main Readme * docs:Updating main Readme * feat:Push the terraform plan as a Pull request comment in ADO * feat:Push the terraform plan as a Pull request comment in ADO * feat:Push the terraform plan as a Pull request comment in ADO * feat:Push the terraform plan as a Pull request comment in ADO * feat:Push the terraform plan as a Pull request comment in ADO * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Feat: Adding a new module to provision Databricks on Azure with Private Link - Standard deployment. And an example using this module * Fix: Removing route_table_id variable as it is not used * Feat: Adding a new module for exfiltration protection. And an example using this module * Feat: Adding a new module for exfiltration protection. And an example using this module * Feat: Adding a new module for exfiltration protection. And an example using this module * Feat: Adding a new module for exfiltration protection with Azure private link. And an example using this module * Feat: Adding a new module for exfiltration protection with Azure private link. And an example using this module
1 parent c7c7ad6 commit 6e3e912

File tree

31 files changed

+392
-169
lines changed

31 files changed

+392
-169
lines changed

‎README.md

+3-1
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ The folder `examples` contains the following Terraform implementation examples :
3737
| Azure | [adb-external-hive-metastore](examples/adb-external-hive-metastore/) | Example template to implement [external hive metastore](https://learn.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore) |
3838
| Azure | [adb-kafka](examples/adb-kafka/) | ADB - single node kafka template |
3939
| Azure | [adb-private-links](examples/adb-private-links/) | Azure Databricks Private Links |
40-
| Azure | [adb-private-links-general](examples/adb-private-links-general/) | Azure Databricks Private Links General Implementation |
40+
| Azure | [adb-private-links-general](examples/adb-private-links-general/) | Provisioning Databricks on Azure with Private Link and [Data Exfiltration Protection](https://www.databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html) |
4141
| Azure | [adb-splunk](examples/adb-splunk/) | ADB workspace with single VM splunk integration |
4242
| Azure | [adb-squid-proxy](examples/adb-squid-proxy/) | ADB clusters with HTTP proxy |
4343
| Azure | [adb-teradata](examples/adb-teradata/) | ADB with single VM Teradata integration |
@@ -59,6 +59,8 @@ The folder `modules` contains the following Terraform modules :
5959
| All | [databricks-department-clusters](modules/databricks-department-clusters/) | Terraform module that creates Databricks resources for a team |
6060
| Azure | [adb-lakehouse](modules/adb-lakehouse/) | Lakehouse terraform blueprints |
6161
| Azure | [adb-with-private-link-standard](modules/adb-with-private-link-standard/) | Provisioning Databricks on Azure with Private Link - Standard deployment |
62+
| Azure | [adb-exfiltration-protection](modules/adb-exfiltration-protection/) | A sample implementation of [Data Exfiltration Protection](https://www.databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html) |
63+
| Azure | [adb-with-private-links-exfiltration-protection](modules/adb-with-private-links-exfiltration-protection/) | Provisioning Databricks on Azure with Private Link and [Data Exfiltration Protection](https://www.databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html) |
6264
| AWS | Coming soon | |
6365
| GCP | Coming soon | |
6466

‎examples/adb-exfiltration-protection/.terraform.lock.hcl

-98
This file was deleted.

‎examples/adb-exfiltration-protection/README.md

+8-5
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# Provisioning Azure Databricks workspace with a Hub & Spoke firewall for data exfiltration protection
22

3+
This example is using the [adb-exfiltration-protection](../../modules/adb-exfiltration-protection) module.
4+
35
This template provides an example deployment of: Hub-Spoke networking with egress firewall to control all outbound traffic from Databricks subnets. Details are described in: https://databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html
46

57
With this setup, you can setup firewall rules to block / allow egress traffic from your Databricks clusters. You can also use firewall to block all access to storage accounts, and use private endpoint connection to bypass this firewall, such that you allow access only to specific storage accounts.
@@ -18,11 +20,12 @@ Resources to be created:
1820
* Associated firewall rules, both FQDN and network rule using IP.
1921

2022

21-
## Getting Started
22-
1. Clone this repo to your local machine running terraform.
23-
2. Run `terraform init` to initialize terraform and get provider ready.
24-
3. Change `terraform.tfvars` values to your own values.
25-
4. Inside the local project folder, run `terraform apply` to create the resources.
23+
## How to use
24+
25+
1. Update `terraform.tfvars` file and provide values to each defined variable
26+
2. (Optional) Configure your [remote backend](https://developer.hashicorp.com/terraform/language/settings/backends/azurerm)
27+
4. Run `terraform init` to initialize terraform and get provider ready.
28+
4. Run `terraform apply` to create the resources.
2629

2730
## How to fill in variable values
2831

‎examples/adb-exfiltration-protection/main.tf

+13-59
Original file line numberDiff line numberDiff line change
@@ -7,63 +7,17 @@
77
* * VNet with public and private subnet
88
* * Databricks workspace
99
*/
10-
provider "azurerm" {
11-
features {}
12-
}
1310

14-
provider "random" {
15-
}
16-
17-
resource "random_string" "naming" {
18-
special = false
19-
upper = false
20-
length = 6
21-
}
22-
23-
data "azurerm_client_config" "current" {
24-
}
25-
26-
data "external" "me" {
27-
program = ["az", "account", "show", "--query", "user"]
28-
}
29-
30-
locals {
31-
// dltp - databricks labs terraform provider
32-
prefix = join("-", [var.workspace_prefix, "${random_string.naming.result}"])
33-
location = var.rglocation
34-
cidr = var.spokecidr
35-
dbfsname = join("", [var.dbfs_prefix, "${random_string.naming.result}"]) // dbfs name must not have special chars
36-
37-
// tags that are propagated down to all resources
38-
tags = {
39-
Environment = "Testing"
40-
Owner = lookup(data.external.me.result, "name")
41-
Epoch = random_string.naming.result
42-
}
43-
}
44-
45-
resource "azurerm_resource_group" "this" {
46-
name = "adb-dev-${local.prefix}-rg"
47-
location = local.location
48-
tags = local.tags
49-
}
50-
51-
output "arm_client_id" {
52-
value = data.azurerm_client_config.current.client_id
53-
}
54-
55-
output "arm_subscription_id" {
56-
value = data.azurerm_client_config.current.subscription_id
57-
}
58-
59-
output "arm_tenant_id" {
60-
value = data.azurerm_client_config.current.tenant_id
61-
}
62-
63-
output "azure_region" {
64-
value = local.location
65-
}
66-
67-
output "resource_group" {
68-
value = azurerm_resource_group.this.name
69-
}
11+
module "adb-exfiltration-protection" {
12+
source = "github.com/databricks/terraform-databricks-examples/modules/adb-exfiltration-protection"
13+
hubcidr = var.hubcidr
14+
spokecidr = var.spokecidr
15+
no_public_ip = var.no_public_ip
16+
rglocation = var.rglocation
17+
metastoreip = var.metastoreip
18+
sccip = var.sccip
19+
webappip = var.webappip
20+
dbfs_prefix = var.dbfs_prefix
21+
workspace_prefix = var.workspace_prefix
22+
firewallfqdn = var.firewallfqdn
23+
}

‎examples/adb-private-links-general/images/adb-private-links-general.png

-996 KB
Binary file not shown.

‎examples/adb-private-links-general/README.md renamed to ‎examples/adb-with-private-links-exfiltration-protection/README.md

+9-6
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# Azure Databricks with Private Links (incl. web-auth PE) and Hub-Spoke Firewall structure (data exfiltration protection).
22

3+
This example is using the [adb-with-private-links-exfiltration-protection](../../modules/adb-with-private-links-exfiltration-protection) module.
4+
35
Include:
46
1. Hub-Spoke networking with egress firewall to control all outbound traffic, e.g. to pypi.org.
57
2. Private Link connection for backend traffic from data plane to control plane.
@@ -8,7 +10,7 @@ Include:
810
5. Private Endpoint for web-auth traffic.
911

1012
Overall Architecture:
11-
![alt text](https://raw.githubusercontent.com/databricks/terraform-databricks-examples/main/examples/adb-private-links-general/images/adb-private-links-general.png?raw=true)
13+
![alt text](https://raw.githubusercontent.com/databricks/terraform-databricks-examples/main/examples/adb-with-private-links-exfiltration-protection/images/adb-private-links-general.png?raw=true)
1214

1315
With this deployment, traffic from user client to webapp (notebook UI), backend traffic from data plane to control plane will be through private endpoints. This terraform sample will create:
1416
* Resource group with random prefix
@@ -17,8 +19,9 @@ With this deployment, traffic from user client to webapp (notebook UI), backend
1719
* Databricks workspace with private link to control plane, user to webapp and private link to dbfs
1820

1921

20-
## Getting Started
21-
1. Clone this repo to your local machine.
22-
2. Run `terraform init` to initialize terraform and get provider ready.
23-
3. Change `terraform.tfvars` values to your own values.
24-
4. Inside the local project folder, run `terraform apply` to create the resources.
22+
## How to use
23+
24+
1. Update `terraform.tfvars` file and provide values to each defined variable
25+
2. (Optional) Configure your [remote backend](https://developer.hashicorp.com/terraform/language/settings/backends/azurerm)
26+
4. Run `terraform init` to initialize terraform and get provider ready.
27+
4. Run `terraform apply` to create the resources.

‎examples/adb-with-private-links-exfiltration-protection/images/adb-private-links-general.png

504 KB
Loading

‎examples/adb-with-private-links-exfiltration-protection/main.tf

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
/**
2+
* Azure Databricks workspace in custom VNet
3+
*
4+
* Module creates:
5+
* * Resource group with random prefix
6+
* * Tags, including `Owner`, which is taken from `az account show --query user`
7+
* * VNet with public and private subnet
8+
* * Databricks workspace
9+
*/
10+
11+
module "adb-with-private-links-exfiltration-protection" {
12+
source = "github.com/databricks/terraform-databricks-examples/modules/adb-with-private-links-exfiltration-protection"
13+
hubcidr = var.hubcidr
14+
spokecidr = var.spokecidr
15+
no_public_ip = var.no_public_ip
16+
rglocation = var.rglocation
17+
metastoreip = var.metastoreip
18+
dbfs_prefix = var.dbfs_prefix
19+
workspace_prefix = var.workspace_prefix
20+
firewallfqdn = var.firewallfqdn
21+
}

‎examples/adb-private-links-general/terraform.tfvars renamed to ‎examples/adb-with-private-links-exfiltration-protection/terraform.tfvars

File renamed without changes.

‎examples/adb-private-links-general/variables.tf renamed to ‎examples/adb-with-private-links-exfiltration-protection/variables.tf

File renamed without changes.

‎examples/adb-with-private-links-exfiltration-protection/versions.tf

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# versions.tf
2+
terraform {
3+
required_providers {
4+
databricks = {
5+
source = "databricks/databricks"
6+
version = ">=0.5.1"
7+
}
8+
9+
azurerm = {
10+
source = "hashicorp/azurerm"
11+
version = ">=2.83.0"
12+
}
13+
}
14+
}

0 commit comments

Comments
 (0)