Add files via upload

Author: adinan-tech

iceberg-table-migration-sample/README.md

@@ -1 +1,234 @@
+# Iceberg Table Migration to OCI
 
+This repository is a Terraform-based demo for migrating an Apache Iceberg table into OCI Object Storage and validating it with Spark.
+
+Current primary run mode:
+
+```text
+local terminal -> terraform apply -> SSH to OCI VM -> run /opt/iceberg scripts
+```
+
+## What It Does
+
+Terraform provisions:
+
+- OCI network
+- OCI compute VM
+- OCI Object Storage bucket
+- cloud-init bootstrap on the VM
+
+Cloud-init installs and prepares:
+
+- Docker
+- Spark
+- Apache Iceberg runtime
+- PostgreSQL
+- Hive Metastore container
+- Iceberg JDBC catalog
+- OCI CLI and AWS CLI
+- helper scripts under `/opt/iceberg`
+
+The demo scripts then:
+
+1. Generate a real tiny Iceberg table using MinIO as fake AWS S3.
+2. Export real Iceberg `data/` and `metadata/` files.
+3. Copy those files into OCI Object Storage.
+4. Register the copied table in Iceberg JDBC catalog.
+5. Validate with Spark.
+6. Optionally validate with Trino.
+
+## Terraform Options
+
+Optional Trino validation is controlled by the Terraform variable `validation_engines`.
+
+Set it in `terraform.tfvars` before `terraform apply`:
+
+```hcl
+validation_engines = ["spark", "trino"]
+```
+
+Default is Spark only:
+
+```hcl
+validation_engines = ["spark"]
+```
+
+The included `terraform.tfvars` may already set `["spark", "trino"]`; change it to `["spark"]` if you want Spark-only validation.
+
+You can also override it for one VM run:
+
+```bash
+export VALIDATION_ENGINES="spark,trino"
+```
+
+## Configure Terraform Inputs
+
+Populate these values before running Terraform.
+
+In `provider.auto.tfvars`:
+
+- `provider_oci.tenancy_ocid`: tenancy OCID from OCI.
+- `provider_oci.user_ocid`: user OCID for the API key owner.
+- `provider_oci.fingerprint`: fingerprint of the uploaded OCI API key.
+- `provider_oci.private_key_path`: local path to the matching API private key.
+- `provider_oci.private_key_password`: private key passphrase, or `""` if the key has none.
+- `provider_oci.region`: target OCI region, for example `eu-frankfurt-1`.
+- `compartment_ids.sandbox`: compartment OCID where Terraform creates the resources.
+
+In `terraform.tfvars`:
+
+- `linux_images`: Oracle Linux 9 image OCID for the selected `provider_oci.region`.
+- `instance_params`: VM availability domain, shape, subnet, image version, OCPUs, and memory.
+- `ssh_public_key`, which must point to the public key Terraform should inject into the VM.
+- `registry`: OCIR region key, for example `fra.ocir.io` or `iad.ocir.io`.
+- `bucket_params`: Object Storage bucket name, compartment name, storage tier, and optional `force_destroy`.
+- Network maps: adjust CIDRs, subnet privacy, route rules, and security list rules if the defaults do not fit your tenancy.
+- `validation_engines`: use `["spark"]` for Spark only or `["spark", "trino"]` to add Trino validation.
+
+## Quick Run
+
+From this repository:
+
+```bash
+terraform init
+terraform validate
+terraform plan
+terraform apply
+```
+
+When Terraform asks for confirmation, enter `yes`.
+
+After Terraform finishes, note the public IP address from the `linux_instances` output.
+
+## Post-Deploy Actions
+
+SSH to the compute VM:
+
+```bash
+ssh opc@<vm_public_ip>
+```
+
+Confirm cloud-init completed and the helper scripts exist:
+
+```bash
+cloud-init status --wait --long
+docker version
+/opt/spark/bin/spark-submit --version
+ls -l /opt/iceberg/
+```
+
+If the helper scripts are missing, inspect the cloud-init log:
+
+```bash
+sudo tail -n 100 /var/log/cloud-init-output.log
+```
+
+Generate the simulated AWS Iceberg source table:
+
+```bash
+/opt/iceberg/generate-simulated-aws-iceberg-table.sh
+```
+
+The default export location is:
+
+```text
+/opt/iceberg/generated_aws_source/iceberg-table-demo/lakehouse/sales/orders/
+```
+
+Confirm the export contains Iceberg `data/` and `metadata/` files:
+
+```bash
+find /opt/iceberg/generated_aws_source/iceberg-table-demo/lakehouse/sales/orders -type f | sort
+find /opt/iceberg/generated_aws_source/iceberg-table-demo/lakehouse/sales/orders/metadata -name "*.metadata.json" -type f | sort
+```
+
+Copy the generated Iceberg files to OCI Object Storage:
+
+```bash
+/opt/iceberg/copy-simulated-source-to-oci.sh
+```
+
+Verify the copied objects:
+
+```bash
+oci os object list \
+  --auth instance_principal \
+  --bucket-name iceberg-table-demo \
+  --prefix lakehouse/sales/orders/ \
+  --fields name \
+  --all
+```
+
+If you changed the bucket or table prefix, use the same `BUCKET` and `TABLE_PREFIX` values for generation, copy, verification, and registration.
+
+Create an OCI Customer Secret Key for S3-compatible Object Storage access, then set it on the VM. Use the Customer Secret Key access key as `OCI_ACCESS_KEY_ID` and the generated secret value as `OCI_SECRET_ACCESS_KEY`.
+
+```bash
+export OCI_ACCESS_KEY_ID="<access-key>"
+export OCI_SECRET_ACCESS_KEY="<secret-key>"
+```
+
+The VM scripts use the Terraform region by default. Set `OCI_REGION` or `OCI_S3_ENDPOINT` only if you need to override the generated endpoint:
+
+```bash
+export OCI_REGION="<oci_region>"
+export OCI_S3_ENDPOINT="https://<namespace>.compat.objectstorage.<oci_region>.oci.customer-oci.com"
+```
+
+Register the copied table and validate it with Spark:
+
+```bash
+/opt/iceberg/register-simulated-oci-table.sh
+```
+
+If `validation_engines` includes `trino`, the same registration script also runs Trino validation. You can enable it for one run with:
+
+```bash
+export VALIDATION_ENGINES="spark,trino"
+/opt/iceberg/register-simulated-oci-table.sh
+```
+
+To run a manual Spark SQL check:
+
+```bash
+/opt/iceberg/spark-sql-oci.sh
+```
+
+```sql
+SHOW TABLES IN oci.sales;
+DESCRIBE oci.sales.orders;
+SELECT * FROM oci.sales.orders;
+exit;
+```
+
+## Input Modes
+
+The scripts support two source modes:
+
+| Mode | Status | How to use |
+| --- | --- | --- |
+| `simulated_aws` / MinIO | Proven default | Run `generate-simulated-aws-iceberg-table.sh`, then copy/register/validate. |
+| `local_export` | Supported as local copy input | Set `SOURCE_DIR` to an existing local Iceberg table export, then run the copy/register flow. |
+
+Inputs must be generated by Iceberg or come from a real exported Iceberg table folder.
+
+### Local Export
+
+The local export folder should contain real Iceberg files:
+
+```text
+/path/to/exported/iceberg/table/
+  data/
+  metadata/
+    *.metadata.json
+    *.avro
+```
+
+Copy it to OCI with:
+
+```bash
+SOURCE_DIR=/path/to/exported/iceberg/table \
+BUCKET=iceberg-table-demo \
+TABLE_PREFIX=lakehouse/sales/orders \
+/opt/iceberg/copy-simulated-source-to-oci.sh
+```

iceberg-table-migration-sample/main.tf

@@ -0,0 +1,41 @@
+# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved.
+# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/
+
+provider "oci" {
+  tenancy_ocid     = var.provider_oci.tenancy_ocid
+  user_ocid        = var.provider_oci.user_ocid
+  fingerprint      = var.provider_oci.fingerprint
+  private_key_path = var.provider_oci.private_key_path
+  region           = var.provider_oci.region
+}
+
+module "network" {
+  source          = "./modules/network"
+  vcn_params      = var.vcn_params
+  compartment_ids = var.compartment_ids
+  igw_params      = var.igw_params
+  ngw_params      = var.ngw_params
+  rt_params       = var.rt_params
+  sl_params       = var.sl_params
+  subnet_params   = var.subnet_params
+}
+
+module "compute" {
+  source             = "./modules/instances"
+  compartment_ids    = var.compartment_ids
+  subnet_ids         = module.network.subnets_ids
+  instance_params    = var.instance_params
+  region             = var.provider_oci.region
+  linux_images       = var.linux_images
+  ssh_public_key     = var.ssh_public_key
+  registry           = var.registry
+  validation_engines = var.validation_engines
+}
+
+module "object-storage" {
+  source        = "./modules/object-storage"
+  compartments  = var.compartment_ids
+  bucket_params = var.bucket_params
+  oci_provider  = var.provider_oci
+  kms_key_ids   = var.kms_key_ids
+}

iceberg-table-migration-sample/modules/instances/main.tf

@@ -0,0 +1,54 @@
+# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved.
+# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/
+
+locals {
+  linux_boot_volumes = [for instance in var.instance_params : instance.boot_volume_size >= 50 ? null : file(format("\n\nERROR: The boot volume size for linux instance %s is less than 50GB which is not permitted. Please add a boot volume size of 50GB or more", instance.hostname))]
+  block_performance = {
+    Low      = "0"
+    Balanced = "10"
+    High     = "20"
+  }
+}
+
+data "oci_identity_availability_domains" "ads" {
+  compartment_id = var.compartment_ids[var.instance_params[keys(var.instance_params)[0]].compartment_name]
+}
+
+
+resource "oci_core_instance" "this" {
+  for_each            = var.instance_params
+  availability_domain = data.oci_identity_availability_domains.ads.availability_domains[each.value.ad - 1].name
+  compartment_id      = var.compartment_ids[each.value.compartment_name]
+  shape               = each.value.shape
+  shape_config {
+    memory_in_gbs = each.value.memory_in_gbs
+    ocpus         = each.value.ocpus
+  }
+  display_name                        = each.value.hostname
+  preserve_boot_volume                = each.value.preserve_boot_volume
+  freeform_tags                       = each.value.freeform_tags
+  is_pv_encryption_in_transit_enabled = each.value.encrypt_in_transit
+  fault_domain                        = format("FAULT-DOMAIN-%s", each.value.fd)
+
+
+  create_vnic_details {
+    assign_public_ip = each.value.assign_public_ip
+    subnet_id        = var.subnet_ids[each.value.subnet_name]
+    hostname_label   = replace(lower(each.value.hostname), "_", "-")
+  }
+
+  source_details {
+    boot_volume_size_in_gbs = each.value.boot_volume_size
+    source_type             = "image"
+    source_id               = var.linux_images[var.region][each.value.image_version]
+  }
+
+  metadata = {
+    ssh_authorized_keys = file(var.ssh_public_key)
+    user_data = base64encode(templatefile("${path.module}/../../userdata/cloudinit.sh.tftpl", {
+      registry           = var.registry
+      region             = var.region
+      validation_engines = join(",", var.validation_engines)
+    }))
+  }
+}

iceberg-table-migration-sample/modules/instances/outputs.tf

@@ -0,0 +1,34 @@
+# Copyright (c) 2024, 2026, Oracle and/or its affiliates. All rights reserved.
+# The Universal Permissive License (UPL), Version 1.0 as shown at https://oss.oracle.com/licenses/upl/
+
+locals {
+  linux_instances = {
+    for instance in oci_core_instance.this :
+    instance.display_name => { "id" : instance.id, "ip" : instance.public_ip != "" ? instance.public_ip : instance.private_ip }
+  }
+  linux_ids = {
+    for instance in oci_core_instance.this :
+    instance.display_name => instance.id
+  }
+
+  linux_private_ips = {
+    for instance in oci_core_instance.this :
+    instance.display_name => instance.private_ip
+  }
+
+
+  all_instances   = merge(local.linux_ids /*,local.windows_ids*/)
+  all_private_ips = merge(local.linux_private_ips /*, local.windows_private_ips*/)
+}
+
+output "linux_instances" {
+  value = local.linux_instances
+}
+
+output "all_instances" {
+  value = local.all_instances
+}
+
+output "all_private_ips" {
+  value = local.all_private_ips
+}