Added comprehensive Kerberos authentication support for Spark 3.1+

[josecsotomorales]

This commit introduces native Kerberos authentication support for secure Hadoop cluster access, optimized for Apache Spark 3.1+ and Kubernetes deployments.

Key Features

• Native Kerberos Integration: Full support for keytab-based authentication
• Spark 3.1+ Optimized: Leverages latest Spark security features including automatic credential renewal and delegation token management
• Service-Specific Control: Configurable Kerberos credentials for HDFS, Hive, HBase, and other Hadoop ecosystem services
• Kubernetes Native: Seamless secret mounting and environment configuration

API Changes

• Add KerberosSpec to SparkApplicationSpec with comprehensive configuration
• New SecretType.KerberosKeytab for automatic keytab file handling
• Support for both keytab and ccache credential renewal strategies
• Configurable service enablement (hadoopfs, hbase, hive, yarn)

Implementation Details

• Automatic Spark/Hadoop configuration generation
• Environment variable setup (KRB5_KEYTAB_FILE, KRB5_CONFIG)
• JVM parameter configuration for driver and executors
• Delegation token retrieval for long-running applications
• HDFS access configuration when Hadoop config is present

Configuration Example

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
spec:
  kerberos:
    principal: "[email protected]"
    keytabSecret: "spark-kerberos-keytab"
    configSecret: "spark-kerberos-config"
    renewalCredentials: "keytab"
    enabledServices: ["hadoopfs", "hbase", "hive"]

Files Changed

• API: Add KerberosSpec and enhance secret types
• Controller: Implement Kerberos configuration logic
• Helm: Add Kerberos configuration options to values.yaml
• CRDs: Generate updated CustomResourceDefinitions
• Docs: Comprehensive documentation and troubleshooting guide
• Examples: Working SparkApplication example with Kerberos
• Tests: Unit tests for Kerberos configuration logic

[josecsotomorales]

@ChenYi015 I think this PR is ready for review! Appreciate any feedback 🚀

[ChenYi015]

Thanks for the great contribution! I have left some comments.

[ChenYi015]

Does it affect .spec.[driver|executor].javaOptions or spark.driver.extraJavaOptions defined in .spec.sparkConf? Since this property can be passed to spark-submit multiple times.

[ChenYi015]

/assign @vara-bonthu @nabuskey @jacobsalway

[ChenYi015]

@josecsotomorales Does this feature also work for Spark 3?

[josecsotomorales]

@josecsotomorales Does this feature also work for Spark 3?

Yes, Kerberos support was added in Spark 3.0.0, and Keytab auth is supported since 3.1.0:
https://issues.apache.org/jira/browse/SPARK-12312?jql=project%20%3D%20SPARK%20AND%20text%20~%20Kerberos

But this PR is only tested with Spark 4.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from jacobsalway. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[josecsotomorales]

Here's the doc for adding it to the Kubeflow website:

Kerberos Authentication Support

The Spark Operator now supports Kerberos authentication for secure access to Hadoop clusters, optimized for Apache Spark. This enables Spark applications to authenticate with Kerberos-enabled services such as HDFS, Hive, HBase, and other components in a secure Hadoop ecosystem using the latest Spark security features.

Overview

Kerberos is a network authentication protocol designed to provide strong authentication for client/server applications. In Hadoop clusters, Kerberos is commonly used to secure access to:

• HDFS (Hadoop Distributed File System)
• Hive Metastore
• HBase
• YARN Resource Manager
• Other Hadoop ecosystem components

The Spark Operator's Kerberos support automates the configuration of Kerberos authentication for Spark applications, making it easier to run secure Spark jobs.

Prerequisites

Before using Kerberos authentication with Spark Operator, ensure:

1. Kerberos Infrastructure: A working Kerberos KDC (Key Distribution Center) is available
2. Keytab Files: Service keytab files are available for the Spark application principal
3. Kerberos Configuration: A properly configured krb5.conf file
4. Hadoop Cluster: The target Hadoop cluster is configured for Kerberos authentication
5. Network Connectivity: Spark pods can reach the Kerberos KDC and Hadoop services

Configuration

1. Create Kerberos Secrets

First, create Kubernetes secrets containing your keytab and Kerberos configuration files:

# Create secret with keytab file
kubectl create secret generic spark-kerberos-keytab \
  --from-file=krb5.keytab=/path/to/your/spark.keytab

# Create secret with Kerberos configuration
kubectl create secret generic spark-kerberos-config \
  --from-file=krb5.conf=/path/to/your/krb5.conf

2. Configure SparkApplication

Add the Kerberos configuration to your SparkApplication specification:

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: spark-pi-kerberos
spec:
  # ... other configuration ...
  
  # Kerberos authentication configuration
  kerberos:
    principal: "[email protected]"          # Kerberos principal
    realm: "EXAMPLE.COM"                    # Kerberos realm (optional)
    kdc: "kdc.example.com:88"              # KDC address (optional)
    keytabSecret: "spark-kerberos-keytab"  # Secret containing keytab
    configSecret: "spark-kerberos-config"  # Secret containing krb5.conf
    
  # Hadoop configuration for Kerberos
  hadoopConf:
    "hadoop.security.authentication": "kerberos"
    "hadoop.security.authorization": "true"
    # Add other Hadoop-specific configuration as needed
    
  driver:
    # Mount Kerberos secrets
    secrets:
      - name: "spark-kerberos-keytab"
        path: "/etc/kerberos/keytab"
        secretType: "KerberosKeytab"
      - name: "spark-kerberos-config"
        path: "/etc/kerberos/conf"
        secretType: "Generic"
        
  executor:
    # Mount Kerberos secrets
    secrets:
      - name: "spark-kerberos-keytab"
        path: "/etc/kerberos/keytab"
        secretType: "KerberosKeytab"
      - name: "spark-kerberos-config"
        path: "/etc/kerberos/conf"
        secretType: "Generic"

Configuration Options

KerberosSpec Fields

Field	Type	Description	Required
principal	string	Kerberos principal name (e.g., [email protected])	Yes
realm	string	Kerberos realm (can be inferred from principal)	No
kdc	string	Key Distribution Center address	No
keytabSecret	string	Name of secret containing keytab file	Yes
keytabFile	string	Filename within keytab secret (default: krb5.keytab)	No
configSecret	string	Name of secret containing krb5.conf	Yes
configFile	string	Filename within config secret (default: krb5.conf)	No
renewalCredentials	string	Credential renewal strategy: keytab (default) or ccache	No
enabledServices	[]string	Services with Kerberos enabled (default: ["hadoopfs", "hbase", "hive"])	No

SecretType

The operator supports a new secret type KerberosKeytab which automatically:

• Sets the KRB5_KEYTAB_FILE environment variable
• Points to the correct keytab file location

Environment Variables

The Kerberos implementation uses standard Kerberos environment variables:

• KRB5_KEYTAB_FILE: Path to the keytab file
• KRB5_CONFIG: Path to the Kerberos configuration file

These are automatically configured when using the KerberosKeytab secret type.

Spark Configuration

Automatic Configuration

The Kerberos configuration automatically adds the following Spark/Hadoop configurations:

# Hadoop-level configuration
spark.hadoop.hadoop.security.authentication=kerberos
spark.hadoop.hadoop.security.authorization=true
spark.hadoop.hadoop.kerberos.principal=<principal>
spark.hadoop.hadoop.kerberos.keytab=<keytab-path>
spark.hadoop.java.security.krb5.conf=<config-path>

# Spark-level Kerberos configuration  
spark.kerberos.principal=<principal>
spark.kerberos.keytab=<keytab-path>
spark.kerberos.renewal.credentials=keytab

# Service credentials (automatically enabled)
spark.security.credentials.hadoopfs.enabled=true
spark.security.credentials.hbase.enabled=true
spark.security.credentials.hive.enabled=true

# Kerberos access configuration
spark.kerberos.access.hadoopFileSystems=hdfs

# JVM configuration for krb5.conf
spark.driver.extraJavaOptions=-Djava.security.krb5.conf=<config-path>
spark.executor.extraJavaOptions=-Djava.security.krb5.conf=<config-path>

Integration with Existing Spark Configuration

JavaOptions Interaction:
When using Kerberos configuration, the operator automatically adds Kerberos-related JVM options to spark.driver.extraJavaOptions and spark.executor.extraJavaOptions.

⚠️ Important: If you specify javaOptions in your driver or executor specification, those options will completely override the Kerberos JVM options, potentially breaking Kerberos authentication. To use both Kerberos and custom JVM options, you have two options:

Option 1: Use sparkConf instead of javaOptions (Recommended)

spec:
  sparkConf:
    # Your custom JVM options - will be merged with Kerberos options
    "spark.driver.extraJavaOptions": "-XX:+UseG1GC -Xms1g"
    "spark.executor.extraJavaOptions": "-XX:+UseG1GC -Xms512m"
  kerberos:
    # Kerberos configuration

Option 2: Include Kerberos options in your javaOptions

spec:
  driver:
    # Include both your options AND the Kerberos krb5.conf option
    javaOptions: "-XX:+UseG1GC -Xms1g -Djava.security.krb5.conf=/etc/krb5-config/krb5.conf"
  executor:
    javaOptions: "-XX:+UseG1GC -Xms512m -Djava.security.krb5.conf=/etc/krb5-config/krb5.conf"
  kerberos:
    # Other Kerberos configuration

SparkConf Interaction:
If you manually specify spark.driver.extraJavaOptions or spark.executor.extraJavaOptions in your sparkConf, the operator will append the Kerberos JVM options to those values, ensuring both your custom options and Kerberos authentication work together.

Credential Renewal Strategies

Keytab-based Renewal (Recommended):

kerberos:
  renewalCredentials: "keytab"  # Default for long-running applications

Ticket Cache Renewal:

kerberos:
  renewalCredentials: "ccache"  # Requires external ticket management

Service-Specific Configuration

Control which services have Kerberos credentials enabled:

kerberos:
  enabledServices: ["hadoopfs", "hbase", "hive", "yarn"]

Troubleshooting

Common Issues

1. Authentication Failures

   • Verify keytab file is valid: klist -k /path/to/keytab
   • Check principal name matches exactly
   • Ensure clocks are synchronized between Spark pods and KDC

2. Network Connectivity

   • Test KDC connectivity: telnet kdc.example.com 88
   • Check DNS resolution for KDC and Hadoop services
   • Verify firewall rules allow Kerberos traffic

3. Configuration Issues

   • Validate krb5.conf syntax and realm configuration
   • Check Hadoop service configurations match SparkApplication
   • Verify secret names and paths are correct

Debug Commands

# Check if secrets are properly mounted
kubectl exec -it <spark-driver-pod> -- ls -la /etc/kerberos/keytab/
kubectl exec -it <spark-driver-pod> -- ls -la /etc/kerberos/conf/

# Test Kerberos authentication
kubectl exec -it <spark-driver-pod> -- kinit -kt /etc/kerberos/keytab/krb5.keytab [email protected]

# View current Kerberos tickets
kubectl exec -it <spark-driver-pod> -- klist

Examples

See examples/spark-pi-kerberos.yaml for a complete example of a SparkApplication with Kerberos authentication configured for secure Hadoop access.

Security Considerations

1. Secret Management: Store keytab files securely using Kubernetes secrets
2. RBAC: Ensure proper RBAC is configured to limit access to Kerberos secrets
3. Network Security: Use network policies to control traffic to/from Kerberos services
4. Key Rotation: Implement processes for regular keytab renewal and rotation
5. Audit Logging: Enable audit logging for Kerberos authentication events

Migration from Manual Configuration

If you were previously configuring Kerberos manually via spark configuration, you can migrate to the new native support:

Before (Manual Configuration)

sparkConf:
  "spark.hadoop.hadoop.security.authentication": "kerberos"
  "spark.hadoop.hadoop.kerberos.principal": "[email protected]"
  "spark.hadoop.hadoop.kerberos.keytab": "/etc/secrets/keytab"
# ... manual secret mounting ...

After (Native Support)

kerberos:
  principal: "[email protected]"
  keytabSecret: "spark-kerberos-keytab"
  configSecret: "spark-kerberos-config"

The native support automatically handles the underlying Spark and Hadoop configurations, secret mounting, and environment variable setup.

Content for Kubeflow Website Kerberos Documentation

This file contains the content that should be added to the Kubeflow website documentation after the Kerberos support PR is merged. This content is based on docs/kerberos-support.md but streamlined for the Kubeflow website format.

Kerberos Authentication Support

The Spark Operator supports Kerberos authentication for secure access to Hadoop clusters and services. This guide explains how to configure Kerberos authentication for Spark applications.

Prerequisites

Before using Kerberos authentication:

1. Kerberos Infrastructure: A working Kerberos KDC (Key Distribution Center)
2. Keytab Files: Service keytab files for the Spark application principal
3. Kerberos Configuration: A properly configured krb5.conf file
4. Hadoop Cluster: The target Hadoop cluster configured for Kerberos authentication
5. Network Connectivity: Spark pods can reach the Kerberos KDC and Hadoop services

Configuration

1. Create Kerberos Secrets

Create Kubernetes secrets containing your keytab and Kerberos configuration:

# Create secret with keytab file
kubectl create secret generic spark-kerberos-keytab \
  --from-file=krb5.keytab=/path/to/your/spark.keytab

# Create secret with Kerberos configuration
kubectl create secret generic spark-kerberos-config \
  --from-file=krb5.conf=/path/to/your/krb5.conf

2. Configure SparkApplication

Add Kerberos configuration to your SparkApplication:

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: spark-kerberos-app
  namespace: default
spec:
  sparkVersion: "4.0.0"
  type: Scala
  mode: cluster
  image: "apache/spark:4.0.0"
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-4.0.0.jar"
  
  # Kerberos configuration
  kerberos:
    principal: "[email protected]"
    keytabSecret: "spark-kerberos-keytab"
    keytabFile: "krb5.keytab"
    configSecret: "spark-kerberos-config" 
    configFile: "krb5.conf"
    renewalCredentials: "keytab"
    enabledServices: ["hadoopfs", "hive", "hbase"]
  
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: "4.0.0"
    serviceAccount: spark-operator-spark
    
  executor:
    cores: 1
    instances: 2
    memory: "512m"
    labels:
      version: "4.0.0"

Configuration Options

Core Kerberos Settings

• principal: Kerberos principal name (e.g., "[email protected]")
• keytabSecret: Name of the secret containing the keytab file
• keytabFile: Filename within the keytab secret (default: "krb5.keytab")
• configSecret: Name of the secret containing krb5.conf
• configFile: Filename within the config secret (default: "krb5.conf")

Advanced Options

• renewalCredentials: Credential renewal strategy ("keytab" or "ccache", default: "keytab")
• enabledServices: List of Hadoop services to enable credentials for (default: ["hadoopfs", "hbase", "hive"])
• realm: Kerberos realm (optional, can be inferred from principal)
• kdc: Key Distribution Center address (optional)

JavaOptions Integration

⚠️ Important: If you use custom javaOptions in driver/executor specifications, they will override Kerberos JVM options. Use sparkConf instead:

spec:
  sparkConf:
    "spark.driver.extraJavaOptions": "-XX:+UseG1GC -Xms1g"
    "spark.executor.extraJavaOptions": "-XX:+UseG1GC -Xms512m"
  kerberos:
    # Kerberos configuration will be merged with the above options

Troubleshooting

1. Authentication Failures

   • Verify keytab: klist -k /path/to/keytab
   • Check principal name matches exactly
   • Ensure clock synchronization between pods and KDC

2. Network Connectivity

   • Test KDC access: telnet kdc.example.com 88
   • Verify DNS resolution
   • Check firewall rules

3. Log Analysis

   • Enable debug logging: Add -Dsun.security.krb5.debug=true to sparkConf
   • Check driver/executor logs for Kerberos errors
   • Verify credential renewal is working

For more examples and detailed configuration, see the spark-operator examples.

[nabuskey]

Some questions:

1. Without this change, is it possible to use Kerberos authentication with the operator?
2. If so, how much of a pain is it to configure it?
3. Are there other authentication methods we may need to support in the future?

It's been a long time since I've worked with Kerberos so it's going to take me some time to review this. And it concerns me in that we are adding support for critical things (like auth) we may not be able to support well in the future.

[josecsotomorales]

Some questions:

1. Without this change, is it possible to use Kerberos authentication with the operator?
2. If so, how much of a pain is it to configure it?
3. Are there other authentication methods we may need to support in the future?

It's been a long time since I've worked with Kerberos so it's going to take me some time to review this. And it concerns me in that we are adding support for critical things (like auth) we may not be able to support well in the future.

Kerberos today: You can make Kerberos work with the operator today by hand (mounting keytabs/krb5, injecting Spark/Hadoop confs), but it’s brittle and undocumented. Multiple issues report failures or complexity around token acquisition, renewal, and path wiring. This PR makes that path first-class in the CRD and examples, so users aren’t reinventing it each time.

How painful is it without this? Fairly high—lots of bespoke YAML, custom images, init scripts, and easy foot-guns around delegation tokens. Upstream Spark already defines the security model on K8s; we’re just wiring to it in a supported way. 

Other auth methods we might need? For object stores, users rely on cloud identities (IRSA, Workload Identity, etc.), which are orthogonal to Kerberos and already work with Spark on K8s. For HiveServer2, LDAP/SIMPLE/PAM exist but are configured at the Hive level; the operator just passes through configs. So adding Kerberos here doesn’t preclude future methods.   

Maintenance risk: The feature is opt-in and leverages Spark’s built-in token/renewal mechanisms. The operator’s surface is limited to CRD fields + secret mounting + conf plumbing, keeping ongoing support modest.

@nabuskey here's the official Spark docs for Kerberos: https://spark.apache.org/docs/latest/security.html#kerberos