Skip to content

Add driver information to the application status #538

@razvan

Description

@razvan

Description

Tools (most notably Airflow) that manage the lifecycle of SparkApplications need to be able to monitor the logs of the application driver.

Currently there is no reference to the driver Pod from the application, which makes the task from above more cumbersome than it should be.

The operator should include such a reference in the SparkApplication status, to enable tools like Airflow's Kubernetes sensor to work out of the box.

Investigate if it's possible to persist logs from k8s executors and spark pods and update the the documentation page accordingly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions