-
-
Notifications
You must be signed in to change notification settings - Fork 3
Open
Description
Description
Tools (most notably Airflow) that manage the lifecycle of SparkApplications
need to be able to monitor the logs of the application driver.
Currently there is no reference to the driver Pod from the application, which makes the task from above more cumbersome than it should be.
The operator should include such a reference in the SparkApplication status, to enable tools like Airflow's Kubernetes sensor to work out of the box.
Investigate if it's possible to persist logs from k8s executors and spark pods and update the the documentation page accordingly.
Metadata
Metadata
Assignees
Labels
No labels