[Bug]

[xc-jianghan]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

WARN AtlasLineageDispatcher: Send lineage to atlas failed. java.lang.IllegalStateException: Shutdown in progress, cannot add a shutdownHook

Affects Version(s)

master

Kyuubi Server Log Output

Kyuubi Engine Log Output

Kyuubi Server Configurations

Kyuubi Engine Configurations

Additional context

Hello, I’m a student studying big data, and recently I’ve been researching the integration of Apache Atlas and Spark lineage. I noticed that the Kyuubi extension you contributed can send Spark lineage to Atlas. However, in my usage, when I execute a single SQL statement using spark-sql -e, I encounter the following error:

WARN AtlasLineageDispatcher: Send lineage to atlas failed. java.lang.IllegalStateException: Shutdown in progress, cannot add a shutdownHook

As a result, the lineage data cannot be uploaded to Atlas. But if I run multiple SQL statements, it seems to “warm up” the Atlas client, and the error does not occur. Do you know how I could optimize this so that lineage can be sent successfully even when running only a single SQL statement?

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix. #7322
• No. I cannot submit a PR at this time.

[github-actions]

Hello @xc-jianghan,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.

[youlibest]

I have encountered this issue as well. Is there a solution