[#11263] feat(authz): Extend credential vending to Hive/Iceberg/Glue/JDBC catalog types by diqiu50 · Pull Request #11264 · apache/gravitino

diqiu50 · 2026-05-27T13:30:51Z

What changes were proposed in this pull request?

Extend credential vending to Glue, Hive, Iceberg and JDBC catalogs
Refactor credential injection into BaseCatalog.addCatalogSpecificCredentialProviders() hook to eliminate duplication
Extract shared S3/OSS/GCS/Azure PropertyEntry definitions to core module
Fix JdbcCredentialProvider to allow empty-string JDBC passwords
Fix DynamicIcebergConfigProvider standalone mode to include hidden JDBC credentials

Why are the changes needed?

Fix: #11263

Does this PR introduce any user-facing change?

Glue, Hive, Iceberg, and Paimon catalogs now support credential vending.

How was this patch tested?

Add unit tests and integration tests.

…, Doris and StarRocks JDBC catalogs

Copilot

Pull request overview

Extends JDBC credential vending (introduced for MySQL in #11149) to PostgreSQL, Doris, and StarRocks JDBC catalogs, and relaxes the credential provider to permit empty-string passwords so StarRocks (which uses an empty password by default) works.

Changes:

Loosen the password check in JdbcCredentialProvider and JdbcCatalog.applyDefaultCredentialProviders to allow non-null empty passwords; add a Trino PostgreSQL connector adapter test and Spark PG IT explicit provider config.
Add new catalog credential ITs for PostgreSQL, Doris, and StarRocks (plus a unit test for the empty-password path in TestJdbcCatalogCredential).
Add the PG_CATALOG_CREDENTIAL_IT test-database enum entry (lowercased for PostgreSQL).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
core/src/main/java/org/apache/gravitino/credential/JdbcCredentialProvider.java	Allow empty-string password (only reject null).
catalogs/catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/JdbcCatalog.java	Register default JDBC credential provider when password is non-null (not non-blank).
catalogs/catalog-jdbc-common/src/test/java/org/apache/gravitino/catalog/jdbc/TestJdbcCatalogCredential.java	Add unit test asserting provider registration with empty password.
catalogs/catalog-jdbc-postgresql/.../CatalogPostgreSqlCredentialIT.java	New PostgreSQL credential vending IT.
catalogs/catalog-jdbc-doris/.../CatalogDorisCredentialIT.java	New Doris credential vending IT.
catalogs/catalog-jdbc-starrocks/.../CatalogStarRocksCredentialIT.java	New StarRocks credential vending IT (exercises empty password).
integration-test-common/.../TestDatabaseName.java	Add `PG_CATALOG_CREDENTIAL_IT` enum entry with lowercase `toString()`.
spark-connector/.../SparkJdbcPostgreSqlCatalogIT.java	Explicitly enable `jdbc-user-password` credential provider for Spark PG IT.
trino-connector/.../TestPostgreSQLConnectorAdapter.java	New unit test verifying vended credentials override Trino PG connector config and fallback when none.

…ns to core module Move S3/OSS/Azure/GCS PropertiesMetadata classes to org.apache.gravitino.cloud.storage in core, so Hive and Iceberg catalog property metadata can reuse them via putAll() instead of defining duplicate inline entries.

…e methods Introduce addCatalogSpecificCredentialProviders() and hiddenCredentialKeys() hooks in BaseCatalog so that propertiesWithCredentialProviders() and properties() backfill are no longer duplicated across Hive, Iceberg, and Jdbc catalogs.

…ckfill Replace the explicit hiddenCredentialKeys() hook with a loop over entity properties that checks catalogPropertiesMetadata().isHiddenProperty(), eliminating duplicate key lists in each catalog subclass.

…ckfill Backfill should only re-add credential properties, not all hidden properties (e.g. PROPERTY_IN_USE is hidden but must not be backfilled). Restore the hiddenCredentialKeys() hook so each catalog declares its own credential keys.

…) and propertiesWithCredentialProviders() When backfill is enabled, properties() starts from the filtered public map and fills in any missing keys from propertiesWithCredentialProviders(), which carries the raw entity properties including hidden credentials. This removes the need for the explicit hiddenCredentialKeys() hook in each catalog subclass.

Delegate properties() directly to propertiesWithCredentialProviders() when backfill is enabled, avoiding redundant merge. Remove temporary design doc that was not intended to be committed.

Mark aws-secret-access-key as hidden. Override addCatalogSpecificCredentialProviders to map Glue AWS credentials to S3 property keys, enabling automatic S3 credential provider registration. Fix NPE in shouldBackfillCredential when config entry returns null.

…f throwing

…ovider and catalogs

…lProviders hook

github-actions · 2026-05-28T14:05:59Z

Code Coverage Report

Overall Project	66.92% `-0.32%`	🟢
Files changed	44.94%	🔴

Module	Coverage
aliyun	1.72%	🔴
api	46.82% `-0.29%`	🟢
authorization-common	85.96%	🟢
aws	3.66%	🔴
azure	2.47%	🔴
catalog-common	10.4%	🔴
catalog-fileset	80.23%	🟢
catalog-glue	66.72% `-0.68%`	🟢
catalog-hive	79.44% `+0.74%`	🟢
catalog-jdbc-clickhouse	80.02%	🟢
catalog-jdbc-common	44.22% `+0.09%`	🟢
catalog-jdbc-doris	80.28%	🟢
catalog-jdbc-hologres	54.03%	🟢
catalog-jdbc-mysql	79.23%	🟢
catalog-jdbc-oceanbase	78.38%	🟢
catalog-jdbc-postgresql	82.29%	🟢
catalog-jdbc-starrocks	78.51%	🟢
catalog-kafka	77.01%	🟢
catalog-lakehouse-generic	58.53%	🟢
catalog-lakehouse-hudi	79.1%	🟢
catalog-lakehouse-iceberg	85.87% `+0.41%`	🟢
catalog-lakehouse-paimon	79.15% `+0.1%`	🟢
catalog-model	77.72%	🟢
cli	44.51%	🟢
client-java	77.91%	🟢
common	49.9% `+0.04%`	🟢
core	82.38% `-0.23%`	🟢
filesystem-hadoop3	77.27%	🟢
flink	0.0%	🔴
flink-common	45.72% `-3.55%`	🟢
flink-runtime	0.0%	🔴
gcp	14.12%	🔴
hadoop-common	10.39%	🔴
hive-metastore-common	53.9%	🟢
iceberg-common	57.41%	🟢
iceberg-rest-server	73.69% `-1.12%`	🟢
idp-basic	86.18%	🟢
integration-test-common	0.0%	🔴
jobs	66.17%	🟢
lance-common	20.81%	🔴
lance-rest-server	60.54%	🟢
lineage	53.02%	🟢
optimizer	82.95%	🟢
optimizer-api	21.95%	🔴
server	85.73%	🟢
server-common	73.28%	🟢
spark	28.57%	🔴
spark-common	41.16% `-2.93%`	🟢
trino-connector	40.13% `+1.03%`	🟢

Files

Module	File	Coverage
api	JdbcCredential.java	0.0%	🔴
catalog-glue	GlueCatalogPropertiesMetadata.java	100.0%	🟢
	GlueCatalog.java	0.0%	🔴
catalog-hive	HiveCatalogPropertiesMetadata.java	100.0%	🟢
catalog-jdbc-common	JdbcCatalog.java	47.83%	🔴
catalog-lakehouse-iceberg	IcebergCatalogPropertiesMetadata.java	97.5%	🟢
	IcebergCatalog.java	87.5%	🟢
catalog-lakehouse-paimon	PaimonCatalog.java	85.0%	🟢
common	CredentialPropertyUtils.java	52.34%	🔴
core	JdbcCredentialProvider.java	90.91%	🟢
	BaseCatalog.java	62.83%	🟢
	AzurePropertiesMetadata.java	0.0%	🔴
	GCSPropertiesMetadata.java	0.0%	🔴
	OSSPropertiesMetadata.java	0.0%	🔴
	S3PropertiesMetadata.java	0.0%	🔴
flink	GravitinoIcebergCatalogFlink118.java	0.0%	🔴
	GravitinoIcebergCatalogFlink119.java	0.0%	🔴
	GravitinoIcebergCatalogFlink120.java	0.0%	🔴
flink-common	GravitinoPaimonCatalog.java	68.32%	🟢
	GravitinoIcebergCatalog.java	13.79%	🔴
	GravitinoHiveCatalog.java	0.0%	🔴
iceberg-rest-server	DynamicIcebergConfigProvider.java	38.84%	🔴
spark-common	GravitinoGlueCatalog.java	38.89%	🔴
	GravitinoHiveCatalog.java	0.0%	🔴
	GravitinoIcebergCatalog.java	0.0%	🔴
	GravitinoPaimonCatalog.java	0.0%	🔴
trino-connector	GlueConnectorAdapter.java	83.87%	🟢
	IcebergConnectorAdapter.java	66.67%	🟢
	IcebergCatalogPropertyConverter.java	64.29%	🟢
	HiveConnectorAdapter.java	53.33%	🔴

…nfigProvider to include hidden JDBC credentials

… usage in DynamicIcebergConfigProvider

…lone mode

…ng credential recovery strategies

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

…o connectors - Add getCredentials/applyIcebergCredentials/applyPaimonCredentials helpers to CredentialPropertyUtils to eliminate duplicated credential injection logic - Flink/Spark Hive: inject S3, OSS, and Azure credentials into catalog config - Flink/Spark Iceberg: inject JDBC and all storage credentials via shared utility - Flink/Spark Paimon: inject JDBC, S3, and OSS credentials via shared utility - Flink Iceberg/Paimon: defer inner catalog creation to open() so credentials can be fetched before initialization - Trino Hive: inject S3 and Azure credentials - Trino Glue: replace broken static property reading with credential vending - Trino Iceberg: inject JDBC, S3, OSS, and Azure credentials - Add WARN logging for unknown credential types and NoSuchCatalogException - Fix stale TestGlueConnectorAdapter test to use S3SecretKeyCredential

…and Paimon catalogs

…al backfill is enabled

mchades · 2026-06-09T02:45:54Z

    Preconditions.checkArgument(StringUtils.isNotBlank(jdbcUser), "JDBC user should not be empty");
-    Preconditions.checkArgument(
-        StringUtils.isNotBlank(jdbcPassword), "JDBC password should not be empty");
+    Preconditions.checkArgument(jdbcPassword != null, "JDBC password should not be null");


JDBC password can be empty string?

Yes，StarRocks required

yuqi1129 · 2026-06-09T03:21:36Z

+        config.put(HIVE_METASTORE_GLUE_SECRET_KEY, s3.secretAccessKey());
+        config.put(HIVE_S3_ACCESS_KEY, s3.accessKeyId());
+        config.put(HIVE_S3_SECRET_KEY, s3.secretAccessKey());
+        return;


Does Glue not support Azure credentials as follows?

S3 is the only supported type?

Glue is typically used on AWS, so it generally works with S3. Other requirements can be support as needed.

yuqi1129 · 2026-06-09T03:26:44Z

+    super.open();
+  }
+
+  static void applyS3Credential(Catalog catalog, Configuration conf) {


This one is almost the same as that in GravitinoHiveCatalog in spark connector.

The logic is similar but the connector APIs differ (Configuration vs Map<String, String>), so they can't share the same method directly.

yuqi1129 · 2026-06-09T03:27:59Z

+   * @return true if backfill is enabled
+   */
+  protected boolean shouldBackfillCredential() {
+    Config serverConfig = GravitinoEnv.getInstance().config();


When will it be null?

remove null check

…/Flink connectors - Add S3SecretKeyProvider ServiceLoader support to catalog-glue via runtimeOnly bundles/aws dep - Override propertiesWithCredentialProviders() in GlueCatalog to always apply aws-* → s3-* key mapping - Mark aws-access-key-id and aws-secret-access-key as hidden in GlueCatalogPropertiesMetadata - Inject vended S3 credentials into GravitinoGlueCatalog for both Hive and Iceberg table paths - Simplify GravitinoIcebergCatalogFlink{118,119,120} by removing redundant createIcebergCatalog helper

…nts in credential classes - Mark aws-access-key-id, s3-access-key-id, oss-access-key-id, azure-storage-account-name as hidden - Extract ICEBERG_JDBC_USER/PASSWORD and PAIMON_S3/OSS key constants in CredentialPropertyUtils - Simplify shouldBackfillCredential() by removing redundant null check on server config

…erge with main

…fillCredential

mchades · 2026-06-09T13:52:17Z

The term "all catalog types" in the PR title may be misleading, as other catalogs (such as Kafka) are not currently supported. right?

…mple auth is default IdpRESTFeature.validateConfiguration() calls System.exit(1) when 'simple' is in gravitino.authenticators. Configure oauth as the base authenticator (matching IdpRESTApiIT) so the server starts cleanly; BasicAuthenticator is registered on top by the IDP plugin.

yuqi1129 · 2026-06-10T03:21:13Z

I have no more comments.

mchades · 2026-06-10T03:40:12Z

@diqiu50 please resolve conflicts

…ending to Hive/Iceberg/Glue/JDBC catalog types (#11264) (#11554) **Cherry-pick Information:** - Original commit: 90c5bbc - Target branch: `branch-1.3` - Status: ✅ Clean cherry-pick (no conflicts) Co-authored-by: Yuhui <hui@datastrato.com>

…end them at the fileset level Hide cloud storage credential properties (S3/OSS/Azure access & secret keys) from the fileset catalog/schema/fileset properties() by reusing the shared hidden PropertyEntry definitions in core, consistent with the JDBC catalog (apache#11149) and Hive/Iceberg/Glue (apache#11264). To keep the "client without credentials, using server-side credentials" flow working after hiding, make fileset-level (path-based) credential vending infer the storage credential provider from static credentials when no explicit credential-providers is set. The detection is extracted into CredentialUtils.getStorageCredentialProviders and reused by BaseCatalog, so it mirrors catalog-level vending: a catalog configured with only static credentials can now vend them at the fileset level. GVFS clients that do not provide credentials must enable credential vending (fs.gravitino.enableCredentialVending=true) to obtain the server-side credentials. When credentials are missing and vending is disabled, GVFS now surfaces an actionable AccessDenied message pointing to the right config.

[apache#11263] feat(authz): Support credential vending for PostgreSQL…

657433e

…, Doris and StarRocks JDBC catalogs

Copilot AI review requested due to automatic review settings May 27, 2026 13:30

Copilot started reviewing on behalf of diqiu50 May 27, 2026 13:31 View session

Remove redundant catalog credential ITs, covered by unit tests

8cf62b9

Copilot AI reviewed May 27, 2026

View reviewed changes

Comment thread core/src/main/java/org/apache/gravitino/credential/JdbcCredentialProvider.java

Comment thread ...apache/gravitino/trino/connector/catalog/jdbc/postgresql/TestPostgreSQLConnectorAdapter.java Outdated

diqiu50 added 14 commits May 27, 2026 21:35

Remove redundant TestPostgreSQLConnectorAdapter

5038385

Remove redundant CREDENTIAL_PROVIDERS from SparkJdbcPostgreSqlCatalogIT

6e1acd5

feat(authz): Support credential vending for Iceberg and Hive catalogs

7623eb3

refactor(authz): Use PropertiesMetadata hidden flag for credential ba…

e890843

…ckfill Replace the explicit hiddenCredentialKeys() hook with a loop over entity properties that checks catalogPropertiesMetadata().isHiddenProperty(), eliminating duplicate key lists in each catalog subclass.

refactor(authz): Simplify credential backfill and remove design doc

1048c25

Delegate properties() directly to propertiesWithCredentialProviders() when backfill is enabled, avoiding redundant merge. Remove temporary design doc that was not intended to be committed.

fix(authz): Return empty credential for blank JDBC password instead o…

a1e713a

…f throwing

fix(authz): Use isBlank to guard empty JDBC password in credential pr…

5f9cdc1

…ovider and catalogs

fix(authz): Allow empty string JDBC password in credential vending

31c3522

refactor(authz): Migrate PaimonCatalog to addCatalogSpecificCredentia…

80f14ca

…lProviders hook

diqiu50 added 4 commits May 29, 2026 14:28

fix(authz): Use propertiesWithCredentialProviders in DynamicIcebergCo…

fe4017f

…nfigProvider to include hidden JDBC credentials

docs(authz): Add comment explaining propertiesWithCredentialProviders…

23e35cc

… usage in DynamicIcebergConfigProvider

fix(authz): Enrich catalog properties with JDBC credentials in standa…

1215ac4

…lone mode

docs(authz): Improve comment in DynamicIcebergConfigProvider explaini…

1de69dc

…ng credential recovery strategies

diqiu50 changed the title ~~[#11263] feat(authz): Support credential vending for PostgreSQL, Doris and StarRocks JDBC catalogs~~ [#11263] feat(authz): Extend credential vending to all catalog types May 29, 2026

mchades added the branch-1.3 Automatically cherry-pick commit to branch-1.3 label Jun 1, 2026

mchades requested a review from Copilot June 1, 2026 06:12

Copilot started reviewing on behalf of mchades June 1, 2026 06:12 View session

Copilot AI reviewed Jun 1, 2026

View reviewed changes

Comment thread core/src/main/java/org/apache/gravitino/connector/BaseCatalog.java

Comment thread core/src/main/java/org/apache/gravitino/cloud/storage/S3PropertiesMetadata.java

diqiu50 added 2 commits June 2, 2026 10:04

Merge origin/main into cv-pg, resolve import conflicts in Flink Hive …

1ef9b0f

…and Paimon catalogs

fix(authz): Ensure PROPERTY_IN_USE default is preserved when credenti…

e088ba1

…al backfill is enabled

diqiu50 self-assigned this Jun 2, 2026

mchades reviewed Jun 9, 2026

View reviewed changes

diqiu50 added 2 commits June 9, 2026 10:49

Fix trino iceberg issues

fee54ad

Merge remote-tracking branch 'origin/main' into cv-pg

d700a9c

yuqi1129 reviewed Jun 9, 2026

View reviewed changes

diqiu50 added 7 commits June 9, 2026 16:41

test(authz): Add unit tests for GravitinoGlueCatalog credential vending

ac6166e

Merge remote-tracking branch 'origin/main' into cv-pg

d14dd97

fix(flink): Remove duplicate open() in GravitinoPaimonCatalog after m…

018a35f

…erge with main

fix(core): Restore null check for GravitinoEnv.config() in shouldBack…

2f19361

…fillCredential

Merge remote-tracking branch 'upstream/main' into cv-pg

cc3034c

diqiu50 changed the title ~~[#11263] feat(authz): Extend credential vending to all catalog types~~ [#11263] feat(authz): Extend credential vending to Hive/Iceberg/Glue/JDBC catalog types Jun 10, 2026

Merge remote-tracking branch 'upstream/main' into cv-pg

d26c2d3

mchades approved these changes Jun 10, 2026

View reviewed changes

mchades merged commit 90c5bbc into apache:main Jun 10, 2026
31 checks passed

yuqi1129 mentioned this pull request Jun 16, 2026

[#11642] improvement(fileset): Hide credential properties from fileset catalog properties #11674

Open

diqiu50 deleted the cv-pg branch June 18, 2026 06:57

Uh oh!

Conversation

diqiu50 commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

diqiu50 Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mchades commented Jun 9, 2026

Uh oh!

yuqi1129 commented Jun 10, 2026

Uh oh!

mchades commented Jun 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

diqiu50 commented May 27, 2026 •

edited

Loading

github-actions Bot commented May 28, 2026 •

edited

Loading

diqiu50 Jun 9, 2026 •

edited

Loading