feat(aws-dataprocessing): add Iceberg format detection, Data Quality, table versions, and partition index tools - #4342
Open
dipayanthedata wants to merge 2 commits into
Conversation
added 2 commits
July 24, 2026 08:52
… table versions, and partition index tools - Add detect-table-format and get-iceberg-table-info operations to manage_aws_glue_tables - Add get-table-versions and batch-delete-table-version operations to manage_aws_glue_tables - Add create-partition-index, get-partition-indexes, delete-partition-index operations to manage_aws_glue_partitions - Add new manage_aws_glue_data_quality tool with full CRUD for rulesets, evaluation runs, and results - Wire up previously dead DataQualityResult model - Fix GetPartitionIndexes Keys field parsing (list of dicts, not strings) - Add unit tests for all new operations (MagicMock convention) - Update README with new tools, IAM policies, and usage examples Refs: awslabs#614
Keep name and GitHub handle only, drop employer affiliation before pushing to a public repo.
dipayanthedata
requested review from
a team,
LiyuanLD,
ckha2000,
naikvaib,
raghav1397 and
yuxiaorun
as code owners
July 24, 2026 16:26
Contributor
|
This pull request is now marked as stale because it hasn't seen activity for a while. Add a comment or it will be closed soon. If you wish to exclude this issue from being marked as stale, add the "backlog" label. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Related to RFC #4341.
aws-dataprocessing-mcp-serveralready covers the vast majority of the AWS Glue Data Catalog surface area, but four specific gaps were verified to be completely absent from the codebase, tests, README, and git history. This PR closes all four, following the server's existing "mega-tool with anoperationparameter" convention rather than introducing a parallel design:detect-table-formatandget-iceberg-table-infooperations onmanage_aws_glue_tables. Classifies a table as Hive, Iceberg, Delta, or Hudi from its Glue table parameters and storage descriptor (best-effort, falls back toUNKNOWNrather than guessing), and surfaces Iceberg metadata location / format version when applicable.manage_aws_glue_data_qualitytool covering ruleset CRUD, starting/inspecting ruleset evaluation runs, and retrieving evaluation results (single, list, and batch-get). This also wires up theDataQualityResultPydantic model inmodels/data_catalog_models.py, which previously existed but was never used by any handler — the README already advertised "data quality validation" as a feature before this PR, without any implementation behind it.get-table-versionsandbatch-delete-table-versionoperations onmanage_aws_glue_tables.create-partition-index,get-partition-indexes, anddelete-partition-indexoperations onmanage_aws_glue_partitions.Notable fix found during testing
While writing tests for
get-partition-indexes, found thatGetPartitionIndexes'Keysfield in the real Glue API response is a list of{Name, Type}dicts, not plain strings. Fixed the manager code to extract key names correctly (awslabs/aws_dataprocessing_mcp_server/core/glue_data_catalog/data_catalog_handler.py).Testing
--allow-write) cases, using this server's existingunittest.mock.MagicMock/patchconvention (not moto — kept consistent with every other test in this server rather than introducing a second testing style).ruff check .andruff format --check .: clean, no changes needed beyond auto-formatting during development.Compatibility
boto3Glue API calls already reachable from the existingglue_client.manage_aws_glue_tables,manage_aws_glue_partitions) or a new standalone tool (manage_aws_glue_data_quality).batch-delete-table-version,create-partition-index,delete-partition-index, and the Data Quality ruleset/evaluation-run mutations) are gated behind--allow-write, consistent with every other mutating operation in this server.Test plan for reviewers
detect-table-format/get-iceberg-table-infoagainst a real Iceberg table to sanity-check the heuristicBy submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.