Skip to content

feat(aws-dataprocessing): add Iceberg format detection, Data Quality, table versions, and partition index tools - #4342

Open
dipayanthedata wants to merge 2 commits into
awslabs:mainfrom
dipayanthedata:feat/aws-glue-catalog-mcp-server
Open

feat(aws-dataprocessing): add Iceberg format detection, Data Quality, table versions, and partition index tools#4342
dipayanthedata wants to merge 2 commits into
awslabs:mainfrom
dipayanthedata:feat/aws-glue-catalog-mcp-server

Conversation

@dipayanthedata

@dipayanthedata dipayanthedata commented Jul 24, 2026

Copy link
Copy Markdown

Summary

Related to RFC #4341.

aws-dataprocessing-mcp-server already covers the vast majority of the AWS Glue Data Catalog surface area, but four specific gaps were verified to be completely absent from the codebase, tests, README, and git history. This PR closes all four, following the server's existing "mega-tool with an operation parameter" convention rather than introducing a parallel design:

  1. Iceberg/table-format detection — new detect-table-format and get-iceberg-table-info operations on manage_aws_glue_tables. Classifies a table as Hive, Iceberg, Delta, or Hudi from its Glue table parameters and storage descriptor (best-effort, falls back to UNKNOWN rather than guessing), and surfaces Iceberg metadata location / format version when applicable.
  2. Glue Data Quality — a new manage_aws_glue_data_quality tool covering ruleset CRUD, starting/inspecting ruleset evaluation runs, and retrieving evaluation results (single, list, and batch-get). This also wires up the DataQualityResult Pydantic model in models/data_catalog_models.py, which previously existed but was never used by any handler — the README already advertised "data quality validation" as a feature before this PR, without any implementation behind it.
  3. Table version history — new get-table-versions and batch-delete-table-version operations on manage_aws_glue_tables.
  4. Partition indexes — new create-partition-index, get-partition-indexes, and delete-partition-index operations on manage_aws_glue_partitions.

Notable fix found during testing

While writing tests for get-partition-indexes, found that GetPartitionIndexes' Keys field in the real Glue API response is a list of {Name, Type} dicts, not plain strings. Fixed the manager code to extract key names correctly (awslabs/aws_dataprocessing_mcp_server/core/glue_data_catalog/data_catalog_handler.py).

Testing

  • Added unit tests for every new operation: happy path, empty results, not-found, and permission-boundary (--allow-write) cases, using this server's existing unittest.mock.MagicMock/patch convention (not moto — kept consistent with every other test in this server rather than introducing a second testing style).
  • Full suite: 1218 passed.
  • ruff check . and ruff format --check .: clean, no changes needed beyond auto-formatting during development.

Compatibility

  • No new dependencies added — everything uses boto3 Glue API calls already reachable from the existing glue_client.
  • No changes to any existing operation's behavior — all new operations are additive to already-shipped tools (manage_aws_glue_tables, manage_aws_glue_partitions) or a new standalone tool (manage_aws_glue_data_quality).
  • Write operations (batch-delete-table-version, create-partition-index, delete-partition-index, and the Data Quality ruleset/evaluation-run mutations) are gated behind --allow-write, consistent with every other mutating operation in this server.
  • README updated: new tool/operation tables, IAM read-only and write policy JSON, and Key Features section.

Test plan for reviewers

  • Review new operation docstrings for clarity/AWS permission accuracy
  • Confirm IAM policy additions in README match the new API calls
  • Optionally exercise detect-table-format / get-iceberg-table-info against a real Iceberg table to sanity-check the heuristic

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.

Dipayan Das added 2 commits July 24, 2026 08:52
… table versions, and partition index tools

- Add detect-table-format and get-iceberg-table-info operations to manage_aws_glue_tables
- Add get-table-versions and batch-delete-table-version operations to manage_aws_glue_tables
- Add create-partition-index, get-partition-indexes, delete-partition-index operations to manage_aws_glue_partitions
- Add new manage_aws_glue_data_quality tool with full CRUD for rulesets, evaluation runs, and results
- Wire up previously dead DataQualityResult model
- Fix GetPartitionIndexes Keys field parsing (list of dicts, not strings)
- Add unit tests for all new operations (MagicMock convention)
- Update README with new tools, IAM policies, and usage examples

Refs: awslabs#614
Keep name and GitHub handle only, drop employer affiliation before pushing to a public repo.
@github-actions

Copy link
Copy Markdown
Contributor

This pull request is now marked as stale because it hasn't seen activity for a while. Add a comment or it will be closed soon. If you wish to exclude this issue from being marked as stale, add the "backlog" label.

@github-actions github-actions Bot added the stale These are items that have been around for a long time without progress label Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale These are items that have been around for a long time without progress

Projects

Status: To triage

Development

Successfully merging this pull request may close these issues.

1 participant