Skip to content

fix(roda-mcp-server): rank search_datasets by relevance, drop stop words - #4463

Open
ariharrisonlab wants to merge 1 commit into
awslabs:mainfrom
ariharrisonlab:fix/roda-search-relevance
Open

fix(roda-mcp-server): rank search_datasets by relevance, drop stop words#4463
ariharrisonlab wants to merge 1 commit into
awslabs:mainfrom
ariharrisonlab:fix/roda-search-relevance

Conversation

@ariharrisonlab

Copy link
Copy Markdown

Fixes #4462

Summary

search_datasets matched query terms by substring (only data/dataset/datasets were filtered) and then ordered results by provider diversity with no relevance score. Broad natural-language queries therefore matched hundreds of datasets, and the most relevant one could be pushed out of the top results by less-relevant datasets from other providers — so a more specific query returned worse results than a single keyword.

Changes

  • Expand stop-word removal so multi-word queries narrow the result set instead of matching nearly everything via substring.
  • Add a field-weighted relevance score (exact tag > name > description, plus a coverage bonus for matching more of the distinct query terms) and sort matches by score before the provider-diversification pass, so diversification applies within relevance order.
  • Keep the permissive substring match gate (no recall regression) and the JSON response shape unchanged; the internal score is stripped from output.
  • Add tests for relevance ordering, stop-word handling, and score omission.

User experience

Against the live registry index (1,177 datasets):

query before (matches / best-match rank) after (matches / best-match rank)
ADMET 1 / #1 1 / #1
cheminformatics 1 / #1 1 / #1
drug discovery molecules with ADMET predictions 746 / not in top 10 281 / #1

Before, adding descriptive words to a query buried the best match; after, a more specific query returns more relevant, better-ranked results (for that example the top 3 are the three actual drug-discovery datasets in the registry). Single-keyword and filtered-query behavior is unchanged.

Checklist

  • I have reviewed the contributing guidelines
  • I have performed a self-review of this change
  • Changes have been tested (pytest tests/ — 93 passed; ruff check + ruff format clean)
  • Changes are documented (code comments + tests)

Is this a breaking change? (N)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.

search_datasets matched query terms by substring (only data/dataset/datasets
were filtered) and then ordered results by provider diversity with no relevance
score. Broad natural-language queries therefore matched hundreds of datasets,
and the most relevant one could be pushed out of the top results by
less-relevant datasets from other providers -- so a more specific query
returned worse results than a single keyword.

- Expand stop-word removal so multi-word queries narrow instead of matching
  nearly everything via substring.
- Add a field-weighted relevance score (exact tag > name > description) with a
  coverage bonus, and sort matches by score before the provider-diversity pass
  so diversification applies within relevance order.
- Keep the permissive substring match gate (no recall regression) and the JSON
  response shape unchanged; the internal score is stripped from output.
- Add tests for relevance ordering, stop-word handling, and score omission.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: To triage

Development

Successfully merging this pull request may close these issues.

search_datasets ranks provider diversity above relevance; broad queries match hundreds via substring + stop words

1 participant