Skip to content

KWIC + n-gram analysis#636

Open
aahafez wants to merge 12 commits into
mainfrom
linguistic-analysis
Open

KWIC + n-gram analysis#636
aahafez wants to merge 12 commits into
mainfrom
linguistic-analysis

Conversation

@aahafez

@aahafez aahafez commented Oct 13, 2025

Copy link
Copy Markdown
Contributor

No description provided.

@aahafez
aahafez requested a review from bigabig October 13, 2025 12:16
@aahafez aahafez added the backend This issue is related to the backend label Oct 13, 2025
@bigabig
bigabig requested a review from Copilot February 10, 2026 10:26

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds backend support for KWIC (keyword-in-context) search and project-level n‑gram frequency analysis backed by Elasticsearch, exposing new analysis endpoints for the frontend.

Changes:

  • Introduces /analysis/kwic endpoint and KWIC snippet extraction/sorting logic.
  • Introduces /analysis/ngrams endpoint and Elasticsearch aggregation to fetch frequent n‑grams.
  • Extends the SDoc Elasticsearch index mapping/settings with unigram/bigram/trigram subfields and analyzers.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 15 comments.

Show a summary per file
File Description
docker/compose.yml Clarifies Elasticsearch JVM heap env var comment.
backend/src/modules/analysis/kwic_analysis.py Implements KWIC snippet extraction and frequency-based sorting.
backend/src/modules/analysis/analysis_endpoint.py Adds /analysis/kwic and /analysis/ngrams endpoints.
backend/src/core/doc/sdoc_kwic_dto.py Adds DTOs/enums for KWIC + n‑gram responses.
backend/src/core/doc/sdoc_elastic_index.py Updates ES index settings/mappings to support n‑grams + fast-vector-highlighter.
backend/src/core/doc/sdoc_elastic_crud.py Adds ES CRUD methods for KWIC search and n‑gram aggregation.

Comment thread docker/compose.yml Outdated
Comment thread backend/src/modules/analysis/kwic_analysis.py Outdated
Comment thread backend/src/modules/analysis/analysis_endpoint.py Outdated
Comment thread backend/src/modules/analysis/analysis_endpoint.py Outdated
Comment thread backend/src/core/doc/sdoc_elastic_crud.py Outdated
Comment thread backend/src/core/doc/sdoc_kwic_dto.py Outdated
Comment thread backend/src/core/doc/sdoc_elastic_crud.py Outdated
Comment thread backend/src/core/doc/sdoc_elastic_crud.py Outdated
Comment thread backend/src/core/doc/sdoc_elastic_index.py
Comment thread backend/src/modules/analysis/kwic_analysis.py Outdated
@bigabig
bigabig force-pushed the linguistic-analysis branch from cbd3139 to 197f272 Compare June 8, 2026 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend This issue is related to the backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants