Skip to content

Commit 3b7a9f2

Browse files
rustyconoverclaude
andcommitted
Enrich catalog doc_md for SEO; bump VGI SDK to latest release
Rewrite the catalog-level vgi.doc_md into SEO-strong, human-readable documentation with links to the underlying library's source and docs, and bump the VGI SDK to the current release so the worker attaches under the latest signed vgi DuckDB extension. Verified locally: worker attaches, vgi-lint runs, and all catalog doc_md links resolve. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent b16efce commit 3b7a9f2

2 files changed

Lines changed: 43 additions & 11 deletions

File tree

pii_worker.py

Lines changed: 37 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -57,12 +57,44 @@
5757
)
5858

5959
_CATALOG_DESCRIPTION_MD = (
60-
"# pii\n\n"
61-
"Detect and redact PII in free text (person names, emails, phone numbers, credit cards, SSNs, "
62-
"locations, URLs, IP addresses, …) directly in SQL, powered by "
60+
"# PII Detection & Redaction in SQL\n\n"
61+
"**Find, redact, and anonymize personally-identifiable information (PII) in free text "
62+
"directly from DuckDB SQL** — person names, email addresses, phone numbers, credit-card "
63+
"numbers, US SSNs, locations, URLs, IP addresses and more — powered by "
6364
"[Microsoft Presidio](https://microsoft.github.io/presidio/).\n\n"
64-
"**Scalars:** `has_pii`, `pii_types`, `redact`, `anonymize`.\n\n"
65-
"**Table functions:** `detect_pii`, `supported_entities`."
65+
"This extension brings privacy scrubbing, data-loss-prevention (DLP) checks, and PII "
66+
"auditing to your data warehouse without exporting a single row to an external service. "
67+
"It is built for data engineers, privacy and security teams, and analysts who need to "
68+
"sanitize logs, user-generated content, support transcripts, and other text columns "
69+
"before sharing, training, or archiving them. Every function runs in-process over Apache "
70+
"Arrow, so detection happens right next to your data and PII never leaves your "
71+
"environment.\n\n"
72+
"Detection is powered by [Microsoft Presidio](https://github.com/microsoft/presidio) "
73+
"(its analyzer and anonymizer engines) backed by a [spaCy](https://spacy.io/) "
74+
"named-entity-recognition pipeline using the pinned `en_core_web_sm` model from the "
75+
"[spaCy models](https://github.com/explosion/spacy-models) collection. Presidio combines "
76+
"this NLP model with pattern recognizers and checksum validation (for example, Luhn "
77+
"validation of credit-card numbers) to flag a broad catalog of entity types, each with a "
78+
"confidence score and character offsets you can inspect or threshold.\n\n"
79+
"## SQL functions\n\n"
80+
"The catalog exposes four scalar functions and two table functions in the `main` schema. "
81+
"Use `has_pii(text)` as a boolean predicate to filter or flag rows that contain sensitive "
82+
"data, and `pii_types(text)` to get the sorted `VARCHAR[]` of distinct entity types "
83+
"present. Use `redact(text)` to replace each detected entity with a `<TYPE>` tag (for "
84+
"example `<PERSON>`, `<EMAIL_ADDRESS>`) and `anonymize(text)` to mask each entity's "
85+
"characters with `*`. For full visibility, the `detect_pii(text)` table function returns "
86+
"one row per entity with its type, start/end offsets and confidence score (with an "
87+
"optional `score_threshold`), and `supported_entities()` lists every entity type the "
88+
"analyzer can detect. All scalars accept an optional ISO `language` argument (defaulting "
89+
"to `'en'`), and NULL or blank input is handled gracefully.\n\n"
90+
"```sql\n"
91+
"SELECT pii.redact('Call John Smith at john@example.com');\n"
92+
"-- 'Call <PERSON> at <EMAIL_ADDRESS>'\n"
93+
"SELECT * FROM pii.detect_pii('Call John Smith at john@example.com') ORDER BY start;\n"
94+
"```\n\n"
95+
"Learn more from the [Presidio documentation](https://microsoft.github.io/presidio/), the "
96+
"[Presidio source repository](https://github.com/microsoft/presidio), and the "
97+
"[spaCy NLP library](https://github.com/explosion/spaCy)."
6698
)
6799

68100
_MAIN_DESCRIPTION_LLM = (

uv.lock

Lines changed: 6 additions & 6 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)