|
57 | 57 | ) |
58 | 58 |
|
59 | 59 | _CATALOG_DESCRIPTION_MD = ( |
60 | | - "# pii\n\n" |
61 | | - "Detect and redact PII in free text (person names, emails, phone numbers, credit cards, SSNs, " |
62 | | - "locations, URLs, IP addresses, …) directly in SQL, powered by " |
| 60 | + "# PII Detection & Redaction in SQL\n\n" |
| 61 | + "**Find, redact, and anonymize personally-identifiable information (PII) in free text " |
| 62 | + "directly from DuckDB SQL** — person names, email addresses, phone numbers, credit-card " |
| 63 | + "numbers, US SSNs, locations, URLs, IP addresses and more — powered by " |
63 | 64 | "[Microsoft Presidio](https://microsoft.github.io/presidio/).\n\n" |
64 | | - "**Scalars:** `has_pii`, `pii_types`, `redact`, `anonymize`.\n\n" |
65 | | - "**Table functions:** `detect_pii`, `supported_entities`." |
| 65 | + "This extension brings privacy scrubbing, data-loss-prevention (DLP) checks, and PII " |
| 66 | + "auditing to your data warehouse without exporting a single row to an external service. " |
| 67 | + "It is built for data engineers, privacy and security teams, and analysts who need to " |
| 68 | + "sanitize logs, user-generated content, support transcripts, and other text columns " |
| 69 | + "before sharing, training, or archiving them. Every function runs in-process over Apache " |
| 70 | + "Arrow, so detection happens right next to your data and PII never leaves your " |
| 71 | + "environment.\n\n" |
| 72 | + "Detection is powered by [Microsoft Presidio](https://github.com/microsoft/presidio) " |
| 73 | + "(its analyzer and anonymizer engines) backed by a [spaCy](https://spacy.io/) " |
| 74 | + "named-entity-recognition pipeline using the pinned `en_core_web_sm` model from the " |
| 75 | + "[spaCy models](https://github.com/explosion/spacy-models) collection. Presidio combines " |
| 76 | + "this NLP model with pattern recognizers and checksum validation (for example, Luhn " |
| 77 | + "validation of credit-card numbers) to flag a broad catalog of entity types, each with a " |
| 78 | + "confidence score and character offsets you can inspect or threshold.\n\n" |
| 79 | + "## SQL functions\n\n" |
| 80 | + "The catalog exposes four scalar functions and two table functions in the `main` schema. " |
| 81 | + "Use `has_pii(text)` as a boolean predicate to filter or flag rows that contain sensitive " |
| 82 | + "data, and `pii_types(text)` to get the sorted `VARCHAR[]` of distinct entity types " |
| 83 | + "present. Use `redact(text)` to replace each detected entity with a `<TYPE>` tag (for " |
| 84 | + "example `<PERSON>`, `<EMAIL_ADDRESS>`) and `anonymize(text)` to mask each entity's " |
| 85 | + "characters with `*`. For full visibility, the `detect_pii(text)` table function returns " |
| 86 | + "one row per entity with its type, start/end offsets and confidence score (with an " |
| 87 | + "optional `score_threshold`), and `supported_entities()` lists every entity type the " |
| 88 | + "analyzer can detect. All scalars accept an optional ISO `language` argument (defaulting " |
| 89 | + "to `'en'`), and NULL or blank input is handled gracefully.\n\n" |
| 90 | + "```sql\n" |
| 91 | + "SELECT pii.redact('Call John Smith at john@example.com');\n" |
| 92 | + "-- 'Call <PERSON> at <EMAIL_ADDRESS>'\n" |
| 93 | + "SELECT * FROM pii.detect_pii('Call John Smith at john@example.com') ORDER BY start;\n" |
| 94 | + "```\n\n" |
| 95 | + "Learn more from the [Presidio documentation](https://microsoft.github.io/presidio/), the " |
| 96 | + "[Presidio source repository](https://github.com/microsoft/presidio), and the " |
| 97 | + "[spaCy NLP library](https://github.com/explosion/spaCy)." |
66 | 98 | ) |
67 | 99 |
|
68 | 100 | _MAIN_DESCRIPTION_LLM = ( |
|
0 commit comments