Simplify duplication and crossref logics

Author: mathpluscode

.claude-plugin/marketplace.json

@@ -16,7 +16,7 @@
         "path": "./"
       },
       "description": "A bibliography toolkit for LaTeX",
-      "version": "1.6.0",
+      "version": "1.7.0",
       "keywords": ["bibtex", "bibliography", "latex", "overleaf", "academic", "reference", "citation"],
       "category": "academic",
       "license": "MIT"

.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "bibtools",
   "description": "A bibliography toolkit for LaTeX",
-  "version": "1.6.0",
+  "version": "1.7.0",
   "author": {
     "name": "Yunguan Fu"
   },

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "bibtools"
-version = "1.6.0"
+version = "1.7.0"
 description = "A bibliography toolkit for LaTeX, built as agent skills"
 requires-python = ">=3.10"
 license = "MIT"

skills/bibtidy/SKILL.md

@@ -28,7 +28,8 @@ Assume standard brace-style BibTeX entries like `@article{...}`. Parenthesized B
 | CrossRef DOI lookup | `python3 $TOOLS_DIR/crossref.py doi <DOI>` |
 | CrossRef title search | `python3 $TOOLS_DIR/crossref.py search "<title>"` |
 | CrossRef bibliographic search | `python3 $TOOLS_DIR/crossref.py bibliographic "<query>"` |
-| Duplicate detection | `python3 $TOOLS_DIR/duplicates.py <file.bib>` |
+| Remove exact duplicates | `python3 $TOOLS_DIR/duplicates.py <file.bib> --exact` |
+| Detect near-duplicates | `python3 $TOOLS_DIR/duplicates.py <file.bib>` |
 | **Apply edits** | `python3 $TOOLS_DIR/edit.py <file.bib> <patches.json>` |
 | Web verification | web search (preferred) or CrossRef scripts (fallback) |
 
@@ -84,20 +85,22 @@ For unchanged entries, do NOT add any comments or URLs.
 1. Read the .bib file, note the file path
 2. Back up for format validation: `cp <file>.bib <file>.bib.orig`
 3. Preserve `@string`, `@preamble`, `@comment` blocks verbatim
-4. Run duplicate detection: `python3 $TOOLS_DIR/duplicates.py <file.bib>`
-5. **Run field comparison**: `python3 $TOOLS_DIR/compare.py <file.bib>` — this programmatically compares every entry against CrossRef and returns exact field-level mismatches. Do NOT skip this step or rely on visual comparison alone. The output is a JSON list; each element has `key`, `versions` (a list of alternative CrossRef candidate matches for the same entry, each with `mismatches`, `url`, `doi`, etc.), and `error`. When multiple versions are returned, choose the best matching candidate; do not combine fields from different versions. **Skip rule**: if an entry has zero mismatches across all versions and no error in the compare.py output, skip it entirely — do NOT investigate, modify, or add comments to it. Only proceed with entries that compare.py flagged (mismatches, errors, or duplicates from step 4).
-6. **Verify every planned modification with web search** — for entries that compare.py flagged with mismatches or errors, and for entries flagged as duplicates, verify the planned action via web search. For `fix` patches, gather one or more source URLs. Entries where `compare.py` returned an error (e.g. "No exact title match") still need full verification — the verification agent should search for the paper and check all fields. **Important: after selecting the best-matching version, verification agents MUST NOT override that selected version's `compare.py` field values.** CrossRef is the authoritative source for metadata (pages, volume, number, etc.) because it receives data directly from publishers via DOI registration. When web search finds a conflicting value (e.g. different page numbers on a conference website), always use the CrossRef value and add `% bibtidy: REVIEW` if desired — but do NOT keep the old value.
+4. **Remove exact duplicates**: `python3 $TOOLS_DIR/duplicates.py <file.bib> --exact` — this comments out entries that are identical (same key, same type, same fields). Safe to auto-remove since no information is lost.
+5. **Run field comparison**: `python3 $TOOLS_DIR/compare.py <file.bib>` — this programmatically compares every entry against CrossRef and returns exact field-level mismatches. Do NOT skip this step or rely on visual comparison alone. The output is a JSON list; each element has `key`, `versions` (a list of alternative CrossRef candidate matches for the same entry, each with `mismatches`, `url`, `doi`, etc.), and `error`. When multiple versions are returned, choose the best matching candidate; do not combine fields from different versions. **Skip rule**: if an entry has zero mismatches across all versions and no error in the compare.py output, skip it entirely — do NOT investigate, modify, or add comments to it. Only proceed with entries that compare.py flagged (mismatches or errors).
+6. **Verify every planned modification with web search** — for entries that compare.py flagged with mismatches or errors, verify the planned action via web search. For `fix` patches, gather one or more source URLs. Entries where `compare.py` returned an error (e.g. "No exact title match") still need full verification — the verification agent should search for the paper and check all fields. **Important: after selecting the best-matching version, verification agents MUST NOT override that selected version's `compare.py` field values.** CrossRef is the authoritative source for metadata (pages, volume, number, etc.) because it receives data directly from publishers via DOI registration. When web search finds a conflicting value (e.g. different page numbers on a conference website), always use the CrossRef value and add `% bibtidy: REVIEW` if desired — but do NOT keep the old value.
 7. **Flag hallucinated/non-existent references** — if compare.py returned an error (e.g. "No CrossRef results found" or "No exact title match in CrossRef results") AND web search also finds no matching paper, the reference likely does not exist. Add `% bibtidy: NOT FOUND — no matching paper on CrossRef or web search; verify this reference exists` above the entry, then comment out the entire entry (prefix every line with `% `). Do NOT add a URL line.
-8. Apply fixes **sequentially** using `edit.py` — do NOT edit the .bib file directly with agent editing tools (for example, Claude Code Edit or Codex `apply_patch`), and do NOT rewrite the entire file. Build a patches.json for each entry (or batch) and run `python3 $TOOLS_DIR/edit.py <file.bib> <patches.json>`. This ensures the commented original, source URLs, and explanation are always included. After selecting the correct version, you MUST apply **every** mismatch from that selected version — do not skip any field (including `number`, `pages`, `volume`). Use the `crossref_value` exactly as given (do NOT rephrase, reformat, or partially apply it). For title mismatches on preprint→published upgrades, replace the entire title with the CrossRef title — do NOT try to edit parts of the old title. Never reject a CrossRef value because another source disagrees. Every patch MUST include `urls` (list of source URLs) and `explanation` (what changed and why). Include the CrossRef URL from compare.py's `url` field when available, plus any other authoritative source (DOI URL, venue page) found via web search.
-9. Run format validation; fix violations and re-run until clean
-10. Delete backup: `rm <file>.bib.orig`
-11. Print a Markdown summary table with headers `Metric | Count` and exactly these rows: total entries, verified, fixed, not found. Do NOT include a separate "needs manual review" row.
+8. Apply fixes **sequentially** using `edit.py` — do NOT edit the .bib file directly with agent editing tools (for example, Claude Code Edit or Codex `apply_patch`), and do NOT rewrite the entire file. Build a patches.json for each entry (or batch) and run `python3 $TOOLS_DIR/edit.py <file.bib> <patches.json>`. This ensures the commented original, source URLs, and explanation are always included. After selecting the correct version, you MUST apply **every** mismatch from that selected version — do not skip any field (including `author`, `number`, `pages`, `volume`). In particular, if the bib entry uses `and others` but CrossRef returns the full author list, you MUST replace the truncated list with the complete one from CrossRef. Use the `crossref_value` exactly as given (do NOT rephrase, reformat, or partially apply it). For title mismatches on preprint→published upgrades, replace the entire title with the CrossRef title — do NOT try to edit parts of the old title. Never reject a CrossRef value because another source disagrees. Every patch MUST include `urls` (list of source URLs) and `explanation` (what changed and why). Include the CrossRef URL from compare.py's `url` field when available, plus any other authoritative source (DOI URL, venue page) found via web search.
+9. **Post-fix exact duplicate removal**: `python3 $TOOLS_DIR/duplicates.py <file.bib> --exact` — entries that were different before fixing may now be identical after metadata corrections. Comment out any new exact duplicates.
+10. **Detect near-duplicates**: `python3 $TOOLS_DIR/duplicates.py <file.bib>` — flag entries that share the same key, DOI, or title (with a shared author), plus likely preprint→published pairs with the same lead author and overlapping significant title words, but are not identical. Apply `duplicate` patches via `edit.py` to add `% bibtidy: DUPLICATE of <other_key>` comments. Do NOT delete or comment out near-duplicates.
+11. Run format validation; fix violations and re-run until clean
+12. Delete backup: `rm <file>.bib.orig`
+13. Print a Markdown summary table with headers `Metric | Count` and exactly these rows: total entries, verified, fixed, not found, exact duplicates removed, near-duplicates flagged. Do NOT include a separate "needs manual review" row.
 
 ## Parallel Verification with Subagents
 
 Use subagents, when available, to verify multiple entries concurrently. This dramatically reduces wall-clock time (e.g., 7 entries: ~1 min parallel vs ~5 min sequential; 100 entries: ~3 min vs ~40 min). If subagents are unavailable, do the same verification work sequentially yourself.
 
-**Step 1 — Dispatch verification agents:** For entries that `compare.py` flagged with mismatches or errors, and any duplicate entries you plan to annotate, launch a subagent that:
+**Step 1 — Dispatch verification agents:** For entries that `compare.py` flagged with mismatches or errors, launch a subagent that:
 - For mismatches: uses web search to confirm the CrossRef data (especially for preprint upgrades and author changes)
 - For errors (e.g. paper not found in CrossRef): uses web search to verify **every** field from scratch — title, author, journal/booktitle, volume, number, pages, year. Do NOT skip number or other fields just because they look plausible.
 - Returns a JSON summary: key, whether each mismatch is confirmed, source URL, CrossRef URL (if there is a CrossRef match), any additional corrections found
@@ -134,19 +137,19 @@ Entry:
 
 ## Duplicate Detection
 
-```
-python3 $TOOLS_DIR/duplicates.py <file.bib>
-```
+Duplicate handling has three phases (see workflow steps 4, 9, 10):
+
+**Exact duplicates** (same key, type, and all field values): `python3 $TOOLS_DIR/duplicates.py <file.bib> --exact` comments them out automatically. Run before and after metadata fixes.
 
-Returns JSON array of duplicate pairs (by key, DOI, or title). For each duplicate, add: `% bibtidy: DUPLICATE of <other_key> — consider removing`
+**Near-duplicates** (same key, DOI, or title with shared author, plus likely preprint→published pairs with the same lead author and overlapping significant title words, but different content): `python3 $TOOLS_DIR/duplicates.py <file.bib>` returns a JSON array of pairs. For each, apply a `duplicate` patch via `edit.py` to add `% bibtidy: DUPLICATE of <other_key>`. Do NOT delete or comment out near-duplicates.
 
 ## Per-Entry Checks
 
 For each `@article`, `@inproceedings`, `@book`, etc.:
 
 **1. Verify existence** — Search for `"<title>" <first author last name>`. If not found: `% bibtidy: NOT FOUND — verify manually`
 
-**2. Cross-check metadata** — Always search via `crossref.py search "<title>"`. If DOI exists, also fetch via `crossref.py doi <DOI>`. If neither finds a match, fall back to `crossref.py bibliographic "<title>"`. Compare title, year, authors, journal, volume, number, pages.
+**2. Cross-check metadata** — `compare.py` runs both `crossref.py search "<title>"` and `crossref.py bibliographic "<title>"` unconditionally, plus `crossref.py doi <DOI>` when a DOI exists, deduplicating results by DOI. Only exact normalized title matches are kept. Compare title, year, authors, journal, volume, number, pages.
 
 **3. Check for published preprints** — If journal contains "arxiv"/"biorxiv"/"chemrxiv", search for published version. Update title, venue, year, volume, pages, entry type. Only update if confirmed via DOI or two independent sources.
 

skills/bibtidy/tools/compare.py

@@ -18,7 +18,8 @@
 import sys
 
 from crossref import fetch_doi, search_bibliographic, search_title
-from duplicates import normalize_doi, normalize_title, parse_bib_entries, split_bibtex_authors
+from duplicates import normalize_doi, normalize_title, split_bibtex_authors
+from parser import parse_bib_entries
 
 
 def _normalize_pages(pages: str) -> str:
@@ -185,34 +186,40 @@ def lookup_and_compare(entry: dict, timeout: int = 10) -> dict:
         result["error"] = "No DOI or title to search"
         return result
 
-    # Collect CrossRef results from multiple strategies.
+    # Collect CrossRef results from all strategies, then deduplicate.
     matches = []
+    seen_dois = set()
     last_error = None
     bib_title_norm = normalize_title(title)
 
-    def _search_and_filter(search_fn, query):
+    def _add(item):
+        item_doi = item.get("doi")
+        if item_doi and item_doi in seen_dois:
+            return
+        if item_doi:
+            seen_dois.add(item_doi)
+        matches.append(item)
+
+    def _search(search_fn, query):
         nonlocal last_error
         cr = search_fn(query, rows=3, timeout=timeout)
         if "error" in cr:
             last_error = cr["error"]
-            return []
-        return [item for item in cr.get("results", [])
-                if normalize_title(item.get("title") or "") == bib_title_norm]
+            return
+        for item in cr.get("results", []):
+            if normalize_title(item.get("title") or "") == bib_title_norm:
+                _add(item)
 
     if title:
-        matches = _search_and_filter(search_title, title)
+        _search(search_title, title)
+        _search(search_bibliographic, title)
 
     if doi:
         cr = fetch_doi(normalize_doi(doi), timeout=timeout)
         if "error" in cr:
             last_error = cr["error"]
         else:
-            existing_dois = {m.get("doi") for m in matches}
-            if cr.get("doi") not in existing_dois:
-                matches.append(cr)
-
-    if not matches and title:
-        matches = _search_and_filter(search_bibliographic, title)
+            _add(cr)
 
     if not matches:
         result["error"] = last_error or "No exact title match in CrossRef results"