feat: consolidate pre-publish event validation gate at upsert_event#420
Merged
Conversation
Contributor
Homeboy Results —
|
…417) Add a single validation boundary in EventUpsert::validateForPublish() that every fetch handler (web scraper, Ticketmaster, Dice, …) passes through before any write. Junk never reaches publish regardless of source. Reuses rather than duplicates the prior scattered guards: - #415 empty/unparseable startDate gate — folded in from the former inline check; getDateTimeConfidence() reused as-is. - #349/#367 junk-marker title guard — isJunkTitle() reused. - #416 junk/test-payload filter — JunkPayloadFilter reused, now evaluated at the upsert boundary for every source instead of only inside the Ticketmaster fetch handler. Adds isPlaceholderTitle() for generic, source-agnostic placeholder/noise title detection ("Test Event", bare "?", etc.), filterable via data_machine_events_placeholder_titles. Disposal policy: SKIP + LOG. Rejected items are neither published nor drafted; each rejection is logged via datamachine_log with a machine-readable rule slug. No human-review queue, no draft fallback. All matching is generic and filterable; no vendor or venue is special-cased in the gate (layer purity).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a single consolidated pre-publish validation gate at the
upsert_eventboundary (EventUpsert::validateForPublish()) so that every fetch handler — web scraper, Ticketmaster, Dice, and any future feed — funnels through one quality bar before any write. Closes #417.What already existed (and how this consolidates rather than duplicates)
Per the site rule "check what already exists before designing new infrastructure", I read the two just-merged fixes first and designed the gate around them:
9001ecf) — added an inline empty-startDaterejection gate insideexecuteUpsert()usinggetDateTimeConfidence(). Folded into the gate — the inline check is removed andgetDateTimeConfidence()is now called fromvalidateForPublish(). The logic is reused unchanged; only its call site moved.bc36b2e) — addedJunkPayloadFilter, a generic, filterable junk/test-payload matcher, currently wired into the Ticketmaster fetch handler only. Reused at the gate — the sameJunkPayloadFilterinstance is now also evaluated atupsert_eventwith the resolvedsource_type, so test/placeholder payloads are caught at the boundary regardless of which feed produced them. This is the key consolidation win: the check now runs for every source, not only inside the Ticketmaster handler.isJunkTitle()guard (Rejected:/Duplicate:/Consolidate:/merged markers, control chars). Reused as-is — called from the gate, unchanged.No parallel reimplementation of any of these.
JunkPayloadFilteris not modified; its contract (source-specific patterns viadata_machine_events_junk_payload_patterns) is preserved.Rules enforced (v1)
startDate(folded in from Event import accepts empty startDate — publishes undated events that render/sort incorrectly #415;getDateTimeConfidence() === 'none'→ reject).isJunkTitle()), and not a generic placeholder/noise title (newisPlaceholderTitle(): exact match against a filterable deny-list, plus a bare-punctuation/noise regex catching?,??,-,…).JunkPayloadFilter::is_junk()evaluated at the upsert boundary with the resolvedsource_type, applying each source's registered deny-list (CCPER-,Standalone Upsell,Upcoming Eventno-artist, etc.).Disposal policy — SKIP + LOG (decided)
Rejected items are skipped entirely and logged. They are neither published nor saved as draft. There is no human-review queue and no draft fallback. Each rejection routes through
gateRejection(), which logs adatamachine_logwarning with a machine-readableruleslug (missing_title,junk_marker_title,placeholder_title,invalid_start_date,junk_payload) and returns the standardsuccess:falsetool response so the item is dropped and the AI moves on.Filters (all generic,
data_machine_events_*prefix)data_machine_events_junk_payload_patterns— existing; source-specific junk buckets (unchanged).data_machine_events_placeholder_titles— new; generic placeholder title deny-list (defaults:Test Event,Upcoming Event).No vendor or venue is special-cased in the gate (layer purity). Patterns are config; matching is generic.
Files changed
inc/Steps/Upsert/Events/EventUpsert.php— addedvalidateForPublish()(the gate),gateRejection()(uniform skip+log),isPlaceholderTitle()(generic placeholder check); refactoredexecuteUpsert()to call the gate before acquiring the advisory lock; added$junk_filterproperty.tests/Unit/EventUpsertTest.php— added gate-level tests: valid event passes, empty date skipped, placeholderTest Eventtitle skipped, bare?title skipped, junk payload skipped from a non-Ticketmaster source (the consolidation win), placeholder deny-list filterable, exact-match-only (non-substring).Lint / test results
php -l— clean on both changed PHP files.phpstan.neonabsent).tests/bootstrap.php/vendor/bin/phpunitlocally); tests will run in CI. Test logic was verified by reasoning against the gate's rule ordering and the existinggetDateTimeConfidence()/JunkPayloadFiltercontracts.