fix: drop Ticketmaster test/placeholder payloads (CCPER-*, Standalone Upsell) during import#418
Merged
Merged
Conversation
… Upsell) during import Add a vendor-agnostic JunkPayloadFilter with a filterable pattern table (data_machine_events_junk_payload_patterns) and wire the Ticketmaster handler to drop known test/placeholder records (CCPER-* IDs, 'Standalone Upsell'/'Test Event' titles, 'Upcoming Event' titles with no artist, and the upstream 'test' boolean flag) before they enter the import pipeline. Closes #416.
Contributor
Homeboy Results —
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
Closes #416.
Ticketmaster's Discovery API returns test/placeholder events — internal QA records with IDs like
CCPER-2756and titles likeUpcoming Event with Standalone Upsell CCPER-2756/Upcoming Event CCPER-2756. These are Ticketmaster's own sandbox/test data leaking through the live discovery feed, and the import pipeline ingested them as real events (draft status).CCPER-2756is a well-known Ticketmaster test event ID. These recurred on every scrape because there was no junk/test filter on the fetch path.Filter approach
Added a vendor-agnostic
JunkPayloadFilter(inc/Steps/EventImport/JunkPayloadFilter.php) and wired it into the Ticketmaster fetch loop. The matcher is generic; the junk patterns are config-driven and fully filterable via a single WordPress filter:Filter name:
data_machine_events_junk_payload_patternsPattern buckets (merged over defaults, all optional):
honor_test_flag(bool) — honor an upstream explicittestflagid(string[]) — substring match on the source record IDtitle(string[]) — substring match on the event titletitle_prefix_no_artist(string[]) — title prefix match only when no real artist/attraction is attachedThe Ticketmaster handler seeds its own defaults via
register_junk_patterns()(hooked into the filter), evaluated only when$source_type === 'ticketmaster'. This satisfies layer purity: the generic matcher contains zero vendor strings; all Ticketmaster-specific junk patterns (CCPER-, Standalone Upsell, Test Event, Upcoming Event) live in the handler that owns them.Drop rules implemented
A Ticketmaster event is dropped during fetch when any of these match:
testboolean flag on the raw record istrue.idcontainsCCPER-.CCPER-,Standalone Upsell, orTest Event.Upcoming Eventand the event has no real artist/attraction.Each dropped item is logged via
$context->log()(Ticketmaster: Dropped test/placeholder payload) with title/source_id/artist so the pattern table can be tuned. The matcher is placed in the sharedEventImportnamespace so Dice and other feeds can reuse it later; only the Ticketmaster wiring is in scope for this PR.Site-tuning example
Files changed
inc/Steps/EventImport/JunkPayloadFilter.php(new) — generic vendor-agnostic matcher with the filterable pattern table.inc/Steps/EventImport/Handlers/Ticketmaster/Ticketmaster.php— instantiates the filter, registers Ticketmaster default patterns viaregister_junk_patterns()hooked todata_machine_events_junk_payload_patterns, and drops junk in the fetch loop (is_junk_payload()), logging each drop.tests/Unit/JunkPayloadFilterTest.php(new) — 15 tests covering: explicittestflag (true/false/absent), CCPER- on id and title, Standalone Upsell, Test Event, Upcoming Event prefix with/without artist, normal-event pass-through, unknown-source no-match, case-insensitivity, and filterability (add a pattern + remove a default).tests/Unit/TicketmasterHandlerTest.php— adds tests forregister_junk_patterns()(TM defaults seeded, other sources ignored), the filter exposure, and end-to-end junk/normal classification through the filter.Test / lint results
php -lclean on all four changed files.homeboy lint, extension-default level 7).php -lis clean and the tests will run in CI.homeboy lintoverall is gated on that unrelated JS tool.Confirms
data_machine_events_junk_payload_patterns, and site code can add, remove, or override any bucket.JunkPayloadFilter) contains no vendor names; the only place Ticketmaster-specific strings appear isTicketmaster::register_junk_patterns().CHANGELOG.mdedits, no version-string hand-bumps (homeboy owns both).Scope
PR only. Not releasing, not deploying, not merging.