Periodic checks to contributor GH profile for new information

[cdolfi]

It would be great if Augur periodically checked contributors for changes in their GH profile information and updated/store the historical data

[MoralCode]

apparrently we have this task - its called process_contributors, it runs hourly, and it updates these fields:

 new_contributor_data = {
            "cntrb_created_at": data["created_at"],
            "cntrb_last_used": data["updated_at"]
        }

🙃

[sgoggins]

Is it ever called?

[shlokgilda]

Dug into this. process_contributors exists and is scheduled hourly via Celery Beat, but it's effectively broken.

Bug: broken SQLAlchemy filter

augur/tasks/github/contributors.py:29:

query = session.query(Contributor).filter(
    Contributor.data_source == data_source,
    Contributor.cntrb_created_at is None,
    Contributor.cntrb_last_used is None
)

Contributor.cntrb_created_at is None uses Python's is operator instead of SQLAlchemy's .is_(None). Since the column attribute is never literally Python None, this evaluates to False before SQLAlchemy even sees it. The query ends up with WHERE ... AND false AND false, so it always returns 0 rows. The task logs "No contributors to enrich...returning..." every hour and does nothing.

Only updates 2 of ~20 available fields

Even with the filter fixed, the task only pulls created_at and updated_at from the GitHub /users/{login} endpoint (lines 56-58). The API returns a lot more that we already have columns for: email, company, location, name, avatar_url, etc. We already have extract_needed_contributor_data() in data_parse.py:647 that maps all these fields - the enrichment task just doesn't use it.

No re-enrichment of existing contributors

The task only targets contributors where both timestamp fields are NULL. Once a contributor is enriched once, they're never rechecked. So if someone changes their company/location/email on GitHub, we never pick that up.

[shlokgilda]

I have a plan to fix this.

1. Fix the .is_(None) filter bug
2. Update all profile fields using the existing extract_needed_contributor_data() mapping
3. Add a cntrb_last_enriched timestamp column to track when we last refreshed a profile
4. We probably should have a tiered priority system: enrich never-enriched contributors first, then refresh stale ones (let's say, >30 days old)
5. Cap at ~500 contributors per hourly run to avoid burning through API rate limits

The bulk_insert_dicts upsert already uses COALESCE(new, existing) so new non-NULL values overwrite old ones without clobbering data with NULLs. Should be a fairly contained change, mostly in contributors.py plus a small migration.

Would love some more eyes on this to verify my understanding of the issue.

[cdolfi]

augur/augur/application/db/data_parse.py

Line 647 in 5dcce83

def extract_needed_contributor_data(contributor, tool_source, tool_version, data_source):

should be updated in the same way this PR was where this is done in facade https://github.com/chaoss/augur/pull/3695/changes -

[MoralCode]

Because we are checking

 Contributor.cntrb_created_at is None,
    Contributor.cntrb_last_used is None

This only runs for new contributors, not refreshing all of them

[MoralCode]

we would probably need to add a new last updated column and limit how many get processed to some number (1k per hour) for large instances (suggested by andrew)

[MoralCode]

The task this pr enables only updates missing data. It doesnt refresh the data once it has successfully been collected