RSS ingestor by eilmiv · Pull Request #1276 · ElixirTeSS/TeSS

eilmiv · 2026-04-07T16:57:10Z

Summary of changes

Added RSS and Atom feed support using rss gem
- Separate ingestor for events and materials (but RSS for events is not as useful)
RSS feeds are optionally discovered from html pages using a link element with application/rss+xml or atom
Support for metadata extentions (not every extension for every rss/atom version)
- RDF metadata (Bioschemas)
- Dublin Core
- iTunes
- yahoo media (e.g. used on YouTube)

Motivation and context

Closes #722

Screenshots

Checklist

I have read and followed the CONTRIBUTING guide.
I confirm that I have the authority necessary to make this contribution on behalf of its copyright owner and agree to license it to the TeSS codebase under the BSD license.

Copilot

Pull request overview

Adds RSS/Atom feed ingestion to TeSS by introducing dedicated ingestors for events and materials, including optional HTML feed discovery and support for several common metadata extensions (Dublin Core, RDF/Bioschemas, iTunes, Yahoo Media).

Changes:

Introduce shared RSS/Atom ingestion helpers (RSSIngestion) plus reusable Dublin Core parsing/building (DublinCoreIngestion).
Add new ingestors for event and material RSS/Atom feeds, including RDF/Bioschemas merge behavior and HTML alternate-feed discovery.
Add RSS Media namespace support for Atom parsing and comprehensive unit tests for RSS/Atom ingestion and extensions.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
test/unit/rss_media_atom_test.rb	Tests Media namespace installation idempotency for Atom.
test/unit/ingestors/material_rss_ingestor_test.rb	Material RSS/Atom ingestion tests (DC, RSS versions, RDF/Bioschemas, HTML discovery, media/iTunes extensions).
test/unit/ingestors/event_rss_ingestor_test.rb	Event RSS/Atom ingestion tests (DC, relative links, RDF/Bioschemas, HTML discovery).
lib/rss/media.rb	Defines Yahoo Media RSS extension wiring + loads Atom-specific patch.
lib/rss/media/atom.rb	Patches Atom classes to support `media:group` parsing and makes namespace installation idempotent.
lib/ingestors/rss_ingestion.rb	Shared feed fetching/parsing + HTML discovery + extraction/merge helpers.
lib/ingestors/dublin_core_ingestion.rb	Centralized DC-to-OpenStruct builders and normalization helpers.
lib/ingestors/material_rss_ingestor.rb	New material RSS/Atom ingestor (RSS/RDF/Atom + Bioschemas LearningResource extraction).
lib/ingestors/event_rss_ingestor.rb	New event RSS/Atom ingestor (RSS/RDF/Atom + Bioschemas Event/Course extraction).
lib/ingestors/oai_pmh_ingestor.rb	Refactors OAI-PMH DC parsing to reuse `DublinCoreIngestion`.
lib/ingestors/ingestor_factory.rb	Registers the new RSS ingestors.
config/initializers/inflections.rb	Adds `RSS` acronym for correct Zeitwerk/inflector naming.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

lib/ingestors/material_rss_ingestor.rb

lib/ingestors/rss_ingestion.rb

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

eilmiv · 2026-04-09T10:29:41Z

Two additional notes:

Depending on the RSS feed, only the most recent entries are imported/updated (e.g. the most recent 15 for YouTube)
Not sure if there exists an rss feed for events, I have no problem removing the rss event ingestor if this is not seen as useful

fbacall

Looks good - very flexible and nice tests. I think some parts can be simplified, and it might be good to split the YouTube functionality into a simple subclass for the sake of clarity.

fbacall · 2026-04-09T13:23:33Z

lib/ingestors/dublin_core_ingestion.rb

+      rescue StandardError
+        nil


Can this rescue be more specific?

fbacall · 2026-04-09T16:12:39Z

lib/ingestors/material_rss_ingestor.rb

@@ -0,0 +1,111 @@
+require 'rss'
+require_relative '../rss/media'


Is this needed? Should be able to autoload

fbacall · 2026-04-09T16:18:06Z

lib/ingestors/event_rss_ingestor.rb

+      else
+        @messages << "Parsing UNKNOWN feed: #{feed_title(feed)}"
+        @messages << 'unsupported feed format'
+      end


What other types of feed are there? Perhaps log what the feed was.

fbacall · 2026-04-09T16:26:06Z

lib/ingestors/rss_ingestion.rb

+    rescue StandardError
+      nil


fbacall · 2026-04-09T16:41:06Z

lib/ingestors/rss_ingestion.rb

+      return nil if candidate.blank?
+
+      URI.parse(candidate)
+      return candidate if URI::DEFAULT_PARSER.make_regexp(%w[http https]).match?(candidate)


Do you need to check the regexp if it's already parsed above? Can just check .scheme on the URI object.

fbacall · 2026-04-09T16:45:14Z

lib/ingestors/rss_ingestion.rb

+      links.map { |link| text_value(link.href) }.find(&:present?)
+    end
+
+    def resolve_feed_url(candidate_url, feed_url)


I'm not entirely sure what this method is for, but if you're trying to get an absolute URL from the candidate_url which may be relative or absolute, you can just use:

Addressable::URI.join(feed_url, candidate_url)

fbacall · 2026-04-09T17:03:56Z

lib/rss/media/atom.rb

@@ -0,0 +1,54 @@
+# Patches RSS::Atom::Feed and RSS::Atom::Entry with Media namespace support (see ../media.rb).
+# Kept as RSS::Media::Atom so Zeitwerk can autoload it from lib/rss/media/atom.rb.


Is anywhere autoloading it?

I think put this in config/initializers, where we already have other patches. It will also save you having to worry about loading twice.

fbacall · 2026-04-09T17:10:01Z

lib/ingestors/rss_ingestion.rb

+      if (url = discover_feed_url_from_youtube_playlist_url(base_url))
+        @messages << "Found Atom feed link from YouTube playlist URL, following: #{url}"
+        return url
+      end


I think it might be nicer if we have a subclassed Ingestor for YouTube ingestion. Then we don't have to explain to users that they need to pick the RSS ingestor for YouTube content.

fbacall · 2026-04-09T17:11:11Z

lib/ingestors/rss_ingestion.rb

+    def discover_feed_url_from_youtube_playlist_url(base_url)
+      uri = URI.parse(base_url)
+      host = uri.host.to_s.downcase
+      return nil unless host == 'youtube.com' || host.end_with?('.youtube.com')


Could perhaps re-use the is_youtube_url? method from lib/renderers/youtube.rb (make it not private)

fbacall · 2026-04-10T13:29:48Z

lib/ingestors/dublin_core_ingestion.rb

+      Array(values).map { |v| dublin_core_text(v) }
+                   .map(&:to_s)
+                   .map(&:strip)


Don't need to map 3 times, just do dublin_core_text(v).to_s.strip in the first block

eilmiv added 8 commits April 7, 2026 18:39

Refactor Dublin Core ingestion from OAI-PMH ingestor

5af12ee

Add RSS ingestion for materials and events

e36dbc2

Add tests for RSS ingestors

be76ff7

Add ingestors to factory

8c2880b

Add support for common extensions

54895a2

Fix Zeitwerk inflection problem with RSS

3cea73b

Add support for relative urls

2c7c05e

Fixes from testing many RSS feeds

a515e46

eilmiv requested a review from Copilot April 8, 2026 11:38

Copilot started reviewing on behalf of eilmiv April 8, 2026 11:39 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

lib/ingestors/material_rss_ingestor.rb Outdated Show resolved Hide resolved

lib/ingestors/rss_ingestion.rb Outdated Show resolved Hide resolved

lib/ingestors/rss_ingestion.rb Outdated Show resolved Hide resolved

eilmiv and others added 6 commits April 8, 2026 13:49

Remove start and end date for events based on date published in rss

cd91db6

Add feed url discovery from youtube url

b8f19c6

Fix error class that was too specific

b2780cf

Fix link handling in atom feeds

0f042e7

Use relative import for loading the custom rss media extention

89e5f53

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Add comment for dublin core to text conversion options

662c450

eilmiv requested a review from fbacall April 9, 2026 10:29

eilmiv marked this pull request as ready for review April 9, 2026 10:30

fbacall requested changes Apr 10, 2026

View reviewed changes

		@@ -0,0 +1,111 @@
		require 'rss'
		require_relative '../rss/media'

		@@ -0,0 +1,54 @@
		# Patches RSS::Atom::Feed and RSS::Atom::Entry with Media namespace support (see ../media.rb).
		# Kept as RSS::Media::Atom so Zeitwerk can autoload it from lib/rss/media/atom.rb.

Conversation

eilmiv commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eilmiv commented Apr 9, 2026

Uh oh!

fbacall left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

eilmiv commented Apr 7, 2026 •

edited

Loading