wish: WARCHdfsBolt with CDX index

[dportabella]

StormCrawler allows to filter web pages and archive them into WARC archives, as follows:

WARCHdfsBolt warcbolt = (WARCHdfsBolt) new WARCHdfsBolt().withFileNameFormat(fileNameFormat);

TopologyBuilder builder = new TopologyBuilder();

builder.setBolt("warc", warcbolt, numWorkers)
  .localOrShuffleGrouping("parse", WarcStreamName)
  .localOrShuffleGrouping("tika",  WarcStreamName);

Would it be possible to create a CDX index (or JCDX index) for the WARC archives at the same time?

[sebastian-nagel]

Should be possible:

• cf. WarcCdxWriter for Nutch
• should ev. exclude most of the dependencies of webarchive-commons which is required to get the SURT key

[jnioche]

Doable indeed.

@dportabella the warc bolt is usually connected to the fetcher, not the parsers

builder.setBolt("warc", warcbolt).localOrShuffleGrouping("fetch");

[sebastian-nagel]

Alternatively, the WARC bolt could add WARC file name, record offset and length to the metadata. An indexer (CDX or anything else) then could store it directly which obsoletes the need to index the CDX files in a separate step.