Solr fulltext indexing (#204)

* Add collective.solr using solr version and config provided there

* pre-commit reformatting of upstream checkin

* Remove intermediate solr/config directory

* Keep results-listing working with collective.solr.

- Backport the index fields from the old deployment.
- No test changes, because no test coverage on this view at all.

This is just a dirty MVP checkpoint. It works but the schema needs
cleaning up.

* Activate fulltext PDF solr search

Also: provide upgrade, adjust tests

* Move solr reindex to a separate upgrade step

* Load collective.solr zcml in zcml

* Remove migration notes from README

* Document Solr gotchas

* Reference upstream ticket on Tika integration

* Fix build regression introduced in dfddb0a

* Use getattr instead of hasattr

* Up Solr to 9.9.0

* Move test solr to a different port

Refs syslabcom/scrum#3974

* Temporary workaround for cross-site Unauthorized

* Move solr config into ./etc/ to avoid confusion

* Update README

* Change CI pre-commit runner

Author: gyst

.github/workflows/plone-package.yml

@@ -30,7 +30,7 @@ jobs:
           restore-keys: |
             ${{ runner.os }}-cache-
       - name: Code Analysis
-        uses: pre-commit/action@v3.0.1
+        uses: pre-commit-ci/lite-action@v1.1.0
         if: ${{ matrix.config[1] == '3.13' }}
       - name: Install dependencies
         run: |

CHANGES.rst

@@ -17,6 +17,8 @@ Changelog
 - Update the pre-commit linters [gyst]
 - Don't error out on main nav on a clean install [gyst]
 - Upgrade ftw.upgrade to the SLC patched version [gyst]
+- Add collective.solr and make the eea.facetednavigation view compatible with that [gyst]
+- Activate fulltext PDF solr search, provide upgrade, adjust tests [gyst]
 - Bugfix PublicationsViewlet for the Plone upgrade [gyst]
 - Add parameter -asxhtml to tidy to fix pdf generation of html reviews [pilz]
 - Fix the ``sehepunkte-import`` view

README.rst

@@ -17,37 +17,20 @@ We need some system packages to be available for this package to be fully functi
 - poppler_utils (pdf)
 - html-tidy (cleans up the html that will be converted to pdf)
 
+Solr: cave canem
+================
 
-Migration
-=========
+This package now works with collective.solr and performs fulltext indexing of PDF documents.
 
-Read `Notes on speed and large migrations <https://github.com/collective/collective.exportimport#notes-on-speed-and-large-migrations>`_ in the collective.exportimport README.
+There are a few gotchas:
 
-Then decide if you want to do an export from Plone 4 and only set blob paths.
-If so, set the `COLLECTIVE_EXPORTIMPORT_BLOB_HOME` variable on import.
+- Fulltext indexing of PDFs on reviews works fine, but fulltext indexing of standalone files will error out with a default Solr install.
+  To get it to work properly, your Solr needs to be started up with special environment variables::
 
-Also disable the recensio event subscribers which would generate PDFs from already generated content.
-To do this, set `RECENSIO_DISABLE_SUBSCRIBERS=true`.
+    SOLR_ENABLE_REMOTE_STREAMING=true SOLR_ENABLE_STREAM_BODY=true SOLR_OPTS="-Dsolr.allowPaths=${instance:blob-storage}"
 
-After successful migration import the page pictures, which were not in the Recensio Archetypes schema in Plone 4 and therefore not exported.
-The blob home environment variable must be set to import the page pictures.
+  For more background, see https://github.com/collective/collective.solr/issues/385
 
-`http://localhost:8080/recensio/@@import-page-pictures`
+- The recensio.plone buildout provided Solr is to be used only for testing; it will be nuked on test runs.
 
-
-Order of import:
-
-1) Create fresh Plone instance "recensio"
-2) Apply profile `recensio.plone:default`
-
-3) Import users (< 5m):           http://localhost:8080/recensio/@@import_members
-4) Import content: (3.5-4h):      http://localhost:8080/recensio/@@import_content
-5) Import relations: (5 minutes): http://localhost:8080/recensio/@@import_relations
-6) Import translations (0):       http://localhost:8080/recensio/@@import_translations
-7) Import local roles (< 5m):     http://localhost:8080/recensio/@@import_localroles
-8) Import default pages (< 5m):   http://localhost:8080/recensio/@@import_defaultpages
-9) Import ordering (~ 20m):       http://localhost:8080/recensio/@@import_ordering
-10) Import portlets (< 5m):        http://localhost:8080/recensio/@@import_portlets
-11) Reset dates (< 5m):            http://localhost:8080/recensio/@@reset_dates
-12) Fix collections (0m):          http://localhost:8080/recensio/@@fix_collection_queries
-13) Fix html (< 5m):               http://localhost:8080/recensio/@@fix_html
+Recensio.buildout provides both a testing solr (via buildout, on :8984) and a development solr (via Makefile, on :8983) to effect this split.

buildout.cfg

@@ -7,6 +7,8 @@ find-links +=
 parts =
     test
     createcoverage
+    solr
+
 show-picked-versions = true
 allow-picked-versions = true
 download-cache = downloads
@@ -23,23 +25,34 @@ eggs =
     Pillow
     recensio.plone [test]
 
-
 [createcoverage]
 recipe = zc.recipe.egg
 eggs = createcoverage
 
+[solr]
+recipe = kitconcept.recipe.solr
+src = https://www.apache.org/dyn/closer.lua/solr/solr/9.9.0/solr-9.9.0.tgz?action=download
+port = 8984
+solr-config = etc/solr
+
+
 [sources]
 guess-language = git https://github.com/syslabcom/guess-language.git
 
 [versions]
 createcoverage = 1.5
 path.py = 12.5.0
 z3c-jbot = 3.1
+kitconcept.recipe.solr = 1.0.0a5
 
 # Required by:
 # ftw.upgrade==3.4.0a3+slc0
 argcomplete = 3.6.2
 
+# Required by:
+# recensio.plone==2.0.0.dev0
+collective.solr = 9.4.0
+
 # Required by:
 # recensio.plone==2.0.0.dev0
 collective.vdexvocabulary = 0.3
@@ -52,6 +65,10 @@ collective.z3cform.datagridfield = 3.0.4
 # recensio.plone==2.0.0.dev0
 eea.facetednavigation = 16.4
 
+# Required by:
+# kitconcept.recipe.solr==1.0.0a5
+hexagonit.recipe.download = 1.7.1
+
 # Required by:
 # recensio.plone==2.0.0.dev0
 ftw.upgrade = 3.4.0a3+slc0