DOCS-194 - Update ignorePatterns, exclude patterns, robots.txt

docs/contributing/remove-doc.md

@@ -59,3 +59,31 @@ If you're deprecating one solution in favor of another (like in the above **Ngin
 ### Hide from search engine results
 
 It can take months for an old URL to drop from Google search engine results. If you need to hide a page immediately from Google, [open a GitHub Issue](https://github.com/SumoLogic/sumologic-documentation/issues/new/choose) and provide the details for our site admin.
+
+### Exclude pages from the site build
+
+You can prevent content from being indexed by excluding it from compilation entirely. Pages that aren't compiled into the build output won't appear on the live site or sitemap, so neither search engines nor internal site search can index them.
+
+To exclude docs, add their paths to the `docs.exclude` setting in `docusaurus.config.js`:
+
+```json title="docusaurus.config.js"
+presets: [
+  [
+    '@docusaurus/preset-classic',
+    ({
+      docs: {
+        exclude: [
+          '**/reuse/**',
+          '**/ja/**',
+        ],
+      },
+    }),
+  ],
+];
+```
+
+Any markdown files in the listed directories are excluded from the build and won't be served or indexed. Note that `reuse` files are still embedded where referenced — they just won't compile as standalone pages.
+
+For finer-grained control — for example, keeping a page compiled but hidden from search — use `robots.txt` or `noindex` meta tags instead. If content should never be published at all, excluding it from compilation is the most reliable approach.
+
+If a document has already been indexed and needs to be urgently removed from Google search results, follow [Google's removal instructions](https://support.google.com/webmasters/answer/9689846?sjid=11985967130976965420-NC).

docs/contributing/style-guide.md

@@ -1317,7 +1317,7 @@ For repeatable content - an identical section that appears in one or more docs -
 
 Headings in the reuse folder will not appear in the right-side nav in docs where they're imported. For this reason, unless the headers are H4 or below, reuse the content only but retain headings in the other docs.
 
-Example: the content under the Cloud-to-Cloud source docs > Restarting Your Source section ([example](/docs/c2c/info/#restarting-your-source)) lives in the `reuse` folder. The H3 headers live in the actual docs, and H4 headers live in [the `reuse` file](/docs/reuse/restart-c2c-source):
+Example: the content under the Cloud-to-Cloud source docs > Restarting Your Source section ([example](/docs/c2c/info/#restarting-your-source)) lives in the `reuse` folder. The H3 headers live in the actual docs, and H4 headers live in the `reuse` file:
 
 <Tabs
   className="unique-tabs"

docusaurus.config.js

@@ -107,12 +107,21 @@ module.exports = {
               'training',
             ],
           },
+          exclude: [
+            '**/reuse/**',
+            '**/ja/**',
+          ],
         },
         sitemap: {
           lastmod: 'date',
           changefreq: 'daily',
           ignorePatterns: [
             '/docs/reuse/**',
+            '/docs/beta/**',
+            '/ja/**',
+            '/files/**',
+            '/release-notes-*/archive/**',
+            '/release-notes-*/2021/**',
             '/tags/**'
           ],
           filename: 'sitemap.xml',

static/robots.txt

@@ -1,19 +1,22 @@
 # Algolia-Crawler-Verif: 4E25C3C253AC39D1
+# Sumo Logic Help Docs
+# AI/LLM crawlers are welcome - see /llms.txt for guidance
 
 User-agent: *
-Disallow: /docs/reuse/
-Disallow: /tags/
+Disallow: /docs/reuse/*
+Disallow: /docs/beta/*
+Disallow: /tags/*
 Disallow: /*?tags=
+Disallow: /ja/*
+Disallow: /files/*
+Disallow: /cid-redirects.json
+Disallow: /release-notes-*/archive
+Disallow: /release-notes-*/2021/
+Disallow: /release-notes-*/rss.xml
 Disallow: /adler32/
 Disallow: /deki/
 Disallow: /crc32/
 Disallow: /zlib/
 Disallow: /inffast/
-Disallow: /ja/docs/
-Disallow: /cid-redirects.json/
-Disallow: /files/
-
-# Sumo Logic Help Center
-# AI/LLM crawlers are welcome - see /llms.txt for guidance
 
 Sitemap: https://www.sumologic.com/help/sitemap.xml