Fix google rankings of docs by using weighting / google sitemaps

[ukd1]

For me I get:

aka, linking to https://crystal-lang.org/api/0.20.1/HTTP/Client.html

I actually want https://crystal-lang.org/api/0.30.0/HTTP/Client.html, or "latest" (https://crystal-lang.org/api/latest/HTTP/Client.html).

This can be done, with sitemap (https://en.wikipedia.org/wiki/Sitemaps) using weighting, aka priority:

<?xml version="1.0" encoding="utf-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">
    <url>
        <loc>https://crystal-lang.org/api/0.20.1/HTTP/Client.html</loc>
        <lastmod>2019-07-10</lastmod>
        <changefreq>daily</changefreq>
        <priority>0.1</priority>
    </url>
    <url>
        <loc>https://crystal-lang.org/api/0.30.0/HTTP/Client.html</loc>
        <lastmod>2019-07-10</lastmod>
        <changefreq>daily</changefreq>
        <priority>1.0</priority>
    </url>
</urlset>

etc...

[straight-shoota]

There is essentially discussed crystal-lang/crystal#5952
The idea was to use canonical references to the latest url. But that also has some issues. And most importantly, it currently doesn't cover doc pages < 0.25.0 so these still show up on the search results (but there are no between 0.25.0 and 0.30.0 because they all point to latest).

Using a sitemap seems like a smart alternative to solve this issue. 👍

[bcardiff]

Does someone know if there is a tool for generating the sitemap does some recursive checks over directories? Unless it can be automated it won't happen.

Doing a pass over old docs to set the canonical seems more likely if that fixes the issue.

[straight-shoota]

This should be fairly simple to automate. Essentially you only need to extract all local links from each version's index.html in the API docs. That's already the URLs for the sitemaps. All but those links from the most recent version get a lower priority.
Each released API docs should be in a distinct sitemap file and all can be combined together in a sitemap index.

I can try to put this together if nobody else is interested.

[ukd1]

@straight-shoota nice, didn't think to check for the issue on the main repo somehow...lol. I'd be down for helping on this, but tbh, I have no idea how the docs or old versions are built - so it might be easier if you do it. If you'd like a hand / pair / lmk.

[RX14]

Doing a pass over old docs to set the canonical seems more likely if that fixes the issue.

this can be done with a simple sed script to adjust the header for old versions. The doc pages don't need to be regenerated.

[RX14]

I'd suggest redoing thse old pages before working on a sitemap

[straight-shoota]

@RX14 The problem with canonical links is that docs for older versions vanish from the search results because the search algorithm treats them as duplicate content. But in fact, they're not duplicate and users might have a need to find documentation for older versions as well. For example, when upgrading to a new version, you may need to read the documentation of deprecated features not available in the current version in order to find a suitable replacement.

Using a sitemap is a superior solution because it allows to assign priorities to individual pages and outdated pages don't vanish completely, they just won't be as prominent as more recent once. I think it should eventually replace the canonical link.

[straight-shoota]

I've put together a simple program to automatically generate sitemaps for https://crystal-lang.org/api

It's available at https://github.com/straight-shoota/crystal_docs_sitemap

Generated output: output.tar.gz

The output contents should be published at https://crystal-lang.org/api/ and search engines need to be informed about the sitemap (see https://www.sitemaps.org/protocol.html#informing).

[straight-shoota]

@bcardiff WDYT?

[bcardiff]

I agree the sitemap is worth having and is a good solution. I don't think having the sitemap checked in the repo is the right thing.

From a workflow point of view what it would make sense is to have a tool to change an existing sitemap with some operations:

1. Add dir content as it will be reached from specific url prefix
2. Set the priority for all routes matching a specific prefix

At least that workflow will play well with the release process, where we have a local dir with the new api documentation to upload.

I am unsure how to keep the sitemap up to date with respect to the content from jekyll itself regarding lastmod of existing pages and new posts. Maybe there is a 3rd action

3. Update last most for a subset of dirs as it will be reached from specific url prefix.

That way we can update those params without iterating the whole content.

So, in essence, is having an approach to update rather than create a sitemap.

[straight-shoota]

I don't think having the sitemap checked in the repo is the right thing.

Agreed. It just needs to be generated and put into an S3 bucket. Ideally, a rebuild should be triggered after the nightly API docs have been updated from master.

I am unsure how to keep the sitemap up to date with respect to the content from jekyll itself regarding lastmod of existing pages and new posts.

Currently, these sitemaps are only for /api, so Jekyll is not even involved. This is perfectly fine, the sitemap doesn't need to incorporate all pages on the domain. Getting the priorities right for the API versions is the main issue here, and I'd like to get that fixed before considering other parts of the website. They can be tackled individually (for example, Jekyll can simply build its own sitemap), we just need to reference all sitemaps from the sitemapindex.

So, in essence, is having an approach to update rather than create a sitemap.

Sure, we can do that. I just figured it would be easier to simply run the generator and push the result to S3 without having to synchronize first.

In practice, there are two events that would require an update to the sitemaps:

1. Every day the updated nightly API docs are published for master. This only needs a rebuild of the sitemap for /api/master.
2. When a new Crystal version is released, we need to build the sitemap for the new release and rebuild for the last x releases in order to update the priority. x is currently 3: The last 2 versions get priorities (0.5, 0.3) and the one after that needs to be set to the default (0.1)

The priority adjustments could actually just be implemented with a simple grep. The contents don't change when a release age, thus there is no need to actually rebuild the sitemap.

Considering all this, it might actually be the best solution to integrate the sitemap generation into the doc generator. This problem is not specific to the stdlib and this way all shards API docs could benefit.
This is really trivial to implement, it just spits out another file. And won't require additional configuration, there is already --canonical-base-url and priority could just be 1.0 by default. Maybe a --sitemap-priority option could be useful, but it's not necessary.

To build the sitemaps for legacy releases, we can just use https://github.com/straight-shoota/crystal_docs_sitemap That's a one-time thing.

With this, updates to master sitemap don't need any additional action because the updated sitemap is already provided by the doc generator.
When a new release is added, we need to add it to the sitemapindex and update the sitemaps for the last releases, but this could just be s/priority="1.0"/priority="0.5"/ etc.

[bcardiff]

Currently, these sitemaps are only for /api, so Jekyll is not even involved

Wouldn't that prevent indexing other pages?

Every day the updated nightly API docs are published for master.

I thought we didn't want sitemap for master. Is mostly used for preview (edit: sorry you mention it at the end)

When a new Crystal version is released,

I'm ok downloading the whole docs for a first time generation (edit: or using the proposed script), but upon a crystal version release I don't have locally all the bucket of docs. And I don't want to require to download it. What I do have is the -doc.tar.gz artifact that is pushed. I was thinking of injecting the new paths there, without actually retrieving them from http or the bucket. Hence the proposed transformations 1 and 2.

[straight-shoota]

Wouldn't that prevent indexing other pages?

No, sitemaps are not used as an exclusive source. Search engines still employ their regular crawling. They just augment the results or help discover pages that would otherwise not be discovered. See https://webmasters.stackexchange.com/questions/114425/if-i-remove-urls-from-an-xml-sitemap-will-google-still-index-them