Stop google from indexing internal URLs

[plexus]

Today when googling for something Google returned a clojurians-log-internal URL. This bypasses CloudFlare, going to the server directly.

We have canonical meta tags on our pages, but somehow Google is still indexing the wrong URL.

Not sure what else we can do about this, but some ideas

• serve a more aggressive robots.txt when accessed via -internal
• redirect from internal to the canonical URL if it looks like a search engine crawler
• register the internal domain as well in google search console, maybe that way we get some control over it