KAFKA-20541 : Move 42/generated from static/ to content/

[muralibasani]

• Adds a catch-all module.mounts entry in hugo.yaml so future Kafka releases that ship docs/generated/ under content/en/<v>/generated/ (the layout used by apache/kafka) are served at /<v>/generated/ URLs automatically
• no per-release edits to hugo.yaml, shortcode, or script changes needed.
• /42/generated/connect_rest.yaml still downloads the original file
• a new config in hugo.yaml to exclude the empty files in 4.2 (these empty files exist only in 4.2). Without this, empty stubs would shadow real files because of catch all config.
• Tested with a new version content/en/99/generated/test.html which is served at /99/generated/test.html

[mimaison]

Thanks for picking this up.

I guess I should put some thoughts in the Jira when opening it. While your PR resolve the duplication, I'm not sure it's the approach we should take to resolve this. When we generate docs for a release, the generated folder is there alongside apis, configurations, etc. This is why it was there under 42.

I wonder if the website should pick generated from that location instead of static.

[muralibasani]

Thanks for picking this up.

I guess I should put some thoughts in the Jira when opening it. While your PR resolve the duplication, I'm not sure it's the approach we should take to resolve this. When we generate docs for a release, the generated folder is there alongside apis, configurations, etc. This is why it was there under 42.

I wonder if the website should pick generated from that location instead of static.

@mimaison pls let me know if the below approach is better to handle this situation.

Currently there are a few empty files in content/en/42/generated too.
We can move files from static/42/generated to content/en/42/generated. (With this, generated dir will be deleted from static)
A mount config change in hugo.yml /42/generated/ pointing to content/en/42/generated/. This will keep the connect_rest.yaml downloadable file url as is.
include-html (shortcode) also to be pointed to new content location and fallback to static//generated/ so versions prior to 4.2 keep working as well.

Finally update the release playbook, so for 4.3 onwards, we can keep the generated/ with the docs. We don't have to copy it to the static folder.
With this, there are no URL changes and everything works the same way. Pages, downloads, urls, etc

I tested this locally and it works.

[muralibasani]

@mimaison pushed the changes too. It's a large PR, but mostly moving of files to content dir, and handling the include-html shortcode.

[mimaison]

Yes I think something along these lines would work. Thanks for figuring it out.

I'm wondering whether we should keep all existing versions as is, at least for now, and only start with the new scheme for newer versions. So we'd do the changes in the kafka repository instead.

Also it seems the new process requires editing hugo.yaml for each release, could we invert the process to include mapping all existing versions, and not require changes for new releases?

[muralibasani]

Yes I think something along these lines would work. Thanks for figuring it out.

I'm wondering whether we should keep all existing versions as is, at least for now, and only start with the new scheme for newer versions. So we'd do the changes in the kafka repository instead.

Also it seems the new process requires editing hugo.yaml for each release, could we invert the process to include mapping all existing versions, and not require changes for new releases?

@mimaison new changes now handle only newer versions and no edits are required for hugo config going fwd.
Added one excludeFiles config to skip empty stubs in 4.2. (if we don't need this config, need to delete those empty files and they only exist in content/4.2/generated)

And for new releases: copy docs/ into content/en// and images/javadoc into static//, catch-all will take care.
Tested with a new version locally.

[mimaison]

I've just tried with 43 (rebasing your branch on top of markdown) and even though generated is only under static it still worked, is it expected?

[muralibasani]

I've just tried with 43 (rebasing your branch on top of markdown) and even though generated is only under static it still worked, is it expected?

Yes, that's expected. It will work in both ways.
It adds support for content/en//generated/ without removing the existing static//generated/ path.

static//generated/ (old way) served by Default static → static mount
content/en//generated/ (new way) served by the catch-all mount entry.

RM can ship generated/ under either static// or content/en// and both work.

[mimaison]

If it's expected to work both ways, can you clarify why 4.2 is treated as a one off?

[muralibasani]

If it's expected to work both ways, can you clarify why 4.2 is treated as a one off?

Both ways work in the sense that files exist in either content/en//generated/ or static//generated/ , pick one.

4.2 is the only version where files exist in both dirs (real files and empty stubs). And to prevent that catch-all from serving those empty stubs, we added excludeFiles config https://github.com/apache/kafka-site/pull/856/changes#diff-5928f4025679c9eee397ca3795367f71f667ce1f25018b7f7ae7c574da1f2f08R524

And per your earlier comment, we left 4.2 untouched in this PR.

[mimaison]

Now that 4.2.1 is out, maybe we can update 4.2 to match 4.3 to get rid of its unique configuration. Sorry for the back and forth, I'm figuring it out myself.

[muralibasani]

Now that 4.2.1 is out, maybe we can update 4.2 to match 4.3 to get rid of its unique configuration. Sorry for the back and forth, I'm figuring it out myself.

Done. Removed content/en/42/generated/ to match 4.3's layout, and dropped the excludeFiles line.
Tested both versions. Look good.

[mimaison]

Sorry but as I said in #856 (comment), when building the docs the generated folder is in the artifact that we put under content/en. So I think it would be better to pick it from there.

Currently generated is picked from static. So we want to:

• delete the generated copy under static/42, 42 currently has a copy under content/en
• move the generated folder of 43 under content/en

Does that make sense?

[muralibasani]

Sorry but as I said in #856 (comment), when building the docs the generated folder is in the artifact that we put under content/en. So I think it would be better to pick it from there.

Currently generated is picked from static. So we want to:

• delete the generated copy under static/42, 42 currently has a copy under content/en
• move the generated folder of 43 under content/en

Does that make sense?

Yes.

1. Moved generated/ from static/42/ and static/43/ into content/en/42/generated/ and content/en/43/generated/. (For 4.2: the previously-existing content/en/42/generated/ was empty stubs that I deleted in the last commit; this step moved the real files from static/ into that location.)
2. Updated the include-html shortcode to read from the content path, with a fallback to /static// for pre-4.2 versions so we don't need to touch their markdown. (older versions (07–41) keep working without any markdown edits.)
3. Kept the catch-all mount so //generated/connect_rest.yaml urls still work.

[mimaison]

Thanks for the updates! This looks good.

Can we also update the README? Section that would need updating are:

• https://github.com/apache/kafka-site#2-setup-static-assets
• https://github.com/apache/kafka-site#supported-features
• https://github.com/apache/kafka-site#helper-script

We also need to update https://github.com/apache/kafka-site/blob/markdown/scripts/replace-version-links.sh

[muralibasani]

Thanks for the updates! This looks good.

Can we also update the README? Section that would need updating are:

• https://github.com/apache/kafka-site#2-setup-static-assets
• https://github.com/apache/kafka-site#supported-features
• https://github.com/apache/kafka-site#helper-script

We also need to update https://github.com/apache/kafka-site/blob/markdown/scripts/replace-version-links.sh

Thanks. Updated all four:

• Setup Static Assets: dropped the generated/ copy line.
• Supported Features: shortcode example reflects content path + static/ fallback.
• Helper Script: added a bullet for the new content/en//generated/ rule.
• replace-version-links.sh: added a matching sed rule and updated the comment.