[Bug] Multithreaded World Generation on Paper 1.21.3

[viktordanov]

Pre-Issue Checklist

• I have checked that I am on the latest version of Terra. (Running on branch ver/6.6.0 commit 1d3c380)
• I have searched the github issue tracker for similar issues, including
  closed ones.
• I have made sure that this is not a bug with another mod or plugin, and it
  is Terra that is causing the issue.
• I have checked that this is an issue with Terra and not an issue with the
  pack I am using.
• I have attached a copy of the latest.log file
• I have filled out and provided all the appropriate information.

Environment

Name	Value
Terra Version	ver 6.6.0 commit 1d3c380 (last commit at the time of writing)
Platform / Platform Version	Paper 1.21.3 Build 15 With MultiVerse
Any External Plugins or Mods	Multiverse-Core (4.3.13)
Terra Packs In Use	https://github.com/PolyhedralDev/TerraOverworldConfig from 2.0 Branch (last commit)
Terra Addons In Use	All default addons which are preinstalled on a fresh Terra run

Issue Description

Terrain Generation used to be extremely quick. A few months ago I upgraded my hardware and reconfigured Paper to try to max out my CPU for the fastest generation.
My paper-global.yml

_version: 29
block-updates:
  disable-chorus-plant-updates: false
  disable-mushroom-block-updates: false
  disable-noteblock-updates: false
  disable-tripwire-updates: false
chunk-loading:
  autoconfig-send-distance: true
  enable-frustum-priority: true
  global-max-chunk-load-rate: -1.0
  global-max-chunk-send-rate: -1.0
  global-max-concurrent-loads: 800.0
  max-concurrent-sends: 24
  min-load-radius: 4
  player-max-chunk-load-rate: -1.0
  player-max-concurrent-loads: 30.0
  target-player-chunk-send-rate: 400.0
chunk-loading-advanced:
  auto-config-send-distance: true
  player-max-concurrent-chunk-generates: 320
  player-max-concurrent-chunk-loads: 320
chunk-loading-basic:
  player-max-chunk-generate-rate: 200.0
  player-max-chunk-load-rate: 400.0
  player-max-chunk-send-rate: 400.0
chunk-system:
  gen-parallelism: 'true'
  io-threads: 24
  worker-threads: 28

My start-up params: java -Xms20G -Xmx20G -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=200 -XX:+UnlockExperimentalVMOptions -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=40 -XX:G1HeapRegionSize=8M -XX:G1ReservePercent=20 -XX:G1HeapWastePercent=5 -XX:G1MixedGCCountTarget=4 -XX:InitiatingHeapOccupancyPercent=15 -XX:G1MixedGCLiveThresholdPercent=90 -XX:G1RSetUpdatingPauseTimePercent=5 -XX:SurvivorRatio=32 -XX:+PerfDisableSharedMem -XX:MaxTenuringThreshold=1 -DPaper.WorkerThreadCount=28 -Dusing.aikars.flags=https://mcflags.emc.gs -Daikars.new.flags=true -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5050 -jar paper-1.21.3-15.jar --nogui

Since my upgrade to 1.21.3 from 1.21.1 Terra Worlds started generating very slowly with minimal CPU usage.

Ablation tests for issue isolation

1. Normal Minecraft world generation is using all allocated cores [Fast]
2. Using other generators with Multiverse-Core (tested random plugins from spigotmc and modrinth [Fast]
3. Tested default.zip with Terra [Slow]
4. Tested the OverworldConfig pack with Terra [Slow]

Steps to reproduce

1. Generate new world / enter existing
2. Fly around to generate new chunks
3. Observe CPU usage and visible chunk generation speed

Expected behavior

All allocated cores should be used.

Actual behavior

Only 1 or no more than 4 are used (difficult to discern).

[viktordanov]

latest.log

[viktordanov]

I identified part of the problem:

In platforms/bukkit/common/src/main/.../PlatformImpl.java

    public PlatformImpl(TerraBukkitPlugin plugin) {
        generationThreads = getGenerationThreadsWithReflection("ca.spottedleaf.moonrise.common.util.MoonriseCommon", "WORKER_THREADS", "Moonrise");
        System.out.println("Generation threads: " + generationThreads);
        if (generationThreads == 0) {
            generationThreads = 1;
            System.out.println("Failed to get generation threads, defaulting to 1.");
        }
        this.plugin = plugin;
        load();
    }

I get these logs:

[11:05:41 WARN]: [com.dfsek.terra.AbstractPlatform] Moonrise found, but WORKER_THREADS field not found this probably means {0} has changed its code and Terra has not updated to reflect that.
[11:05:41 INFO]: [STDOUT] [com.dfsek.terra.bukkit.PlatformImpl] Generation threads: 0
[11:05:41 INFO]: [STDOUT] [com.dfsek.terra.bukkit.PlatformImpl] Failed to get generation threads, defaulting to 1.
[11:05:41 INFO]: [com.dfsek.terra.AbstractPlatform] Initializing Terra...
[11:05:41 INFO]: [com.dfsek.terra.AbstractPlatform] Loading config.yml
[11:05:41 INFO]: [com.dfsek.terra.config.PluginConfigImpl] Loading config values from config.yml
[11:05:41 INFO]: [com.dfsek.terra.AbstractPlatform] Loading 39 Terra addons:
        - biome-provider-extrusion@1.0.0-BETA+65e4c9a14
        - biome-provider-image@1.0.1-BETA+65e4c9a14
        ```

Check https://github.com/Tuinity/Moonrise/blob/ea50ba38eada4c852a52f9bc3fd0960bcf57c39b/src/main/java/ca/spottedleaf/moonrise/common/util/MoonriseCommon.java#L47

[OakLoaf]

I'll try to take a look into this shortly!

[solonovamax]

also:

• https://noflags.sh/
• please try bumping up the IO threads. this may possible be caused by the IO thread pool being saturated and then generation blocking on those. unsure. setting it to like 2-4x the worker threads probably should be fine? highly unlikely that this causes performance to degrade. (though if it does, try reverting back)

[OakLoaf]

The moonrise code has just changed, I'll hopefully have a fix for it soon

[viktordanov]

Any updates?