Multithreaded file scan

[wgbusch]

Description

• When I scan files using the "Scan Directory" feature, retroarch doesn't seem to do any parallelization or efficiently use the CPU. This should be a highly parallelizable task, Im not sure why is implemented linearly. I'm a developer but I have zero experience with the retroarch code, if someone could point me out to where I have to look to implement this feature I will gladly take a look myself and try to implement it.
  

Expected behavior

Scan Directory should be parallelizable.

Actual behavior

Very low CPU usage, making the Scan Directory extremely slow.

Steps to reproduce the bug

1. Scan a directory with thousands of small files (ideally more than 10k).

Bisect Results

[Try to bisect and tell us when this started happening]

Version/Commit

You can find this information under Information/System Information

• RetroArch: 1.10.2

Environment information

• OS: Windows 10
• Compiler: [In case you are running local builds]

[hizzlekizzle]

I'm very much not an expert in this sort of thing, but I believe the code in question lives in tasks/task_database.c. The "tasks" are designed to be nonblocking, but I think a fully multithreaded implementation would need to be behind an ifdef HAVE_THREADS, at least.

[LibretroAdmin]

What @hizzlekizzle said.

We'd very much appreciate somebody taking a stab at this. Just design it so that there is still the possibility to do it singlethreaded as well.

[zoltanvb]

Some thoughts on this, as I was working on #18341.

The basic (auto)scan functionality is iterated in the following manner:

1. a file list is assembled (which may be extended later, if archives are encountered)
2. another RA task is created for each file in turn, while the main scanning task (that displays the progress) stays alive during the whole process
3. checksum (or serial) is read, and checked against each database with a query, which in turn reads each .rdb
4. when a match is found, result is written to playlist immediately

Spawning parallel tasks for (2) may be possible, but it will require consideration because of (4), and also I would be a bit worried by loading too much to RA task system. Implementing and debugging low-level parallelization issues is not much fun, at least to me.

Improving scan performance can still be done without this, I do plan to implement some of it in the future.

[warmenhoven]

(4) is a problem also because it turns scanning into an O(n^2) algorithm, having to re-read and re-write the entire playlist each time there's a match. For people scanning thousands of entries in a playlist this becomes quite noticeable. Having each subtask return its result to the parent task that does the aggregation and final playlist write(s) would fix that as well.