Scanning optimization #18341
Merged
Scanning optimization #18341
+255
−41
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Extend scanning logic by filtering databases, so that queries that will certainly be unsuccessful, are skipped. Filtering criteria: - size of scanned file - presence of serial (in case of serials) Two extra queries for min and max are run for each database on the first scan, this causes single-file scans to be somewhat slower, but it pays off with large scans. Archive size is also checked. The built-in shortcut of scanning both the archive (zip file) and the first (possibly only) file inside the archive is retained, it makes logic more complicated, but the speed difference is significant for sets that contain one zipped file per title.
Collaborator
|
I think this commit is what broke 3DS scanning. |
Contributor
Author
|
Scanning on 3DS, or scanning of 3DS content? |
Collaborator
|
Sorry, scanning 3DS content. I think the max size comes back as 0 and it takes it literally. |
Merged
Collaborator
|
Scanning F355 Challenge - Passione Rossa (Europe) (En,Fr,De,Es,It).chd for Dreamcast is also failing and if I remove the sizehint check it succeeds. The rdb has the size of Track 3 which is large, and Track 1 is what is being compared against, and is tiny relatively. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Extend scanning logic by filtering databases, so that queries that will certainly be unsuccessful, are skipped.
Filtering criteria:
Two extra queries for min and max are run for each database on the first scan, this causes single-file scans to be somewhat slower, but it pays off with large scans (~20% on a desktop system when scanning a NES set).
Archive size is also checked. The built-in shortcut of scanning both the archive (zip file) and the first (possibly only) file inside the archive is kept unchanged, it makes logic more complicated, but the speed difference is significant for sets that contain one zipped file per title.
Tested with several different content types, produces same results as previously (just a bit faster, in case of large sets).
Very verbose logs added to debug builds - they actually slow the scanning down, so I did not want it to be part of regular builds.
Related Issues
Fixed one issue when some playlist entries were added to playlist with previously matched serial number instead of the valid CRC (not sure if it had any effect on usage).
Removed an undocumented shortcut: if scanned folder matched the database name exactly, scan was limited to that database. I plan to re-introduce a similar option later (use first match).
Serial scan inside archives does not work primarily because of a shortcut (line 1557), but even if it was removed, there are some remaining issues. This PR does not change anything in that respect.
Could not reproduce #15712 during all the experiments.
Related Pull Requests
Preparatory work:
libretro/libretro-database#1581
#18314
#18315