Use mapper-provided romanized names (ja-Latn, ko-Latn) instead of machine transliteration#25425
Open
MiMoHo wants to merge 2 commits into
Open
Use mapper-provided romanized names (ja-Latn, ko-Latn) instead of machine transliteration#25425MiMoHo wants to merge 2 commits into
MiMoHo wants to merge 2 commits into
Conversation
name:ja-Latn (165k uses) and name:ko-Latn (139k uses) were dropped during map creation because the language list did not contain them. Also harden EntityParser to accept BCP 47 capitalized language suffixes: production parsing lowercases tag keys, but paths using putTagNoLC (e.g. the OSM editing download with setConvertTagsToLC(false)) preserve the original case and previously failed the langsSet lookup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When name:en is missing and transliteration is enabled, use a stored romanized name (name:ja-Latn, name:ko-Latn, name:sr-Latn, ...) before falling back to Junidecode, which e.g. renders Japanese kanji with Chinese readings (大山阿夫利神社 -> "Da Shan A Fu Li Shen She" instead of the mapper-provided "Oyama Afuri Jinja"). Fixes osmandapp#25267 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #25267
Problem
For objects without
name:en, OsmAnd falls back to machine transliteration (Junidecode), which produces wrong results especially for Japanese — kanji get Chinese readings. Example from the issue, way 1175336500 (大山阿夫利神社,name:ja-Latn= "Oyama Afuri Jinja"):The mapper-provided names can't even be used today because
name:ja-Latn(165,044 uses) andname:ko-Latn(138,850 uses) are not inMapRenderingTypes.langs, so they are dropped during map creation (unlikesr-latn/zh-hans/zh-hant, which are already indexed).Changes
MapRenderingTypes.langs: addja-latnandko-latn(placed next toja/ko, matching the existingsr-latnstyle). Takes effect once maps are regenerated.EntityParser.parseMapObject(): lowercase the language suffix before thelangsSetlookup and store names under the lowercase code. Production parsing already lowercases tag keys (Entity.putTag), but code paths usingputTagNoLC— e.g. the OSM editing download, which setssetConvertTagsToLC(false)— preserve OSM's BCP 47 capitalization (name:ja-Latn,name:sr-Latn) and previously failed the lookup.MapObject.getEnName(true): prefer a stored Latin-script name (key ending in-latn) over machine transliteration whenname:enis missing. Explicitname:enstill wins; behavior with transliteration disabled is unchanged; objects without a romanized name fall back to Junidecode exactly as before.int_name(907k uses) could be a further fallback candidate, but it is not indexed today and not guaranteed to be Latin script, so it is left out of this PR.Testing
Compiled the patched classes against the current master snapshot jar and ran a standalone test (class-shadowing), 14 checks — all pass with the patch; the unpatched jar shows the current behavior:
name:ja-Latnindexed (lowercased keys, production path)ja-latn✓name:ko-Latnindexedko-latn✓name:ja-Latn/name:sr-Latnvia case-preserving path (putTagNoLC)getEnName(true)for 大山阿夫利神社 withja-Latnname:sr-latn,name:zh-Hans,name:de(production path)name:enprecedencegetEnName(false)withoutname:enClaude Code — Claude Fable 5, xhigh