refactor(normalizer): 모듈 수준 deprecated 함수 래퍼 추가 (#232) - #247
Merged
Conversation
- normalize(), remove_doublespace(), repeat_normalize(): DeprecationWarning 추가 (이전 커밋) - only_hangle(), only_hangle_number(), only_text(): DeprecationWarning 추가 - emoticon_normalize(): repeat_normalize() 대신 RepeatCharacterNormalizer 직접 호출로 이중 경고 방지 - __init__.py: EmojiNormalizer export 추가 - test_normalizer.py: 6개 deprecated 함수 경고 발생 테스트 추가 (15 tests all pass) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
lovit
added a commit
that referenced
this pull request
Mar 9, 2026
* fix(normalizer): RemoveLongspaceNormalizer 공백 정규화 버그 수정 (#242) - 기존: 2개 이상 공백을 두 공백으로 치환 (정규화 효과 없음) - 수정: \s+ 패턴으로 탭·개행 포함 모든 연속 공백을 단일 공백으로 치환 - 영향 받는 테스트(test_normalizer_builder) 기대값 갱신 Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(normalizer): emoticon_normalize deprecated 처리 및 HangleEmojiNormalizer로 통합 (#243) - emoticon_normalize(): DeprecationWarning 발생 후 HangleEmojiNormalizer로 위임 - 기존 구현의 버그 문서화: 완성형 음절 뒤 자음에서 종성 불일치 시 음절 묵소 삭제 (예: 'ㅋ크ㅋ' → 'ㅋㅋ', HangleEmojiNormalizer는 'ㅋ크ㅋ'로 올바르게 유지) - soynlp.hangle.compose/decompose import 제거 (normalizer 모듈에서 미사용) - test_emoticon_normalize_deprecated 테스트 추가 Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(normalizer): EmojiNormalizer 추가 — Unicode 이모지 처리 (#245) - EmojiNormalizer(Normalizer) 신규 추가 - emoji 패키지 기반으로 단일·복합(ZWJ/Skin tone/Regional indicator) 이모지 처리 - replace="" (기본, 제거) / replace="[EMOJI]" (토큰 치환) 지원 - emoji 미설치 시 명확한 ImportError 메시지 반환 - pyproject.toml: optional-dependencies [emoji] 섹션 추가 - dev 의존성에 emoji>=2.0 추가 (lint/test 환경용) - test_emoji_normalizer 테스트 추가 (단일 이모지, ZWJ 시퀀스, Skin tone, Regional indicator, 치환 옵션 검증) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(normalizer): 모듈 수준 deprecated 함수 래퍼 추가 (#247) - normalize(), remove_doublespace(), repeat_normalize(): DeprecationWarning 추가 (이전 커밋) - only_hangle(), only_hangle_number(), only_text(): DeprecationWarning 추가 - emoticon_normalize(): repeat_normalize() 대신 RepeatCharacterNormalizer 직접 호출로 이중 경고 방지 - __init__.py: EmojiNormalizer export 추가 - test_normalizer.py: 6개 deprecated 함수 경고 발생 테스트 추가 (15 tests all pass) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(pipeline): task_normalize() deprecated 처리 및 normalize pipeline 예시 추가 (#248) - task_normalize(): DeprecationWarning 추가 (ReadTextTask + NormalizeTask + WriteTextTask 조합 안내) - examples/normalize/pipeline.yaml: NormalizeTask를 사용하는 예시 파이프라인 추가 NormalizeTask는 이미 soynlp/pipeline/tasks/normalize.py에 구현되어 있으며 pipeline/__init__.py 및 TASK_REGISTRY에도 등록되어 있음 Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(pipeline): 개별 XxxNormalizer pipeline Task 6종 추가 (#249) - PassCharacterNormalizeTask, HangleEmojiNormalizeTask, EmojiNormalizeTask - RepeatCharacterNormalizeTask, RemoveLongspaceNormalizeTask, PaddingSpaceNormalizeTask - TASK_REGISTRY에 6종 등록: PassCharacterNormalize, HangleEmojiNormalize 등 - 공통 헬퍼 _apply_normalizer()로 코드 중복 제거 - 각 Task별 단위 테스트 추가 (총 13 tests 통과) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
normalize(),remove_doublespace(),repeat_normalize(),only_hangle(),only_hangle_number(),only_text()에DeprecationWarning추가emoticon_normalize()가repeat_normalize()를 호출하여 이중 경고가 발생하는 문제 수정 →RepeatCharacterNormalizer직접 호출로 변경__init__.py에EmojiNormalizerexport 추가Test plan
uv run pytest tests/unit/test_normalizer.py— 15 passeduv run pre-commit run --all-files— all passedCloses #232
🤖 Generated with Claude Code