Skip to content

Incorrect tokenization of plain scalars #122

Description

@RecursiveError

Related issue: #87

The tokenizer incorrectly handles plain scalars in cases like the following:

foo: some "text" more text
bar: some [text] more text

The second case fails because the tokenizer only checks in_flow when encountering opening flow indicators ([ and {), but not when encountering the corresponding closing tokens (] and }).
(

'\r', '\n', ' ', '\'', '"', ']', '}' => {
)

As a result, the tokenizer breaks the literal flow even when it is not actually inside a flow sequence.

I noticed that #91 attempted to address this issue, but it appears to have been closed without much explanation.

The first case involving " is more complex due to the different contexts in which it can appear. However, introducing a state similar to in_flow (or extending the existing state handling) might help resolve this case as well.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions