Skip to content

Add Token.as_str() to decode bytes token values#1617

Open
apoorva-01 wants to merge 1 commit into
lark-parser:masterfrom
apoorva-01:bytes-token-as-str
Open

Add Token.as_str() to decode bytes token values#1617
apoorva-01 wants to merge 1 commit into
lark-parser:masterfrom
apoorva-01:bytes-token-as-str

Conversation

@apoorva-01

Copy link
Copy Markdown

float(token) breaks with use_bytes=True: could not convert string to float: Token('NUMBER', b'1'). Token subclasses str, but under use_bytes the value is bytes, so the string is the repr "b'1'", not "1".

Added Token.as_str() that decodes bytes (else returns as-is), so float(token.as_str()) works either way. Used latin-1 since that's what lark uses for lexer patterns and it won't choke on odd bytes. Can switch if you want.

This is the small fix. It doesn't touch the deeper thing where token.value == token is false under use_bytes. Making Token bytes-aware would fix that too, but it's a bigger semantics change, so I kept it contained. Happy to go that way instead. Test parses bytes and checks float(token.as_str()), fails without the fix.

Closes #1526

Under use_bytes=True a token's value is bytes, so float(token) and other
str ops fail. as_str() decodes it so callbacks work the same either way.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parsing bytestrings and token values

1 participant