Need to use only a single thread to parse this file#1
Open
jimhester wants to merge 1 commit intowarint:masterfrom
Open
Need to use only a single thread to parse this file#1jimhester wants to merge 1 commit intowarint:masterfrom
jimhester wants to merge 1 commit intowarint:masterfrom
Conversation
This file has some quoted fields with escaped quotes inside of them, which causes problems when doing multi-threaded reading in readr 2.0.0. Forcing only a single thread allows the file to be parsed as intended.
|
@jimhester I've updated the readr package, I do see a warning message when using the spir_indicator() function, but the function runs and the data frame looks good. Do you know by any chance why escaped quotes would result in such multi-threading issues? |
Author
|
You need to use the current development version of readr to see the issue. The way the multi-threading works the other threads don't know if they are in a quoted field or not. There is automatic detection with a fallback to single threaded mode if an unexpected newline is encountered, but in this case the issue is with the embedded quotation, not a newline so the fallback does not trigger. |
|
Thanks for the details! Wouldn't it be cleaner to patch readr to detect embedded quotations as well? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This file has some quoted fields with escaped quotes inside of them,
which causes problems when doing multi-threaded reading in readr 2.0.0.
Forcing only a single thread allows the file to be parsed as intended.