[SYNPY-1749]Allow quote, apostrophe and ellipsis in store_row_async

[danlu1]

Problem:

A JSON serialization issue occurs when a DataFrame passed to store_row_async contains a list or dictionary with strings that include both double quotes and apostrophes.

Solution:

1. Add default value "escapechar": "\\" to to_csv_kwargs in store_rows_async
2. Add to_csv_kwargs to _stream_and_update_from_df so it can take the passed to_csv_kwargs values for downstream data processing.

Testing:

Unit test and integration test have been added.

[andrewelamb]

@danlu1 Is this still WIP, or are you looking for reviews?

[danlu1]

@andrewelamb sorry I should have marked this a draft.

[danlu1]

The integration test failures are in the recordset and submission modules and do not appear to be related to my changes.

[linglp]

I think overall it looks good. The tests can be consolidated a bit to test all the edge cases in fewer integration tests to improve performance, and the docstring can be updated to reflect the new state of the code since json.dumps() was removed. There's also some logic that can be simplified in the redundant checks where sample_values is created but never actually used. The function name could also be more descriptive of what it actually does now (sanitizing special values rather than just converting dtypes)

[BryanFauble]

This is looking great, once we get the last few items handled (and develop merged in), I can approve!

[BryanFauble]

Nice work on this fix -- the backslash-escaping approach for embedded quotes makes sense, and the Ellipsis/pd.NA handling is solid. I flagged one issue that I think needs to be addressed before merge, plus a few cleanup nits.

Note: This comment was drafted with AI assistance and reviewed by me for accuracy.

[linglp]

Hi @danlu1 ! Thanks for your hard work. I think I found two bugs after a few rounds of testing myself:

Bug 1: Nested np.nan not handled

df = pd.DataFrame({"val": [[1.0, np.nan], [2.0, 3.0]]})

The nested np.nan would currently pass through to JSON serialization, which could cause issues since np.nan is not valid JSON.

This happened becauseconvert_dtypes() only converts top-level np.nan to pd.NA but nested np.nan would remain unchanged. And even though in your code, you have:

if obj is pd.NA:

But this won't handle np.nan. The fix would just be:

def _reformat_special_values(obj):
    if pd.isna(obj):      # Catches pd.NA, np.nan, and None
        return None

Bug 2: Top-level missing values not handled

def _serialize_json_value(x):
    if isinstance(x, (list, dict)):
        # pd.NA handling is only here, inside _reformat_special_values
        ...
    if x is ...:
        return "..."
    return x  # <-- Top-level pd.NA, np.nan, None just pass through

Top-level pd.NA (or np.nan) isn't converted to None. It only works now because .replace({pd.NA: None}) is called later.

[BryanFauble]

This is great progress on a real user-reported issue, nice work on the recursive handling of Ellipsis and pd.NA in nested structures! There's one item that needs to be fixed before we can merge.

Note: This comment was drafted with AI assistance and reviewed by me for accuracy.

[Copilot]

Pull request overview

This PR addresses JSON/CSV serialization failures when uploading DataFrames via store_rows_async, specifically when cells contain JSON-like dict/list values with embedded quotes/apostrophes and when Ellipsis/pd.NA values appear in nested structures.

Changes:

• Adds a default escapechar ("\\ ") to to_csv_kwargs in store_rows_async.
• Threads to_csv_kwargs through the DataFrame streaming/upload path so the same CSV settings are used downstream.
• Introduces convert_dtypes_to_json_serializable and adds unit + integration coverage around quotes/apostrophes/ellipsis handling.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

File	Description
synapseclient/models/mixins/table_components.py	Adds default escapechar, threads to_csv_kwargs, and adds convert_dtypes_to_json_serializable used before chunked DF upload.
synapseclient/core/upload/multipart_upload_async.py	Updates docstring to describe to_csv_kwargs usage for DataFrame multipart uploads.
tests/unit/synapseclient/mixins/unit_test_table_components.py	Adds unit tests for convert_dtypes_to_json_serializable.
tests/integration/synapseclient/models/async/test_table_async.py	Adds an integration test covering quotes/apostrophes/ellipsis round-tripping through table storage/query.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[BryanFauble]

The last round of changes with my final patches looked good to me.

@danlu1 is going to give this a final review as well before we merge in.

[danlu1]

lgtm