feat(RND-13624): Migrate WithS3File to AWS Data Wrangler#88
Draft
Christos-Hadjinikolis wants to merge 2 commits intomasterfrom
Draft
feat(RND-13624): Migrate WithS3File to AWS Data Wrangler#88Christos-Hadjinikolis wants to merge 2 commits intomasterfrom
WithS3File to AWS Data Wrangler#88Christos-Hadjinikolis wants to merge 2 commits intomasterfrom
Conversation
Christos-Hadjinikolis
changed the title
WithS3File to AWS Data Wrangler
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #88 +/- ##
==========================================
+ Coverage 88.21% 89.40% +1.18%
==========================================
Files 16 16
Lines 1256 1321 +65
Branches 142 153 +11
==========================================
+ Hits 1108 1181 +73
+ Misses 110 98 -12
- Partials 38 42 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Christos-Hadjinikolis
requested review from
arturk-vortexa,
deaglancrew,
joloppo and
matteo-pallini
as code owners
| @@ -1,4 +1,6 @@ | |||
| """Invokes dynamicio cli.""" | |||
|
|
|||
| # Application Imports | |||
Collaborator
Author
There was a problem hiding this comment.
Black does this now...
Comment on lines
+22
to
+30
| """Handles I/O operations for AWS Athena. | ||
|
|
||
| Note: | ||
| The `__abstractmethods__ = frozenset()` is used to silence false positives from pylint, | ||
| which might wrongly assume this class has abstract methods due to NotImplementedError. | ||
| This class is *not* an abstract base class and does not use `abc.ABC`. | ||
| """ | ||
|
|
||
| __abstractmethods__ = frozenset() |
Collaborator
Author
There was a problem hiding this comment.
Fixes a pylint false positive on abstract methods not implemented...
Comment on lines
+142
to
+153
| # 🧼 Check if this is a single-record json file | ||
| if is_single_record: | ||
| # Re-wrap as single dict row — i.e., rehydrate the record | ||
| df = pd.DataFrame([{df.columns[0]: dict(zip(df.index, df.iloc[:, 0]))}]) | ||
| elif ( | ||
| df.shape[1] == 1 | ||
| and df.columns.dtype == "object" | ||
| and df.index.dtype == "object" | ||
| and all(isinstance(i, str) for i in df.index) | ||
| and all(isinstance(v, (str, int, float, bool, type(None))) for v in df.iloc[:, 0]) | ||
| ): | ||
| logger.warning("[local-json-read] File appears to be a single-record JSON object. Pass 'single_record=True' in options to handle this case.") |
Collaborator
Author
There was a problem hiding this comment.
Necessary for alignment with wrangler.read_json() behaviour.
|
refactor: Manage reference data test: Warning docs: Update feat: Add single-record option for local read_json files refactor: Make sure Local and s3 reads are consistent refactor: Linting fixes refactor: remove support for index orientation wip refactor: Allow index orientation for json read/writes test: HDF args test: Make sure json kwargs don't cause discrepancies test: COnfirm allow options config works as expected fixes feat: Migrate to aws-wrangler for WithS3 mixin only # Conflicts: # dynamicio/__init__.py # dynamicio/config/pydantic/config.py # dynamicio/core.py # dynamicio/mixins/utils.py # dynamicio/mixins/with_kafka.py # dynamicio/validations.py # poetry.lock # pyproject.toml # tests/test_mixins/test_postgres_mixins.py # tests/test_mixins/test_s3_mixins.py
Christos-Hadjinikolis
force-pushed
the
rnd-13624
branch
from
9cb7282 to
ab0030a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Migrate
WithS3Fileto AWS Data WranglerLinks
RND-13624
Summary
This PR refactors the
WithS3Filemixin to use[awswrangler](https://github.com/aws/aws-sdk-pandas)for all S3 I/O operations (CSV, JSON, Parquet, and HDF), removing reliance on local temp files,boto3, and the AWS CLI. HDF files are now streamed in-memory usingboto3.client().download_fileobjand PyTables.Key Changes
awswrangler:wr.s3.read_*/to_*calls.BytesIO._read_s3_{type}_file/_write_s3_{type}_filemethods.allow_optionsdecorator to:pandas,pyarrow,wrangler, etc).pandas_kwargs,pyarrow_additional_kwargs,s3_additional_kwargs.convert_dates=Trueis ignored to ensure consistency.single_record=Trueoption to force wrapping a single JSON object into a tabular DataFrame.pandas.read_jsonandawswrangler.read_jsonfororient="records"andlines=Trueusage.Benefits
awswrangler.Notes
_s3_reader,_s3_writer,_s3_named_file_reader) has been fully removed.awswrangleris now a required dependency.single_record=True.