EIR Data + JSON Script addition#44
Open
zainahmed7771 wants to merge 30 commits into
Open
Conversation
… the overflow hub
The input folders carried a _data suffix and two of them had spelling variants (northumbria/northumbrian, south_west_water/_data), which resolve_company_folder existed solely to paper over by trying each candidate in turn. Rename every folder to match its COMPANIES key exactly, so the input path is just INPUT_ROOT / company. resolve_company_folder and the per-company "folders" lists are deleted; COMPANIES collapses from a dict of single-key dicts to a plain company -> API URL mapping, and enrich_company / load_company_csvs take the values they actually use. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
api_id_field_used, api_x_field_used, api_y_field_used and api_watercourse_field_used recorded which API property the field detection heuristics had chosen for each company. Since the schema was hardcoded those four columns always report the same literals, so they convey nothing and are dropped from both the per-company summary and the empty-company summary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
valid_lonlat and valid_bng each coerced their arguments with to_number before range-checking, so convert_lonlat_to_bng coerced lon/lat via valid_lonlat, coerced them again with float(), then coerced the transformer's float output a third time via valid_bng. Split the range checks into in_lonlat_bounds and in_bng_bounds, which take numbers, so each value is coerced exactly once at the boundary. valid_bng keeps its coercing form for validate_output_json, which reads untyped values back from JSON. valid_lonlat had no callers left and is removed. Verified coordinate-identical against the previous commit on 1600 live API features across anglian, southern_water, south_west_water and yorkshire. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All 29 print() calls become logger calls at an appropriate level. LevelPrefixFormatter emits INFO lines bare, so ordinary progress output is unchanged, and prefixes WARNING/ERROR with the level name, so the manual "WARNING: " strings inside the messages are no longer needed. The bare `except Exception` in main() logged only str(exc), which hid where a company actually failed. It now passes exc_info=True, so the traceback reaches the operator while the run still continues to the next company. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The script imports pandas, pyproj and requests, but nothing in the repo declared them and no requirements file existed, despite both READMEs telling the reader to pip install one. .gitignore excludes *.txt, so the file needs an explicit negation. The existing "!requirements-company-json.txt" guarded a filename that was never created; replace it with "!requirements.txt", which is the name the script's own docstring already tells you to install. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The file still documented standardised_data/ as the input tree and
outputs/{csv_clean,json,qc,api_cache}/ as the outputs, none of which
exist any more, and an unmatched code fence at line 34 swallowed the
formatting of everything below it.
Describe the actual contract: input_stopstart_data/{company}/ in, one
outputs/{company}.json out, nothing else written. Document exact-only
permit matching and the loud unmatched-ID warnings, and state plainly
that json_validation_passed checks the shape of the JSON and not
whether any rows matched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The README documented scripts/build_company_json.py with --company, --refresh-api, --incremental and --allow-low-match flags, a requirements-company-json.txt, and a 95% match-rate floor. None of that was ever written. Describe the script as it is: ONLY_COMPANIES at the top of json_cleaning_python_pipeline/build_water_company_json.py, one JSON per company out, no intermediates and no API cache. State explicitly that json_validation_passed checks the JSON's shape rather than whether its rows matched, since that distinction is the easiest way to ship a file of nulls. The PR had also deleted the "Output data" section, taking with it the CC-BY-SA licence statement, the table of public geojson download URLs, and the Thames Water historical data links. Restore them verbatim; the Scottish Water and Thames API additions to "Source data" are kept. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ning
Coordinates were guarded by two bounds predicates. in_lonlat_bounds
rejected a lon/lat pair before projection, which silently nulled any
site a company had mislocated, and in_bng_bounds re-checked the result.
Keep only in_bng_bounds, and work in BNG from the point of projection.
convert_lonlat_to_bng no longer range-checks its result: a site that
converts outside the grid is returned as published and reported by name
("btw, SVT02707 converts to (394704, 2865498), outside the British
National Grid"). Where a company puts its outfalls is not ours to
correct. It still returns None for values that cannot be projected at
all, which pyproj signals with a non-finite result and which round()
would otherwise raise OverflowError on -- that safety, rather than
tidiness, is what the lon/lat bounds check was really providing.
validate_output_json keeps reporting xy_bng but drops it from `passed`,
since an out-of-grid coordinate is now a deliberate pass-through rather
than a defect in the file.
first_geometry_lonlat is deleted. It fell back to the GeoJSON geometry
when the lon/lat properties were unusable, but across all 11183 features
of the eight endpoints every feature carries Latitude and Longitude, and
the geometry is always present and exactly equal to them.
Verified against the previous commit on 13183 live features: 13182
identical, the one change being SVT02707, which is now emitted rather
than nulled.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Timestamp handling was two functions: a vectorised parser and a 40-line
scalar fallback it handed off to when a column mixed tz-aware and naive
values. On pandas 2.2 that fallback is unreachable for the shapes it was
written for, and worse, the mixed case it meant to catch is coerced to
NaT before the handoff can happen. Both are replaced by one function.
pd.to_datetime was called without a format, so it inferred one from the
first value in each column. Wessex publishes fractional seconds on some
rows and not others ("2025-01-05T03:16:00.05" alongside
"2025-02-01T00:00:00"), so every row not matching the inferred format
was coerced to NaT and written to the JSON with a null timestamp.
Passing format="ISO8601" accepts both. Across all input files this
recovers 443239 timestamps, 220788 of them Wessex's, with no
previously-parsed value changing.
The companies publish naive local times, so we localise to Europe/London
to say so explicitly, then convert to UTC. The two hours a year that do
not map 1:1 onto UTC are resolved so no event is dropped:
ambiguous=True reads the repeated autumn hour as BST, and
nonexistent="shift_forward" nudges the skipped spring hour onto the hour
that exists. Previously ambiguous="NaT" discarded 368 real events.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pd.read_csv inferred column types. An all-numeric permit column types as int64, which stringifies cleanly, but the moment it contains a single blank row pandas widens it to float64 and permit 103283 becomes the string "103283.0", which matches no API Id. No current input file trips this: Southern Water's numeric permits share a column with SWS-prefixed ones, so it already infers as object. The guard is preventive, and costs nothing. duration_minutes is still coerced with pd.to_numeric and the timestamps with pd.to_datetime, both of which take text. Output is byte-identical for yorkshire and united_utilities. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
enrich_company walked every event row, calling match_keys_to_api once per row and building a dict per row. Yorkshire has 694915 rows but only 2088 distinct permits, so each lookup was repeated about 333 times. build_api_lookup now returns a one-row-per-permit DataFrame, which enrich_company left-joins onto the events. match_keys_to_api and the per-row match_report are deleted; match_status is derived from whether the join found a partner, and the summary counts come off the merged frame. Output is byte-identical for yorkshire and united_utilities. An API ID appearing more than once was previously excluded from the lookup entirely, so every event for that permit silently lost its coordinates. Report it instead: warn, print each occurrence in full with its projected X/Y, and take the first. Anglian publishes AWS00528 twice, the two rows differing only in ObjectId, which recovers coordinates for 158 Anglian event rows that were null before. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
json_cleaning_python_pipeline/ becomes clean_EIR_stopstartdata/, the script becomes build_water_company_jsons.py, and outputs/ becomes output_jsons/. OUTPUT_ROOT still named "outputs", so a run would have created a second output folder alongside the renamed one. The .gitignore rule and the links in the top-level README pointed at the old paths too. README_json.md and data_transparency.md are merged into a single clean_EIR_stopstartdata/README.md, so the folder carries one document covering how to run the pipeline and where its source data comes from. No content is lost: the EDM source URLs and StormHub endpoints are carried over as tables. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tion The module docstring was a numbered list of how to run the script in VS Code, which said nothing about what the script does and had drifted from the code. Replace it with an account of the pipeline: what the two data sources are, what the join produces, and the three things it does on the way through. State the contract it now keeps -- that it transcribes rather than corrects, reporting unmatched permits, mislocated sites and duplicate API IDs by name rather than silently dropping them. The how-to-run steps move to README.md, alongside the input format and the data provenance, so the operator instructions live in one place. Every function and class now carries at least a one-line docstring, including the nested format() and numeric_or_null(). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pyflakes and vulture report the module clean, because what is dead here is computed rather than merely declared. Thirteen of the summary's eighteen keys were never surfaced. They fed the per-company QC CSVs, which were deleted when the pipeline was cut back to writing JSON and nothing else; the closing table only ever printed five. Drop the thirteen and the values computed to fill them: matched_permits, unmatched_permits, missing_watercourse, the duplicate tally returned by build_api_lookup, and the csv_files list threaded through load_company_csvs and empty_company_summary purely to name themselves in input_csv_files. bad_start and bad_stop also fed only dead keys, so deleting them would have removed the pipeline's one detection of an unparseable timestamp. Report them instead, in the same voice as the other transcription warnings, and keep them. fetch_arcgis_geojson wrapped its features in a GeoJSON FeatureCollection whose "type" and "metadata" no caller read; it now returns the feature list. load_company_csvs tagged every row with a _source_file column that was written and never used. All eight company JSONs are byte-identical. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
unmatched_rows counts event rows, which says how much data is affected but not how many distinct sites are behind it. Anglian's 413616 unmatched rows come from 2771 permits; Southern's 41785 come from 1038. Add unmatched_ids, counted over the same population report_unmatched_permits uses, so the table and the per-permit warnings can never disagree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What I Added:
-Standardised CSV datasets for all water companies. Each dataset links the company’s unique IDs to the corresponding StormHub API IDs, allowing us to retrieve the X and Y coordinates and the receiving watercourse for each discharge point.
-A JSON script that processes the standardised datasets, retrieves the relevant API data for each water company, and generates a company-specific JSON file using the same reference structure as thames.json. These files are ready to be uploaded to the AWS server, enabling the discharge history data to be deployed on the front end of the sewage map website.