Fixes permafrost endpoints in API#719
Conversation
…ata and the current variable naming structure.
There was a problem hiding this comment.
Pull request overview
Updates permafrost-related API requests/packaging to align with backend Rasdaman coverage changes (GIPL outputs axis rename and variable naming), so permafrost endpoints return correctly shaped data again.
Changes:
- Switch WCPS/WCS subsetting for
crrel_gipl_outputs_ncfromyear(...)totime(...). - Introduce a variable-name LUT to translate Rasdaman variable identifiers to the API/CSV-expected keys.
- Normalize preview CSV decoding by using Flask
Response.get_data(as_text=True).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| nullified_data = nullify_nodata(temp, "permafrost") | ||
| if nullified_data is not None: | ||
| di = {"year": year, "depth": depth, "temp": temp} | ||
| di = {"time": year, "depth": depth, "temp": temp} |
There was a problem hiding this comment.
I think Copilot did catch a stray find-and-replace here, I think reverting is correct? Or, update the template. Your thoughts @BobTorgerson? I don't see any issues with the actual responses though, they look fine.
…re responding as expected.
charparr
left a comment
charparr
left a comment
There was a problem hiding this comment.
Tested multiple routes here for a broad set of lat-lon slices and time slices, and the summarize and CSV parameters as well, and the responses are looking good. See my one comment where I tagged you and proceed there as you see fit. I think the usage of the simple look-up table is a good patch here -- easy to trace what is happening. Hopefully this banishes the GIPL ghost for another few months...
| nullified_data = nullify_nodata(temp, "permafrost") | ||
| if nullified_data is not None: | ||
| di = {"year": year, "depth": depth, "temp": temp} | ||
| di = {"time": year, "depth": depth, "temp": temp} |
There was a problem hiding this comment.
I think Copilot did catch a stray find-and-replace here, I think reverting is correct? Or, update the template. Your thoughts @BobTorgerson? I don't see any issues with the actual responses though, they look fine.
| preview_string = [r.data.decode("utf-8") for r in preview] | ||
| preview_string = [r.get_data(as_text=True) for r in preview] |
There was a problem hiding this comment.
Looks like this is the more canonical way to decode content
There was a problem hiding this comment.
Ran this locally, and the test passed as expected:
pytest -v tests/test_eds_all.py
==================================================================================================================================================== test session starts =====================================================================================================================================================
platform darwin -- Python 3.11.14, pytest-9.0.2, pluggy-1.6.0 -- /Users/cparr/workspace/data-api/.venv/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.11.14', 'Platform': 'macOS-15.7.3-arm64-arm-64bit', 'Packages': {'pytest': '9.0.2', 'pluggy': '1.6.0'}, 'Plugins': {'metadata': '3.1.1', 'html': '4.2.0'}}
rootdir: /Users/cparr/workspace/data-api
configfile: pytest.ini
plugins: metadata-3.1.1, html-4.2.0
collected 1 item
tests/test_eds_all.py::test_eds_all PASSED [100%]
====================================================================================================================================================== warnings summary ======================================================================================================================================================
routes/vectordata.py:43
routes/vectordata.py:43
routes/vectordata.py:43
routes/vectordata.py:43
routes/vectordata.py:43
routes/vectordata.py:43
/Users/cparr/workspace/data-api/routes/vectordata.py:43: DeprecationWarning: The 'unary_union' attribute is deprecated, use the 'union_all()' method instead.
region_geom = gdf_extent.unary_union
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=============================================================================================================================================== 1 passed, 6 warnings in 16.27s ===============================================================================================================================================
This PR updates the permafrost endpoints to utilize the new dimensionality of the crrel_gipl_outputs_nc coverage. The coverage has changed on the backend, where a dimension previously called "year" is now displayed as "time". This PR changes the calls to the WCS commands to Rasdaman to use that dimension.
Additionally, this PR has a LUT to translate the coverage's variable names to the expected API outputs, as all of the endpoints were returning the incorrect keys for the JSON output.
Testing
This impacts all of our websites that use permafrost data so we need to verify that all of the endpoints are working as expected again after this change.
The following endpoints need to be tested: