Chunked data error in wrong format

[krowvin]

Attempting to store a larger amount of data (5 years worth of 1h data) of a single timeseries.

Looks like the error format is expecting a 2d list as usual but there is a wrapper function that converts the data object to something else:
https://github.com/HydrologicEngineeringCenter/cwms-python/blob/main/cwms/timeseries/timeseries.py#L528-L534

Wrote this code

 ts_ids = cwms.get_timeseries_group(
            group_id=ts_group,
            category_id=ts_group_category_id,
            office_id=source_office,
            category_office_id=ts_group_category_office_id,
        )
        logging.info(
            f"Found {len(ts_ids.json.get('assigned-time-series', []))} timeseries in group {ts_group}."
        )
        logging.info(f"Storing TSID from begin: {begin} to end: {end}")
        with open("ts_ids.json", "w") as f:
            f.write(json.dumps(ts_ids.json))
        for ts in ts_ids.json.get("assigned-time-series", []):
            try:
                if dry_run:
                    click.echo(
                        f"Would store timeseries data for {ts['timeseries-id']}({ts['office-id']})"
                    )
                else:
                    cwms.init_session(api_root=source_cda, api_key=None)
                    ts_values = cwms.get_timeseries(
                        ts_id=ts["timeseries-id"],
                        office_id=ts["office-id"],
                        begin=begin,
                        end=end,
                    )
                    cwms.init_session(api_root=target_cda, api_key=target_api_key)
                    cwms.store_timeseries(
                        data=ts_values.json,
                        store_rule="REPLACE_ALL",
                        override_protection=False,
                    )
                    click.echo(
                        f"Wrote {len(ts_values.json.get("values", []))} values to {ts['timeseries-id']}."
                    )
            except Exception as e:
                logging.warning(
                    f"Error storing timeseries ({ts['timeseries-id']}) data: {e}",
                    exc_info=True,
                )

Got this error:

2026-02-13 22:04:37;INFO;Storing TSID from begin: 2020-01-01 00:00:00-06:00 to end: 2026-02-13 22:04:35-06:00
2026-02-13 22:04:40;WARNING;Error storing timeseries (TULA.Flow.Inst.1Hour.0.Ccp-Rev) data: 0
Traceback (most recent call last):
  File "\Code\cwms-python\cwms\timeseries\timeseries.py", line 617, in store_timeseries
    responses.append({"success:": future.result()})
                                  ~~~~~~~~~~~~~^^
  File "\AppData\Local\Programs\Python\Python313\Lib\concurrent\futures\_base.py", line 449, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "AppData\Local\Programs\Python\Python313\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
  File "\AppData\Local\Programs\Python\Python313\Lib\concurrent\futures\thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File "\Code\cwms-python\cwms\api.py", line 374, in post
    _post_function(endpoint=endpoint, data=data, params=params, api_version=api_version)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\Code\cwms-python\cwms\api.py", line 342, in _post_function
    data = json.dumps(data)
  File "\AppData\Local\Programs\Python\Python313\Lib\json\__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "\AppData\Local\Programs\Python\Python313\Lib\json\encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "\AppData\Local\Programs\Python\Python313\Lib\json\encoder.py", line 261, in iterencode
    return _iterencode(o, 0)
  File "\AppData\Local\Programs\Python\Python313\Lib\json\encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
                    f'is not JSON serializable')
TypeError: Object of type Timestamp is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "\cwms-cli\cwmscli\load\timeseries\timeseries_data.py", line 83, in _load_timeseries_data
    cwms.store_timeseries(
    ~~~~~~~~~~~~~~~~~~~~~^
        data=ts_values.json,
        ^^^^^^^^^^^^^^^^^^^^
        store_rule="REPLACE_ALL",
        ^^^^^^^^^^^^^^^^^^^^^^^^^
        override_protection=False,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "\Code\cwms-python\cwms\timeseries\timeseries.py", line 619, in store_timeseries
    start_time = chunk["values"][0][0]
                 ~~~~~~~~~~~~~~~~~~^^^
KeyError: 0

Root case

Changing the logging statement to

start_time = chunk["values"][0]["date-time"]
end_time = chunk["values"][-1]["date-time"]

I could see this error

Error storing chunk from 2025-12-24 00:00:00+00:00 to 2026-02-13 22:00:00+00:00: Object of type Timestamp is not JSON serializable

If you look into one of the chunks the date-time is of the form:
Timestamp('2025-12-27 16:00:00+0000', tz='UTC')

I had an assumption I could feed the data from get_timeseries into store_timeseries and that might be where i'm going wrong?

[msweier]

Is it the same error for just a couple days of data? I've had trouble storing the json directly from a read too but I figured it was related to the date time format. Does it work if you convert to dataframe and convert to to json via timeseries_df_to_json?

[msweier]

There could be a couple things going on with the json. Trying the following works for me:

data_new = cwms.timeseries_df_to_json(data=data.df,ts_id = data.json['name'],units=data.json['units'],office_id=data.json['office-id'])
cwms.store_timeseries(data_new)

The json return from timeseries_df_to_json is much cleaner than the raw json return from CDA. If you try to post a timeseries json return direct from cda from swagger or curl it will fail too. There's a lot of stuff in the json header from the raw return.

{
  "begin": "2026-02-18T10:01:24.146047Z",
  "date-version-type": "UNVERSIONED",
  "end": "2026-02-18T17:05:01.874331Z",
  "interval": "PT0S",
  "interval-offset": -2147483648,
  "name": "McGregor.Elev.Inst.~15Minutes.0.best-MSL1912",
  "office-id": "MVP",
  "page": "MTc3MTQwOTcwMDAwMHx8M3x8NTAw",
  "page-size": 500,
  "time-zone": "US/Central",
  "total": 3,
  "units": "ft",
  "value-columns": [
    {
      "name": "date-time",
      "ordinal": 1,
      "datatype": "java.sql.Timestamp"
    },
    {
      "name": "value",
      "ordinal": 2,
      "datatype": "java.lang.Double"
    },
    {
      "name": "quality-code",
      "ordinal": 3,
      "datatype": "int"
    }
  ],
  "values": [
    [
      1771409700000,
      613.2099999999999,
      0
    ],
    [
      1771410600000,
      613.2099999999999,
      0
    ],
    [
      1771411500000,
      613.2099999999999,
      0
    ]
  ],
  "vertical-datum-info": {
    "office": "MVP",
    "unit": "ft",
    "location": "McGregor",
    "native-datum": "OTHER",
    "elevation": 600,
    "local-datum-name": "",
    "offsets": []
  }
}

[Enovotny]

This is caused by the data chunks. If the call is less than the chunk length it pull correctly. when chunks are used it will not store correctly. I am guessing that the code to recreate the json when merging the chunks together is printing a timestamp instead of the MS.