Pydantic validation for the v1.2 API by Yannicked · Pull Request #53 · iterorganization/SimDB

Yannicked · 2026-03-12T10:00:55Z

This PR introduces a @pydantic_validate decorator that wires up typed request/response handling for all v1.2 API endpoints, replacing the previous mix of manual jsonify, try/except DatabaseError, and ad-hoc error returns.

This also automatically adds the expected request and response data to the autogenerated swagger API docs:

deepakmaroo

There is exception when Dashboard request for list of simulations

127.0.0.1 - - [13/Mar/2026 09:50:27] "OPTIONS /v1.2/simulations?code.name=&status=&uploaded_by= HTTP/1.1" 200 -
Unhandled exception in SimulationList.get
Traceback (most recent call last):
  File "/home/ITER/marood/Desktop/redhat-workspace/testing/SimDB_Test/src/simdb/remote/core/pydantic_utils.py", line 287, in wrapper
    result = f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
  File "/home/ITER/marood/Desktop/redhat-workspace/testing/SimDB_Test/src/simdb/remote/apis/v1_2/simulations.py", line 206, in get
    return PaginatedResponse[SimulationListItem].model_validate(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ITER/marood/Desktop/redhat-workspace/testing/SimDB_Test/venv/lib/python3.11/site-packages/pydantic/main.py", line 716, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for PaginatedResponse[SimulationListItem]
count
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=110.66666666666667, input_type=float]
    For further information visit https://errors.pydantic.dev/2.12/v/int_from_float
127.0.0.1 - - [13/Mar/2026 09:50:27] "GET /v1.2/simulations?code.name=&status=&uploaded_by= HTTP/1.1" 500 -

No sure if below error occurred due to local database migration applied

(venv) [marood@98dci4-srv-1002 SimDB_Test]$ simdb --debug remote sdcc list -l 10
Error: Corrupted data - non-optional metadata is None.
Traceback (most recent call last):
  File "/home/ITER/marood/Desktop/redhat-workspace/testing/SimDB_Test/venv/bin/simdb", line 10, in <module>
    sys.exit(main())

  File "/home/ITER/marood/Desktop/redhat-workspace/testing/SimDB_Test/src/simdb/cli/remote_api.py", line 584, in list_simulations
    return [Simulation.from_data(sim) for sim in data["results"]]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ITER/marood/Desktop/redhat-workspace/testing/SimDB_Test/src/simdb/cli/remote_api.py", line 584, in <listcomp>
    return [Simulation.from_data(sim) for sim in data["results"]]
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ITER/marood/Desktop/redhat-workspace/testing/SimDB_Test/src/simdb/database/models/simulation.py", line 340, in from_data
    metadata = checked_get(data, "metadata", list)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ITER/marood/Desktop/redhat-workspace/testing/SimDB_Test/src/simdb/database/models/utils.py", line 69, in checked_get
    raise ValueError(f"Corrupted data - non-optional {key} is None.")
ValueError: Corrupted data - non-optional metadata is None.

Yannicked · 2026-03-16T14:47:48Z

There is exception when Dashboard request for list of simulations

127.0.0.1 - - [13/Mar/2026 09:50:27] "OPTIONS /v1.2/simulations?code.name=&status=&uploaded_by= HTTP/1.1" 200 -
Unhandled exception in SimulationList.get
Traceback (most recent call last):
  File "/home/ITER/marood/Desktop/redhat-workspace/testing/SimDB_Test/src/simdb/remote/core/pydantic_utils.py", line 287, in wrapper
    result = f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
  File "/home/ITER/marood/Desktop/redhat-workspace/testing/SimDB_Test/src/simdb/remote/apis/v1_2/simulations.py", line 206, in get
    return PaginatedResponse[SimulationListItem].model_validate(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ITER/marood/Desktop/redhat-workspace/testing/SimDB_Test/venv/lib/python3.11/site-packages/pydantic/main.py", line 716, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for PaginatedResponse[SimulationListItem]
count
  Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=110.66666666666667, input_type=float]
    For further information visit https://errors.pydantic.dev/2.12/v/int_from_float
127.0.0.1 - - [13/Mar/2026 09:50:27] "GET /v1.2/simulations?code.name=&status=&uploaded_by= HTTP/1.1" 500 -

Hi deepak, the first problem seems to be very odd. The count variable in the result seems to be 110.66666666666667 somehow. This is supposed to return how many entries can be returned. I've tested using the database dump you've provided, and there it seems to work correctly:

➜  ~ curl "http://localhost:5000/v1.2/simulations?code.name=&status=&uploaded_by=" -i           
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 35680
Vary: Accept-Encoding
Access-Control-Allow-Origin: *
Server: Werkzeug/2.0.3 Python/3.13.5
Date: Mon, 16 Mar 2026 14:44:20 GMT

{"count":1320,"page":1,"limit":100,"results":[{"uuid":{"_type":"uuid.U.....

Is it possible to share the database you're using for me to debug with?

deepakmaroo · 2026-03-17T08:57:33Z

Thank you @Yannicked for your comment,
I investigate and found one of simulation is missing code.name element in my test database and because of that return query.count() / len(meta_keys), list(data.values()) returning 110.6666666 for query count: 332 meta_keys size: 3 meta_keys: ['code.name', 'status', 'uploaded_by']. After removing invalid simulation Dashboard is working fine.

On second issue Error: Corrupted data - non-optional metadata is None.

When requesting list of simulations list from local (applied database migration) database it complained error.

(venv) [marood@98dci4-srv-1002 SimDB_Test]$ simdb remote sdcc list -l 5 5 simulations found on remote {'count': 110, 'page': 1, 'limit': 5, 'results': [{'uuid': UUID('634979e0-00f0-11f0-bbc5-9440c9e76fd0'), 'alias': 'artolaj/jorek/ITER_DISRUPTIONS/111202/mar0925/seq-1', 'datetime': '2025-08-14T11:04:20.362220', 'metadata': None}, {'uuid': UUID('f709a9b8-00e6-11f0-b9fb-9440c9e76fd0'), 'alias': 'artolaj/jorek/ITER_DISRUPTIONS/111211/mar0925/seq-1', 'datetime': '2025-08-14T11:04:26.512249', 'metadata': None}, {'uuid': UUID('8ffc3f73-00eb-11f0-b788-9440c9e76fd0'), 'alias': 'artolaj/jorek/ITER_DISRUPTIONS/111213/mar0925/seq-1', 'datetime': '2025-08-14T11:04:41.783699', 'metadata': None}, {'uuid': UUID('17393ed5-00eb-11f0-a7fd-9440c9e76fd0'), 'alias': 'artolaj/jorek/ITER_DISRUPTIONS/111212/mar0925/seq-1', 'datetime': '2025-08-14T11:05:17.551891', 'metadata': None}, {'uuid': UUID('54c90fd0-00ee-11f0-8720-9440c9e76fd0'), 'alias': 'artolaj/jorek/ITER_DISRUPTIONS/111215/mar0925/seq-1', 'datetime': '2025-08-14T11:05:27.083715', 'metadata': None}]}. Simulations data: {'uuid': UUID('634979e0-00f0-11f0-bbc5-9440c9e76fd0'), 'alias': 'artolaj/jorek/ITER_DISRUPTIONS/111202/mar0925/seq-1', 'datetime': '2025-08-14T11:04:20.362220', 'metadata': None} Error: Corrupted data - non-optional metadata is None.
Where as when requesting simulations from production server, it is working fine

(venv) [marood@98dci4-srv-1002 SimDB_Test]$ simdb remote iter list -l 5
5 simulations found on remote {'count': 1320, 'limit': 5, 'page': 1, 'results': [{'alias': '100001/2', 'datetime': '2025-08-11T13:46:25.682813', 'uuid': UUID('1e237707-76a8-11f0-bd82-d4f5ef75e918')}, {'alias': '101024/2', 'datetime': '2025-08-11T13:54:16.722362', 'uuid': UUID('0845d270-00fa-11f0-9661-9440c9e76fd0')}, {'alias': '100002/1', 'datetime': '2025-08-11T13:47:05.636035', 'uuid': UUID('e4c30267-76a8-11f0-8012-d4f5ef75e918')}, {'alias': '100003/1', 'datetime': '2025-08-11T13:47:11.187199', 'uuid': UUID('e8023860-76a8-11f0-9cc6-d4f5ef75e918')}, {'alias': '100007/1', 'datetime': '2025-08-11T13:47:16.668763', 'uuid': UUID('eb4d0a5d-76a8-11f0-a79e-d4f5ef75e918')}]}.
Simulations data: {'alias': '100001/2', 'datetime': '2025-08-11T13:46:25.682813', 'uuid': UUID('1e237707-76a8-11f0-bd82-d4f5ef75e918')}
Simulations data: {'alias': '101024/2', 'datetime': '2025-08-11T13:54:16.722362', 'uuid': UUID('0845d270-00fa-11f0-9661-9440c9e76fd0')}
Simulations data: {'alias': '100002/1', 'datetime': '2025-08-11T13:47:05.636035', 'uuid': UUID('e4c30267-76a8-11f0-8012-d4f5ef75e918')}
Simulations data: {'alias': '100003/1', 'datetime': '2025-08-11T13:47:11.187199', 'uuid': UUID('e8023860-76a8-11f0-9cc6-d4f5ef75e918')}
Simulations data: {'alias': '100007/1', 'datetime': '2025-08-11T13:47:16.668763', 'uuid': UUID('eb4d0a5d-76a8-11f0-a79e-d4f5ef75e918')}
5 simulations found on remote iter.
alias    
--------
100001/2 
101024/2 
100002/1 
100003/1 
100007/1

It seems in the local server is adding metadata: None where as production server is not. So may be have to add validation on the client side or needs to correct server response or have to recheck database migration on my local.

deepakmaroo · 2026-03-17T14:19:05Z

src/simdb/remote/apis/v1_2/simulations.py

+        return result


 @api.route("/simulation/<path:sim_id>")


API call http://localhost:5000/v1.2/simulation/0eb5e8586c8d11f087edd4f5ef75e918 is failing with error message:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <html> <head> <title>TypeError: Object of type SimulationDataResponse is not JSON serializable // Werkzeug Debugger</title> <link rel="stylesheet" href="?__debugger__=yes&cmd=resource&f=style.css" type="text/css">  <link rel="shortcut icon" href="?__debugger__=yes&cmd=resource&f=console.png"> <script src="?__debugger__=yes&cmd=resource&f=debugger.js"></script> <script type="text/javascript"> var TRACEBACK = 139895977337424, CONSOLE_MODE = false, EVALEX = true, EVALEX_TRUSTED = false, SECRET = "t8CJKJPeke4iTJlQZOPo"; </script> </head> <body style="background-color: #fff"> <div class="debugger"> <h1>TypeError</h1> <div class="detail"> <p class="errormsg">TypeError: Object of type SimulationDataResponse is not JSON serializable</p>

My local database is migrated with alembic_version: 22360ccb4154, does I missed upgrade or downgrade?

I've pushed an update to address this, there was an issue with serializing numpy arrays in the metadata

Yannicked · 2026-03-17T15:22:17Z

Thank you @Yannicked for your comment, I investigate and found one of simulation is missing code.name element in my test database and because of that return query.count() / len(meta_keys), list(data.values()) returning 110.6666666 for query count: 332 meta_keys size: 3 meta_keys: ['code.name', 'status', 'uploaded_by']. After removing invalid simulation Dashboard is working fine.

On second issue Error: Corrupted data - non-optional metadata is None.

When requesting list of simulations list from local (applied database migration) database it complained error.

(venv) [marood@98dci4-srv-1002 SimDB_Test]$ simdb remote sdcc list -l 5 5 simulations found on remote {'count': 110, 'page': 1, 'limit': 5, 'results': [{'uuid': UUID('634979e0-00f0-11f0-bbc5-9440c9e76fd0'), 'alias': 'artolaj/jorek/ITER_DISRUPTIONS/111202/mar0925/seq-1', 'datetime': '2025-08-14T11:04:20.362220', 'metadata': None}, {'uuid': UUID('f709a9b8-00e6-11f0-b9fb-9440c9e76fd0'), 'alias': 'artolaj/jorek/ITER_DISRUPTIONS/111211/mar0925/seq-1', 'datetime': '2025-08-14T11:04:26.512249', 'metadata': None}, {'uuid': UUID('8ffc3f73-00eb-11f0-b788-9440c9e76fd0'), 'alias': 'artolaj/jorek/ITER_DISRUPTIONS/111213/mar0925/seq-1', 'datetime': '2025-08-14T11:04:41.783699', 'metadata': None}, {'uuid': UUID('17393ed5-00eb-11f0-a7fd-9440c9e76fd0'), 'alias': 'artolaj/jorek/ITER_DISRUPTIONS/111212/mar0925/seq-1', 'datetime': '2025-08-14T11:05:17.551891', 'metadata': None}, {'uuid': UUID('54c90fd0-00ee-11f0-8720-9440c9e76fd0'), 'alias': 'artolaj/jorek/ITER_DISRUPTIONS/111215/mar0925/seq-1', 'datetime': '2025-08-14T11:05:27.083715', 'metadata': None}]}. Simulations data: {'uuid': UUID('634979e0-00f0-11f0-bbc5-9440c9e76fd0'), 'alias': 'artolaj/jorek/ITER_DISRUPTIONS/111202/mar0925/seq-1', 'datetime': '2025-08-14T11:04:20.362220', 'metadata': None} Error: Corrupted data - non-optional metadata is None. Where as when requesting simulations from production server, it is working fine
(venv) [marood@98dci4-srv-1002 SimDB_Test]$ simdb remote iter list -l 5
5 simulations found on remote {'count': 1320, 'limit': 5, 'page': 1, 'results': [{'alias': '100001/2', 'datetime': '2025-08-11T13:46:25.682813', 'uuid': UUID('1e237707-76a8-11f0-bd82-d4f5ef75e918')}, {'alias': '101024/2', 'datetime': '2025-08-11T13:54:16.722362', 'uuid': UUID('0845d270-00fa-11f0-9661-9440c9e76fd0')}, {'alias': '100002/1', 'datetime': '2025-08-11T13:47:05.636035', 'uuid': UUID('e4c30267-76a8-11f0-8012-d4f5ef75e918')}, {'alias': '100003/1', 'datetime': '2025-08-11T13:47:11.187199', 'uuid': UUID('e8023860-76a8-11f0-9cc6-d4f5ef75e918')}, {'alias': '100007/1', 'datetime': '2025-08-11T13:47:16.668763', 'uuid': UUID('eb4d0a5d-76a8-11f0-a79e-d4f5ef75e918')}]}.
Simulations data: {'alias': '100001/2', 'datetime': '2025-08-11T13:46:25.682813', 'uuid': UUID('1e237707-76a8-11f0-bd82-d4f5ef75e918')}
Simulations data: {'alias': '101024/2', 'datetime': '2025-08-11T13:54:16.722362', 'uuid': UUID('0845d270-00fa-11f0-9661-9440c9e76fd0')}
Simulations data: {'alias': '100002/1', 'datetime': '2025-08-11T13:47:05.636035', 'uuid': UUID('e4c30267-76a8-11f0-8012-d4f5ef75e918')}
Simulations data: {'alias': '100003/1', 'datetime': '2025-08-11T13:47:11.187199', 'uuid': UUID('e8023860-76a8-11f0-9cc6-d4f5ef75e918')}
Simulations data: {'alias': '100007/1', 'datetime': '2025-08-11T13:47:16.668763', 'uuid': UUID('eb4d0a5d-76a8-11f0-a79e-d4f5ef75e918')}
5 simulations found on remote iter.
alias    
--------
100001/2 
101024/2 
100002/1 
100003/1 
100007/1 
It seems in the local server is adding metadata: None where as production server is not. So may be have to add validation on the client side or needs to correct server response or have to recheck database migration on my local.

I've added exclude_none=False to the pydantic json serialization. This should omit the metadata: None fields in the returned data :)

deepakmaroo · 2026-03-18T15:59:38Z

Current Dashboard expecting array data in numpy.ndarray type but here it is Array type.
So moving forward, we have to release new Dashboard version to support with Array and backward compatible numpy.ndarray.

Yannicked · 2026-03-19T09:41:12Z

Current Dashboard expecting array data in numpy.ndarray type but here it is Array type. So moving forward, we have to release new Dashboard version to support with Array and backward compatible numpy.ndarray.

Good catch, I'll implement the numpy ndarray type for now for compatibility with the current dashboard

deepakmaroo

LGTM

Yannicked added 9 commits February 26, 2026 09:41

Use pydantic to validate request and response data

3321482

Raise errors instead of returning them

a28cfa8

Cleanup endpoints

6e0f8b5

Add header and query params to docs

6c41588

Merge branch 'develop' into feature/v1.2-pydantic-validation

78b6231

Add validation to staging directory endpoint

138c556

Add validation to metadata endpoints

6c8ea68

Fix issue with sim children and parents

c686e47

Add seperate return codes for client and server errors

af97aca

Yannicked requested review from SimonPinches, deepakmaroo, maarten-ic and olivhoenen as code owners March 12, 2026 10:00

deepakmaroo reviewed Mar 13, 2026

View reviewed changes

deepakmaroo reviewed Mar 17, 2026

View reviewed changes

Yannicked added 3 commits March 17, 2026 15:54

Fix numpy array serialization

19959e1

Fix uuid serialization

ffae694

Exclude none values in JSON

38d1ed4

Fix ndarray handling correctly

f63c2de

Yannicked force-pushed the feature/v1.2-pydantic-validation branch from 85dee87 to f63c2de Compare March 19, 2026 16:56

deepakmaroo approved these changes Mar 20, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pydantic validation for the v1.2 API#53

Pydantic validation for the v1.2 API#53
Yannicked wants to merge 13 commits intoiterorganization:developfrom
Yannicked:feature/v1.2-pydantic-validation

Yannicked commented Mar 12, 2026

Uh oh!

deepakmaroo left a comment

Uh oh!

Yannicked commented Mar 16, 2026 •

edited

Loading

Uh oh!

deepakmaroo commented Mar 17, 2026

Uh oh!

deepakmaroo Mar 17, 2026

Uh oh!

Yannicked Mar 17, 2026

Uh oh!

Yannicked commented Mar 17, 2026

Uh oh!

deepakmaroo commented Mar 18, 2026

Uh oh!

Yannicked commented Mar 19, 2026

Uh oh!

deepakmaroo left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Yannicked commented Mar 12, 2026

Uh oh!

deepakmaroo left a comment

Choose a reason for hiding this comment

Uh oh!

Yannicked commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deepakmaroo commented Mar 17, 2026

Uh oh!

deepakmaroo Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

Yannicked Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

Yannicked commented Mar 17, 2026

Uh oh!

deepakmaroo commented Mar 18, 2026

Uh oh!

Yannicked commented Mar 19, 2026

Uh oh!

deepakmaroo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Yannicked commented Mar 16, 2026 •

edited

Loading