fix: reconcile NWB export handler to deployed schema#90
Draft
tabedzki wants to merge 1 commit into
Draft
Conversation
The NWB export handler and helpers referenced columns and tables that do
not exist in the deployed nwb_production schema. This reconciles them to
the authoritative schema (u19_pipeline/nwb_production.py).
Changes:
- status_nwb_id -> status_id: the active-jobs query and per-job status read
in pipeline_handler_main, and the update1 in update_status_pipeline, now use
the real NwbExportJob.status_id (FK to NwbExportStatus).
- Part-tables -> NwbExportModality: process_data_validation (handler) and
estimate_total_size (nwb_production_utils) no longer query the nonexistent
NwbExportJob.BehaviorExport/.EphysExport/.ImagingExport part-tables. They now
query NwbExportModality & {"nwb_job_id": ...} and branch on modality_name
('behavior'/'ephys'/'imaging'), parsing probe_numbers/fov_numbers (JSON-array
strings) and deriving session/recording/scan keys from the NwbExportJob record.
- Log column names: NwbExportLogStatus insert now uses status_old/status_new/
status_timestamp (was status_nwb_id_old/status_nwb_id_new).
- Non-unique id fetch: submit_nwb_export_job no longer fetch1("nwb_job_id") by
session_key alone (a session can have multiple jobs). It now fetches by
{**session_key, "job_name": job_name} ordered by nwb_job_id DESC.
State-machine behavior is otherwise unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The NWB export handler and helpers referenced columns and tables that do not exist in the deployed
nwb_productionschema (u19_pipeline/nwb_production.py, the authoritative source). This PR reconciles the handler and helpers to that real schema. State-machine behavior is otherwise unchanged.Changes
status_nwb_id->status_id: The active-jobs query and per-job status read inpipeline_handler_main, plus theupdate1inupdate_status_pipeline, now use the realNwbExportJob.status_id(FK toNwbExportStatus).NwbExportModality:process_data_validation(handler) andestimate_total_size(nwb_production_utils) no longer query the nonexistentNwbExportJob.BehaviorExport/.EphysExport/.ImagingExportpart-tables. They now queryNwbExportModality & {"nwb_job_id": ...}and branch onmodality_name('behavior'/'ephys'/'imaging'), parsingprobe_numbers/fov_numbers(stored as JSON-array strings) and deriving the session/recording/scan keys from theNwbExportJobrecord.NwbExportLogStatusinsert now usesstatus_old/status_new/status_timestamp(wasstatus_nwb_id_old/status_nwb_id_new).submit_nwb_export_jobno longer doesfetch1("nwb_job_id")filtered bysession_keyalone (a session can have multiple jobs). It now fetches by{**session_key, "job_name": job_name}ordered bynwb_job_id DESC, taking the most recent.Testing
tests/nwb_exportno-db suite passes (193 passed); thewith_dbtests require a live DataJoint database not available in this environment, but they mock at module boundaries and already use the correct schema columns, so they are consistent with these fixes.Closes BrainCOGS/tech_calendar_streamlit#70
🤖 Generated with Claude Code