chore: user office experiment workflow

[yoganandaness]

Description

This PR implements comprehensive workflow improvements for experiment safety management and integrates the new workflow data structures across the application. It includes support for manual status changes, enhanced workflow engine with guards, and updates to frontend components for better workflow status handling.

Motivation and Context

The experiment safety workflow needs to support more sophisticated status transitions and manual interventions similar to the proposal workflow. This requires integrating the new workflow data structures (from SWAP-4949) into the experiment safety module and implementing proper authorization checks, event handling, and UI components for managing experiment safety workflow statuses.

Related Issue: SWAP-5222

How Has This Been Tested

• Unit tests updated and fixed for new workflow structures
• E2E tests updated for:
  • Experiment safety review workflows
  • Instrument settings with new workflow statuses
  • Settings workflow management
• Integration tests for workflow guards and state transitions
• Manual testing of experiment safety status changes
• Fixes applied to resolve flaky tests

Changes

Backend Changes

• Database Migrations:

  • 0208_MigrateOldWorkflowsToNewDataStructures.sql - Migrates existing workflows to new data structures
  • 0211_ExperimentSafetyWorkflowStatusNotNull.sql - Makes experiment_safety.workflow_status_id non-nullable with AWAITING_ESF backfill

• Workflow Engine & Guards:

  • Added experiment workflow engine (src/workflowEngine/experiment.ts)
  • Implemented new guards for experiment ESF status checking
  • Enhanced workflow state machine with proper transitions

• Data Sources:

  • Updated ExperimentDataSource to support workflow status management
  • Enhanced WorkflowDataSource for experiment safety workflows
  • Updated mock implementations for testing

• Event Handlers:

  • Implemented experiment workflow event handlers
  • Added email and RabbitMQ handlers for experiment status changes
  • Added status action engine for experiment workflows
  • Enhanced proposal workflow event handlers

• Authorization:

  • Updated ExperimentSafetyAuthorization for workflow-based access control
  • Enhanced authorization checks in workflow operations

• Mutations:

  • Added ChangeExperimentsSafetyStatusMutation for manual status changes
  • Enhanced experiment and proposal mutations

Frontend Changes

• Experiment Safety Components:

  • ChangeExperimentSafetyStatus.tsx - New component for manual status changes
  • ExperimentSafety.tsx, ExperimentSafetyPage.tsx, ExperimentSafetyReview.tsx - Updated for new workflow status handling
  • ExperimentSafetyNotification.tsx - Enhanced notifications for workflow changes

• Workflow Management:

  • WorkflowEditor.tsx, WorkflowView.tsx - Enhanced workflow visualization
  • StatusEventsAndActionsDialog.tsx - Updated for workflow actions
  • workflowUtils.ts - Utility functions for workflow status handling

• UI Updates:

  • Updated status picker and proposal status components
  • Enhanced questionary step factory for workflow integration
  • Improved experiment and proposal status display tables

E2E Tests

• Updated Cypress tests for experiment safety review workflows
• Fixed instrument and settings tests with new workflow structures
• Enhanced test coverage for workflow status transitions

Depends on

This PR builds upon the workflow overhaul from SWAP-4949. The new workflow data structures and state machine implementation are integrated into the experiment safety module.

Tests included/Docs Updated?

• I have added tests to cover my changes.
• All relevant doc has been updated

[jekabs-karklins]

suggestion: Can we group both workflows together?

[jekabs-karklins]

I am still getting this error running the branch against our staging database

[2026-06-09T14:17:55.821Z] ERROR - Failed to initialize db 
 {"exception":{"name":"error","message":"DO\n$$\nDECLARE\n    v_updated BIGINT := 0;\nBEGIN\n    IF register_patch(\n       'ExperimentSafetyWorkflowStatusNotNull',\n       'Yoganandan Pandiyan',\n       'Make experiment_safety.workflow_status_id NOT NULL after backfilling with AWAITING_ESF status from experiment workflow.',\n       '2026-05-29'\n     ) THEN\n\n        -- Backfill existing experiment_safety rows that have NULL workflow_status_id.\n        -- Chain: experiment_safety → experiments (experiment_pk) → proposals (proposal_pk)\n        --        → call (call_id) → call.experiment_workflow_id → workflow_has_statuses\n        --        (where status_id = 'AWAITING_ESF')\n        --\n        -- NOTE: If a call has no experiment_workflow_id assigned, those rows are skipped.\n        -- This means experiment_safety rows linked to such calls will retain NULL\n        -- workflow_status_id, which will cause an error when SET NOT NULL is applied.\n        UPDATE experiment_safety es\n        SET workflow_status_id = whs.workflow_status_id\n        FROM experiments e\n        JOIN proposals p ON p.proposal_pk = e.proposal_pk\n        JOIN call c ON c.call_id = p.call_id\n        JOIN workflow_has_statuses whs\n          ON whs.workflow_id = c.experiment_workflow_id\n         AND whs.status_id = 'AWAITING_ESF'\n        WHERE es.experiment_pk = e.experiment_pk\n          AND es.workflow_status_id IS NULL\n          AND c.experiment_workflow_id IS NOT NULL;\n\n        GET DIAGNOSTICS v_updated = ROW_COUNT;\n\n        IF v_updated = 0 THEN\n          RAISE NOTICE 'No experiment_safety rows required workflow_status_id backfill.';\n        ELSE\n          RAISE NOTICE USING MESSAGE = format('%s experiment_safety rows updated with AWAITING_ESF workflow_status_id.', v_updated);\n        END IF;\n\n        -- Make the column non-nullable now that existing rows have been backfilled\n        ALTER TABLE experiment_safety ALTER COLUMN workflow_status_id SET NOT NULL;\n\n    END IF;\nEND;\n$$\nLANGUAGE plpgsql;\n - column \"workflow_status_id\" of relation \"experiment_safety\" contains null values","stack":"error: DO\n$$\nDECLARE\n    v_updated BIGINT := 0;\nBEGIN\n    IF register_patch(\n       'ExperimentSafetyWorkflowStatusNotNull',\n       'Yoganandan Pandiyan',\n       'Make experiment_safety.workflow_status_id NOT NULL after backfilling with AWAITING_ESF status from experiment workflow.',\n       '2026-05-29'\n     ) THEN\n\n        -- Backfill existing experiment_safety rows that have NULL workflow_status_id.\n        -- Chain: experiment_safety → experiments (experiment_pk) → proposals (proposal_pk)\n        --        → call (call_id) → call.experiment_workflow_id → workflow_has_statuses\n        --        (where status_id = 'AWAITING_ESF')\n        --\n        -- NOTE: If a call has no experiment_workflow_id assigned, those rows are skipped.\n        -- This means experiment_safety rows linked to such calls will retain NULL\n        -- workflow_status_id, which will cause an error when SET NOT NULL is applied.\n        UPDATE experiment_safety es\n        SET workflow_status_id = whs.workflow_status_id\n        FROM experiments e\n        JOIN proposals p ON p.proposal_pk = e.proposal_pk\n        JOIN call c ON c.call_id = p.call_id\n        JOIN workflow_has_statuses whs\n          ON whs.workflow_id = c.experiment_workflow_id\n         AND whs.status_id = 'AWAITING_ESF'\n        WHERE es.experiment_pk = e.experiment_pk\n          AND es.workflow_status_id IS NULL\n          AND c.experiment_workflow_id IS NOT NULL;\n\n        GET DIAGNOSTICS v_updated = ROW_COUNT;\n\n        IF v_updated = 0 THEN\n          RAISE NOTICE 'No experiment_safety rows required workflow_status_id backfill.';\n        ELSE\n          RAISE NOTICE USING MESSAGE = format('%s experiment_safety rows updated with AWAITING_ESF workflow_status_id.', v_updated);\n        END IF;\n\n        -- Make the column non-nullable now that existing rows have been backfilled\n        ALTER TABLE experiment_safety ALTER COLUMN workflow_status_id SET NOT NULL;\n\n    END IF;\nEND;\n$$\nLANGUAGE plpgsql;\n - column \"workflow_status_id\" of relation \"experiment_safety\" contains null values\n    at Parser.parseErrorMessage (/mnt/data/discipline/wealth/work/ess/repos/user-office-core/apps/backend/node_modules/pg-protocol/src/parser.ts:369:69)\n    at Parser.handlePacket (/mnt/data/discipline/wealth/work/ess/repos/user-office-core/apps/backend/node_modules/pg-protocol/src/parser.ts:188:21)\n    at Parser.parse (/mnt/data/discipline/wealth/work/ess/repos/user-office-core/apps/backend/node_modules/pg-protocol/src/parser.ts:103:30)\n    at Socket.<anonymous> (/mnt/data/discipline/wealth/work/ess/repos/user-office-core/apps/backend/node_modules/pg-protocol/src/index.ts:7:48)\n    at Socket.emit (node:events:519:28)\n    at addChunk (node:internal/streams/readable:561:12)\n    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)\n    at Socket.Readable.push (node:internal/streams/readable:392:5)\n    at TCP.onStreamRead (node:internal/stream_base_commons:189:23)"},"initDbFailed":2}
[2026-06-09T14:17:56.823Z] INFO - Applying patches started 
 {"timestamp":"2026-06-09T14:17:56.823Z"}
[2026-06-09T14:17:56.980Z] ERROR - QUERY ERROR 
 {"message":"DO\n$$\nDECLARE\n    v_updated BIGINT := 0;\nBEGIN\n    IF register_patch(\n       'ExperimentSafetyWorkflowStatusNotNull',\n       'Yoganandan Pandiyan',\n       'Make experiment_safety.workflow_status_id NOT NULL after backfilling with AWAITING_ESF status from experiment workflow.',\n       '2026-05-29'\n     ) THEN\n\n        -- Backfill existing experiment_safety rows that have NULL workflow_status_id.\n        -- Chain: experiment_safety → experiments (experiment_pk) → proposals (proposal_pk)\n        --        → call (call_id) → call.experiment_workflow_id → workflow_has_statuses\n        --        (where status_id = 'AWAITING_ESF')\n        --\n        -- NOTE: If a call has no experiment_workflow_id assigned, those rows are skipped.\n        -- This means experiment_safety rows linked to such calls will retain NULL\n        -- workflow_status_id, which will cause an error when SET NOT NULL is applied.\n        UPDATE experiment_safety es\n        SET workflow_status_id = whs.workflow_status_id\n        FROM experiments e\n        JOIN proposals p ON p.proposal_pk = e.proposal_pk\n        JOIN call c ON c.call_id = p.call_id\n        JOIN workflow_has_statuses whs\n          ON whs.workflow_id = c.experiment_workflow_id\n         AND whs.status_id = 'AWAITING_ESF'\n        WHERE es.experiment_pk = e.experiment_pk\n          AND es.workflow_status_id IS NULL\n          AND c.experiment_workflow_id IS NOT NULL;\n\n        GET DIAGNOSTICS v_updated = ROW_COUNT;\n\n        IF v_updated = 0 THEN\n          RAISE NOTICE 'No experiment_safety rows required workflow_status_id backfill.';\n        ELSE\n          RAISE NOTICE USING MESSAGE = format('%s experiment_safety rows updated with AWAITING_ESF workflow_status_id.', v_updated);\n        END IF;\n\n        -- Make the column non-nullable now that existing rows have been backfilled\n        ALTER TABLE experiment_safety ALTER COLUMN workflow_status_id SET NOT NULL;\n\n    END IF;\nEND;\n$$\nLANGUAGE plpgsql;\n - column \"workflow_status_id\" of relation \"experiment_safety\" contains null values","error":{"length":343,"name":"error","severity":"ERROR","code":"23502","where":"SQL statement \"ALTER TABLE experiment_safety ALTER COLUMN workflow_status_id SET NOT NULL\"\nPL/pgSQL function inline_code_block line 41 at SQL statement","schema":"public","table":"experiment_safety","column":"workflow_status_id","file":"tablecmds.c","line":"6111","routine":"ATRewriteTable"},"obj":{"__knexUid":"__knexUid1","__knexTxId":"__knexUid1","method":"raw","sql":"DO\n$$\nDECLARE\n    v_updated BIGINT := 0;\nBEGIN\n    IF register_patch(\n       'ExperimentSafetyWorkflowStatusNotNull',\n       'Yoganandan Pandiyan',\n       'Make experiment_safety.workflow_status_id NOT NULL after backfilling with AWAITING_ESF status from experiment workflow.',\n       '2026-05-29'\n     ) THEN\n\n        -- Backfill existing experiment_safety rows that have NULL workflow_status_id.\n        -- Chain: experiment_safety → experiments (experiment_pk) → proposals (proposal_pk)\n        --        → call (call_id) → call.experiment_workflow_id → workflow_has_statuses\n        --        (where status_id = 'AWAITING_ESF')\n        --\n        -- NOTE: If a call has no experiment_workflow_id assigned, those rows are skipped.\n        -- This means experiment_safety rows linked to such calls will retain NULL\n        -- workflow_status_id, which will cause an error when SET NOT NULL is applied.\n        UPDATE experiment_safety es\n        SET workflow_status_id = whs.workflow_status_id\n        FROM experiments e\n        JOIN proposals p ON p.proposal_pk = e.proposal_pk\n        JOIN call c ON c.call_id = p.call_id\n        JOIN workflow_has_statuses whs\n          ON whs.workflow_id = c.experiment_workflow_id\n         AND whs.status_id = 'AWAITING_ESF'\n        WHERE es.experiment_pk = e.experiment_pk\n          AND es.workflow_status_id IS NULL\n          AND c.experiment_workflow_id IS NOT NULL;\n\n        GET DIAGNOSTICS v_updated = ROW_COUNT;\n\n        IF v_updated = 0 THEN\n          RAISE NOTICE 'No experiment_safety rows required workflow_status_id backfill.';\n        ELSE\n          RAISE NOTICE USING MESSAGE = format('%s experiment_safety rows updated with AWAITING_ESF workflow_status_id.', v_updated);\n        END IF;\n\n        -- Make the column non-nullable now that existing rows have been backfilled\n        ALTER TABLE experiment_safety ALTER COLUMN workflow_status_id SET NOT NULL;\n\n    END IF;\nEND;\n$$\nLANGUAGE plpgsql;\n","bindings":[],"options":{},"__knexQueryUid":"czK7wGPLynLdmtFVn5ctY"},"QueryName":"DO\n$$\nDECLARE\n    v_updated BIGINT := 0;\nBEGIN\n    IF register_patch(\n       'ExperimentSafetyWorkflowStatusNotNull',\n       'Yoganandan Pandiyan',\n       'Make experiment_safety.workflow_status_id NOT NULL after backfilling with AWAITING_ESF status from experiment workflow.',\n       '2026-05-29'\n     ) THEN\n\n        -- Backfill existing experiment_safety rows that have NULL workflow_status_id.\n        -- Chain: experiment_safety → experiments (experiment_pk) → proposals (proposal_pk)\n        --        → call (call_id) → call.experiment_workflow_id → workflow_has_statuses\n        --        (where status_id = 'AWAITING_ESF')\n        --\n        -- NOTE: If a call has no experiment_workflow_id assigned, those rows are skipped.\n        -- This means experiment_safety rows linked to such calls will retain NULL\n        -- workflow_status_id, which will cause an error when SET NOT NULL is applied.\n        UPDATE experiment_safety es\n        SET workflow_status_id = whs.workflow_status_id\n        FROM experiments e\n        JOIN proposals p ON p.proposal_pk = e.proposal_pk\n        JOIN call c ON c.call_id = p.call_id\n        JOIN workflow_has_statuses whs\n          ON whs.workflow_id = c.experiment_workflow_id\n         AND whs.status_id = 'AWAITING_ESF'\n        WHERE es.experiment_pk = e.experiment_pk\n          AND es.workflow_status_id IS NULL\n          AND c.experiment_workflow_id IS NOT NULL;\n\n        GET DIAGNOSTICS v_updated = ROW_COUNT;\n\n        IF v_updated = 0 THEN\n          RAISE NOTICE 'No experiment_safety rows required workflow_status_id backfill.';\n        ELSE\n          RAISE NOTICE USING MESSAGE = format('%s experiment_safety rows updated with AWAITING_ESF workflow_status_id.', v_updated);\n        END IF;\n\n        -- Make the column non-nullable now that existing rows have been backfilled\n        ALTER TABLE experiment_safety ALTER COLUMN workflow_status_id SET NOT NULL;\n\n    END IF;\nEND;\n$$\nLANGUAGE plpgsql;\n"}

[TCMeldrum]

Quite a large pr so have put my comments in for the backend will look through the rest later this week

[yoganandaness]

suggestion: Can we group both workflows together?

Good point. I would like to look @ellen-wright.

@ellen-wright We would like to bring the exp workflow along with proposal workflow. As I understand, stfc decided to move the proposal workflow up in the menu. So I wanted to check if you could help us with this.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (2)

apps/backend/src/eventHandlers/workflowEntities/proposal/statusActionEngine.ts:25

• Proposal doesn't have workflowId or statusId fields (see apps/backend/src/models/Proposal.ts), so grouping by ['workflowId', 'statusId'] collapses everything into one group (both keys are undefined). That can result in executing status actions using the first proposal’s workflowStatusId for all proposals, even when they differ.
  apps/backend/src/eventHandlers/workflowEntities/proposal/statusActionEngine.ts:33
• Looking up a workflow connection only by next workflow status id is ambiguous when multiple transitions lead into the same status (allowed by the schema; only (workflow_id, prev, next) is unique). In those cases this can fetch the wrong connection and run the wrong status actions. The status action engine should use the actual workflow_status_connection_id from the workflow transition (or at minimum include prevWorkflowStatusId in the lookup).

[TCMeldrum]

Looked through the frontend/e2e, seemed fine to me.