Error State

How are error states handled in drunc?

The intiial design is that once an application goes into error, only that application is in an error state. Once another stateful (FSM) transition is attempted, the segment controller goes into error. Once another stateful transition is attempted, the root controller goes into error. Once a controller goes into error, it no longer accepts stateful commands.

The philosophy of this is rooted in the days of collider experimenmts. In an error state, we still want to be able to write all the data that is being collected to a file before we shut down a given run. Once in an error state, this can be corrected and the error state can be removed with recompute_status once the source of the error has been addressed, or the given application has been excluded.