README

Workflow Automation Tool (WhAT)

Workflow Instance Creation

1. Workflow creation
   1. workflow copied
   2. Ids replaced and template bool flipped
   3. State: started
   4. Flowstate created
2. NodeInstances copied (by finding all nodeinstances in the template workflow: workflow.node_instances)
   1. workflow assignment changed to new workflow
3. Start on Start node is sent to arq

Workflow Processing

1. Called via arq process
2. Filter all waiting workflows
   1. Get all nodeinstances tied to workflow
   2. ForEach: Filter all waiting nodes
      1. Check depends/depends_on to determine if node can run
      2. Run node if appropriate(send node id to arq)
      3. IF action node send to arq
      4. IF decision process it
      5. Continue to next node

Ingestion

• Ingestion Start nodes need an entry in ingestion url table site/w/my-name-friendly-goes-here This would clone and start the workflow and add the relevant data to flowstate
• If not a start node needs workflow instance as well site/ingest/id/some-workflow-uuid4-goes-here
  • Maybe this could just be the id of the ingestion node
  • Might not need /ingest at all, just check all URL variables for uuid or string
• If next node is a response node process it and respond

Notes

• https://testdriven.io/blog/fastapi-and-celery/
• Does workflow need a metadata to get name/description/version? or include in workflow body
• workflow: 21d3d9aa-2ba8-11ed-81b0-c3b7eaeab847
• workflow template: 550ef5da-36a6-11ed-a892-bb8818cce9dc
• node: 9a465bea-2bd2-11ed-b25b-ab922a707b03
• node instance: 1f37564c-2bd3-11ed-9214-97a3e3a9190a

Node Instance Relationships

• NIs know parents : list[dict[str, UUID]]
• NI.children: list[dict[id, UUID]] is done via reverse lookup of NI.parents
• NI.depends: int how many parents a child should wait for (may be less than len(NI.parents))
  • -1: all parents
  • 0: no parents
  • >1: N parents
• NI.depends_on: list[dict[id, UUID]] of specific parents that must be awaited

ToDo [6/6]

• Important [9/16]
  • [X] Arq to actually run the workflows
  • [X] Re-enqueue on callback and interactive response
  • [ ] Worker status viewer [0/1]
    • Is this actually important?
    • [ ] Track running workflows / nodes
  • [X] interactive node
  • [X] interactive node calls shouldn’t run on blocked
  • [ ] validate interactive node config input on creation/update (lots of that necessary for workflows in general)
    • [ ] Pydantic schema exists for all known config fields now
  • [X] Ingestion friendly name table/path (see notes above)
    • Notes above mention a start node ingest, but /car already takes a json body for start state.
      • Takes query param for wf_id and body with “start” key
  • [X] lanes and a way to id users (trusts?)
  • [X] Polling
    • [X] API to check for desired response to continue (maybe including fetching the results)
  • [X] Callbacks
    • How does a node wait for a callback? Set state to waiting
    • Poll state to continue a workflow? or Does the callback handler set this and execute wf again? (this seems to make more sense)
      • [ ] Parse the cb data using a model stored in NI config
  • [-] GUI to configure/build
    • [-] Shows required states for all node instances and allows selecting which to add to wf start attribs
      • [X] Shows required states
      • [ ] Allows selecting which to add to wf start attributes
  • [ ] GUI Dashboard
  • [ ] workflow starting other workflows [0/3]
    • [ ] check workflow and node status endpoints / svc functions
    • [ ] workflows should return configured state to parent workflow / API caller
    • [ ] Options to wait for it or to be happy with queuing
  • [X] Interactive node should only work on pending nodes
  • [ ] Interactive node should only allow users on lane to use it.
  • [ ] Get workflows (or ni) based on lane token
• Fixes [0/4]
  • [ ] cleanup tx/client issues
  • [ ] use generated queries
  • [ ] Fix node instances need sequence number, workflows need name/version
  • [ ] Workflows mark themselves complete (I think this works if I had one with a finish)
• Nice to Have [0/9]
  • [ ] Math Decision Node processing (nodes.decision NotImplementedError) # min/max
  • [ ] Validation on Attributes
  • [ ] api node needs to validate whatever it wants to store into flowstate. is this the same as validation on attributes now?
  • [ ] dynamic validation system for api node to flowstate
  • [ ] CLI?
  • [ ] Workflow instance creation fills in a template_id to know what template it came from
  • [ ] Dynamic models should include friendly names and maybe descriptions for interactive node uses
  • [ ] Graphlib to replace my code for graph?
  • [ ] Run all pending nodes simultaneously?
• Dev Only [0/4]
  • [ ] system to do inserts for db start state (required attributes and such)
  • [ ] system to populate db for tests
  • [ ] functional tests
  • [ ] contract testing
• Consider [0/1]
  • [ ] Drop Node, moving name, version, base, and type to NodeInstances to simplify things
    • NIs need a name to go with the new template fields anyway.
• [X] saving state and changes
• [X] Try individual query files and/or directories (edgedb-py doesn’t innately do different directories)
• [X] Validation on Start Requirements
• [X] start node needs to take validated inputs to enter into flowstate (workflow does this instead)
• [X] fastapi endpoint for starting workflow should take a dict of state for start
• [X] Decision Node

GUI Notes

• [X] Location Coordinates in Workflow {id: {x: int, y: int}}
• [X] NI Name
• [ ] Sidebar has nodes/node instances to choose from, drag/drop them into place
• [ ] Configure node instances (nodes?)
• [ ] Connecting nodes saves parent/child relationships
• [ ] Viewing a workflow shows node instance states, and they are immediatly apparent (change color?)
• [X] Show flowstate in wf view
• [ ] Arq status viewer
• [ ] Saving NI configs