Flowlord Dashboard

.circleci/config.yml

@@ -1,10 +1,14 @@
-version: 2
+version: 2.1 
 
-jobs:
-  build:
+executors:
+  go-executor:
     docker:
       - image: cimg/go:1.23
-    working_directory: ~/task-tools
+    working_directory: ~/task-tools 
+
+jobs:
+  test:
+    executor: go-executor 
     steps:
       - checkout
       - run: go install github.com/jstemmer/go-junit-report/v2@latest
@@ -23,4 +27,36 @@ jobs:
       - store_artifacts:
           path: ~/task-tools/junit
       - store_artifacts:
-          path: tests.out
+          path: tests.out
+
+  build:
+    executor: go-executor
+    steps:
+      - checkout
+      - setup_remote_docker:
+          docker_layer_caching: false
+      - run: 
+          name: "docker login" 
+          command: echo ${DOCKERHUB_TOKEN} | docker login -u ${DOCKERHUB_USERNAME} --password-stdin
+      - run:
+          name: "Push Docker Image"
+          command: make docker
+
+workflows:
+  version: 2
+  test_and_build: 
+    jobs: 
+      - test: 
+          filters: 
+            tags: 
+              only: 
+                - /.*/
+      - build: 
+          requires: 
+            - test
+          context: 
+            - DOCKER
+          filters: 
+            tags: 
+              only: 
+                - /.*/

.gitignore

@@ -18,5 +18,7 @@ apps/utils/file-watcher/file-watcher
 */stats/stats
 apps/workers/sql-load/sql-load
 build
+tasks.db 
+*_preview.html
 
 coverage

README.md

@@ -88,9 +88,16 @@ func (tm *taskMaster) Run(ctx context.Context) error {
 ## Pre-built Apps 
 
 ### **Flowlord**
-an all-purpose TaskMaster that should be used with workflow files to schedule when tasks should run and the task hierarchy. It can retry failed jobs, alert when tasks fail and has an API that can be used to backload/schedule jobs and give a recap of recent jobs run. 
+Production-ready task orchestration engine for managing complex workflow dependencies with intelligent scheduling, automatic retries, and real-time monitoring. Features include:
 
-See Additional [docs](apps/flowlord/README.md).  
+- **Workflow Management** - Multi-phase workflows with parent-child task dependencies
+- **Intelligent Scheduling** - Cron-based scheduling with template-based task generation
+- **Optional SQLite Cache** - Task history, alerts, and file tracking for troubleshooting (non-critical, stateless operation)
+- **Web Dashboard** - Real-time monitoring UI with filtering, pagination, and date navigation
+- **Batch Processing** - Generate multiple tasks from date ranges, metadata arrays, or data files
+- **RESTful API** - Comprehensive API for backloading, monitoring, and workflow management
+
+See detailed [documentation](apps/flowlord/README.md).  
 
 
 ### Workers

apps/flowlord/README.md

@@ -1,11 +1,64 @@
 # flowlord taskmaster
-flowlord schedules and coordinates task dependency across workflows. Flowlord reads tasks from the done topic, failed tasks can be configured to retry a set number of times before being sent to slack and/or a retry_failed topic. Successful tasks will start children tasks.
 
-![](flowlord.drawio.svg)
+<img src="handler/static/favicon.svg" align="left" style="margin-right: 20px;" />
+
+**Flowlord** is a production-ready task orchestration engine that manages complex workflow dependencies with intelligent scheduling, automatic retries, and real-time monitoring. Built on the [task](https://github.com/pcelvng/task) ecosystem, it coordinates distributed workers through message bus communication while providing visibility into task execution through a web dashboard.
 
+**Key Features:**
+- **Workflow Management** - Define multi-phase workflows with parent-child task dependencies
+- **Intelligent Scheduling** - Cron-based scheduling with template-based task generation
+- **Automatic Retries** - Configurable retry logic with exponential backoff and jitter to prevent thundering herd
+- **File Watching** - Trigger tasks automatically when files are written to specified paths
+- **Batch Processing** - Generate multiple tasks from date ranges, metadata arrays, or data files
+- **Alerting** - Slack notifications for failed tasks and incomplete jobs with smart frequency management
+- **RESTful API** - Web UI and API for monitoring workflows, viewing task history, and managing alerts
 
 [![Static Badge](https://img.shields.io/badge/API%20Docs-green)](https://github.com/pcelvng/task-tools/wiki/Flowlord-API)
 
+<br clear="all"/>
+
+## Overview 
+
+![](flowlord.drawio.svg)
+
+## Monitoring & Troubleshooting
+
+SQLite is used to store phases and provides troubleshooting convenience by recording task history. As flowlord is stateless task management system, the cache persistent is not required for flowlord to work, but it convenient to review historical tasks and alerts. It is recommended to backup the task and alert logs for long term storage. 
+
+- **Task Records** - Full execution lifecycle with timing metrics
+- **Alert History** - Failed task tracking for debugging
+- **File Processing Audit** - Which files triggered which tasks
+- **Workflow State** - Phase configuration and dependencies
+
+The database is optional and non-critical. If deleted, Flowlord continues normally and rebuilds fresh data. Features:
+- Automatic backup/restore from remote paths (S3/GCS)
+- 90-day default retention with automatic cleanup
+- WAL mode for concurrent access
+
+**Configuration:**
+```toml
+[cache]
+  local_path = "./tasks.db"              # required local cache
+  backup_path = "gs://bucket/tasks.db"   # Optional backup location
+  retention = "2160h"                    # 90 days
+  task_ttl = "4h"                        # Alert deadline from task to complete (creation to complete)  
+```
+
+## Web Dashboard
+
+Built-in web UI for monitoring workflows and troubleshooting. Uses Go templates to render HTML dashboards with:
+- Task execution history with filtering and pagination
+- Alert summaries grouped by task type
+- File processing history
+- Workflow phase visualization
+- System statistics
+
+Access at `http://localhost:8080/` (or configured port)
+
+| Files View | Tasks View | Alerts View | Workflow View |
+|:----------:|:----------:|:-----------:|:-------------:|
+| [![Files View](../../internal/docs/img/flowlord_files.png)](../../internal/docs/img/flowlord_files.png) | [![Tasks View](../../internal/docs/img/flowlord_tasks.png)](../../internal/docs/img/flowlord_tasks.png) | [![Alerts View](../../internal/docs/img/flowlord_alerts.png)](../../internal/docs/img/flowlord_alerts.png) | [![Workflow View](../../internal/docs/img/flowlord_workflow.png)](../../internal/docs/img/flowlord_workflow.png) |
+
 
 ## workflow 
 A workflow consists of one or more phases as a way to define of how a set of task is to be scheduled and run and the dependencies between them.