feat: Integrate Workflow Manger V2

[sthanikan2000]

Summary

This PR brings the integration of Workflow Manger package defined in the https://github.com/OpenNSW/go-temporal-workflow. This package introduces the Node and Edges based graph representation for the workflow management with token passing through the edges to unlock new workflow nodes. And it make use of temporal.io for managing states of the workflow nodes.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)
• Performance improvement
• Other (please describe):

Changes Made

• Added configuration option to enable the Workflow Manager V2 in the application.
• Updated the TaskManager interface to support callbacks for task completion and updates using the v2 workflow manager.
• Implemented logic in the ConsignmentService to handle workflow initialization and retrieval for both v1 and v2 workflows.
• Introduced a new JSON template for the Customs workflow.
• Enhanced unit tests to accommodate the workflow manager v2 integration.
• Add docker compose to run the temporal locally

Testing

• I have tested this change locally
• I have added unit tests for new functionality
• I have tested edge cases
• All existing tests pass

Testing Guidelines

1. Run cp .env.exampl .env in the root
2. Set the USE_WORKFLOW_MANAGER_V2 as,
   • false to validate v1 workflow manager functioning properly without any breaking
   • 'true` to check the functionality of the v2 workflow manager
3. Run ./start-dev.sh. This will brings up all the services to validate end-to-end flow

Additional Notes

This only bring the new version of the workflow manager such that previous functionality also exists. You need to setup the boolean environment USE_WORKFLOW_MANAGER_V2 as true to make use of new version of Workflow Manger package

Deployment Notes

• Need to deploy Self-hosted Temporal Service if we setup USE_WORKFLOW_MANAGER_V2 as true. Guide available at https://docs.temporal.io/self-hosted-guide

Limitations need to be address in future PR

• The current workflow management is only wired for Consignment Workflow. And it's still not stable.

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have checked that there are no merge conflicts

Related Issues

No specific issues created for this PR

Screenshots/Demo

N/A (backed change)

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant upgrade to the application's workflow management capabilities by integrating a new, more robust Workflow Manager V2, powered by Temporal.io. This new system allows for a more flexible and scalable approach to defining and executing complex workflows using a graph-based model. The changes are implemented with a feature flag, allowing for a phased rollout and ensuring that existing functionalities remain operational. This lays the groundwork for enhanced workflow orchestration and improved handling of business processes.

Highlights

• Workflow Manager V2 Integration: Integrated a new Workflow Manager (V2) based on the OpenNSW/go-temporal-workflow package and temporal.io, enabling node and edge-based graph representation for workflow management with token passing.
• Conditional Activation: Introduced a USE_WORKFLOW_MANAGER_V2 configuration option and environment variable to allow toggling between the old and new workflow managers, ensuring backward compatibility during the migration phase.
• Task Manager Interface Updates: Modified the TaskManager interface to support new callback mechanisms (RegisterUpstreamDoneCallback, RegisterUpstreamUpdateCallback) specifically for the V2 workflow manager, handling task completion and updates.
• Consignment Service Logic: Updated the ConsignmentService to conditionally initialize and retrieve workflow instances using either the V1 or V2 workflow manager, depending on the active configuration.
• Temporal Local Development Setup: Added a docker-compose.yml file and start-dev.sh script modifications to easily set up and run a local Temporal server, PostgreSQL, and Temporal UI for development and testing of the V2 workflow manager.
• New Customs Workflow Template: Introduced a new JSON template (CustomsWorkflowJSON) for the Customs workflow, designed for use with the V2 workflow manager.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request integrates a new Workflow Manager v2 based on Temporal, using a feature flag to switch between the existing v1 and the new v2 implementation. The changes are extensive, touching configuration, application bootstrap, task management, and workflow services. The use of a feature flag is a sound strategy for this migration. My review highlights several areas for improvement, particularly in logging practices, error handling, and the completeness of the v2 implementation path. I've identified instances where logging levels are inappropriate, error handling could be more robust, and parts of the v2 implementation appear incomplete or may lead to information loss in the API response. For some identified gaps in the v2 implementation, I've noted that deferral is acceptable if tracked, aligning with repository guidelines.