[DSIP-107][Master] Add separate workflow instance logs for UI viewing similar to task logs.

[njnu-seafish]

Search before asking

• I had searched in the DSIP and found no similar DSIP.

Motivation

Currently, all workflow execution logs are mixed within the Master service logs. This requires operations staff to log in to the server to access them, making it impossible for regular users to view logs related to their specific workflows.

Design Detail

The core of this design relies on Logback's SiftingAppender, which dynamically creates separate log files for each distinct workflow instance. This ensures that logs for different workflows are isolated from one another, facilitating easier debugging and UI presentation.

Here is the detailed breakdown of the mechanism:

1. Dynamic Routing with SiftingAppender
   The WORKFLOWLOGFILE is defined as a ch.qos.logback.classic.sift.SiftingAppender. Unlike standard appenders that write to a single file, this component acts as a router. It evaluates every log event and decides which specific file to write to based on runtime data.

2. Context-Based Discrimination
   The separation logic is driven by the block:
   Key: workflowInstanceLogFullPath
   Mechanism: The system extracts this key from the MDC (Mapped Diagnostic Context). In DolphinScheduler, the WorkflowLogDiscriminator retrieves the unique path for the current workflow instance from the thread's context.
   Result: Each unique workflowInstanceLogFullPath triggers the creation of a distinct log file.

3. Filtering Logic
   Before routing, a specific filter is applied:
   Filter Class: org.apache.dolphinscheduler.server.master.log.WorkflowLogFilter
   Function: This ensures that only logs relevant to a workflow instance are processed by this appender. System-level logs (which lack workflow context) are ignored here, preventing them from polluting the instance-specific files.

4. File Storage Strategy
   The actual file writing is handled by the nested FileAppender inside the block:
   Base Directory: Defined by ${log.base}/workflows.
   File Naming: The final path is constructed dynamically using ${workflowInstanceLogFullPath}.
   Append Mode: Set to true, ensuring that new logs are added to the existing file rather than overwriting it, which is crucial for long-running workflows.

5. Log Content & Format
   The logs written to these separated files use a specific pattern:
   Pattern: %d{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%thread] %logger{10}:[%line] - %message%n
   Note: Unlike the MASTERLOGFILE, the workflow-specific logs here do not explicitly repeat the [WI-...] prefix in the pattern, as the file name itself identifies the workflow instance.

Compatibility, Deprecation, and Migration Plan

No response

Test Plan

I have already tested and verified this solution in my local environment.

The log_path field in the t_ds_workflow_instance table:

Isolated workflow instance log files on the Master service:

Code of Conduct

• I agree to follow this project's Code of Conduct

[ruanwenjun]

+1,
We need to optimize the log content. We may need to add a new WorkflowInstanceLogEmitter or WorkflowInstanceLogCollector to collect logs and write them to a log file, so that more intuitive log content is generated.

The current logs are more server-oriented and difficult for users to understand.