Does count of workflows stuck in delayed queue , impact performance of ST2 execution?

[zsmanjot]

I have an environment where we have tons of alerts and those are actioned upon by ST2. But when alert flow is more and we have delayed policy in place , the workflows get stuck in delayed queue and the execution of other workflows becomes very slow.

For example:

And here below one execution started at 19 sept 16:38 UTC but completed at 20 Sept 00:15 UTC

Almost after 8 hours, the execution got completed.

Anyone who can suggest here?

[orangelzh]

how many action runner do you have
and how many workflowengine process do you run
suggest imcrease this two process number。

[zsmanjot]

Have 20 action runners, How can we check workflow engine process?

[orangelzh]

ps auxf|grep workflowengine
The recommended ratio of workflowengine to actionrunner is 1:3
If the number of action tasks to be processed concurrently is 100, it is recommended that the number of action runners is 50 (assuming that it takes 30 seconds to execute an action), and the number of workflowengines is 20.

the following script is start workflowengine process：
#!/bin/bash
while [[ $retry_i -lt 20 ]]
do
now=date +%Y%m%d%H%M
retry_i=expr $retry_i + 1
if [[ ps auxf|grep st2workflowengine |wc -l -gt 20 ]]
then
exit
fi
nohup /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2workflowengine --config-file /etc/st2/st2.conf >> /var/log/st2/st2workflowengine_${now}_${retry_i}.log 2>&1 &
sleep 2
done

[zsmanjot]

Hey i have 200 actionrunners but just 2 workflowengine running. How can i increase the st2workflowengine? Please help.

[orangelzh]

#!/bin/bash
retry_i=1
while [[ $retry_i -lt 60 ]]
do
now=date +%Y%m%d%H%M
retry_i=expr $retry_i + 1
if [[ ps auxf|grep st2workflowengine |wc -l -gt 60 ]]
then
exit
fi
nohup /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2workflowengine --config-file /etc/st2/st2.conf >> /var/log/st2/st2workflowengine_${now}_${retry_i}.log 2>&1 &
sleep 2
done

[zsmanjot]

@orangelzh

Could you explain how it is going to help? Where i have to place this script and how it will be invoked?

[orangelzh]

Just execute this script directly on your st2 machine, and then ps auxf | grep workflowengine to see if the number of processes is correct.