Huge Performance Degredation in version 5.0.1

[ewmcsweeney]

Recently upgraded from v4.0.0 to v5.0.6 and noticed a huge bump in CPU usage when using the ExecuteAllRulesAsync routine.

After going back through the change log, I can see this being attributed to the Memory Cache removal that was done in release 5.0.1

I did a pretty basic performance test by running the routine 1000x in a row (with 30 rules defined in my workflow). v5.0.0 will run all 1000 iterations in total in ~1.5 seconds, whereas v 5.0.1 runs an order of magnitude longer at ~16.0 seconds.

[pbhal]

@ewmcsweeney - will look into the issues causing this performance degradation and update the thread.

[donnytian]

Without rule changes, I observed a cpu increase in our service from 3% to 30%. P95 increased from 5ms to 300ms. We just upgraded from v5.0.2 to v6.0.0.
Unfunately, we have to revert our release.

[dario-griffo-paytently]

Without rule changes, I observed a cpu increase in our service from 3% to 30%. P95 increased from 5ms to 300ms. We just upgraded from v5.0.2 to v6.0.0. Unfunately, we have to revert our release.

A change in the compilation setting? If you don't have custom types you should be good, or don't even update
https://github.com/microsoft/RulesEngine/pull/675/files

[pbhal]

@donnytian @dario-griffo-paytently - With an upgrade to System.Linq.Dynamic.Core library, the existing FastExpressionCompiler runs in with a runtime exception when using custom types. If your use case doesn't involve using CustomTypes, you can toggle the UseFastExpressionCompiler option to true. I am currently looking at ways to enhance this degrade in performance with custom types.

[donnytian]

@pbhal Thanks for the response! We are using custom types to contain some complex logic. Any advise in this case?

[pbhal]

@donnytian - Currently we have prioritized fixing the security bugs over the System.Linq.Dynamic.Core library and upgrading it to latest version. We are currently looking into possible ways of mitigating the performance degradation when CustomTypes are being used. Will keep the thread updated with the findings this week.

[donnytian]

@pbhal any updates? Thanks for your attention!

[YogeshPraj]

@donnytian review #727