Experiments with NN models having >1000 parameters & simulation freeze in OpenModelica

[awecefil]

Hi, thanks for your great work on this project!

I have a question regarding the use of neural networks with a large number of parameters in OpenModelica.

I modified the provided Dense layer so that it can directly read weights and biases from a .mat file (instead of manually pasting parameters into the model because it is not possible to paste all parameters into the parameter field in OpenModelica due to size limitations). The implementation works correctly for small networks.

However, when I try using a neural network with more than ~1000 parameters, I encounter two issues:

When the number of parameters becomes large, the simulation does not start.
As soon as I press Simulate, OpenModelica freezes and never proceeds to the simulation stage.

So I’d like to ask:

1. Have you ever tested your NN implementation with networks containing more than 1000 parameters?
2. Are there known limitations in OpenModelica regarding large NN structures?

Any guidance would be greatly appreciated. Thank you!

[phannebohm]

Glad to hear you're using the library. Thanks for the feedback!

I modified the provided Dense layer so that it can directly read weights and biases from a .mat file (instead of manually pasting parameters into the model because it is not possible to paste all parameters into the parameter field in OpenModelica due to size limitations). The implementation works correctly for small networks.

Would you like to contribute with a PR?

1. Have you ever tested your NN implementation with networks containing more than 1000 parameters?

Not really, we tried a few hundred parameters (see this paper with around 600 parameters). I don't know the upper limit of what a modern Modelica compiler can do.

However, the intended use case has been for PeN-ODEs mainly. The idea is to train a NN within a larger system model so the NN has to learn only small parts of the overall system dynamics, so to speak.

The NeuralNetwork library is mostly interesting if you want to make the internal structure available to the compiler, be it for exporting to eFMI or training the NN. If you want to include large NNs and don't care about pure Modelica equations, you can try the SMArtInt library that uses external functions to hide the internal complexity of the NN from the Modelica compiler.

2. Are there known limitations in OpenModelica regarding large NN structures?

The OpenModelica Compiler (omc), and I suspect any Modelica compiler, performs a structural analysis of the equation system resulting from the model. This system also includes the equations of the NN. The old backend of omc scalarizes all equations and (parameter) variables, so that matrix-vector products are expanded into fully connected subgraphs. The time is mostly spent just building that structure from the equations and traversing it. That will probably make the compilation take forever for even medium sized NNs, which is why the simulation doesn't start.

The new backend of omc is currently under development and looks promising, see coverage of NeuralNetwork. It still has many issues but the overall coverage is going up almost daily. If your models work with it (compiler flag --newBackend) you can expect some major speed improvement with the new causalization module. However, that is still very much experimental, so no guarantees.

[jgoppert]

I just found this library and we are considering how to handle it in the rumoca Modelica compiler: CogniPilot/rumoca#266. I think there could be some good synergies. We are also tinkering with neural ODEs in our lab: https://cognipilot.github.io/mocap_sysid/