Option to setup direct solver once for entire simulation and re-use matrices assembly and factorization

[jibril-b-coulibaly]

Problem Statement / Feature Request

I have been profiling FEA simulations that use HHT timestepper and PardisoMKL solver (see kcachegrind image below).
I noticed the ChDirectSolverLS::Setup() function, in particular the factorization ChSolverPardisoMKL::FactorizeMatrix() takes up a very large majority of the computational time.

The system I am simulating is subject to slow and small transformations, so that the stiffness and mass matrix is nearly constant throughout the simulation. Therefore, a lot of time is wasted factorizing a matrix that is nearly constant.

This simulation already uses the modified_newton flag in the HHT timestepper, which calls Setup once per step (called 4x in the image below) and reuses the factorization for each iteration (called 13x in the image below).
My proposal/request is to implement an additional flag that only calls setup once at the beginning of the simulation, and stores persistent initial matrices, and re-uses them throughout the entire simulation.

The tradeoff for using initial matrices can be higher memory costs and a smaller convergence rate / larger number of iterations, but given the high cost of factorization I believe this would be beneficial for FEA simulations where these matrices are almost constant (small transformations), or where using the initial elastic stiffness is commonplace (e.g. fracture mechanics simulations where softening and loss of positive definiteness can cause numerical difficulties).

Implementation ideas

I create a draft pull request #567 so we can collaborate on implementing the feature. Here are some ideas so far:

• Implementing a new flag at the timestepper level, similar to the modified_newton flag, seems to be the simpler approach because of the similarities with existing capabilities..
• We can start with HHT timestepper and once we have it tested and working, generalize it to all timesteppers.
• If this is Solver-agnostic, it could work for MINRES as well. If not we might start with Pardiso and later consider how to interface various solvers.

[rserban]

@jibril-b-coulibaly - yes, this feature is planned, but will be implemented in a more robust manner than what you suggest here.

[jibril-b-coulibaly]

@rserban

Thank you for a prompt response.

I should have brought more context to this issue.
This is a feature I have been discussing with @tasora @dariofusai93 @dariomangoni over the past month as it is a big bottleneck in some of the work we do.

We met today and decided making an issue and PR would be the best way to move forward in a collaborative way.

I am glad to hear you have ideas for implementing such feature that go beyond what we first outlined and are more robust. If you would like to work together on a permanent solution that would be great.

I understand you must be busy with other things, especially if the solution you have in mind would be a bigger lift. In the meantime, this is a high priority for some of my projects so I, @tasora @dariofusai93 @dariomangoni could work on a short term solution on a different fork in the meantime.

Either way, I think we could re-open this issue if this is a planned feature. That allows others developers to know it is planned. I could not find an open issue that addressed that so I had no way to know you were already working on it.

Please let me know how you would like to proceed.

[rserban]

Hi @jibril-b-coulibaly - Dario M. just informed me that this is something you have discussed with the Parma group. As you can see, there is miscommunication (or lack of communication) internally as well :-)

I understand this is a bottleneck for you, so you may want to implement something in your local clone of Chrono for now. But I'd rather not put this in Chrono like this, since I would prefer to have the more general solution.

If you are interested in helping with and working together on this, we can have a separate conversation on how to best proceed. But I will only be available to start discussing this in a couple of weeks.