investigate moving remaining global data to libctx for atexit mitigation

[nhorman]

Based on our discussion today, we discussed the possibility of moving remaining global heap data into the libctx structure. Doing should limit our potential memory leakage to the single default libctx. As we offer guidance to our users that libraries should always use self-allocated libctx structures (and free them on completion), this should significantly limit our memory leakage, which can then be cleaned up via an explicit call to OPENSSL_cleanup within the application.

This task is to investigate the possibility of migrating our remaining global heap data to be within the libctx structure.

Output should be a report here as to the feasibility of this approach, and is timeboxed to 7 days.

[t8m]

I think you will see all sort of breakage from this. Besides the affected APIs do not have libctx argument so you would either have to always act upon the thread local default libctx. And you will need at least to keep the global default library context as a global data unless you want to break all apps that are not using library contexts explicitly everywhere.

[esyr]

That's the current plan: leave the default libctx (with its application lifetime) to applications, and provide a way to work without any global allocations for libraries, which will also require adding interface variants that allow providing libctx.

[t8m]

There are for example the memory allocation callbacks. They are currently global. Do we really want these to be per-libctx? How would that even work when the libctx by itself needs to be allocated.

So no, I do not think we should aim for absolutely no global data. We need to list all the current global data and think through if the particular thing is really something that should be in libctx or whether we can either say that libraries cannot use the related APIs (that might be suitable for things like the memory allocation callbacks) or that having just one global database in a process is actually OK (that might or might not be true for the OBJECT db depending on how we make the APIs behave).

In the OBJECT db example, I do not see how having different object dbs in different libctxes would be actually beneficial. We just need to ensure that adding objects is thread-safe and idempotent, i.e. you can add an object that is already there, it would be just a no-op.

Some of the global data might probably need to be just deprecated. I.e., string table configuration.

[esyr]

For the purposes of atexit removal, I was talking about global allocations (to essentially prevent any memory leaks by libraries that use explicitly allocated libctx), not global data. Moving all of the global state inside libctx is an interesting experiment, but I agree that it is superfluous and likely unnecessary, and you're right, we will probably end up with a list of APIs libraries are not supposed to touch as well.

[mbroz]

I think it is worth an experiment - as I provided that libcryptsetup library where we can easily check global allocations in valgrind.

At least it should be documented ("what to not expect from libctx").
For example, I would kind of expect the memory callback to be per libctx - not saying I need it, just the idea someone overengineer application/library to do own memory management is not so rare as it should be :-)