Move Lua scripting engine into a Valkey module

[rjd15372]

This PR restructures the Lua scripting functionality by extracting
it from the core Valkey server into a separate Valkey module. This change
enables the possibility of a backwards compatible Lua engine upgrade, as well
as, the flexibility in building Valkey without the Lua engine.

Important: from a user's point of view, there's no difference in using
the EVAL of FUNCTION/FCALL scripts. This PR is fully backward compatible
with respect to the public API.

The main code change is the move and adaptation of the Lua engine source
files from src/lua to src/modules/lua. The original Lua engine code is
adapted to use the module API to compile and execute scripts.

The main difference between the original code and the new, is the
serialization and deserialization of Valkey RESP values into, and from,
Lua values. While in the original implementation the parsing of RESP values
was done directly from the client buffer, in the new implementation the
parsing is done from the ValkeyModuleCallReply object and respective API.

The Makefile and CMake build systems were also updated to build and
integrate the new Lua engine module, within the Valkey server build
workflow.
When the Valkey server is built, the Lua engine module is also built,
and, the Lua module is loaded automatically by the server upon startup.
When running make install the Lua engine module is installed in the
default system library directory.
There's a new build option, called BUILD_LUA, that if set to no allows to
build Valkey server without building the Lua engine.

This modular architecture enables future development of additional Lua
engine modules with newer Lua versions that can be loaded alongside the
current engine, facilitating gradual migration paths for users.

Fixes: #1627

[rjd15372]

This PR is rebased on top of #2836 . When #2836 is merged, I'll update this PR.

[zuiderkwast]

Wow!

Some initial comments:

• For the build option, why not use the name BUILD_LUA (default yes), to align with the build options BUILD_TLS and BUILD_RDMA?
• For backward compatibility, the lua module needs to load by default. To disable this auto-loading, we need a new config, don't we? I can't see it mentioned here.
• Can the lua module be unloaded using MODULE UNLOAD?

[rjd15372]

• For the build option, why not use the name BUILD_LUA (default yes), to align with the build options BUILD_TLS and BUILD_RDMA?

Sure, that makes more sense. I'll update the PR.

• For backward compatibility, the lua module needs to load by default. To disable this auto-loading, we need a new config, don't we? I can't see it mentioned here.

I was thinking that if the lua module is built then it would always be loaded by default. But we can add a new config to disable auto-loading even if the lua module is built.

• Can the lua module be unloaded using MODULE UNLOAD?

Yes

[zuiderkwast]

I was thinking that if the lua module is built then it would always be loaded by default. But we can add a new config to disable auto-loading even if the lua module is built.

Yeah, we can discuss it with the core team.

I believe in many contexts, such as in pre-built containers, the module is already built. Many users don't build their own binaries.

[rjd15372]

I think it makes sense to have that config option to disable auto-load.

[rjd15372]

I think it makes sense to have that config option to disable auto-load.

@zuiderkwast I vote for implementing such option in a follow up PR. This PR is already huge to review.

[codecov]

Codecov Report

❌ Patch coverage is 81.03448% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.91%. Comparing base (05540af) to head (02586d0).
⚠️ Report is 9 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/module.c	74.28%	9 Missing ⚠️
src/scripting_engine.c	92.30%	1 Missing ⚠️
src/server.c	87.50%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #2858      +/-   ##
============================================
+ Coverage     72.41%   73.91%   +1.50%     
============================================
  Files           128      124       -4     
  Lines         70414    68723    -1691     
============================================
- Hits          50987    50796     -191     
+ Misses        19427    17927    -1500     

Files with missing lines	Coverage Δ
src/config.c	78.76% <ø> (+0.32%)	⬆️
src/eval.c	87.46% <100.00%> (ø)
src/module.h	0.00% <ø> (ø)
src/replication.c	85.81% <ø> (-0.12%)	⬇️
src/script.c	89.03% <ø> (+8.89%)	⬆️
src/scripting_engine.c	56.88% <92.30%> (+3.51%)	⬆️
src/server.c	88.72% <87.50%> (+0.26%)	⬆️
src/module.c	25.48% <74.28%> (+15.71%)	⬆️

... and 26 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[zuiderkwast]

Just some initial comments. I'll do another pass later. The change is not as huge as the +/- numbers indicate. Many lines are moved to other files.

The build options should be mentioned in the README.md.

[zuiderkwast]

This looks pretty solid in general.

My main concerns are about the build-time changes. Is there a risk that it will break compilation or installation for any user?

If Valkey is build with BUILD_LUA=no, then it doesn't automatically load a lua module even if such module is installed in the system later. Maybe it's not bad, but I just want to mention it.

@valkey-io/valkey-committers Does anyone else want to take a look, especially on the compile-time changes (Makefiles etc.)?

[rjd15372]

My main concerns are about the build-time changes. Is there a risk that it will break compilation or installation for any user?

I tried my best to not break it, but a another pair of eyes is needed to improve confidence on the changes.

If Valkey is build with BUILD_LUA=no, then it doesn't automatically load a lua module even if such module is installed in the system later. Maybe it's not bad, but I just want to mention it.

Right. But if later someone installs the module separately, it can add a loadmodule line to valkey.conf to load it automatically.