Infiltrator fails to hook into the REPL when the user-facing REPL package is different than its system counterpart

[serenity4]

With @topolarity and @KristofferC, we noticed a failure mode that prevents Infiltrator from correctly hooking into the REPL functionality.

Currently, standard libraries may be loaded twice in a running session: one used by Julia (Base and other stdlibs), and one used by packages, if both do not resolve to the same version. This is good because then a different stdlib version on the user side cannot break Base/stdlib code, but it brings its lot of issues here given how Infiltrator uses the REPL stdlib.

You can see it in action:

julia> println.(Base.loaded_modules_array());
Core
Base
Main
FileWatching
Libdl
Artifacts
SHA
Sockets
LinearAlgebra
OpenBLAS_jll
libblastrampoline_jll
Random
Base64
StyledStrings
JuliaSyntaxHighlighting
Markdown
InteractiveUtils
Unicode
REPL

julia> using REPL

julia> println.(Base.loaded_modules_array());
Core
Base
Main
FileWatching
Libdl
Artifacts
SHA
Sockets
LinearAlgebra
OpenBLAS_jll
libblastrampoline_jll
Random
Base64
StyledStrings
JuliaSyntaxHighlighting
Markdown
InteractiveUtils
Unicode
REPL
StyledStrings
JuliaSyntaxHighlighting
Base64
Markdown
InteractiveUtils
Unicode
REPL

julia> mods = Base.loaded_modules_array()[findall(x -> nameof(x) === :REPL, Base.loaded_modules_array())]
2-element Vector{Module}:
 REPL
 REPL

julia> pathof(mods[1])
"/home/serenity4/julia/master/stdlib/REPL/src/REPL.jl"

julia> pathof(mods[2])
"/home/serenity4/julia/master/stdlib/REPL/src/REPL.jl"

julia> mods[1] === mods[2]
false

Which, in this case, causes Infiltrator to fail:

julia> using Infiltrator

julia> @infiltrate
Infiltrating top-level frame

ERROR: MethodError: no method matching run_interface(::REPL.Terminals.TTYTerminal, ::REPL.LineEdit.ModalInterface)
The function `run_interface` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState)
   @ REPL ~/julia/master/stdlib/REPL/src/LineEdit.jl:2845
  run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface)
   @ REPL ~/julia/master/stdlib/REPL/src/LineEdit.jl:2845
  run_interface(::REPL.LineEdit.Prompt)
   @ REPL ~/julia/master/stdlib/REPL/src/LineEdit.jl:2830
   ...

julia> typeof(Base.active_repl.t)
REPL.Terminals.TTYTerminal

julia> ans === Infiltrator.REPL.Terminals.TTYTerminal
false

The main cause here is that Infiltrator depends on REPL which may be different from the REPL used to populate Base.active_repl.

To solve this, we must make sure that we use the correct REPL module, for example by removing it from package dependencies and using the internal version with

const REPL = Base.require_stdlib(PkgId(UUID(0x3fa0cd96_eef1_5676_8a61_b3b8758bbffb), "REPL"))

The problem is that precompilation explicitly forbids this: JuliaLang/julia#56233

A possible solution would be to disable precompilation, using

__precompile__(false)
module Infiltrator

But this would virally prevent precompilation for all packages depending on Infiltrator, which we'd probably want to avoid at all costs.

Another workaround would be to defer evaluation to runtime, with

# gigantic `eval` on all of the code that depends on `REPL`.
__init__() = eval(quote
      const REPL = Base.require_stdlib(PkgId(UUID(0x3fa0cd96_eef1_5676_8a61_b3b8758bbffb), "REPL"))
      # Use this REPL module to define new methods, types, etc.
      # ...
    end)

which, on the other hand, would be more likely to cause invalidations (and will prevent precompilation for Infiltrator itself).

Note that currently, Infiltrator still works in many cases, but when it fails due to this, it is not clear why nor what can be done about it. An alternative would therefore be to give up on hooking into the REPL when such a state is detected (two different REPL stdlibs loaded), however this case is likely to happen every time REPL is explicitly added to a manifest (which is outside our control here).