darktable starts importing folder before lua scripts loaded

[jdchristensen]

Is there an existing issue for this?

• I checked and did not find my issue in the already reported ones

Describe the bug

I have enabled a lua script that adds a post-import-image hook. If I add a folder to darktable after it has already started, all images are processed by the script. But if a run darktable /path/to/folder, only some of the images are processed by the script (in one test, 52 out of 96 are processed). I suspect that darktable starts importing the images before the script has finished loading.

Steps to reproduce

1. I have attached an example script that simply counts the images it has loaded and prints the count and filename of each one. Start darktable and enable the script. Then exit darktable and run darktable -d lua /path/to/folder where the folder has around 100 images and see what the last count shown is.

Expected behavior

Darktable should wait until the lua scripts have all loaded before doing any imports.

Logfile | Screenshot | Screencast

Here is the script. Rename as .lua.
count_imports.txt

Commit

No response

Where did you obtain darktable from?

OBS build

darktable version

5.4.1

What OS are you using?

Linux

What is the version of your OS?

Ubuntu 24.04

Describe your system

32GB, nvidia

Are you using OpenCL GPU in darktable?

Yes

If yes, what is the GPU card and driver?

No response

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

No response

[jdchristensen]

Tagging @wpferguson as this came up during discussion of darktable-org/lua-scripts#645

[wpferguson]

Reproduced

darktable 5.5.0+816~g014e7b636b-dirty
Copyright (C) 2012-2026 Johannes Hanika and other contributors.

Compile options:
  Bit depth              -> 64 bit
  Exiv2                  -> 0.27.5
  Lensfun                -> 0.3.2
  Debug                  -> DISABLED
  SSE2 optimizations     -> ENABLED
  OpenMP                 -> ENABLED
  OpenCL                 -> ENABLED
  Lua                    -> ENABLED  - API version 9.6.0
  Colord                 -> ENABLED
  gPhoto2                -> ENABLED  - Camera tethering is available
  OSMGpsMap              -> ENABLED  - Map view is available
  GMIC                   -> ENABLED  - Compressed LUTs are supported
  GraphicsMagick         -> ENABLED
  ImageMagick            -> DISABLED
  libavif                -> ENABLED
  libheif                -> ENABLED
  libjxl                 -> DISABLED
  LibRaw                 -> ENABLED  - Version 0.22.0-Release
  OpenJPEG               -> ENABLED
  OpenEXR                -> ENABLED
  WebP                   -> ENABLED
  AI                     -> DISABLED

See https://www.darktable.org/resources/ for detailed documentation.
See https://github.com/darktable-org/darktable/issues/new/choose to report bugs.

[dt starting] as : /opt/dttest_20718_collections/bin/darktable -d lua -d pwstorage --cachedir /home/bill/.cache/dttest_20718_collections --configdir /home/bill/.config/dttest_20718_collections /home/bill/Pictures/dttest_20718_collections/Seniors/Aubrey/
     0.1532 [pwstorage_new] Creating new context 0x622f0c78b690
     0.1532 [pwstorage_new] using libsecret backend for username/password storage
importing directory
     1.7890 LUA WARN: callback: no matching branch found for API-9.6.0 
     1.8282 LUA ERROR: script_manager.lua: get_script_doc: 496: can't read from development/create_hdr 
     1.8283 LUA ERROR: script_manager.lua: get_script_metadata: 453: cant read from development/create_hdr 
     1.8303 LUA ERROR: script_manager.lua: get_script_doc: 496: can't read from development/file_tether 
     1.8304 LUA ERROR: script_manager.lua: get_script_metadata: 453: cant read from development/file_tether 
lua fully initialized
    30.2269 [pwstorage_new] Destroying context 0x622f0c78b690

I added a print statement to darktable.c where it called the directory import, right after the comment that said not to import until Lua is fully initialized.

I also added another print statement in lua/init.c when Lua says initialization is complete.

EDIT: Here is the section from darktable.c starting at line 2014.

    // load image(s) specified on cmdline.  this has to happen after
    // lua is initialized as image import can run lua code
    if(argc == 2 && !_is_directory(argv[1]))
    {
      // If only one image is listed, attempt to load it in darkroom
      fprintf(stderr, "importing image\n");
      dt_splash_screen_set_progress(_("importing image"));
      dt_load_from_string(argv[1], TRUE, NULL);
    }
    else if(argc >= 2)
    {
      // when multiple names are given, or we are given a directory,
      // fire up a background job to import them after switching to
      // lighttable showing the filmroll for the first one
      fprintf(stderr, "importing directory\n");
      _switch_to_new_filmroll(argv[1]);
      dt_control_add_job(DT_JOB_QUEUE_USER_BG, dt_pathlist_import_create(argc,argv));
    }

[wpferguson]

I did some digging last night. Lua gets started at line 1991. The problem is that there is no check to make sure Lua finished initializing before the image import starts at line 2014.

lua/init.c sets an _lua_fully_initialized static int when initialization is complete. We could add a getter function and poll waiting for initialization. Or we could detach it as a job and poll from there. Or we could add a signal?

My thought is poll. My concern is that it could slow startup a couple of seconds, though it would only happen if you're importing images.

@ralfbrown, @TurboGit, @jenshannoschwalm thoughts?

[ralfbrown]

Poll would be simplest to implement. The additional time would be needed when importing no matter what if there are any Lua scripts hooking the post-import action.

[wpferguson]

Poll would be simplest to implement

My thought too, but I didn't know if there was something that I wasn't aware of

[wpferguson]

This is fun 😲. Tried a couple of simple fixes and got nowhere fast. Then I started thinking and realized that there is somewhat of a circular dependency so I'm going to have to do some thinking...

I tried wrapping the image load in an if block and polled for Lua initialization complete but it wouldn't complete. Added print statements to the Lua initialization code and nothing was happening leading me to believe that the Lua initialization was happening in the same thread where I was waiting.

Wrote a quick script to count imported images. Ran it while loading an image straight into darkroom and observed the script not starting because _scripts_installer and script_manager wont start until we are in lighttable view.

So, I need to do some thinking....

[wpferguson]

Ok, figured it out 😄

Lua initializes before the import functions (single/folder) are called. There is nothing forcing a wait until Lua finishes initializing so the way it's implemented it's up to Lua to initialize itself before the image import happens.

This worked OK until we "improved" the Lua experience

• We added _scripts_installer to the system luarc file to make it easier for users to install the scripts
  • _scripts_installer does system checks for git, the user luarc file, the user lua directory
  • _scripts_installer checks the view and wont install until lighttable is active. It adds an event waiting on the view change to lighttable, then finishes installing, then installs when the event fires.
• We added script_manager to bypass the need to manually edit the luarc file.
  • script_manager does system checks for git.
  • script_manager does a check against the lua-scripts repository to see if an update is available
  • script_manager reads each script and extracts the metadata block
  • script_manager restarts the previously running scripts
  • script_manager checks the view and wont install until lighttable is active. It adds an event waiting on the view to change to lighttable, then finishes installing when the event fires.

So, if I comment out _scripts_installer, and require my count script from the user luarc file I get;

[dt starting] as : /opt/dttest_20718_collections/bin/darktable -d lua -d pwstorage --cachedir /home/bill/.cache/dttest_20718_collections --configdir /home/bill/.config/dttest_20718_collections ./Robert/_7D_4165.CR2
     0.1515 [pwstorage_new] Creating new context 0x5cb200690a60
     0.1515 [pwstorage_new] using libsecret backend for username/password storage
path set, going to run early
Lua omp_thread_num is 0
ran system luarc
     1.4936 LUA starting count
     1.4936 LUA started count and all events in place
     1.4974 LUA starting script_manager
number of threads is 24omp thread num is 0
importing image
loader omp_thread_num is 0
actually importing image
     1.5181 LUA count is 1
     2.2829 LUA ERROR: script_manager.lua: get_script_doc: 498: can't read from development/create_hdr 
     2.2829 LUA ERROR: script_manager.lua: get_script_metadata: 455: cant read from development/create_hdr 
     2.2855 LUA ERROR: script_manager.lua: get_script_doc: 498: can't read from development/file_tether 
     2.2856 LUA ERROR: script_manager.lua: get_script_metadata: 455: cant read from development/file_tether 
lua_initialized is 0
     2.5958 LUA script_manager waiting on lighttable view
ran user luarc
lua fully initialized
_lua_fully_initialized is set to 1

and if I import a folder

[dt starting] as : /opt/dttest_20718_collections/bin/darktable -d lua -d pwstorage --cachedir /home/bill/.cache/dttest_20718_collections --configdir /home/bill/.config/dttest_20718_collections ./Will
     0.1508 [pwstorage_new] Creating new context 0x5b9e02a6f720
     0.1508 [pwstorage_new] using libsecret backend for username/password storage
path set, going to run early
Lua omp_thread_num is 0
ran system luarc
     1.4666 LUA starting count
     1.4666 LUA started count and all events in place
     1.4690 LUA starting script_manager
number of threads is 24omp thread num is 0
importing directory
     1.4992 LUA count is 1
     1.5018 LUA count is 2
     1.5044 LUA count is 3
     1.5068 LUA count is 4
     1.5091 LUA count is 5
     1.5115 LUA count is 6
     1.5258 LUA count is 7
     1.5284 LUA count is 8
     1.5307 LUA count is 9
     1.5332 LUA count is 10
     1.5355 LUA count is 11
     1.5379 LUA count is 12
     1.5403 LUA count is 13
     1.5518 LUA count is 14
     1.5543 LUA count is 15
     1.5570 LUA count is 16
     1.5595 LUA count is 17
     1.5621 LUA count is 18
     1.5922 LUA ERROR: script_manager.lua: get_script_doc: 498: can't read from development/create_hdr 
     1.5923 LUA ERROR: script_manager.lua: get_script_metadata: 455: cant read from development/create_hdr 
     1.5949 LUA ERROR: script_manager.lua: get_script_doc: 498: can't read from development/file_tether 
     1.5949 LUA ERROR: script_manager.lua: get_script_metadata: 455: cant read from development/file_tether 
     1.6184 LUA count is 19
     1.6220 LUA count is 20
     1.6246 LUA count is 21
     1.6271 LUA count is 22
     1.6296 LUA count is 23
     1.6320 LUA count is 24
     1.6370 LUA count is 25
     1.6397 LUA count is 26
     1.6426 LUA count is 27
     1.6452 LUA count is 28
     1.6477 LUA count is 29
     1.6502 LUA count is 30
     1.6532 LUA count is 31
     1.6558 LUA count is 32
lua_initialized is 0
     1.6756 LUA count is 33
     1.6780 LUA count is 34
     1.6802 LUA count is 35
     1.6825 LUA count is 36
     1.6848 LUA count is 37
     1.7118 LUA script_manager installed
     1.7119 LUA count is 38
     1.7143 LUA count is 39
     1.7165 LUA count is 40
     1.7186 LUA count is 41
     1.7207 LUA count is 42
     1.7230 LUA count is 43
     1.7252 LUA count is 44
     1.7273 LUA count is 45
     1.7293 LUA count is 46
     1.7314 LUA count is 47
     1.7338 LUA count is 48
     1.7397 LUA count is 49
     1.7419 LUA count is 50
     1.7440 LUA count is 51
     1.7460 LUA count is 52
ran user luarc
lua fully initialized
_lua_fully_initialized is set to 1
     1.7481 LUA count is 53
     1.7502 LUA count is 54
Took out the boring line count...
     2.4147 LUA count is 300
     2.4166 LUA count is 301
     2.4185 LUA count is 302
     2.4204 LUA count is 303
     2.4223 LUA count is 304
     2.4243 LUA count is 305
     2.4263 LUA count is 306
     2.4673 LUA import count is 306

You'll notice there is not a lot of time between when count starts and the import starts

---

Fix

• In 5.6 _scripts_installer will go away and become a separate script in the tools section.
• In 5.6 the scripts will be bundled so script_manager should be faster with less system checks to perform
• This still might not be enough so I'm thinking of adding a script to the system luarc to capture import events and save the collection for any scripts that need it. Since it's in the system luarc file the results can be stored in global variables accessible to any script. We can use inter-script-communication to turn the script off after Lua has finished initializing and all the scripts have had a chance to get the imported images.

[jdchristensen]

@wpferguson Thanks for the hard work tracking this down!

Can you explain your "Fix" section a bit more? Right now, script_manager waits until the lighttable is ready before finishing initialization. Is that going to be changed in 5.6? If not, won't the problem persist?

Also, the idea of caching import events for later use seems fragile and awkward for user scripts to make use of. If the deadlock issue is fixed, will the busy wait idea work, delaying imports until the lua scripts are initialized? It would be nice to be 100% sure that the scripts are loaded before imports happen.

Is it possible that the part of the script initialization that involves adding the module to the UI could be delayed until the lighttable is ready, but that the scripts could still indicate to darktable that they are otherwise ready to handle imports?

So, if I comment out _scripts_installer, and require my count script from the user luarc file I get;

This sounds like it could be a temporary workaround. Did you mean that you comment out the single script_manager line in the user luarc file? Or all of the lines related to _scripts_installer in the system luarc?

[wpferguson]

Can you explain your "Fix" section a bit more?

If you look at the logs of the runs, you'll see ran system luarc, which was a print statement after the system luarc ran. I added

--[[
  images and folders can be specified for import when darktable launches.  Lua can't do a full
  initialization and start a script int the amount of time necessary to capture the import events 
  so we need a capture script here to capture the events and images imported.  They weill be stored in
  global variables available to any scripts that need them.
]]

darktable_imported_images = {}

and

_scripts_install.dt.register_event("capture_import_images", "post-import-image",
  function(event, image)
    table.insert(darktable_imported_images, image)
  end
)

_scripts_install.dt.register_event("capture_import_images", "post-import-film",
  function(event, image)
    _scripts_install.dt.print_log("darktable_imported_images contains " .. #darktable_imported_images .. " images")
  end
)

to the system luarc file which means it will run before the import of either image or directory and capture all the images.

---

The above code runs early enough to catch all the images.

Now, with this, I can do a compare between what a script running after script_manager started catches with what got caught on startup. If I have less, then I use the darktable_imported_images table which captured them all.

This isn't a final solution, just something I played with to figure out ways around it.

---

Or, you can edit the user luarc file and add the scripts that need to import on startup manually so they start before script_manager.

---

If the deadlock issue is fixed

It's not a deadlock, it's just timing. script_manager can't execute fast enough, at present, to get everything running before the import starts.

will the busy wait idea work, delaying imports until the lua scripts are initialized?

That would be the "correct" way to fix it so that we don't rely on timing (as I've already proved I can make that not work 😄). That also adds back the difficulty:hard label. It's a lot of work, a learning curve (I haven't played with threading in years and my darktable job knowledge is sketchy at best), and a lot of time that I don't have right now if I'm to complete the things I had scheduled for 5.6.

---

Is it possible that the part of the script initialization that involves adding the module to the UI could be delayed until the lighttable is ready, but that the scripts could still indicate to darktable that they are otherwise ready to handle imports?

Right now I don't start the scripts until script_manager is installed. That can change.

---

This sounds like it could be a temporary workaround. Did you mean that you comment out the single script_manager line in the user luarc file? Or all of the lines related to _scripts_installer in the system luarc?

I commented out all the _scripts_installer lines in the system luard, since it's going to go away in 5.6

I added a line before the require "tools/script_manager" to start the script that I needed to run for importing.

---

I guess I look at this as a problem I caused by adding lots of Lua code that runs early in an attempt to improve the Lua experience. I didn't use the import at run time, so I didn't realize the issue I created. For the present I'd prefer to solve it with Lua since I now understand it and later when I have more time provide a "bulletproof" solution. I'm already working with script_manager and the system luarc for the bundled scripts so I can just add this in to that work.