How to implement deferred dependency loading in bevy

[apriori]

Hello, first thank you for this great crate.

I am currently using your project to implement scripting support in a project using bevy. However, I got the following issue:

• Scripts depend on each other in an unknown (or lets say cumbersome to maintain) way
• All scripts are loaded buy an AssetIO override that maps the path to a path into a zip file

So in all loading, the asset server is involved - and that would include loading dependencies.
Now I got the the issue that e.g. a lua snippet wants to run

dofile '/some/path/to/a/script.lua'

and would require to be paused, until the resource is available and the string contents are injected. The lua engine used (Luajit) claims to be fully resumable, so it should be possible to yield across the language barriers, however that exactly is not working with the error:

attempt to yield across C-call boundary

Is there another approach I should take when lua wants to load a resource that involves async loading from the asset server?

[makspll]

Hi @apriori! Thanks for opening this issue!
As for why that's not working, I am not entirely sure, I would start with creating an issue/question in the mlua repo, It looks like this should work. For context are you using the unsafe_lua_modules feature? It may be worth trying that out as well, although I don't think that's the problem since it seems you have the dofile function loaded in. It seems to me that the root of the issue may stem from there.

As for alternative approaches, you can always define an API for running scripts, for example run_script('script.lua') or run_script_with_env('script.lua', _ENV) (maybe you can pass the environment down with https://docs.rs/mlua/latest/mlua/struct.Chunk.html#method.set_environment) which could take care of setting up the script context as you would like to. This is of course a more involved solution.

@khvzak

[apriori]

Hello. Thanks for your swift response.

The context is of course a lot more involved. My current approach loads a an entire script without executing it in a coroutine that is stored in a known global member.
What I would like to do now have is to have a globally available override function for "dofile etc." that will suspend the lua vm execution until the respective file has been loaded by the asset loader and its value/content is available for injection.

So far I came up with the following solution:

• Replace LuaScripting host with a variant that does the following:
  • Attach APIs before executing the script snippet
  • Attach the following snippet before script loading:

                __active_async_func = {}
                function doscript(arg)
                    __active_async_func['doscript'] = arg
                    coroutine.yield()
                end

  • Pack the script snippet in a coroutine and assign to e.g. __mod_coroutine
  • Resume that coroutine immediately

After that an exclusive system can look in the global variable __mod_courtine and the __active_async_func for asset loading dispatch and only call resume after the script has been injected via mlua::Lua::load.

While this will work, it is quite involved. So far this solution is only working for depth 1, so I guess I need to create a stack of coroutines on the lua side and always drive the stack head further.

The thing is, I cannot really change the available lua code too much, because I am loading game assets of an old game in an unchanged way.

Do you have async functions for the API provider on the agenda?

[makspll]

Hmm, thanks for the detailed explanation! I am currently prioritizing completing language and Bevy API support, but I am happy to take on PR's! Did you try the mlua_async feature yet? To help me implement a good solution from your perspective, how would you like to see async support for APIProviders implemented ?

[apriori]

Honestly, I can't tell yet, how to do this in an acceptable way. Bevy ergonomics around async is a bit lacking. However, I was able to so solve my usecase using your proposed mlua_async. It is ... quite verbose. Not sure how to generalize this to more general async functions that require some kind of world access.

You will get the general idea looking at this snippet (not a fully runnable example. if you want one, I could create it)
https://gist.github.com/apriori/08c7c9e909dd4e9701cd9b460ad23763

Of course one could completely bypass all the mumbo jumbo involving PriorityEvent by directly refering to a world AssetIO and call its async functions directly, however I did not want to do that. At least using mlua_async prevents having to pretty much suspend and resume using coroutine magic on both the rust and lua end. It is completely done by mlua.

[makspll]

I see! Glad to see you got something working, the gist is much appreciated, it's always good to see how the consumers of this crate use it in order to improve it! I will look into this, can't guarantee when but hopefully should get something into next release that will help with your use case!

I noticed a new reflect type data ReflectAsset is present on AssetIO so it may well be possible to directly expose that in the Bevy API

[khvzak]

The lua engine used (Luajit) claims to be fully resumable, so it should be possible to yield across the language barriers

The "fully resumable" term in this context relates to language features, such as (x)pcall/metamethods/iterators/etc (which lua 5.1 does not support). Unfortuantely LuaJIT cannot yield across C boundaries (this is not part of the VM) and generates the error you see.

The mlua's Rust async integration seems the best way to achieve this. It allows to yield a coroutine and then resume it based on an external event (channel message / networking call / etc). You just need an executor to do this (tokio/async-std/etc).

[makspll]

Ah! Thanks @khvzak, that's very good to know!