Prefect as local process without server

[JaktensTid]

Hi, I want Prefect to be an in-app built orchestration workflow engine in my open source project and leverage its entire functionality (like saving io of tasks, saving logs, etc) (flows will be executed from my application code as well). Is this possible, or I still need to install prefect server?

[cicdw]

Hey @JaktensTid - as of right now you'll need to setup a server - note that servers can be run "ephemerally", meaning Prefect will automatically start the server in a subprocess so you don't need a longstanding API. Whether or not this is a good idea for your app I can't say. That being said, we are interested in supporting an execution mode that doesn't persist metadata to a server at all; we haven't planned this work yet but if others are interested we can consider picking it up sooner rather than later.

[rama-reddy]

Hi @cicdw,
I have a similar requirement: my use case is fully local (no distributed execution or external server) but I still need task dependency resolution and runtime context (logging, retries, caching). Since the ephemeral server adds time overhead, is there any recommended approach to handle this? I will be using a fork of Prefect as I need it urgently. Would patching PrefectClient be the way to go, or is there any other recommened approach to achieve this? Thanks.

[cicdw]

Hi @rama-reddy - orchestration fundamentally relies on state preservation so you're going to need the Prefect API somehow for all of that. The lowest latency choice is to use Prefect Cloud, the second lowest latency choice is to self-host the API backed by PostgreSQL.

[rama-reddy]

Hi @cicdw, thanks for clarifying. When you say “state preservation”, do you specifically mean persistence (e.g. saving flow/task states to a DB/API)?
I was thinking about creating a new client class that handles everything in-memory (hashmap or SQLite). The idea would be to implement the client functions against the in mem store, and since my use case is fully local I could safely skip things like deployments, work-queue, etc.
Do you see any major pitfalls with this approach?

[cicdw]

Yea, the biggest pitfall is that it's attempting to extend the system in a way that it isn't designed for, so my guess is that it's going to be a headache to build and maintain.

If you really don't care at all about any of the server-side state, I guess you could implement a client that does nothing at all and that would be easier than implementing a client with an arbitrary subset of server functionality that it only stores in memory.