Reuse Desktop Model Adapters in X-AnyLabeling-Server (Shared Model Layer)

[MrBean2016]

First of all, thank you for the great work on X-AnyLabeling and X-AnyLabeling-Server. The desktop client has an impressive and well-structured model zoo, while the server provides a clean and decoupled inference architecture.

While evaluating X-AnyLabeling-Server, I noticed that the number of supported models on the server side is currently much smaller than in the desktop client. From reading the code, it appears that each model must be implemented separately as a server-side class/adapter, even though very similar (or identical) model logic already exists in the desktop application.

This raises the question whether the existing model adapter structure in the desktop client could be reused by the server, instead of reimplementing the same logic twice.

Would it make sense to introduce a shared, UI-free model layer that can be used by both:
X-AnyLabeling
X-AnyLabeling-Server

For example:
Extract the existing model adapters (or their core logic) into a common Python package/module (e.g. xanylabeling_models)
Define an explicit, stable base interface (e.g. load(), predict()) for all model adapters
Let the desktop client focus on UI, interaction, and visualization, the server focus on API, authentication, concurrency, GPU/device management, and request/response handling

Reuse the same adapters for both local inference and remote inference

This could avoid duplication of model logic
Make it much easier to expose all desktop-supported models via the server
Improve long-term maintainability as new models are added

An alternative design, would it be feasible to move all model implementations into X-AnyLabeling-Server and let the desktop client always call the server, even for local installations (e.g. via localhost)?

In this setup, the server would be the single inference runtime, responsible for loading models, managing dependencies, GPU usage, and producing standardized outputs, while the desktop client would act as a thin UI layer for annotation and visualization.

This could eliminate duplication entirely and ensure that local and remote inference behave identically. The client could automatically start and manage a local server instance when needed, so the workflow would remain seamless for end users.

I’m curious whether this “server-first / server-only inference” approach has been considered, or if there are architectural or usability reasons why local in-process inference in the client is preferred.

[CVHub520]

Hey there! @MrBean2016,

Thanks for the thoughtful analysis and detailed proposal. This is a very reasonable question, and it touches one of the core architectural trade-offs in X-AnyLabeling.

To give some background: X-AnyLabeling did not start as a platform-oriented project. It was originally a personal productivity tool I built to support my own daily research and engineering work, mainly because I couldn’t find an existing tool that really fit my workflow. After using it for a while and seeing clear efficiency gains, I gradually refined it and decided to open source it.

As the project evolved and more users joined, feedback started to converge on a common issue: many users are working on laptops or limited local hardware, where running heavier models is either inefficient or not feasible at all. X-AnyLabeling-Server was introduced primarily to address this problem.

At the moment, the overall positioning is still client-first, server-optional.
The desktop client aims to be an out-of-the-box tool for individuals and small research / engineering teams, with fast iteration and easy experimentation with new models. The server acts as a complementary component, mainly for models with heavier compute or more complex environments (including some experimental or large-model-based features).

On the server side, I intentionally try to keep the codebase small and clean, providing relatively atomic inference capabilities. This also makes it easier for advanced users to extend or integrate additional models themselves. Based on current feedback, this split seems acceptable for most users.

Regarding the idea of a shared, UI-free model layer or a server-only inference architecture: from a pure engineering perspective, these are both clean and attractive designs, and I generally agree with their long-term benefits. However, there are also practical constraints. This is a personally driven open-source project, and doing a full abstraction and refactor of the model layer would require significant time and effort. Additionally, many users explicitly value a zero-configuration, “just install and use” experience, without needing to run or manage a local server or even a Python environment.

For now, some degree of duplication between client and server is a conscious trade-off in favor of usability and iteration speed, rather than an oversight. That said, your proposal is very valuable and aligns well with possible future directions if and when the project grows further.

Thanks again for raising this discussion — it’s exactly the kind of input that helps clarify long-term architectural decisions.

[MrBean2016]

Thanks for the detailed background on the project’s history and the reasoning behind the current architecture.

I see your point regarding the 'zero-configuration' experience. For many users, the ability to run an out-of-the-box desktop client without managing a Python environment or a local server is a significant advantage. I can appreciate how a 'server-only' approach might add friction for that specific use case.

While a unified model layer would offer long-term maintenance benefits, I totally understand that as a solo-driven project, your time is best spent on high-impact features rather than a massive structural refactor.

I'll keep using the current setup and may look into porting some of the client-side adapters to the server as needed. Thanks for the clarification and for the work on these tools.