When Gunicorn workers are set to more than 1, some workers will fail to load sqlalchemy.dialects:doris

[stgztsw]

Superset version 4.0.2
When Gunicorn workers are set to more than 1, some workers will fail to load sqlalchemy.dialects:doris, but anything is ok when work =1 .
When Gunicorn workers are set to more than 1, some chart can not be loaded whith "DB engine Error"

error log like this:
2025-09-26 01:50:19,170:ERROR:superset.views.base:Can't load plugin: sqlalchemy.dialects:doris
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/usr/local/lib/python3.9/site-packages/flask_appbuilder/security/decorators.py", line 95, in wraps
return f(self, *args, **kwargs)
File "/app/superset/views/base_api.py", line 127, in wraps
raise ex
File "/app/superset/views/base_api.py", line 121, in wraps
duration, response = time_function(f, self, *args, **kwargs)
File "/app/superset/utils/core.py", line 1470, in time_function
response = func(*args, **kwargs)
File "/app/superset/utils/log.py", line 255, in wrapper
value = f(*args, **kwargs)
File "/app/superset/charts/data/api.py", line 262, in data
return self._get_data_response(
File "/app/superset/charts/data/api.py", line 418, in _get_data_response
result = command.run(force_cached=force_cached)
File "/app/superset/commands/chart/data/get_data_command.py", line 45, in run
payload = self._query_context.get_payload(
File "/app/superset/common/query_context.py", line 100, in get_payload
return self._processor.get_payload(cache_query_context, force_cached)
File "/app/superset/common/query_context_processor.py", line 647, in get_payload
query_results = [
File "/app/superset/common/query_context_processor.py", line 648, in
get_query_results(
File "/app/superset/common/query_actions.py", line 227, in get_query_results
return result_func(query_context, query_obj, force_cached)
File "/app/superset/common/query_actions.py", line 103, in _get_full
payload = query_context.get_df_payload(query_obj, force_cached=force_cached)
File "/app/superset/common/query_context.py", line 121, in get_df_payload
return self._processor.get_df_payload(
File "/app/superset/common/query_context_processor.py", line 128, in get_df_payload
cache_key = self.query_cache_key(query_obj)
File "/app/superset/common/query_context_processor.py", line 208, in query_cache_key
extra_cache_keys = datasource.get_extra_cache_keys(query_obj.to_dict())
File "/app/superset/connectors/sqla/models.py", line 2000, in get_extra_cache_keys
sqla_query = self.get_sqla_query(**query_obj)
File "/app/superset/models/helpers.py", line 1510, in get_sqla_query
template_processor = self.get_template_processor(**template_kwargs)
File "/app/superset/connectors/sqla/models.py", line 1414, in get_template_processor
return get_template_processor(table=self, database=self.database, **kwargs)
File "/app/superset/jinja_context.py", line 665, in get_template_processor
return template_processor(database=database, table=table, query=query, **kwargs)
File "/app/superset/jinja_context.py", line 466, in init
self.set_context(**kwargs)
File "/app/superset/jinja_context.py", line 509, in set_context
dialect=self._database.get_dialect(),
File "/app/superset/models/core.py", line 957, in get_dialect
return sqla_url.get_dialect()()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/url.py", line 679, in get_dialect
entrypoint = self._get_entrypoint()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/url.py", line 661, in _get_entrypoint
cls = registry.load(name)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 343, in load
raise exc.NoSuchModuleError(
sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:doris

[stgztsw]

@dosu
Hi dosu, can you help to resolve this problem?

[dosubot[bot]]

This error is usually caused by environment or packaging issues: each Gunicorn worker is a separate process and must have access to the Doris dialect plugin via the pydoris package. If some workers can't load sqlalchemy.dialects:doris, it's likely that pydoris isn't installed or registered correctly in the Python environment used by all workers, or the entry_point for the dialect is missing or broken. Superset relies on pydoris to register the dialect via Python packaging metadata, not on its own code—so this is typically an environment or packaging problem, not a Superset bug (source).

To troubleshoot:

1. In the same environment where Superset runs, check that the Doris dialect is registered:

   from importlib.metadata import entry_points
   eps = entry_points()
   dialects = eps.select(group='sqlalchemy.dialects') if hasattr(eps, 'select') else eps.get('sqlalchemy.dialects', [])
   print([f"{ep.name} = {ep.value}" for ep in dialects if 'doris' in ep.name])

   You should see something like doris = pydoris.sqlalchemy.doris:DorisDialect. If not, reinstall pydoris with pip install --force-reinstall pydoris and make sure you're using the same Python environment as Superset.

2. Make sure all Gunicorn workers use the same Python environment and have access to the same installed packages. If you're using Docker, virtualenv, or system Python, confirm that pydoris is installed in the environment where Gunicorn launches Superset.

3. If the dialect is registered but you still see errors, check your Doris SQLAlchemy URI. Superset expects a catalog in the URI; if it's missing, it defaults to internal, which can cause connection issues. Try a URI like doris://user:pass@host:port/internal.schema (source).

4. After making changes, restart all Gunicorn workers to ensure they pick up the correct environment and packages.

If you follow these steps, it should resolve the plugin loading error for all workers. If the problem persists, there may be a deeper packaging or environment issue.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[stgztsw]

This error is usually caused by environment or packaging issues: each Gunicorn worker is a separate process and must have access to the Doris dialect plugin via the pydoris package. If some workers can't load sqlalchemy.dialects:doris, it's likely that pydoris isn't installed or registered correctly in the Python environment used by all workers, or the entry_point for the dialect is missing or broken. Superset relies on pydoris to register the dialect via Python packaging metadata, not on its own code—so this is typically an environment or packaging problem, not a Superset bug (source).

To troubleshoot:

1. In the same environment where Superset runs, check that the Doris dialect is registered:

   from importlib.metadata import entry_points
   eps = entry_points()
   dialects = eps.select(group='sqlalchemy.dialects') if hasattr(eps, 'select') else eps.get('sqlalchemy.dialects', [])
   print([f"{ep.name} = {ep.value}" for ep in dialects if 'doris' in ep.name])

   You should see something like doris = pydoris.sqlalchemy.doris:DorisDialect. If not, reinstall pydoris with pip install --force-reinstall pydoris and make sure you're using the same Python environment as Superset.

2. Make sure all Gunicorn workers use the same Python environment and have access to the same installed packages. If you're using Docker, virtualenv, or system Python, confirm that pydoris is installed in the environment where Gunicorn launches Superset.

3. If the dialect is registered but you still see errors, check your Doris SQLAlchemy URI. Superset expects a catalog in the URI; if it's missing, it defaults to internal, which can cause connection issues. Try a URI like doris://user:pass@host:port/internal.schema (source).

4. After making changes, restart all Gunicorn workers to ensure they pick up the correct environment and packages.

If you follow these steps, it should resolve the plugin loading error for all workers. If the problem persists, there may be a deeper packaging or environment issue.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other    

Hi @dosu
Thanks for your replay, but your answer can't reslove my problem.
I tried the solution you proposed，and no problem in my enviroment. (When Gunicorn workers are set to 1, everything works normally. This also proves that the environment is problem-free)
I have an idea, but I'm not sure if it's correct.
Pydoris is a lazily loaded package. When Superset starts, Pydoris has not yet been loaded. At this point, Gunicorn begins forking processes. Therefore, the forked processes do not have Pydoris loaded. However, the main process loads the Pydoris library afterward. This results in a situation where the main process functions normally, while the forked processes encounter exceptions.

[dosubot[bot]]

Your reasoning is correct: if pydoris is lazily loaded and not imported before Gunicorn forks worker processes, the dialect plugin may not be registered in the child processes, causing the error you see with multiple workers.

The most reliable solution is to force the import of pydoris before Gunicorn forks any workers. You can do this by adding import pydoris at the very top of your superset/app.py file (right after the existing imports). This ensures the Doris dialect is registered in the master process and inherited by all workers, regardless of your deployment method (Docker or not). No changes to Gunicorn or entrypoint scripts are needed—just the import statement.

After making this change, restart Superset so all Gunicorn workers start with the correct environment. This should resolve the plugin loading error for all workers. If you still see issues, double-check your Doris SQLAlchemy URI to ensure the catalog is specified as required (e.g., doris://user:pass@host:port/internal.schema) source.

Let me know if you need help with the exact code change or if the problem persists after this!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}