Is there any way to use duckdb ui in offline environment?

[coloraven]

The local HTTP server fetches the files for the UI from a remote HTTP server(https://ui.duckdb.org/) so they can be kept up-to-date.
But I need to use it in an offline environment.
Is there any way to achieve this?

[Y--]

Hi @coloraven this is not possible at the moment, but something we are thinking about adding. Stay tuned :-)

[mlafeldt]

There are a few ways to achieve offline mode right now. One of them involves traffic capture/replay. Here's my short guide on how to use the DuckDB UI without internet access.

Install mitmproxy.

Start mitmdump to capture network requests to ui.duckdb.org:

mitmdump --mode reverse:https://ui.duckdb.org -w hatchling.dump

Start the DuckDB UI in a different terminal:

ui_remote_url=http://localhost:8080 duckdb -ui -unsigned

Click around the UI to make sure that all assets are loaded.

Press Ctrl+C to stop mitmdump. The resulting hatchling.dump file should be ~10M in size.

Once captured, you can replay the traffic forever, even offline:

mitmdump --mode reverse:https://ui.duckdb.org \
    --set server_replay=hatchling.dump \
    --set server_replay_reuse=true \
    --set connection_strategy=lazy

Just remember to always start the DuckDB UI like this:

ui_remote_url=http://localhost:8080 duckdb -ui -unsigned

(-unsigned is required for ui_remote_url to work)

[jraymakers]

Thanks for this! We are researching ways to accomplish this more easily & robustly, but this may be a reasonable workaround in the meantime.

Please note that, when using the above, some features may not work properly, and you will not get updates & fixes unless/until you refresh your capture.

[nojetlag]

There are a few ways to achieve offline mode right now. One of them involves traffic capture/replay. Here's my short guide on how to use the DuckDB UI without internet access.

Install mitmproxy.

Start mitmdump to capture network requests to ui.duckdb.org:

mitmdump --mode reverse:https://ui.duckdb.org -w hatchling.dump

Start the DuckDB UI in a different terminal:

ui_remote_url=http://localhost:8080 duckdb -ui -unsigned

Click around the UI to make sure that all assets are loaded.

Press Ctrl+C to stop mitmdump. The resulting hatchling.dump file should be ~10M in size.

Once captured, you can replay the traffic forever, even offline:

mitmdump --mode reverse:https://ui.duckdb.org \
    --set server_replay=hatchling.dump \
    --set server_replay_reuse=true \
    --set connection_strategy=lazy

Just remember to always start the DuckDB UI like this:

ui_remote_url=http://localhost:8080 duckdb -ui -unsigned

(-unsigned is required for ui_remote_url to work)

That might be an option in a small shop or in a private environment, but in a corp environment, especially in the finance sector, well...

[jraymakers]

That might be an option in a small shop or in a private environment, but in a corp environment, especially in the finance sector, well...

Since we're considering the best way to support this feature, which aspects of the above approach do you think would be a problem for a corporate environment? @nojetlag

[nojetlag]

The best in my environment is a single download that contains all required files, resources etc. So i can go to an isolated download station, get the package, bring it inside, have it approved via security checks and then install it in an isolated environment that has no internet access at all.

duckdb -ui would be used in a development environment to work with data and part of duckdb is also used in a python context in testing, integration and production environments, which are strictly separated and also don't have any direct internet access.

I hope i can give you an idea. For me it is always surprising to hear that nowadays there still seam to be large corps where people can access internet from any machine. At home, sure, but at work.

[jraymakers]

Thanks for the context! There are indeed many different environments with varying constraints. The one you describe - where the machine is never connected to the Internet - is a particularly challenging one.

[nojetlag]

I think even if you find scenarios where a machine has internet access often they will have browser isolation nowadays. Totally agree that makes things very complicated but that is today's work environments.

[jraymakers]

I'm curious if you've actually tried using duckdb in such a locked-down environment (without the UI). DuckDB downloads extensions from the internet; I'm curious if you've encountered that and whether you've worked around it somehow.

[im5h]

I'm curious if you've actually tried using duckdb in such a locked-down environment (without the UI). DuckDB downloads extensions from the internet; I'm curious if you've encountered that and whether you've worked around it somehow.

I have tried it on an offline machine.

1. First, manually download the extension file. eg:http://extensions.duckdb.org/v1.2.1/linux_arm64/excel.duckdb_extension.gz

2. Copy the extension file to the offline machine and store it in the following directory structure.

    extensions
    └── v1.2.1
        └── linux_arm64
            ├── excel.duckdb_extension.gz
            └── ui.duckdb_extension.gz

3. Modify the host file on the offline machine, "127.0.0.1 extensions.duckdb.org".

4. Use Python to start a web service. python -m http.server 80 at extensions folder.

Then, duckdb can auto download extensions.
This works, but it's cumbersome.

[mlafeldt]

@im5h Just a quick tip: you may not need a local extension repo as INSTALL works with local files too - see https://duckdb.org/docs/stable/extensions/advanced_installation_methods

[daviewales]

It seems strange to me that the UI is not included within the extension itself. Perhaps because it's not open source yet? The docs state:

The local HTTP server fetches the files for the UI from a remote HTTP server so they can be kept up-to-date.

But I'd be quite happy if they just updated when I update my local copy of the extension with FORCE INSTALL ui;.

[nojetlag]

I'm curious if you've actually tried using duckdb in such a locked-down environment (without the UI). DuckDB downloads extensions from the internet; I'm curious if you've encountered that and whether you've worked around it somehow.

I use it with python, getting python packages is more or less easy with an internal mirror repo (yes sometimes packages get blocked due to know vulnerabilities, but in most cases such an approach works). However everything from duckdb that doesn't come with the python package is a challenge as described above.

[jraymakers]

It is challenging to make the UI work in such locked-down environments, given both the architecture of DuckDB (which downloads extensions dynamically) and the UI itself (which was originally built for the web, and later adapted for this local use case). But it's useful to hear about these constraints. We do plan to support offline mode, and we'll try to support as many environments as possible - but we can't promise we'll be able to make them all work well.

[jraymakers]

It seems strange to me that the UI is not included within the extension itself.

There are several reasons for this:

• We want the UI extension to remain general enough that it could be used to support other UIs.
• The UI is likely to be updated far more frequently than the extension.
• The deployment of the extension currently involves manual steps, so the overhead of updating it for each new UI version is high. (New UI versions are published publicly roughly weekly, but much more often internally as we develop and test it.)
• Separating the extension from the UI is useful for local development and testing purposes; it allows running local and internal builds of the UI without rebuilding the extension (which is relatively slow).
• Embedding files in the extension isn't trivial, because it's (necessarily) a C++ binary. There are technique for doing this, but they all have tradeoffs.

Of course, it would be possible to accomplish many of the above while still including a (perhaps outdated) copy of the UI in the extension. But this doesn't seem like a good cost/benefit tradeoff, given the above.

Our current thinking is to support offline mode by saving (to disk) the UI assets that are fetched by the extension. So you'd have to be online for the initial download, just as you have to be online to download the extension itself initially. But, once downloaded, the files (both the extension and the UI assets) could be moved to another environment if desired (e.g. by copying the .duckdb directory).

(Since the UI was originally built for the web, it depends on some dynamic endpoints that are not simple files. So, making it run using a set of downloaded files is not as easy as it might sound. It's possible, but it'll take some work. That's why this isn't supported yet.)