Preliminary work for Seafile 10.0

[ChatDeBlofeld]

Hi folks,

I've started the work to build Seafile 10. Compilation dependencies were straightforward to update (see ChatDeBlofeld/seafile-rpi@321057d) but python packages have been completely reworked upstream and I don't know how to merge them with the custom requirements we have here (which still have pending changes #120)

What I can tell so far:

• wsgidav requires markupsafe==2.0.1 #113 is still needed, no update here.
• Cryptography downgrade (Build script stops with building cryptography (and some more errors...) #99) obviously still needed for Buster.
• Golang minimal version is 1.17 since haiwen/seafile-server@df78d10, which is higher than ones provided in Debian Buster (1.11), Bullseye (1.15) and Ubuntu 20.04 (1.13). Honestly, I don't think it would be that bad not to build the fileserver when the go version is too old...

By disabling the go fileserver when needed, I was able to build something usable for:

• Debian Buster
• Debian Bullseye
• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 22.10

Tested features (22.04 only):

• Web UI (upload/download)
• Sync
• Webdav
• Memcached
• sqlite backend
• MariaDB backend
• MySQL backend

[lanmarc77]

Thank you for maintaining this here.
Not really understanding a lot of the seafile compiling process, might I throw into the ring the idea to build the go part on e.g. a docker based environment and then statically link the output of the compiler? Doesn't the go part generate binaries anyways? This way general usage binaries could be created which not even rely on a locally libc of a specific version or OS. Additionally it might also be easy to generate arch64 binaries.

[Duke690]

I simply upgraded the golang version for my raspberrypi (bullseye) with this tutorial https://www.e-tinkers.com/2019/06/better-way-to-install-golang-go-on-raspberry-pi/

I was then able to build seafile 10.0.1 with the build.sh script. I had to remove the line in build.sh where golang is installed via apt. I used the same requirements as for v9.0.9. for seafdav and the ones ChatDeBlofeld mentioned for seahub (https://github.com/haiwen/seahub/blob/master/requirements.txt)

I second lanmarc77's suggestion to use a docker based build environment.

[nihilistaX]

I was able to build seafile 10.0.1, golang is not a problem as you can install and activate every version with gvm.
So build runs fine, but after installing i got internal server error. Seems something went wrong with seahub.

[Duke690]

I was able to build seafile 10.0.1, golang is not a problem as you can install and activate every version with gvm. So build runs fine, but after installing i got internal server error. Seems something went wrong with seahub.

Where do you get that internal server error? I got an error starting seahub when using the v9.0.9 requirements to build. But after using the ones mentioned above my installation works flawless so far.

[nihilistaX]

Seahub starts without error, but web interface shows internal error. Used same requirements as above. And stopping seahub shows "failed to stop seahub", but it is stopped.....Maybe some problem with some requirement or somewhat.

[ChatDeBlofeld]

@nihilistaX Try to set deamon = False in gunicorn.conf.py. If exceptions arise in Seahub, stack trace will be displayed.

@lanmarc77 I won't work on such an integration but yes it certainly works. Universal binaries will be harder though since the server depends on libc (see below), I don't even know if it possible to static link it... However, I built a fileserver on buster (which has oldest versions of dependencies) which should work fine on later versions.

$ ldd fileserver 
        linux-vdso.so.1 (0x0000007f8b365000)
        libresolv.so.2 => /lib/aarch64-linux-gnu/libresolv.so.2 (0x0000007f8b30e000)
        libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007f8b2fa000)
        libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007f8b2c9000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f8b154000)
        /lib/ld-linux-aarch64.so.1 (0x0000007f8b335000)

---

I've added support for notification server (ChatDeBlofeld@23acec9), which is written in go too 🙃. Pretty ugly, backward compatibility is broken (but it was already the case when the go fileserver was added and it seemed ok at the time) and I patch the build-server.py script on the fly to copy the new server properly. I will open a PR (haiwen/seahub#5453) to try to improve the later.

[Gustl22]

@ChatDeBlofeld thanks for your work! I wrote the maintainers to merge your PR (but don't know, if they actually do). Also if you feel ready you can start a PR for v10. I also wrote you a message on LinkedIn.

[ChatDeBlofeld]

PR was merged, thanks!

I will start a PR for 10.0.1, maybe should we write down somewhere the minimal go requirements to build the server?

[ChatDeBlofeld]

Hi @lanmarc77 @Duke690 @nihilistaX

Static linking libc in go happened to be very easy, so I set up an experimental docker pipeline to cross-compile seafile from amd64. You'll find some compiled packages for seafile 10 here. If some of you want to test them it would be very helpful!

[lanmarc77]

Hi @ChatDeBlofeld , @Gustl22
thank you very much for your work. I integrated the docker pipeline with github actions. This way packages can be built online. You find the the finally complete build here: https://github.com/lanmarc77/seafile-rpi/actions/runs/5031423956/jobs/9024494684
I takes around 1hr per distribution and fits very well into the free 2000 github actions minutes per month.

I would like to add automated tests of these builds to get rid of the human more or less completely for the testing process.
So I am thinking of starting a container installing the build package in it probably with different database backends and then test if all the processes start and keep running.
Are there more sophisticated tests needed?

And how about renaming seafile-rpi into seafile-arm?
All the best and thank you again.

[lanmarc77]

Small Update, went down the rabbit hole kinda...

I am now using the newest Go version 1.20.4 which is downloaded and installed inside the build container during creation. This allows compiling also for older distributions directly in the container type it is meant to run later.
Except for focal arm/v7 everything can be build and compiled using Github actions. Focal arm/v7 I think is a bug inside Github actions itself as it does compile locally just not online with the same script.

I am currently compiling only with the requirement that come with the original repos. Are there any reasons beside maybe historical ones that there are version specific requirements which you apply ahead? It did compile without them, also for tests I did with 9.0.9 and 8.0.7.

I released 10.0.1 for all major distributions for everyone who wants to test: https://github.com/lanmarc77/seafile-arm/releases

No tests have been conducted so far.

[Gustl22]

Wow @ChatDeBlofeld @lanmarc77 that looks very promising, if you feel comfortable, you can start a Merge Request, if you want, or even start it as Draft, if you don't feel ready to merge yet ;D

[lanmarc77]

Hi @Gustl22 , I am unsure before I could implement automated tests.
Can you help with some of my questions?

[nihilistaX]

Hi, tested bullseye aarch64 and armv7l builds, as upgrade and as fresh install, but seahub won't start.
That's what i got on fresh aarch64 install:
Starting seahub at port 8000 ... Traceback (most recent call last): File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/bin/gunicorn", line 8, in <module> sys.exit(run()) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/wsgiapp.py", line 67, in run WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/base.py", line 231, in run super().run() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/base.py", line 72, in run Arbiter(self).run() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/arbiter.py", line 58, in __init__ self.setup(app) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/arbiter.py", line 118, in setup self.app.wsgi() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/base.py", line 67, in wsgi self.callable = self.load() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/wsgiapp.py", line 58, in load return self.load_wsgiapp() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp return util.import_app(self.app_uri) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/util.py", line 359, in import_app mod = importlib.import_module(module) File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 790, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/opt/seafile/seafile-server-10.0.1/seahub/seahub/wsgi.py", line 25, in <module> application = get_wsgi_application() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/django/core/wsgi.py", line 12, in get_wsgi_application django.setup(set_prefix=False) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/django/__init__.py", line 24, in setup apps.populate(settings.INSTALLED_APPS) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/django/apps/registry.py", line 122, in populate app_config.ready() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/statici18n/apps.py", line 8, in ready from . import conf # noqa File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/statici18n/conf.py", line 3, in <module> from appconf import AppConf ImportError: cannot import name 'AppConf' from 'appconf' (/usr/local/lib/python3.9/dist-packages/appconf/__init__.py) Error:Seahub failed to start.

[abutre]

Hi, tested bullseye aarch64 and armv7l builds, as upgrade and as fresh install, but seahub won't start. That's what i got on fresh aarch64 install: Starting seahub at port 8000 ... Traceback (most recent call last): File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/bin/gunicorn", line 8, in <module> sys.exit(run()) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/wsgiapp.py", line 67, in run WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/base.py", line 231, in run super().run() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/base.py", line 72, in run Arbiter(self).run() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/arbiter.py", line 58, in __init__ self.setup(app) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/arbiter.py", line 118, in setup self.app.wsgi() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/base.py", line 67, in wsgi self.callable = self.load() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/wsgiapp.py", line 58, in load return self.load_wsgiapp() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp return util.import_app(self.app_uri) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/util.py", line 359, in import_app mod = importlib.import_module(module) File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 790, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/opt/seafile/seafile-server-10.0.1/seahub/seahub/wsgi.py", line 25, in <module> application = get_wsgi_application() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/django/core/wsgi.py", line 12, in get_wsgi_application django.setup(set_prefix=False) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/django/__init__.py", line 24, in setup apps.populate(settings.INSTALLED_APPS) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/django/apps/registry.py", line 122, in populate app_config.ready() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/statici18n/apps.py", line 8, in ready from . import conf # noqa File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/statici18n/conf.py", line 3, in <module> from appconf import AppConf ImportError: cannot import name 'AppConf' from 'appconf' (/usr/local/lib/python3.9/dist-packages/appconf/__init__.py) Error:Seahub failed to start.

Solved for me in Ubuntu by installing python3-django-appconf