ci: reduce build time

[zoli]

This started to reduce Appflowy desktop app build time but in the middle, It became more about enhancements on CI. I have some ideas about reducing user build time but first, it should be decided what is the best option overall.

Changes I made

Common

Flutter build

Building "lib_super_native_extension.so" static library which is used for "super_clipboard" dependency took noticeably time during the flutter build. In the new version, there is an option available for downloading a precompiled static library. I updated to new version and added "cargokit_options.yaml" file to make downloading precompiled libraries as default option.
Time-saving: this saves about 1:20

Code generation

First I removed generate_env script because it was doing the same as generate freezed script (am I right?).
Second in bash scripts, I made a flag available for not running flutter packages pub get each time. Only run it once before running "generate.sh". For "generate.cmd" I didn't know how to make a flag available so I just commented out the flutter packages pub get.
Time-saving: this saves about 1:50

Flutter CI

The changes I'll mention made it possible to merge the whole flutter ci into one workflow. And run flutter build, unit tests, integration tests job parallel.

Cache rust build and code generation

Created an artifact from libdart_ffi and code generations so to eliminate the need for running "appflowy-core-dev" and code generation task for each of flutter unit tests and integration tests.
I added "dart_unit_test_no_build" task to make it possible just copy the built dart ffi static library to ".sandbox" directory which is used for unit tests.
Note, I couldn't do this for mac os unit tests because somehow it wasn't possible to make macos use ffi ".a" file it had to be ".dylib" file. So for macos unit tests, cargo make runs two times.
Also added "appflowy-make-product-dev" task to enable running the flutter build part separate from the "appflowy-core-dev" and code generation task.

Use prebuilt dockscript binary

Used "taiki-e/install-action" github action to install duckscipt_cli and avoid compiling it each time.
Time-saving: this saves about 2:20 for each job.

Docker CI

Use precompiled rocksdb, zstd static libraries

I will discuss this in more details later. But as Appflowy dockerfile is based on Arch Linux image. I changed it so it wouldn't compile librocksdb, libzstd. Couldn't make it to use ssl precompiled static library because we directly ask it to use vendored one as a cargo feature.
Time-saving: docker ci workflow is about 19 minutes faster now

What are other options to reduce build time, especially for first-time users?

Building from scratch still takes a lot of time. Just take a look Docker CI workflow it takes about 43 minutes on average and 35 minutes of it is "appflowy-linux" task. I think the dev version even takes a bit more.
Here is how much time each of "appflowy-linux" sub tasks takes (these are before my changes):

1. 24 minutes "appflowy-core-release" which is cargo build
2. 7.5 minutes "code_generation"
3. 3 minutes flutter build

Cargo build

If you run cargo build with timing flag (cargo build —timings) you will see that about 65 percent (14-15 minutes) of the 24 minutes is spend on building librocksdb, libzstd, libssl. Providing them as pre-compiled libraries in the repo or in a web server and downloading them would do the job. Or at least putting it in documentation like here which has done it. They are available as package in Arch Linux and probably Arch based distros. But the version on Ubuntu packages doesn't match what Appflowy requires (Openssl 3.1.2, ZSTD 1.5.5, Rocksdb 8.1.1). I don't know how is the case for Windows and MacOS.
Nathan suggested creating a precompiled library for collab-* which also Rocksdb and ZSTD are from there. Cargo doesn't support precompiled dependencies as far as I know (look here). I think the only available option for this is creating a static library and maintaining some c code to define specified functionalities.

Code generation

Removing generate_env script and redundant flutter packages pub get saved about 2 minutes. But I suggest another option for better performance.
Why not commit generated freezed files? Also ".g.dart" files. This saves another 3-4 minutes. And in CI we can check if there is a not committed change in generated files. We just need to run "code_generation" task and check it with git status that the code does not get dirty. So if the CI passes we will be sure the generated files are rightly generated from the latest changes.

Flutter build

I don't see any other enhancement than I made for using the precompiled library for super_clipboard dependency which reduced the time by 1:20.

PR Checklist

• My code adheres to the AppFlowy Style Guide
• I've listed at least one issue that this PR fixes in the description above.
• I've added a test(s) to validate changes in this PR, or this PR only contains semantic changes.
• All existing tests are passing.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (8e5b6b6) 64.64% compared to head (3c1d244) 64.99%.
Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3519      +/-   ##
==========================================
+ Coverage   64.64%   64.99%   +0.35%     
==========================================
  Files         542      568      +26     
  Lines       25171    25995     +824     
==========================================
+ Hits        16271    16896     +625     
- Misses       8900     9099     +199     

Flag	Coverage Δ
appflowy_flutter_integrateion_test	63.01% <ø> (-1.63%)	⬇️
appflowy_flutter_unit_test	11.71% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

see 76 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[zoli]

Just wanted to note this is not review-ready yet. I didn't make it as a draft because I wanted to see how it affects github actions.
It is an attempt to reduce build time for developers and the overall CI pipeline time in github actions.

[zoli]

@LucasXu0, @richardshiue Kindly wanted to ask for a review on this and your opinion on the options I suggested for lowering user build time.

[richardshiue]

Wow, very impressive

[LucasXu0]

@zoli Really appreciate that. I retriggered the CI jobs to check the reduced time.

[richardshiue]

Note, I couldn't do this for mac os unit tests because somehow it wasn't possible to make macos use ffi ".a" file it had to be ".dylib" file. So for macos unit tests, cargo make runs two times.

Oh.. that's a bottleneck then

[rileyhawk1417]

I wonder would this improvement help with local builds or its just mainly for the CI?

[zoli]

I wonder would this improvement help with local builds or its just mainly for the CI?

Not that much on the local build only 3 minutes. Which are saved because of code generation enhancement and using a prebuilt super clipboard library. I've suggested some options for reducing local build in the PR.

[zoli]

Note, I couldn't do this for mac os unit tests because somehow it wasn't possible to make macos use ffi ".a" file it had to be ".dylib" file. So for macos unit tests, cargo make runs two times.

Oh.. that's a bottleneck then

Yes, Its possible to separate it in another workflow. But I guess maybe there is a way to use .a library without problem I think on running appflowy binary Mac OS uses .a library not the dylib. I don't use Mac OS and haven't tried to check it on sort of a VM yet.

[zoli]

@zoli Really appreciate that. I retriggered the CI jobs to check the reduced time.

The Flutter-CI workflow is worse and that's because of caching I think. So it's hard to make any comparison. The prepare job in my fork repo didn't take more than 20m on average but here it took 47m and the difference is in "appflowy-core-dev" task (cargo build) which is affected by rust cache.
The Docker-CI workflow is good for making comparisons because it doesn't use any sort of cache. It takes 24m on this branch but the average on main is 40m.

[zoli]

Also, any comments on the options I suggested that enhance local build time (committing freezer files, providing precompiled rocksdb, ...)?

[LucasXu0]

Also, any comments on the options I suggested that enhance local build time (committing freezer files, providing precompiled rocksdb, ...)?

I agree with the second option, which involves providing a precompiled RocksDB. And the frozen files may sometimes conflict because we're using different Flutter versions(different linter).

[LucasXu0]

Before
integration test, linux, total 38m3s

After
integration test, linux, prepare 13m4s, test 26m29s, total 39m33s

Actually, for the longest time-consuming tasks, this PR doesn't optimize them; it only reduces the duplicated steps. @zoli, right?

[zoli]

Actually, for the longest time-consuming tasks, this PR doesn't optimize them; it only reduces the duplicated steps. @zoli, right?

In theory, I expect 3-4 minutes of saving on integration tests (because of removing generate_env script and precompiled super_clipboard static library). But as far as I see it's not showing itself. The comparison should be on average of multiple workflows. A commit that changes the rust code versus the dart code can have a notable difference in workflow time because of the rust cache.
Overall the flutter_ci is more about integrating all in a single workflow and removing repetitive tasks.

I agree with the second option, which involves providing a precompiled RocksDB.

I will work on this.

Also, any IDEA how the flutter integration tests can get faster?

[LucasXu0]

Also, any IDEA how the flutter integration tests can get faster?

Not yet.

@zoli I believe it would be beneficial to segregate the tests across different platforms. Although the unit tests, integration tests, and build tasks all depend on the 'prepare' task, the time required for the 'prepare' tasks varies across different platforms. Consequently, other tasks are held up by the longest time-consuming 'prepare' task.

For instance, in the 'prepare' task, Linux takes 10 minutes, macOS takes 15 minutes, and Windows takes 20 minutes. Consequently, the remaining tasks can only be executed after the Windows task has been completed.

As a solution, I suggest that the other macOS tasks should only be dependent on the completion of the macOS 'prepare' task, and likewise for other platforms.

[zoli]

As a solution, I suggest that the other macOS tasks should only be dependent on the completion of the macOS 'prepare' task, and likewise for other platforms.

@LucasXu0 I was wary of creating a separate workflow for each platform but your solution is better. Creating separate jobs (e.g. prepare-macos) for each OS and just depending on that in the specific OS next job (e.g. integration-test-macos). I will work on this too.

Also using a docker image with pre-installed flutter and rust will enhance Windows CI time by about 3-4 minutes.

[zoli]

Hi @LucasXu0. While working on providing static libraries, I was wondering if the flowy_codegen result (dart, protobuf, ts) also depends on user OS or STH like that?