Irmin as a CMS Database

[eWert-Online]

Hello 👋

I am working on a CMS and I am trying to use irmin as the database. I have a few questions about it. 😄 I am putting each question as a comment in this discussion, so further discussion can happen in threads. I hope this is ok.

[eWert-Online]

Is irmin a good choice for potentially large datasets. I am specifically asking, because there seems to not be a concept of "paging" in irmin (e.g. getting only the first 100 nodes of a path).

[art-w]

Yes definitively, the paging interface is there to support arbitrarily large datasets. The irmin-pack backend is specifically optimized in that regard, using a probabilistic B-tree representation internally for large collections and efficient paging (on the other end, irmin-fs relies on the underlying filesystem so it can run into issues since standard filesystems generally dislike large directories). This is for example how Octez stores the Tezos blockchain, with millions of files and multiple hundred of gigabytes of data on disk.

[eWert-Online]

I am currently using Json_value contents, as I have differently shaped data structures under different paths. I read somewhere, that there was some work using gadts to make working with different data types in the same store possible. Is that still being worked on or should I not get my hopes up of something like that coming in the future?

[art-w]

I don't think we are planning to work on this in the near future, but we would welcome a PR and we would be happy to provide guidance if you are interested :)
In the mean time with the current API, the design pattern to get the same guarantees is to define a sum type with all the possible shapes, accompanied by (your) domain-specific functions to wrap/unwrap your typed values in the right constructor to ensure that different paths store the expected types.

[eWert-Online]

I want to have different "workspaces" in the CMS. The main workspace is the content thats live. Other workspaces can be created to make changes in a "draft" stage, which can later be published to live. Irmin branches seem to be a perfect fit for that. I am wondering though, if it is also possible to merge only a subset of commits or even a subset of changes from a commit. Also: Other branches would have to be rebased, as soon as one is merged into main. Is there a concept for that in irmin or are long lived branches not really a thing?

[art-w]

Yes branches sound perfect for this! They can be as short or long lived as you want (just like git): a cold inactive branch only consumes disk space but doesn't impact the performances of the database otherwise (and writing/reading from a branch is the same as operating on the main branch; also they share the same storage for any identical file contents, so it's very cheap to branch).

In order to merge only a subset of the commits/changes, you will need some custom logic to traverse the commit history (using the Commit.parents or lcas function) to select the changes you want. I don't think we provide pre-defined operations like "rebase" or "cherrypick", so that might be a good first PR :) (I'll leave some general idea on your question below on how to support "undo commit")
In order to automatically trigger a rebase on a branch whenever the main one is updated, you'll want to use the watch function on the main branch (to trigger a custom callback whenever the watched branch has changed).

[eWert-Online]

I would like to have a history feature in the cms, with which the user can roll back changes made previously. I guess going back linearly is no problem, as thats just pointing the HEAD of the workspace to an old commit. Is it also possible to "undo" a specific commit in the middle of the commit history?

[art-w]

Yes that's possible although it's not predefined. The general idea behind any git-like operation on history (undo, rebase, cherry-pick, etc) is that they are all defined in terms of the three-way merge function (but with some smart choices for the ~old least-common ancestor, which is picked differently than you would expect from the commit history).

First notice that if we want to undo a commit A->B, then all we need to do is restore the state A to get an undo commit:

flowchart LR
    X[...] --> A
    A --> |ab| B
    B --> |undo ab| A2[A]

Loading

Then suppose we want to undo the commit A->B in the following history: (even though we made some further changes after B to reach the state C)

flowchart LR
    X[...] --> A
    A --> |ab| B
    B --> |...many commits...| C
    C -.-> |undo ab?| D[C without ab changes]

Loading

Even though it doesn't respect the commit historical order, it turns out that a merge ~old:b a c yields the undo commit for A->B applied after our others commits:

flowchart LR
    B ==> |...many commits...| C
    A -.-> |...many commits...'| E[C without ab changes]
    C -.-> |undo ab'| E

    B ==> |undo ab| A
    A --> |ab| B

Loading

That's because in a three-way merge, the top path B --undo-ab--> A --many-commits'--> C-without-ab must yield the same result as the bottom path B --many-commits--> C --undo-ab'--> C-without-ab. In other words, the three-way merge computes how to reorder the commits.

In Irmin you can compute the outcome of merges on your trees/contents in-memory, then choose how to create the commit to represent the operation (with which commit parents, so you don't have to remember how the undo commit was initially computed, in order to keep your commit history as linear as desired, just like in git -- i.e. you can hide the construction lines I've drawn).

I don't know how clear this explanation was, so please let me know if I should add further details regarding how this works for rebase/cherrypick :) (but I think it's a fun exercise so I don't want to spoil it!)

[eWert-Online]

Thank you for the detailed answer 🙂
I will look into it and come back, if I have any further questions.

[eWert-Online]

I am currently implementing the media upload and am considering saving the uploaded files in irmin directly, so deleting a file can also potentially be reverted. I am worried about the size of the database though, as it would never be possible to really delete a file. So if I upload a 100GB file and don't need that anymore, I would not be able to free up that disk space, right?

I had a look into irmin-pack (currently I am using irmin-fs.unix), but I wasn't able to get it running reliably. I always got errors in String.blit while trying to decode a value and later got dangling hashes while trying to construct a concrete tree. I have to investigate a bit if thats maybe because I am not using string branches and would open an issue as soon as I know more.

[art-w]

• While there are no hard limits on file sizes, one caveat of storing a 100gb file in Irmin is that the file contents will be fully in memory when (de)serializing the contents from/to disk (so that would require a beefy server with lots of ram!). I'm guessing the 100gb size is a bit extreme for a CMS, but in case this is critical you will want to try @kayceesrk 's mergeable rope, to break a large file into smaller chunks (that can be accessed with granularity, instead of reading/writing the whole file at once). This is similar to Git LFS in a way, but with a simpler setup since everything still lives in the same store.
• Only the irmin-pack backend supports freeing up disk space, using its "garbage collector" feature to truncate the commits history (and delete any old files that are not part of the kept history). There are some plans to build upon this with more precise removals... please get in touch with @balat if you are interested :)

Regarding the errors you saw with the irmin-pack backend, could you try again using a Configuration with let contents_length_header = None ? (instead of Some ``Varint, which should not show up as often as it does in our Irmin examples, since it's a low-level optim for string-values)

[eWert-Online]

Unfortunately I already tried to set the contents_length_header to None.
I am using the following configuration to be exact:

module StoreMaker = Irmin_pack_unix.Maker (struct
    let entries = 32
    let stable_hash = 256
    let contents_length_header = None
    let inode_child_order = `Seeded_hash
    let forbid_empty_dir_persistence = false
  end)

module Store = StoreMaker.Make (struct
    module Hash = Irmin.Hash.BLAKE2B
    module Info = Irmin.Info.Default
    module Branch = Branch
    module Path = Irmin.Path.String_list
    module Metadata = Irmin.Metadata.None
    module Contents = Irmin.Contents.Json_value
    module Node = Irmin.Node.Generic_key.Make (Hash) (Path) (Metadata)
    module Commit = Irmin.Commit.Generic_key.Make (Hash)
  end)

The Branch module is a custom branch type, which uses variants with records.
Inserting data seems to be fine, but trying to read them again, yields the error message: Irmin.Tree.to_concrete: encountered dangling hash ab6a424af22507531d8e3ca6e98e5a885773b19f40743213ab5e55c7a8289360c6717881db7af0b7ff39e1b225d9e015605b5612de7d0c84fc5c6cc3d67df4e9.

I currently don't have much time, but I will try to look into it this weekend to find what else the issue might be.

[eWert-Online]

I had a look into this on another small project where I am using irmin.
It seems to be the contents. If I switch them to be string contents everything is working. (Created issue: #2369)

[eWert-Online]

I see, that Irmin is currently in the process of switching the concurrency library from lwt to eio. That has me worried a bit, because the author of Dream said, that they will probably never switch to eio, but stay on lwt (direct).

Is the option considered, to make irmin compatible with both lwt and eio? As a lot in irmin is configurable with functors, I am wondering, if the same thing would be possible with the concurrency library.

[art-w]

While Irmin's interface is migrating to Eio, it should still work well with other Lwt libraries/frameworks like Dream thanks to lwt_eio. This is for example how the Irmin git backend will keep working after the switch to direct-style, as it's still based on ocaml-git which is a large piece of lwt code. We used a similar technique to bridge Lwt with Eio to run irmin-pack in direct-style within a Mirage unikernel (where syscalls to write to disk are provided by Lwt).

(Otherwise you are right that there are no big blockers to support other direct-style I/O schedulers, since most of Irmin is already functorized including the I/O layers... but we don't currently have a use-case requiring an other scheduler than Eio)