Create a decent archive page for the mailing list

[Drup]

It's a well known fact that the archives for the mailing list are quite terrible (to put it mildly). It is possible to retrieve the content as an mbox file (see this script by @marklrh). It would be nice to have some decent plain text search (elastic search or the like) hooked up with it.

cc @gasche

[objmagic]

Here is a Python script used to generate shell commands that are used to download files
For those two hardcoded numbers start and end, start number is fixed at 3399 and end number can be found in the latest reply of the first thread on the first page of gmane archive. One can see the latest number on the bottom of the page, which is 63349.

s = "wget http://download.gmane.org/gmane.comp.lang.caml.inria/{}/{}"

start = 3399
end = 63349 # has to manually get this ending number
interval = 3000
sleep_time = 60

# plus one here since every time a interval is {x | start < x < end}
end = end + 1

while (start < end): 
    l = start
    u = start + interval
    if u > end:
        u = end
    print(s.format(str(l), str(u)))
    print("sleep {}".format(sleep_time))
    start = u

print("cat * > caml-list.mbox")

Be aware that a single HTTP connection cannot last more than 30 seconds or server will cut connection. This means you should decrease interval if you have slow internet speed. 3000 works fine with 200 ~ 300 KBps

The above script gives me a mbox file of 255 MB

[gasche]

@xavierleroy told me he can get an archive of the mailing-list messages (before and after the migration to Sympa) at the source, which would probably be a better starting base if some messages are missing from Gmane.

[avsm]

I could inject an mbox archive into lists.ocaml.org

On 25 Sep 2015, at 14:25, gasche notifications@github.com wrote:

@xavierleroy told me he can get an archive of the mailing-list messages (before and after the migration to Sympa) at the source, which would probably be a better starting base if some messages are missing from Gmane.

—
Reply to this email directly or view it on GitHub.

[UnixJunkie]

related to ocaml/ocaml.org#699 and ocaml/ocaml.org#381. Looks like the situation has not improved since then.