RFC: Repo organization

[jlblancoc]

Hi all: (specially, @berndpfrommer )

How should we organize the contents of this repo?

My first idea was to simply put a /debian directory on the root, but we should do it as git-buildpackage recommends. On the other hand, @berndpfrommer mentioned that this same repo could be used to store packaging stuff for other non-debian formats, but: how could that be done? in different branches, for example?

[berndpfrommer]

I started to read about git-buildpackage, but quite frankly couldn't wrap my mind around the documentation. It almost seemed like the whole system is meant to operate on the original source tree, with the debian packaging on a separate branch. I will need to look more at it during this week.

[berndpfrommer]

I was able to successfully build a source package using git buildpackage.

I will produce a README.md that describes how the building works. PR to follow.

[berndpfrommer]

One thing that I'm not clear about: how do we handle all the different flavors of libgtsam that only differ by some compile options? Like for instance with or without MKL support. We can make two different packages, and give them different names, but they would not be able to co-exist.

[jlblancoc]

One thing that I'm not clear about: how do we handle all the different flavors of libgtsam that only differ by some compile options? Like for instance with or without MKL support. We can make two different packages, and give them different names, but they would not be able to co-exist.

In principle, I was not thinking of supporting such a feature... but it would be great! From other packages, they do it normally as you suggest: e.g. libgtsam-dev vs libgtsam-mkl-dev, etc.
We will have to add

Package: foo-v1
Breaks: foo-v2
Replaces: foo-v2
...

Package: foo-v2
Breaks: foo-v1
Replaces: foo-v1
...

to debian/control to let apt know about the need to pick only one.

[berndpfrommer]

OK, I added instructions on how to get the basics going.
Next:

• figure out how to build for other distros (bionic/focal etc) and upload all of this to a ubuntu PPA without conflicts
• figure out how to make snapshots rather than full releases,
• update the debian/control per your suggestions above

[jlblancoc]

#2 is brilliant!! 😄 It was far more complicated than I expected, but using gbp seems to be systematic enough to be a good choice.

Just a couple of follow-up doubts and thoughts:

• Then, should we keep the debian/ directory in upstream?
• How to handle nightly builds? I guess one way could be creating temporary local tags adding the date and/or last git commit hash to the version, just as done now in gtsam/package_scripts/. It should be done such that NOT all those unreleased versions' pristine tarballs are kept in the repo, or the size would grow out of control... perhaps only those tarballs of released versions should be "git push"ed, and the rest, just discarded?

[berndpfrommer]

• we should definitely remove the debian/ directory. Right now I need an extra commit to remove it, just to avoid conflicts with the debian/ directory from the packaging branch. I'd love to have that done before the next release.
• nightly builds: I will have to spend some more quality time with git buildpackage to figure that one out. It will be pretty much the same way that your script was doing it, except that the "Debian" way of generating the version numbers is different. Ideally the pristine tar ball will stay the same until the next release, and the differences between last release and current snapshots will be captured by patches in the debian source package.
  I will be busy with other stuff the next few days, just a heads up.

[jlblancoc]

Hi @berndpfrommer !

Should we add the debian directory to this repo?

[berndpfrommer]

Yes! Please give me til the weekend to work on it.

[berndpfrommer]

I finally finished the mother of all build scripts, using gbp to build a nightly snapshot of gtsam.
Here is a prototype of it:
https://github.com/berndpfrommer/gtsam-packaging/blob/master/build_ubuntu/snapshot_script.bash

The final version is in another workspace. Unfortunately this repo has to be private because it must hold the necessary keys to sign the package for upload to the PPA. These keys cannot be in a public repo since anybody with the keys can then upload to the PPA.

As of tonight that script is running and building a gtsam snapshot to

ppa:bernd-pfrommer/gtsam-snapshot

I will watch it for a couple of days to make sure it works as it should. Then I'll create a public repo with the build script and instructions how to set it up, but with the keys removed.

[jlblancoc]

wow, awesome!! 💯
Looking forward to test it within a few days.

[berndpfrommer]

as you can probably see from the commit history I've been chipping away on this for the last few days and weeks.
I've hit some serious security issues because the packages need to be signed before they can be uploaded to the ppa. This becomes an issue because (so I thought) you would have to keep a secret key unencrypted in a repo. The workaround I devised became incredibly awkward and complicated.
But I have come to learn now that one can probably use a GPG key deposited with github to sign a package. I will be pursuing this approach in the following days.
Bottom line: the nightly script is not working in a github environment, so hold off on testing it.

[dellaert]

I have seen this trickle in, and this is supercool :-)

[berndpfrommer]

Quite an odyssey to figure out how to do Ubuntu packaging. Sorry it's taken so long, but it works now, and the setup is reasonably sound.
I've pushed all the documentation and nightly build scripts. We cannot build in this repository because signing the packages requires depositing a private key with the repo that does the nightly build. So the repo needs to be cloned to somebody's personal repo, and built from there. We should however consider having an official Ubuntu PPA, see #6

[jlblancoc]

Awesome work, Bernd!

I'm eager to get gtsam-4.0.3 into Debian, so I'm preparing a manually-edited version of 4.0.3, removing the only 3rdparty library that is taken from the system at present (Eigen3), and leaving the rest as is, for now.

Also, I needed to edit the debian/* files quite a lot, and futher changes are probably yet to show up after the review by Debian sponsors.
My changes to debian/* are in this branch: https://github.com/borglab/gtsam-packaging/tree/debian

Any ideas on a better approach to this?

[jlblancoc]

Note to ourselves: these are the remaining lintian warnings/errors:

Now running lintian gtsam_4.0.3-1_amd64.changes ...
E: gtsam changes: bad-distribution-in-changes-file unstable
W: libgtsam4: dev-pkg-without-shlib-symlink usr/lib/x86_64-linux-gnu/libmetis-gtsam.so usr/lib/x86_64-linux-gnu/libmetis-gtsam.so
W: libgtsam-dev: new-package-should-close-itp-bug
W: libgtsam-unstable-dev: new-package-should-close-itp-bug
W: libgtsam-unstable4: new-package-should-close-itp-bug
W: libgtsam4: new-package-should-close-itp-bug
W: libgtsam4: shlib-without-versioned-soname usr/lib/x86_64-linux-gnu/libmetis-gtsam.so libmetis-gtsam.so

the so-file version needs cmake changes, that could be integrated into develop, then added to the 4.0.3 release as a patch.
We should also create the ITP bug.

PS: the "unstable" error is not really a bug, can be ignored.

[jlblancoc]

One more issue we need to discuss: I think we might need a "custom export" script anyway for gtsam (like the first one I added time ago, later removed) since gtsam will keep shipping many 3rd-party libraries (I think that's @dellaert 's idea) but that's not allowed for an official Debian release. It's OK for our private Ubuntu PPA repos, but Debian folks won't like that solution at all.
So, the alternatives are:

• Remove all 3rd party sources: pros=clean source tree; cons=harder to set up for OSX and Windows users. I think we can then rule out this option.
• Automate the stripping out of those 3rd party sources that can be installed from Ubuntu official packages.

Right now, this is WIP addressed in borglab/gtsam#292
And only Eigen3 is, I think, solved. So this script would take a given gtsam release, export it (git archive, or the tool I just found git deborig), remove the 3rd party sources that are already supported as system libraries, and add the debian/ directory from this repo.
I think gbp cannot handle this "especial" situation on its own, right?

My idea is submitting gtsam 4.0.3 + some patches to fix lintian errors, without Eigen3, then ourselves or someone else will report the Debian "wish-list" bugs regarding the need to get rid of 3rd party embedded libraries and we can work more on that in the future, but at least we would take gtsam within Debian and Ubuntu official repos and that would be super useful! ;-)

[berndpfrommer]

My preference would be to first establish an official borglab branded PPA before we shoot for anl inclusion into Ubuntu outright. I think we can get feedback from users and iron out a lot of kinks more easily in a private PPA.

[berndpfrommer]

Closing this issue because the main topic has been addressed now. Other steps necessary towards getting a full, official Ubuntu release working should be addressed with separate issues.