Experiment with libmambapy 2.0 alpha (WIP)

[jaimergp]

Description

Checklist - did you ...

• Add a file to the news directory (using the template) for the next release's release notes?
• Add / update necessary tests?
• Add / update outdated documentation?

[jaimergp]

The remaining failures are due to the same underlying issue, I think. It all comes to this diff:

-  'channel-4/win-64::yaml-0.1.7-had09818_2',
?                                  ^^^^^^
+  'channel-4/win-64::yaml-0.1.7-h014fa73_2',
?                                 ++++ ^^

These two yaml versions have the following metadata:

  "yaml-0.1.7-had09818_2.tar.bz2": {
    "build": "had09818_2",
    "build_number": 2,
    "depends": [
      "libgcc-ng >=7.2.0"
    ],
    "license": "MIT",
    "md5": "8bc3d68df049640c74941c52156a2e7a",
    "name": "yaml",
    "sha256": "fdb7ff699dd2ec13677601b4f01c023ca03e0683c917db30556f9cfafde5e0d2",
    "size": 87051,
    "subdir": "linux-64",
    "timestamp": 1510609996778,
    "version": "0.1.7"
  },
  "yaml-0.1.7-h014fa73_2.tar.bz2": {
    "build": "h014fa73_2",
    "build_number": 2,
    "depends": [
      "libgcc-ng >=7.2.0"
    ],
    "license": "MIT",
    "md5": "e723d423ade454d40d09840b1d61ea95",
    "name": "yaml",
    "sha256": "a280801f71ca749954a5683dcba311d6e006cf9b7597753cad9433b83da2ab27",
    "size": 86890,
    "subdir": "linux-64",
    "timestamp": 1510100858096,
    "version": "0.1.7"
  }

Identical except for the timestamp, size and hash. The only part relevant to the solver is the timestamp. had09818_2 is more recent than h014fa73_2. It's build string also happens to sort alphabetically "later".

The solution preferred by libmamba v2 is the same as conda classic used to do before conda/conda#8067 reintroduce timestamp optimization, so I wonder if something changed in the v2 refactor. I'll see if I can provide a reproducer with micromamba so upstream folks can take a look.

[jaimergp]

I created this script to fetch the synthetic channel-4 repodata and create it locally:

# Create synthetic channel
import os
import json
from urllib.request import urlretrieve


SUBDIR = os.environ.get("CONDA_SUBDIR", "linux-64")
channel_4 = "https://github.com/conda/conda/raw/main/tests/data/index4.json"
urlretrieve(channel_4, "index4.json")

with open("index4.json", "r") as f:
    pkgs = json.load(f)

subdir_pkgs, noarch_pkgs = {}, {}
for fn, pkg in pkgs.items():
    pkg["channel"] = "channel-4"
    if pkg.get("subdir", "") == "noarch":
        noarch_pkgs[fn] = pkg
    else:
        pkg["subdir"] = SUBDIR
        subdir_pkgs[fn] = pkg

subdir_repodata = {
    "info": {"subdir": os.environ.get("CONDA_SUBDIR", "linux-64")},
    "packages": subdir_pkgs,
}
noarch_repodata = {
    "info": {"subdir": "noarch"},
    "packages": noarch_pkgs,
}

os.makedirs(f"channel-4/{SUBDIR}", exist_ok=True)
with open(f"channel-4/{SUBDIR}/repodata.json", "w") as f:
    json.dump(subdir_repodata, f)
os.makedirs("channel-4/noarch", exist_ok=True)
with open("channel-4/noarch/repodata.json", "w") as f:
    json.dump(noarch_repodata, f)

print("Channel-4 repodata available at:", os.path.abspath("channel-4"))

Then we can do the following:

# Choose your subdir if not linux-64
CONDA_SUBDIR=osx-arm64 python script.py

micromamba create --dry-run --name unused --override-channels -c ./channel-4 conda conda-build --json | grep yaml-0.1.7

We obtain the correct yaml (had09818_2) with both micromamba 1.5.6 and micromamba 2.0.0alpha1. So it doesn't reproduce.

Interestingly (!), I can't reproduce either with the current release of conda-libmamba-solver OR this PR! We only obtain the wrong yaml when in pytest 🤷

[jaimergp]

Ok, yes, it's timestamp related, but the problem is on the conda.testing.helpers side, not the solver. I don't know why it only popped up now, but that's not too important. Submitted fix at conda/conda#13483

[jaimergp]

Only tests/core/test_solve.py::test_current_repodata_usage[libmamba] is failing consistently now. Apparently, zlib-1.2.11-h62dcd97_3.tar.bz2 is picked when the .conda equivalent should have been picked instead. To be investigated.

[jaimergp]

Committed a fix, but we'll need conda/conda#13490 to fully address that test error. It was passing in libmamba 1.x by chance, because that test is supposed to only use current_repodata.json and that was being actively ignored and replaced with repodata.json. libmamba 1.x does prefer the .conda artifact in case of a tie (not sure if it's by choice or by chance).

[jaimergp]

For tests/test_create.py::test_install_tarball_from_local_channel[libmamba], the error is a mismatch in channel encoding (with or without %-codes). Internally, the conda channels will have the decoded unicode symbol, but make_temp_channel returns a URL-encoded path, so package_is_installed fails. However, the package is correctly installed:

where None = package_is_installed(
    'C:\\\\Users\\\\runneradmin\\\\AppData\\\\Local\\\\Temp\\\\8d2e \xea\xf4\xdf\u03b3\xe7\ud55c\xe1\u0452\u014d\u56fa\xf1\u5bb6',
    (('file:///C:/Users/runneradmin/AppData/Local/Temp/f6db%20%CE%B3%C3%A1%C3%B4%C3%A7%C5%8D%C3%AA%C3%B1%ED%95%9C%E5%AE%B6%D1%92%C3%9F%E5%9B%BA' + '::') + 'flask')
)

Unclear why this is a Windows-only error. I'll debug more in depth when I get access to a VM. In-CI debugging with tmate is a bit messy.

[jaimergp]

• There's a build issue with the 2nd alpha, reported at libmambapy v2.0.0alpha2 pins to libmamba v2.0.0alpha1 conda-forge/mamba-feedstock#212

That's why we can't provision the CI.

[jaimergp]

Continues in #457