Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use standard cache directories for nce cache. #49

Merged
merged 1 commit into from
Nov 29, 2022

Conversation

jsirois
Copy link
Collaborator

@jsirois jsirois commented Nov 29, 2022

Fixes #47

PathBuf::from(base)
} else if let Some(base) = &lift.base {
base.clone()
} else if let Some(dir) = dirs::cache_dir() {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is all the heavy lifting. I had already added the dirs crate a good while back but never actually used it.

@jsirois
Copy link
Collaborator Author

jsirois commented Nov 29, 2022

I'll plow ahead and merge this by Wednesday afternoon. I'll always be happy for review feedback that comes later, but that's also just a bonus and I don't expect that either.

@jsirois
Copy link
Collaborator Author

jsirois commented Nov 29, 2022

Thanks for flagging this @kamalmarhubi.

@sureshjoshi
Copy link
Collaborator

sureshjoshi commented Nov 29, 2022

So, running the tests creates the nce directory at ~/Library/Caches which is great. When I go into the lift and add "base": "~/.nce" - I get an error. What did I do wrong? I wiped out the nce caches, and the .jar

⏺ jump/examples % git diff

diff --git a/examples/cat/lift.macos-x86_64.json b/examples/cat/lift.macos-x86_64.json
index b722393..3b9baee 100644
--- a/examples/cat/lift.macos-x86_64.json
+++ b/examples/cat/lift.macos-x86_64.json
@@ -1,6 +1,7 @@
 {
   "scie": {
     "lift": {
+      "base": "~/.nce",
       "name": "java",
       "files": [
         {
⏺ jump/examples % RUST_LOG=debug sh run.sh --no-gc cat

   Compiling package v0.1.1 (/Users/sj/Developer/scratch/jump/package)
    Finished release [optimized] target(s) in 4.23s
     Running `/Users/sj/Developer/scratch/jump/target/release/package /Users/sj/Developer/scratch/jump/dist`
Wrote the scie-jump to /Users/sj/Developer/scratch/jump/dist/scie-jump-macos-x86_64
scie-jump-macos-x86_64: OK
-e
-e *** Running cat example ***
-e
Fetching https://repo1.maven.org/maven2/com/github/ricksbrown/cowsay/1.1.0/cowsay-1.1.0.jar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1683k  100 1683k    0     0  11.2M      0 --:--:-- --:--:-- --:--:-- 11.4M
Fetching https://download.java.net/java/GA/jdk19.0.1/afdd2e245b014143b62ccb916125e3ce/10/GPL/openjdk-19.0.1_macos-x64_bin.tar.gz ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  183M  100  183M    0     0  5817k      0  0:00:32  0:00:32 --:--:-- 7830k
[DEBUG TimerFinished] prepare_boot(), Elapsed=38.148µs
[DEBUG TimerFinished] assemble(), Elapsed=33.321µs
[DEBUG TimerFinished] digest(), Elapsed=12.583µs
[DEBUG TimerFinished] load_lift(), Elapsed=190.051µs
[DEBUG TimerFinished] load_manifest(), Elapsed=211.287µs
[DEBUG TimerFinished] pack(), Elapsed=332.129152ms
lift.macos-x86_64.json: /Users/sj/Developer/scratch/jump/examples/cat/java
[DEBUG TimerFinished] assemble(), Elapsed=13.592µs
[DEBUG TimerFinished] digest(), Elapsed=10.722µs
[DEBUG TimerFinished] load_scie(), Elapsed=190.316µs
[DEBUG TimerFinished] new(), Elapsed=7.464µs
[DEBUG TimerFinished] digest_reader(), Elapsed=1.337034651s
[DEBUG jump::installer] The tar.gz destination /Users/sj/.nce/469af195906979f96c1dc862c2f539a5e280d0daece493a95ebeb91962512161/openjdk-19.0.1_macos-x64_bin.tar.gz of size 192577932 had expected hash
[DEBUG TimerFinished] unpack_tar(), Elapsed=1.789279159s
[DEBUG TimerFinished] unpack_archive(), Elapsed=3.126854049s
[DEBUG TimerFinished] digest_reader(), Elapsed=11.706816ms
[DEBUG jump::installer] The blob destination /Users/sj/.nce/212ee64546eb8b5074572fed4107d850eb90bc462aa3099c0ac8ea63fdca7811/cowsay-1.1.0.jar of size 1724250 had expected hash
[DEBUG TimerFinished] unpack_blob(), Elapsed=14.670095ms
[DEBUG TimerFinished] install(), Elapsed=3.141545519s
[DEBUG TimerFinished] prepare_boot(), Elapsed=3.145649272s
 _____________________
< scie-jump boot-pack >
 ---------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

[DEBUG TimerFinished] assemble(), Elapsed=14.26µs
[DEBUG TimerFinished] digest(), Elapsed=10.673µs
[DEBUG TimerFinished] load_scie(), Elapsed=197.242µs
[DEBUG TimerFinished] prepare_boot(), Elapsed=254.936µs
[DEBUG TimerFinished] assemble(), Elapsed=58.513µs
[DEBUG TimerFinished] digest(), Elapsed=12.841µs
[DEBUG TimerFinished] load_scie(), Elapsed=596.521µs
[DEBUG TimerFinished] prepare_boot(), Elapsed=829.114µs
[DEBUG TimerFinished] assemble(), Elapsed=17.054µs
[DEBUG TimerFinished] digest(), Elapsed=14.157µs
[DEBUG TimerFinished] load_scie(), Elapsed=224.571µs
[DEBUG TimerFinished] new(), Elapsed=8.976µs
[DEBUG jump::atomic] The atomic directory at /Users/sj/.nce/469af195906979f96c1dc862c2f539a5e280d0daece493a95ebeb91962512161/openjdk-19.0.1_macos-x64_bin.tar.gz has already been established.
[DEBUG TimerFinished] unpack_archive(), Elapsed=18.031µs
[DEBUG jump::atomic] The atomic file at /Users/sj/.nce/212ee64546eb8b5074572fed4107d850eb90bc462aa3099c0ac8ea63fdca7811/cowsay-1.1.0.jar has already been established.
[DEBUG TimerFinished] unpack_blob(), Elapsed=12.56µs
[DEBUG TimerFinished] install(), Elapsed=39.361µs
[DEBUG TimerFinished] prepare_boot(), Elapsed=389.598µs
 __________
< scie cat >
 ----------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Hashes of java and java.cat were not the same.


⏺ jump/examples % cat cat/java.sha256
1573e093b616f3060502896c0dee8c355e8b6de7dc87c8fbaa3f0af4d0eed834  java

⏺ jump/examples % cat cat/java.cat.sha256
2f06d2f09b65241abe96878af75f973e225aec287be7757d03c0a413aeda36a0  java.cat
SCIE=inspect ./java (identical to java.cat)

{
  "scie": {
    "lift": {
      "name": "java",
      "base": "~/.nce",
      "files": [
        {
          "name": "openjdk-19.0.1_macos-x64_bin.tar.gz",
          "key": "jdk",
          "size": 192577932,
          "hash": "469af195906979f96c1dc862c2f539a5e280d0daece493a95ebeb91962512161",
          "type": "tar.gz"
        },
        {
          "name": "cowsay-1.1.0.jar",
          "key": "cowsay.jar",
          "size": 1724250,
          "hash": "212ee64546eb8b5074572fed4107d850eb90bc462aa3099c0ac8ea63fdca7811",
          "type": "blob"
        }
      ],
      "boot": {
        "commands": {
          "": {
            "exe": "{jdk}/jdk-19.0.1.jdk/Contents/Home/bin/java",
            "args": [
              "-jar",
              "{cowsay.jar}"
            ],
            "env": {
              "=JAVA_HOME": "{jdk}/jdk-19.0.1.jdk/Contents/Home",
              "=PATH": "{jdk}/jdk-19.0.1.jdk/Contents/Home/bin:{scie.env.PATH}"
            }
          }
        }
      }
    },
    "jump": {
      "size": 1341152,
      "version": "0.3.9"
    }
  },
  "fetch": [
    "https://download.java.net/java/GA/jdk19.0.1/afdd2e245b014143b62ccb916125e3ce/10/GPL/openjdk-19.0.1_macos-x64_bin.tar.gz"
  ]
}

@sureshjoshi
Copy link
Collaborator

🤦🏽

Putting the "base" below the "name" to match the expected output passes. During testing I was comparing with SCIE=inspect, which I guess always gives the same key ordering?

⏺ jump/examples % git diff

diff --git a/examples/cat/lift.macos-x86_64.json b/examples/cat/lift.macos-x86_64.json
index b722393..96eb0eb 100644
--- a/examples/cat/lift.macos-x86_64.json
+++ b/examples/cat/lift.macos-x86_64.json
@@ -2,6 +2,7 @@
   "scie": {
     "lift": {
       "name": "java",
+      "base": "~/.nce",
       "files": [
         {
           "name": "openjdk-19.0.1_macos-x64_bin.tar.gz",

@jsirois
Copy link
Collaborator Author

jsirois commented Nov 29, 2022

Keep in mind what's being tested, that a hand assembled scie using cat is identical to a scie assembled with the scie-jump boot-pack. The former is manual and the latter semi-automated. To get bytewise identical scies from both forms of assembly you need bytewise identical lift manifests. Although key order is not significant for other purposes, it is significant for this artificial test. The only real constraint that I can see is that the boot-pack, split, boot-pack is always automatically bytewise identical. Another test tests that already. You just mucked with the most titchy test of the most artificial property.

Let me know if any part of this doesn't wash with you, I may be missing a meaningful problem here or something that needs documentation at least.

@sureshjoshi
Copy link
Collaborator

sureshjoshi commented Nov 29, 2022

Let me know if any part of this doesn't wash with you

Nah, it's fine - it was just a 🤦🏽 moment. I was running some random tests, and it failed on a key ordering while I was manipulating something. Just took a while to debug, even though I knew it was directly related to that extra key.

@jsirois
Copy link
Collaborator Author

jsirois commented Nov 29, 2022

Ah, ok. Gotcha. Thanks for giving this a spin.

@jsirois jsirois merged commit f13033b into a-scie:main Nov 29, 2022
@jsirois jsirois deleted the issues/47 branch November 29, 2022 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Consider moving CAS to platform-standard cache directories
2 participants