CIPD checksum error results in multiple unzips of all packages, no re-download

[erock2112]

See this task: https://chromium-swarm.appspot.com/task?id=350ea8dd07d23c10&refresh=10&show_raw=1
There was a zip checksum error for one of the three packages. It looks like there were three attempts in which all three packages were unzipped, but the broken package was never re-downloaded.

[maruel]

+Vadim in case he can investigate.

[vadimsht]

There are no 3 attempts, it's the same log repeated 3 times :(

What bothers me, is that some other tasks on this bot that use exact same package file succeed: https://chromium-swarm.appspot.com/task?id=350c1a3ae1e90710&refresh=10&request_detail=true

Not sure how to explain it. I'll need to login to the bot to check what's wrong with it.

[erock2112]

That's strange. FWIW I deleted the CIPD cache this morning to get the issue resolved.

[vadimsht]

Can you please login to skia-e-win-025 and check SHA1 hash of c:\b\s\cipd_cache\cache\instances\6720f19ba2c618f8541da7f4eacbc32a6ed42b91 (it should be 6720f19ba2c618f8541da7f4eacbc32a6ed42b91).

Alternatively, point me to instructions how do it myself...

[erock2112]

I'm not sure whether these machines are accessible outside of the local network. @dogben can verify. I don't know how to get the SHA1 sum on Windows, since I can't just do sha1sum.

[vadimsht]

You have python there. It should be easy to write a small script using hashlib module.

[erock2112]

Okay, using hashlib.sha1 I got: e63e58397ea6f01ccad8c81a6840a722f3750673. FTR none of the three files had hashes matching the file names.

[vadimsht]

Did you use open(..., 'rb') mode in Python? :) By default it opens files in text mode, and on Windows its important.

[vadimsht]

(If the hash really doesn't match, would it be possible to copy this file somewhere?)

[erock2112]

Sigh. I've forgotten or repressed all of my Python and Windows experience. Yes, with 'rb' the hashes match.

[vadimsht]

Then I'm puzzled :(

The locally cached file is not corrupt (it matches whatever was uploaded to the CIPD backend). And the file in the cipd storage is correct (I can install it locally on my machine, and other bots have no problem with it).

Most likely it's something related to this particular bot. Maybe the disk is faulty and sometimes file reads return wrong result?..