FileNotFoundError error in tests.py

[martinbudden]

I'm sometimes getting a FileNotFoundError in tests.py when doing a make_all, see error trace below. I haven't been able to track down what exactly causes this to happen, but do have a fix, see at end.

magick BC200/assemblies/main_assembled.png -trim -resize 280x280 -background #F8F8F8 -gravity Center -extent 280x280 -bordercolor #F8F8F8 -border 10 tmp.png
Traceback (most recent call last):
  File "../nopscadlib/scripts/make_all.py", line 47, in <module>
    views(target)
  File "C:\Users\martin\Code\GitHub\OpenSCAD\nopscadlib\scripts\views.py", line 199, in views
    update_image(tmp_name, tn_name)
  File "C:\Users\martin\Code\GitHub\OpenSCAD\nopscadlib\scripts\tests.py", line 68, in update_image
    shutil.copyfile(tmp_name, png_name)
  File "C:\Users\martin\AppData\Local\Programs\Python\Python38\lib\shutil.py", line 259, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: 'tmp.png'

The addition of two checks of if os.path.isfile(tmp_name): fix the problem.

def update_image(tmp_name, png_name):
    """Update an image only if different, otherwise just change the mod time"""
    # print("in update_image\n")
    # print("tmp name =", tmp_name, "\n")
    # print("png name =", png_name, "\n")
    diff_name = png_name.replace('.png', '_diff.png')
    pixels = compare_images(png_name, tmp_name, diff_name)
    if pixels < 0 or pixels > threshold:
        if os.path.isfile(tmp_name):
            shutil.copyfile(tmp_name, png_name)
        print(Fore.YELLOW + png_name + " updated" + Fore.WHITE, pixels if pixels > 0 else '')
    else:
        os.utime(png_name, None)
        os.remove(diff_name)
    if os.path.isfile(tmp_name):
        os.remove(tmp_name)
    print("\n")

[nophead]

I think it means your scad file has an warning in it. Have a look at openscad.log.

I pushed a change recently that makes warnings hard errors and terminates openscad but there is a bug in openscad that returns a zero exit status in that case, so the script continues. I reported the bug and I think that it was fixed in master.

[martinbudden]

It turns out that I did have a warning, but I fixed that and the problem still remains. I tried another project of mine and that also now suffers the same problem.

So I tried the MainsBreakoutBox example project and that also suffers the same problem, so I think something has been inadvertently broken. Here's the output from MainsBreakoutBox:

C:\Users\martin\Code\GitHub\OpenSCAD\NopSCADlib\examples\MainsBreakOutBox> python ../../scripts/make_all.py
openscad --hardwarnings -D $bom=2 -D $preview=true -o openscad.echo -d bom/bom.deps scad/bom.scad
Generating bom ... done
  5.2   0.0 foot.stl
 12.5   0.0 socket_box.stl
 17.7
openscad --hardwarnings --colorscheme=Nature --projection=p --imgsize=4096,4096 --camera=0,0,0,70,0,315,500 --preview --autocenter --viewall -o tmp.png png.scad
magick tmp.png -trim -resize 280x280 -background #F8F8F8 -gravity Center -extent 280x280 -bordercolor #F8F8F8 -border 10 tmp.png
magick compare -metric AE -fuzz 5% stls/socket_box.png tmp.png stls/socket_box_diff.png
stls/socket_box.png updated 6709870
command line options changed
openscad --hardwarnings -D$show_threads=true -D$pose=1 -D$explode=0 --colorscheme=Nature --projection=p --imgsize=4096,4096 --autocenter --viewall -d deps/bob_main.deps -o tmp.png png.scad
magick tmp.png -trim -resize 1004x1004 -bordercolor #F8F8F8 -border 10 tmp.png
magick compare -metric AE -fuzz 5% assemblies/base_assembled.png tmp.png assemblies/base_assembled_diff.png
assemblies/base_assembled.png updated 7988960
magick assemblies/base_assembled.png -trim -resize 280x280 -background #F8F8F8 -gravity Center -extent 280x280 -bordercolor #F8F8F8 -border 10 tmp.png
magick compare -metric AE -fuzz 5% assemblies/base_assembled_tn.png tmp.png assemblies/base_assembled_tn_diff.png
Traceback (most recent call last):
  File "../../scripts/make_all.py", line 47, in <module>
    views(target)
  File "C:\Users\martin\Code\GitHub\OpenSCAD\NopSCADlib\scripts\views.py", line 199, in views
    update_image(tmp_name, tn_name)
  File "C:\Users\martin\Code\GitHub\OpenSCAD\NopSCADlib\scripts\tests.py", line 66, in update_image
    pixels = compare_images(png_name, tmp_name, diff_name)
  File "C:\Users\martin\Code\GitHub\OpenSCAD\NopSCADlib\scripts\tests.py", line 59, in compare_images
    pixels = int(float(f.read().strip()))
ValueError: could not convert string to float: "compare: unable to open image 'tmp.png': No such file or directory @ error/blob.c/OpenBlob/3496."
PS C:\Users\martin\Code\GitHub\OpenSCAD\NopSCADlib\examples\MainsBreakOutBox>

[nophead]

It runs fine on my machine but none of the images have changed because it is not long since I last ran it. They did change then due to the change in teardrop_plus which makes a few pixels change around holes and countersunk screw heads.

It looks like a problem in this command:

magick assemblies/base_assembled.png -trim -resize 280x280 -background #F8F8F8 -gravity Center -extent 280x280 -bordercolor #F8F8F8 -border 10 tmp.png

For some reason it doesn't create tmp.png but we know assemblies/base_assembled.png was updated.

Does base_assembled.png contain a valid image?

What happens if you run the magick command manually? Does it make tmp.png?

[martinbudden]

So if I run that command manually, it takes a few seconds and then does not produce tmp.png. It doesn't produce an error message. And base_assembled.png contains a valid image.

[nophead]

Can you attach the image and I will try it here. It seems like a bug in magick. Checking that tmp.png exists and is not really a work around because you will end up with a missing image.

[martinbudden]

Here it is.

By the way, my magick version is: Version: ImageMagick 7.0.9-16 Q16 x64 2020-01-12

[nophead]

Works fine for me.

I have an older version: Version: ImageMagick 7.0.8-14 Q16 x64 2018-10-24

Did you recently update it and is that when it started to fail?

[martinbudden]

No, I haven't changed it since I originally installed it.

I'll upgrade to the latest version and try that...

[nophead]

Odd it should suddenly stop working. Does it work for any small images?

Note it takes less than 1s to process that file on my laptop.

[nophead]

I pushed a change to give an explicit error message when png.tmp is not generated instead of a carrying on and generating an exception.

[martinbudden]

I've been playing around with this a while now. It seems there is something odd going on with the file's path.

If I type:

magick BC200/assemblies/base_assembled.png -trim -resize 280x280 -background #F8F8F8 -gravity Center -extent 280x280 -bordercolor #F8F8F8 -border 10 tmp.png

then it will fail silently (note, BC200 is the variant of the project).
However if I copy BC200/assemblies/base_assembled.png to tmp.png and then type:

magick tmp.png -trim -resize 280x280 -background #F8F8F8 -gravity Center -extent 280x280 -bordercolor #F8F8F8 -border 10 tmpA.png

it works.

Two other points of note:

1. If I add -identify to the non-working form, it clearly identifies the input file, so the problem is not that it cannot find the file
2. When the command works, it completes quickly, when it fails, it takes quite a while to complete (5 or more seconds) - so it is clearly doing something.

I have recently had a Windows update, that is the only thing I can think of that may have stopped this working, but I can't see anything in the update notes about this.

So a workaround could be to copy the file to the local directory before running magick on it.

[nophead]

Weird. One advantage of Win7 is I don't get updates!

Does it still work if you copy it to tmpA.png and then do

magick tmpA.png -trim -resize 280x280 -background #F8F8F8 -gravity Center -extent 280x280 -bordercolor #F8F8F8 -border 10 tmp.png

I can't imagine a Windows update being so broken. Is the filesystem on a normal drive, not networked or a folder synced to GoogleDrive? Is it a virus checker issue?

I don't really want to add a workaround for paths not working.

[martinbudden]

I was mistaken when I said it works when you copy the file to the local directory. What does work is:

magick BC200/assemblies/Base_assembled.png aa.png
magick aa.png -trim -resize 280x280 -gravity Center -extent 280x280 -border 10 bb.png

Not sure why splitting the operation in two helps. Note that magick BC200/assemblies/Base_assembled.png is 248KB, aa.png is 97KB and bb.png is 21KB, so the first line must be stripping some information out of the original file.

I also tried doing

magick BC200/assemblies/Base_assembled.png -resize 280x280  bb.png

ie, just the resize on its own, and this fails. So it looks like the resize on the full size file is what causes the problem, so it seems like some sort of out of memory problem, but this does seem very odd in what is a mature program.

[nophead]

Is it possible to get older releases? E.g. the same as I have to rule it out.

I don't see how magick can go from working to broken if you didn't change it's version. Is it perhaps a hardware problem on your PC. Do you have anything else to try it on?

It only seems to go wrong making the small image from the bigger one that it created itself. It seems to be able to make the bigger one from the giant one exported from OpenSCAD, so I don't think it runs out of memory.

I don't understand how running it with no parameters makes the file smaller, when the source is a file it has just created itself. When I do that here the file is exactly the same size and the contents only differ by 10 bytes, which I think is a timestamp.

[martinbudden]

I knew this issue would be a rabbit hole when I reported it...

I agree, the issue is probably something to do with my PC, but I haven't changed anything except for auto-updates.

I thought that BC200/assemblies/Base_assembled.png was created by OpenSCAD, not image magick, so running magick BC200/assemblies/Base_assembled.png aa.png could create a smaller file?

I have an old mac that I haven't really used since I went back to Windows, so I could try it out on there. I'll start doing that as a background task.

[nophead]

No OpenSCAD makes an image four times bigger and it is downsized by Magick to get anti-aliasing.

What happens if you use windows paths, i.e. \BC200\assemblies\Base_assembled.png ?

[martinbudden]

I tried windows paths and also putting quotes around the path, both without success. It does seem that there is something about my machine that is causing Magick to fail.

[martinbudden]

I found some old Magick versions at https://imagemagick.en.uptodown.com/windows/versions

7.0.6-9, Aug 22nd, 2017 works and 7.0.9-16, Jan 14th, 2020 fails. Unfortunately they have nothing in between those versions.

I can't believe that if this is a bug in Magick, that it hasn't been found in all that time. It is possible that they have become more strict on the chaining of operations though - is it possible there are some restrictions on or prerequisites for applying multiple operations on an image that you have overlooked?

[nophead]

So I revisited the image you posted and there is something wrong about it. It is 4096 by 4096 and is not centred vertically. Processing with magick with no parameters does shrink it from 248K to 97K. So it does look like an image from OpenSCAD, not one that has been processed with magick.

So it looks like the first call to magick also does nothing. That has tmp.png as source and destination so if it does nothing it presumably leaves the file unchanged and update_image() then copies it to the correct place.

[martinbudden]

Odd about the image, but I don't think the image is the problem. I may have generated that image from within OpenSCAD, since it is quicker than running a build.

I made a dummy project with a single assembly which consisted of a single stl, a 10x10x10 cube, and that failed. Going back to Magick 7.0.6-9 seems to have fixed my problems. I've just kicked off a full build to verify.

[martinbudden]

By the way, at line 193 of views.py there is a resize to 1004x1004. Is that deliberate, or is it a typo of 1024x1024?

[nophead]

No it must have been made from the command line to get an image that big. Basically I thought it was only failing to make the small image but it also fails to make the large one as well. The script just does not notice because it is done in place.

When did you last update OpenSCAD. I wonder if that is what started it failing.

[nophead]

The size is deliberate to allow for the 10 pixel border all the way round to get 1024.

[martinbudden]

I last updated OpenSCAD as a result of looking into this problem - because you said it wasn't dealing with warnings correctly.

See my previous post, the problem seems to be with Magick. I've just reinstalled Magick 7.0.9-16, and my full build has just completed and everything now works fine.

I guess I must have upgraded Magick fairy recently, but I don't remember doing so.

[nophead]

Your previous version of Magick was a year old though and the next version only two weeks later, so it is unlikely you updated since Jan last year.

I was mistaken when I suggested it could be an OpenSCAD warning because I didn't notice it was the second magick call that was failing. Seems like the first one does not like the output from OpenSCAD and silently leaves it unchanged and that gets copied to assemblies unchanged. The second call to magick does not like it either and so does not create tmp.png and then the script falls over.

If you run magick with no arguments it looks like it compresses the file and then running it with arguments on the compressed version works. So it looks a bit like magick can't handle a 4096 x 4096 uncompressed image if it has to do those operations on it. Odd though because you would think the first thing it would do is uncompress it before it starts trimming and shrinking.

When did it last work OK?

[nophead]

Actually 4096 x 4096 uncompressed would be a huge file, so it must be compressed by OpenSCAD but magick can compress it further somehow.

[martinbudden]

I'm not sure when it stopped working. I'd say sometime in the last month, but I couldn't be sure. I think it was working when you added the pixels = int(float(f.read().strip())) line to tests.py, but again couldn't be sure.

I first noticed a problem a while ago in the the images in my readme.html were very large. I put it aside as something that could be looked at later, but now it seems clear this was because the image resizings were failing.

[nophead]

Yes the big images would be too big but the small ones are missing and the script would have stopped until you added the change in this PR. Then I presume it would carry on leaving a readme with no small assembly pictures and the big ones huge.

[nophead]

If it had been built before then the small images would probably exist but be out of date,

[martinbudden]

Yes, the small images would have still been hanging around, so a readme would still have been produced.

[nophead]

Can we close this as it seems to be a bug in Magick and should now cause the build to stop with a more obvious pointer to Magick?

[martinbudden]

Well, we both seem to have work arounds, but it might hit new users. But I guess it can be closed.

Perhaps we should report the bug to Magick?

[nophead]

Yes if they accept bug reports.