prevents files from being opened/read in callback?

[blaenk]

I'm trying to track down a bug in static site generator Hakyll's master branch (i.e. unreleased) which now uses fs-notify for automatic recompilation in conjunction with a preview server, i.e. to be able to preview one's site while editing it. As a consequence, Hakyll's preview functionality (which depends on this package) is now broken in Windows.

I believe I have found two problems which can be traced back to win32-notify.

1. Programs using win32-notify (most likely indirectly through fs-notify) have to be compiled with -threaded or win32-notify will block all other threads. If a file under watch (with watchTree) is saved such that it triggers the callback, the main/other threads are afforded a time slice. I'm not familiar with the implications of -threaded, but I don't think it's a big deal. I'm just noting here because it took me a while to figure out why my program was being blocked (wouldn't even respond to CTRL+C; had to terminate through task manager).

2. The file for which the event was triggered cannot be opened in the callback (e.g.
   with readFile). Trying to do so leads to an error.

   readfile.exe: provider\pages\about.markdown: openFile: permission denied (Permission denied)

   However, any other file being watchTree'ed can be opened in the callback (as long as it's not the one for which the event was fired).

I'm not familiar with the ReadDirectoryChangesW function/API. However, win32-notify does seem to use flags (fILE_SHARE_READ .|. fILE_SHARE_WRITE) which I've read in two different places is what you would want in order to be able to open the files being watched.

The following code replicates the issue:

import qualified Filesystem.Path.CurrentOS as FP
import System.FSNotify
import Control.Monad
import Control.Monad.Trans (liftIO)
import System.FilePath
import Control.Concurrent

main :: IO ()
main = do
    let provider = FP.decodeString "provider"
    manager <- startManagerConf (Debounce 0.1)

    watchTree manager provider (const True) $ \event -> do
        putStrLn "EVENT ==========================="
        let path = FP.encodeString $ eventPath event
        putStrLn $ "      " ++ path
        contents <- readFile ("provider" </> path)
        putStrLn $ "      " ++ contents

    threadDelay 36000000
  where
    preServeHook _ = return ()

I'm trying to find the cause of this but I'm not very familiar with the Win32 API, so I'd appreciate any leads or ideas.

[gregwebs]

what version of ghc are you on? The issue with the program blocking got better with ghc 7 I think, although maybe it was 7.4

Thanks for digging into this. This project needs some help to fix these last bugs. I will look at things while at the hackathon on Sunday. One thing I want to do is change the chan based API to something that is perhaps more like a blocking read. You could try using the Chan api for reading the file right now.

[blaenk]

@gregwebs Thanks for the reply! I didn't have much hope since I noticed the last commit was five months ago. I'm on version 7.4.2, as was installed by the haskell-platform.

I'll look into the Chan API, thanks.

[blaenk]

So I tried it with watchTreeChan and I'm getting the same results. For reference, I used this program:

import qualified Filesystem.Path.CurrentOS as FP
import System.FSNotify
import Control.Monad
import Control.Monad.Trans (liftIO)
import System.FilePath
import Control.Concurrent

main :: IO ()
main = do
    let provider = FP.decodeString "provider"
    manager <- startManagerConf (Debounce 0.1)

    chan <- newChan

    watchTreeChan manager provider (const True) chan

    forkIO $ forever $ do
        event <- readChan chan
        putStrLn "EVENT =============="
        let path = FP.encodeString $ eventPath event
        putStrLn $ "      " ++ path
        contents <- readFile ("provider" </> path)
        putStrLn $ "      " ++ contents

    threadDelay 36000000
  where
    preServeHook _ = return ()

The output is:

EVENT ==============
      posts\im-back.markdown
readfile.exe: provider\posts\im-back.markdown: openFile: permission denied (Permission denied)

[gregwebs]

In your chan code, if you put a large enough delay in before reading the file, does the issue go away?

Also, what happens if you instead try something roughly like:

contents <- system $ "cat " <> toString filepath

except it is windows, I don't know what the cat equivalent is.

[blaenk]

Both things worked!

In your chan code, if you put a large enough delay in before reading the file, does the issue go away?

I added a threadDelay 5000000 (five seconds) and it went through successfully.

Also, what happens if you instead try something roughly like:

Apparently Windows has cat. I did more or less this:

system $ ("cat " ++ "provider" </> path)

I have a feeling the first one working (delay) is more important. Perhaps this hints at something?

[blaenk]

For what it's worth (though I imagine this is pretty dependent on my hardware) I've gotten the delay down to about 100 milliseconds (100000 microseconds, as passed to threadDelay). Haven't bothered to try any lower, since like I said I imagine this is dependent on my hardware.

[blaenk]

I found an answer on stackoverflow that seems to touch on this issue:

Something else you will have to deal with is that, at the time the notification is generated, the process that writes the file is very likely to still have a lock on the file. Which prevents you from doing anything useful with the file yourself. Like reading the file or copying it is likely to fail. You have to wait until the process is done with the file and has closed its handle to the file. There's no way to discover this, other than by trying to open the file yourself and deny any sharing. This requires a timer, periodically trying to acquire a lock on the file yourself. Once you got that plumbing in place, getting more than one change notification for the file doesn't matter anymore.

I think this might very well be the case. I'm not very familiar with the win32 API, but in skimming through the code for win32-notify it doesn't seem to me like win32-notify is somehow acquiring locks on files (though that was my suspicion), maybe you guys can look through the source for that to verify. This leads me to believe that indeed the problem is not in win32-notify but in the ReadDirectoryChangesW API itself.

However, pursuing a fix from this direction led me to simply create a function that tries to openFile the file with ReadMode, and if it receives a permission denied exception, it sleeps for 100 milliseconds before it tries again.

This is similar to what seems to be standard practice for this very same issue. Here is the code I came up with:

import qualified Filesystem.Path.CurrentOS as FP (encodeString, decodeString)
import System.FSNotify (WatchConfig(Debounce), eventPath, watchTree, startManagerConf)
import System.FilePath ((</>))
import System.IO (IOMode(ReadMode), Handle, hGetContents, openFile, hClose)
import System.IO.Error (isPermissionError)
import Control.Concurrent (threadDelay)
import Control.Exception (IOException, throw, try)

main :: IO ()
main = do
    let provider = FP.decodeString "provider"
    manager <- startManagerConf (Debounce 0.1)

    watchTree manager provider (const True) $ \event -> do
        let path = FP.encodeString $ eventPath event
            providerPath = "provider" </> path

        waitOpen providerPath ReadMode $ \handle -> do
            contents <- hGetContents handle
            putStrLn contents

    threadDelay 36000000
  where
    waitOpen :: FilePath -> IOMode -> (Handle -> IO r) -> IO r
    waitOpen path mode handler = do
        res <- try $ openFile path mode :: IO (Either IOException Handle)
        case res of
            Left ex -> do
                if isPermissionError ex
                then do
                    threadDelay 100000
                    waitOpen path mode handler
                else throw ex
            Right h -> do
                res <- handler h
                hClose h
                return res

This workaround works perfectly in my tests so far.

Notice that waitOpen is similar to withFile from System.IO.

Resources I've found that lead me to believe this is a flaw in the ReadDirectoryChangesW API:

• http://stackoverflow.com/a/14040978/101090 (last paragraph)
• http://stackoverflow.com/questions/50744/wait-until-file-is-unlocked-in-net (uses ReadDirectoryChangesW under the hood)
• http://stackoverflow.com/questions/6537368/delphi-wait-until-a-file-copy-process-is-complete
• http://blogs.msdn.com/b/oldnewthing/archive/2011/08/12/10195186.aspx#10195293
• http://blogs.msdn.com/b/oldnewthing/archive/2011/08/12/10195186.aspx#10195468

I don't believe a fix can be provided in in this package (win32-notify) because I don't think it can be assumed that everyone who uses the library wants to open the file for which the event occurred. It could be that the user is simply logging all events to a separate file. That said, perhaps you guys might want to expose my workaround in the package as a helper function for others (or a variation of it, if you could suggest improvements). If so, let me know and I'll submit a pull request; I'd love to contribute. Let me know what you think.

So I guess now I'm gonna go back to Hakyll and see what solution they would prefer: this fix or using the polling component of fs-notify. I believe this fix is better and lighter, despite the workaround, so hopefully they prefer that. The only thing is that this will inevitably introduce preprocessor checks for a win32 environment.

Hope this helps someone out there.

[gregwebs]

That first link is very informative. It seems that win32-notify should at least have an option to poll the file to wait until writing is done. Does setting a high debounce delay also work?

[blaenk]

I experimented with higher debounce + same delay. Without a delay, it crashed instantly as soon as an event was fired. With a higher debounce (0.5 and 1.0), everything was fine most of the time but eventually one event would trigger the crash. I'm not using this as a 'solution' since the times seem to be pretty arbitrary and may end up being completely different for someone else.

By the way, I was looking into testing the polling 'fallback' functionality in fs-notify, but there doesn't seem to be a way to force its use instead of the native interface. The module doesn't export the sufficient types/functions required. Do you know if there is a way? I figure it'd be a nice option to have to compare the native interface against.

[blaenk]

I just noticed that when I delete a file by either right click -> delete (sends to recycle bin) or shift+delete (bypasses recycle bin, 'permanently deletes'), a 'modified' event is received for that file. I'll have to add a 'file exists' kind of check to make sure it hasn't been removed.

[gregwebs]

@blaenk Thanks for figuring out this issue. To make the changes needed this project needs your help. I was hoping to have time to fix things now, but I just broke my arm. I added you to the haskell-fswatch project where there is a fork of this repo and you have access

[blaenk]

@gregwebs Sorry to hear about your broken arm :( Hope it heals quickly.

Sure, I'll contribute the changes I can come up with. I'm not sure which ones you would like me to implement though.

That last tidbit about a modified event also being received, I believe is the inherent nature of the ReadDirectoryChangesW API and not a fault in win32-notify.

I was originally thinking to at least contribute the waitOpen function or some variant of it. However, I've given that pause for consideration. One consequence of doing this is that it would be specific to Windows. Users of just win32-notify would of course be able to use this normally, but it would require special consideration if it is to be integrated into fs-notify, since it would likely only be useful in Windows. Perhaps it'd be best to implement it in win32-notify, and then users of fs-notify could use preprocessor conditionals or System.Info.os to decide whether to use it?

My point is that integrating it into fs-notify would require special consideration. It wouldn't make sense to stick it into watchTree and friends because that would embed the assumption that the users want to open or read the file itself, which not everyone might want to do (thereby incurring unnecessary overhead). If waitOpen is instead implemented in win32-notify, then users could continue to use watchTree normally, but would then have to use some sort of conditional to determine whether or not they need to use it.

I'm just not sure where or how this should be implemented. What do you think?

[gregwebs]

so this seems like a configuration setting much like the debounce. we can vary the default settings per OS, and even make the setting a no-op on some OSes. Other thing we can do is try to see how these issues are solved in the ruby, python, javascript, and other language variants.