Repository navigation
Terminating build part way through book leaves many intermediate files lying around #986
Description
Activity
By Terminating build part part way through book, you are meaning stopping the build ? Or after an error occurs ?
I'll need to try reproduce this in order to see what is happening.
BTW there is a
bookdown::clean_book()function available - but this may not be for the files you are seingGenerally, I mean pressing cmd + c. I suspect the problem is that the cleanup code is just run after the book is successfully built, rather than being wrapped in an
on.exit().There are a bunch of
on.exit()call already. For example, the one for the rds file:
Lines 168 to 169 in 7ace59f
I'll try to reproduce and see why the cleaning is not happening
In theory, these files should be move/cleaned since I did use
on.exit()to do that. Unless R crashed very badly, it shouldn't leave the intermediate files in the root dir.Hmmm, weird. I double checked that you always used
add = TRUE(since that's another easy mistake) but it looked ok to me.Another thing worth checking if it you're using relative paths and the working directory changes at some point.
Yes in theory they should be removed. However, while investigating #920 , I stumble upon this issue with intermediary md files still present. If there is an error in the book (using
new_session: true), the rendering will stop but there still be the .md file present in the directory. They are not deleted.Reprex:
dir.create(temp_proj <- tempfile()) old <- setwd(temp_proj) bookdown:::bookdown_skeleton('.') #> [1] TRUE files1 <- fs::dir_ls() files1 #> 01-intro.Rmd 02-literature.Rmd 03-method.Rmd 04-application.Rmd #> 05-summary.Rmd 06-references.Rmd book.bib index.Rmd #> preamble.tex README.md style.css _bookdown.yml #> _output.yml content <- xfun::read_utf8("02-literature.Rmd") content <- c(content, "```{r}", "stop('Will be an error')", '```') xfun::write_utf8(content, "02-literature.Rmd") yml <- yaml::read_yaml("_bookdown.yml") yml <- c(yml, list(new_session = TRUE)) yaml::write_yaml(yml, "_bookdown.yml") bookdown::render_book("index.Rmd", "bookdown::gitbook") #> Error in Rscript_render(f, render_args, render_meta, add1, add2): Failed to compile 02-literature.Rmd files2 <- fs::dir_ls() setdiff(files2, files1) #> [1] "01-intro.md" "01-intro.utf8.md" "02-literature.knit.md" #> [4] "index.md" "index.utf8.md" "packages.bib" #> [7] "_bookdown_files" setwd(old) unlink(temp_proj)
However, the
.rdsfile is not present so I guess there is some place where this does not go well and other (like which rds) which are ok.I'll double check this tomorrow, specificaly the relative path.
Reacted by Julia SilgeI often have extra files left when the build fails because of an error, fwiw. (I make a lot of errors. lol)
The problem is that the intermediate files are left behind by:
for (f in files[rerun]) Rscript_render(f, render_args, render_meta, add1, add2)
And those keep intermediate files because
cleanis explicitly set toFALSE:saveRDS( list(output_format = output_format, ..., clean = FALSE, envir = envir), render_args )- The intermediate file
render*.rdsis cleaned up atIf R is terminated by clicking the redLine 191 in ca0145f
Stopbutton in RStudio, for some reason thison.exit()is not called. @hadley I've verified that even with your fix in Improved intermediate cleanup #1079, the.rdswill still be left inside thetempdir(). We need to figure out why the Stop button in RStudio kills R so brutally. - For the intermediate
*.mdfiles, they are left there if a certain Rmd fails to compile, but they will be cleaned up again the next time after you fix the error in the failing Rmd document and recompile it:Lines 222 to 223 in ca0145f
- The intermediate file
I find that I am left with many intermediate files (
.mdfiles,render.rdsfiles, etc) when a bookdown render fails due to an error and that I have to manually delete them, or else the next time I try torender()it fails again. I have assumed this is because of being in an uncomfortable cached situation.I am not clicking the red 🛑 RStudio button; the render is erroring out part of the way through.
I find that I am left with many intermediate files (
.mdfiles,render.rdsfiles, etc) when a bookdown render fails due to an error and that I have to manually delete them, or else the next time I try torender()it fails again.Hmm... After you fix the error in the Rmd, bookdown should clean up the intermediate files the next time you build the book. I wasn't able to reproduce this problem the last time, but let me try again.
Unfortunately I still can't reproduce the problem. Here I add an error chunk to
index.Rmd:d = tempfile() dir.create(d) owd = setwd(d) bookdown:::bookdown_skeleton('test') setwd('test') cat('new_session: true\n', file = '_bookdown.yml', append = TRUE) cat('```{r} stop("Error") ``` ', file = 'index.Rmd', append = TRUE) bookdown::render_book()
Rendering
index.Rmdwill fail, and leave an intermediateindex.knit.md:cat(list.files(), sep = '\n')
_bookdown.yml _output.yml 01-intro.Rmd 02-literature.Rmd 03-method.Rmd 04-application.Rmd 05-summary.Rmd 06-references.Rmd book.bib index.knit.md # <- the intermediate file index.Rmd packages.bib preamble.tex README.md style.cssThen I remove the error chunk and re-render:
xfun::process_file('index.Rmd', function(x) head(x, -3)) bookdown::render_book() cat(list.files(), sep = '\n')
_book _bookdown_files _bookdown.yml _output.yml 01-intro.Rmd 02-literature.Rmd 03-method.Rmd 04-application.Rmd 05-summary.Rmd 06-references.Rmd book.bib index.Rmd packages.bib preamble.tex README.md style.css test.rdsThe intermediate
.knit.mdfile is gone.Same thing when building the book with the RStudio IDE.
So I still need a reproducible example...
Let me see if I can put one together soon; I think the problem may be related to how caching is working when some chunks depend on something that has errored.
Reacted by Yihui XieTo help with this, @apreshill suggested to look into https://github.com/rstudio-education/stat545 where this problem was also encountered while collaborating on the book.
Well, I spent some time today trying to generate this type of problem in a small bookdown repo, and I was not successful. 😔 The small bookdown would always build successfully after I removed the error (even playing with caching and chunk dependencies).
Unfortunately I can't offer a small reproducible example of the problems I experienced for now, but I will update you all in the future if/when I come upon it again!
Thank you anyway! When we resolve the rmarkdown issue rstudio/rmarkdown#2086, this bookdown issue should also be resolved.
@yihui I have took some time to look into this issue.
A reminder: there was a weird interaction between rlang and bookdown in #920
It required a fix in rlang and it is adviced to use rlang 0.4.9+ if you setoptions(error = rlang::entrace)in . The initial issue was causing wrong exit and, even if I am not sure it could be linked, it worth checking the rlang version if you have a fixed one because using renv.Based on your examples, and a more complex project like mentioned in #986 (comment), I went further.
What I did is I added an error in the first chunk of one of the file, not the first. The file with the error has a.knit.mdleftover where others have*.ut8.mdleftover. It means each rendering withnew_sessionis not cleaned up as it should for this file. About,*.utf8.mdit is solved by rstudio/rmarkdown#2086 and this file won't exist anymore. Initially I think those files were removed because the clean call has not been made andon.exithandler for those has not been registered yet.But there will still be the intermediates .md files as the merged has not happened yet (in knit and merge approach)
d = tempfile() dir.create(d) owd = setwd(d) bookdown:::bookdown_skeleton('test') setwd('test') cat('new_session: true\n', file = '_bookdown.yml', append = TRUE) cat('```{r} stop("Error") ``` ', file = '05-summary.Rmd', append = TRUE) bookdown::render_book('.') xfun::raw_string(list.files())
_bookdown.yml _bookdown_files _output.yml 01-intro.md <-- unmerged md file 01-intro.Rmd 01-intro.utf8.md 02-literature.md <-- unmerged md file 02-literature.Rmd 02-literature.utf8.md 03-method.md <-- unmerged md file 03-method.Rmd 03-method.utf8.md 04-application.md <-- unmerged md file 04-application.Rmd 04-application.utf8.md 05-summary.knit.md <-- error during knitting 05-summary.Rmd 06-references.Rmd book.bib index.md <-- unmerged md file index.Rmd index.utf8.md packages.bib preamble.tex README.md style.css test.rds <-- is this expected ?It seems currently those
.mdfile are moved to the output folder when no error.xfun::process_file('05-summary.Rmd', function(x) head(x, -3)) bookdown::render_book(".")
> xfun::raw_string(list.files()) _book _bookdown.yml _bookdown_files _output.yml 01-intro.Rmd 02-literature.Rmd 03-method.Rmd 04-application.Rmd 05-summary.Rmd 06-references.Rmd book.bib index.Rmd packages.bib preamble.tex README.md style.css test.rds > xfun::raw_string(list.files("_book")) 01-intro.md <-- moved 01-intro_files 02-literature.md <-- moved 03-method.md <-- moved 04-application.md <-- moved 05-summary.md <-- moved 06-references.md <-- moved applications.html final-words.html index.html index.md intro.html libs literature.html methods.html reference-keys.txt references.html search_index.json style.css
I believe on suggestion in #1079 is to remove the
.mdfiles on error during the process left because the on.exit clause where not called. Same asutf8.md, theon.exit()handler has not yet been registered for those file to be moved.Why are those .md files moved to output dir by the way ? I am not sure to follow the purpose.
If an error occurs during
Rscript_render(), it seems ok to remove all those unused file which currently does not happen. As suggested in https://github.com/rstudio/bookdown/pull/1079/files#diff-0d68ae244e4a8c5ddc391c48f5399a91f18bc2a955f098e0e66069c607b444f4R210
Would that be ok ? It could depend on why we move them.render*.rdsfiles are fromrender_argssave and should be clean whenrender_new_session()exists. I can't yet reproduce this file to still be present on error. If something cause this, it is during the mainrender().I believe most of the issue here is to be sure the correct
on.exit()handler will be register as soon as possible, or should be moved closed to where the intermediates happens. Currently we need to do a lot of cleaning because we useclean = FALSEin the call torender()usingRscript_render()- is callingclean = TRUEhere an issue ?Reacted by Alison Presmanes Hill
e.g. all of the
.mdfiles of the chapters built so far, and a number ofrender*.rds