Optimize build

[glittle]

What problem does this feature solve?

Between two builds, the entire set of files is recreated, even when nothing was changed. This is very inefficient and poorly affects end users.

As it is now, when 1 character is modified in a simple JS file, the user's browser must re-download the entire website rather than just one of the files from the site. This might be acceptable for a site that is mostly static. However, for a site that is updated frequently, this slows user experience significantly.

The build process should compare its current build with the previous build and only output/replace the old file if there were changes.

This might imply that links must be created in a second pass (recursively?) so that previously generated (unchanged) files can be reused.

What does the proposed API look like?

I'd be happy to see a configuration setting that allowed optimization to be done as it is now or optimized for minimal reloads (as requested above).

[Akryum]

Are you implying there is a bug with the file names with the content hash changing even though the actual content of the file didn't change?

[glittle]

If a file name changes, the browser will be forced to reload it. If the content didn't change, the file name should not be changed.

[Akryum]

Yeah that's the point of vue-cli using content hashes in the file names.

[glittle]

Yes, and that is very good. However, the hash should not be changed when the content has not changed between builds.

[glittle]

For instance, in a "simple" web app with one font, the "chunk-vendors" file is 300 KB and the font files are 3.9 MB. Those are not changing as I work on my project or make a minor tweak to my content.

However, because of the bundling and naming process, those files are re-issued each time the project is built. This slows down all the users of the site and slows down my deployment processes.

[Akryum]

Are you sure the file names change each time, even if the files are re-emitted to the disk?

[glittle]

After further review, I'm happy to report that the file names are the same.

So, this may be easier to solve.

The process deletes the "dist" folder and recreates all the files with the same names. So, the timestamps on the files are new, and deployment processes therefore treat them as new files.

If the build process would not delete and recreate the files, it could check to see if the new file is identical to the previously built file and simply leave the file as is. By the end of the process, any extraneous files could be removed.

[LinusBorg]

If the build process would not delete and recreate the files, it could check to see if the new file is identical to the previously built file and simply leave the file as is. By the end of the process, any extraneous files could be removed.

@glittle While that sounds like an optimization on paper, it won't have any noteworthy effect. 99% of build time is spend with webpack actually creating those files. Writing them to disk only takes a few milliseconds.

And to compare old and new files like you propose, the new files still have to be actually built by webpack.

Walking through all generated files, cleaning up those not originating from the latest built could actually be slower than simply whiping the whole directory before building the new version, so I don't think this is a worthwhile thing to implement.

[glittle]

I'm not interested in the minute or so that is spent building the files.

After those files are created, they need to be deployed to the hosting environment. Everyone's process will be different, and this is where the problem is.

I'm my case, for this project, I deploy to local shared testing location for UAT. From there, the files are deployed to the production environment. IT has processes in place that copy up all changed files and that includes checking for timestamp changes. Because the files are recreated (with the same file names! 😀) they are considered to be new files.

My workaround, until the build process can be optimized, will be to develop a deployment process that ignores each file's timestamp and checks only the file name.

For the build process to do this, I'd suggest leaving the current dist folder alone and build into a temp folder. When done, replace any changed files and remove the temp folder.

[LinusBorg]

I see the requirement that your deployment process creates, in a way, but still don't see why it's a problem if some additional files are deployed to production even though they don't contain any changes.

IT has processes in place that copy up all changed files and that includes checking for timestamp changes. Because the files are recreated (with the same file names! grinning) they are considered to be new files.

If it has processes set up to do this automatically, why is this a problem? Does it take significantly longer to copy a dozen or even a hundred more files than required?

Don't get me wrong, I totally believe that this is a challenge for you in some way, I just have problems understanding in what way, and consequently can't see why we should implement this, making the build process more complicated and brittle in the process.

[glittle]

In one case, there was a multi-megabyte sized file that had to be uploaded over a slow link from the dev environment to the production environment. Because all changed files were copied, it was being recopied each time, making the deployment noticeably slower.

In light of this discussion, I'm now manually using a tool (Synchronize It!) to deploy files as selected by changes to filename/content and ignoring the timestamps on the files. This doesn't allow a good scripting solution but is acceptable for small projects.

[glittle]

I'd still recommend building the dist into a temp folder rather than wiping the dist folder at the start of the process. This makes the system a bit more robust, as the dist folder is not affected if the build fails. It would then be a "simple" process to compare the two folders and update the dist folder to match the temp folder, adding/removing/updating files, but not touching any files that have not been changed.

Or, if the output files can be managed in memory before writing to the output location, each file could be compared to the existing file and if there are no changes, it can be discarded leaving the old file untouched. This would speed the process and eliminate the need for a temp folder.

The end result is that the dist folder will eventually have files in it that have old timestamps that indicate when they were last modified.