Server: Take less memory during Webpack build

[samuel-duhaime]

Issue

Problems

During an OVH maintenance restart, all our servers went down, because we had memory problems.

Translate from @greenscientist

The problem is the Webpack step at launch. This takes like 3gig of ram per pod, so we don't have enough memory for it all to run at the same time. I can try to find a way to prevent webpack from running on too many pods at the same time on the infra side, but if on your side you can see if there is a way to reduce the memory used by this step that would help too

Description

Take less memory during Webpack build

[greenscientist]

Here's a few idea from meta.ai:
is it possible to reduce how much memory webpack use when it runs
Yes, it is possible to reduce the memory usage of Webpack when it runs. Here are some strategies to help minimize Webpack's memory footprint:

1. Optimize Node.js memory usage
   Update Node.js to the latest version, as newer versions often include memory-related improvements.
   Set the --max-old-space-size flag when running Webpack to limit the heap size. For example: node --max-old-space-size=2048 node_modules/webpack/bin/webpack.js
2. Reduce the number of resolved modules
   Limit the number of dependencies: Minimize the number of dependencies in your project.
   Use resolve.modules: Specify the directories where Webpack should look for modules, reducing the number of directories to search.
   Use resolve.mainFields: Specify the fields to look for in package.json files.
3. Minimize plugin usage
   Remove unnecessary plugins: Only use plugins that are essential for your build process.
   Use lighter alternatives: Choose plugins with smaller footprints.
4. Optimize loaders
   Use include and exclude: Limit the files processed by loaders.
   Use loader options: Optimize loader settings, such as babel-loader's cacheDirectory option.
5. Use caching
   Use cache option: Enable Webpack's caching to store compiled modules.
   Use hard-source-webpack-plugin: A more efficient caching solution.
6. Split your build
   Use webpack-merge: Split your configuration into smaller files.
   Use multiple configurations: Run separate builds for different parts of your application.
7. Monitor memory usage
   Use memory-inspector: A tool to inspect Webpack's memory usage.
   Use node --inspect: Inspect Node.js memory usage.
   Example configuration
   JavaScript
   // webpack.config.js
   module.exports = {
   // ...
   resolve: {
   modules: [path.resolve(__dirname, 'src'), 'node_modules'],
   mainFields: ['main', 'module'],
   },
   module: {
   rules: [
   {
   test: /.js$/,
   include: path.resolve(__dirname, 'src'),
   use: {
   loader: 'babel-loader',
   options: {
   cacheDirectory: true,
   },
   },
   },
   ],
   },
   plugins: [
   // Only essential plugins
   new webpack.optimize.ModuleConcatenationPlugin(),
   ],
   cache: {
   type: 'memory',
   },
   };
   By implementing these strategies, you can reduce Webpack's memory usage and improve build performance.

[tahini]

Not really related to the build itself, but more to the content, here's what our builds look like (in od_nationale) with the help of https://github.com/webpack-contrib/webpack-bundle-analyzer

Here's a couple of general idea. Our json/geojson files maybe should not be part of the builds but could be downloaded on demand if necessary.

Also, why do we have 2 different versions of free-solid-svg-icons? (I suppose it's 2 different ones, but they do take a lot of space!)

I think the locales include all languages preloaded. We should preload the language on demand only and fetch the locales file as necessary, like the configuration that we have in Transition (though this one also downloads all files for all langues).

Why turf-jsts.min.js and another @Turf block on the left?

[samuel-duhaime]

@tahini
Are we importing fortawesome free-solid-svg-icons twice? It seems we are, both in the evolution-generator and node_modules.

[samuel-duhaime]

@greenscientist Did we fix this issue?

[tahini]

@samuel-duhaime not really, we mitigated by capping the memory at build time, but did not change anything in the build process itself, which means it's still building the whole too big application in one package.