Set higher default ulimit in Docker Compose/K8s deployments

[dadlerj]

Reported by https://app.hubspot.com/contacts/2762526/company/947819054

They are deployed using Docker Compose

We had a user receive the following error from the following search “lang:typescript type:commit after:"1 week ago"”

1 error occurred: 

 commit search failed: failed to search commit log <REPO REDACTED>@: git command [git log --no-prefix --max-count=31 --extended-regexp --regexp-ignore-case --since=1 week ago -z --no-abbrev-commit --format=oneline --no-color --source --no-patch --no-merges --extended-regexp --max-count=500 --] failed (output: ""): open /dev/null: too many open files (stderr: "")

They can reproduce reliably using structured search, and users have seen it intermittently since then.

This issue coincides with a lot of new users coming on to Sourcegraph at the company.

We are debugging and helping them set higher ulimit, but we should update defaults to prevent this.

[emidoots]

Quick and easy way to check that ulimit is set correctly, run the following command:

docker run -it --rm alpine sh -c "ulimit -S -n && ulimit -H -n"

This should show two numbers - both of which MUST be 262,144 or greater. Quick assessment of Linux distros:

• EC2 Amazon Linux 2 does not have this configured out of the box correctly, and https://docs.sourcegraph.com/admin/install/docker-compose/aws sets it too low for any really large deployments.
• Ubuntu 18.04 with the latest docker-ce package defaults to 1,048,576 - plenty enough. So our GCP and DigitalOcean guides are both OK.

This means we should:

1. Update the EC2 guide to use 262,144
2. Add monitoring which fires a critical alert if ulimit reports lower, so that existing EC2 admins know this is a problem on their instance. The solutions doc should indicate how to solve and test this clearly in Amazon Linux 2 at the least.

[emidoots]

Additionally, all guides seem to be wrong with setting fs.inotify.max_user_watches. To confirm it is set properly, run:

docker run -it --rm alpine sh -c "sysctl -n fs.inotify.max_user_watches"

This should show at least 128,000, otherwise search indexing may fail.

To configure this and have it persist across machine restarts, you must:

echo -e "\nfs.inotify.max_user_watches = 128000\n" >> /etc/sysctl.d/local.conf

Restart the machine and confirm it applied.

We should similarly add a monitor / critical alert for this.