Load Balancer

[zdehasek]

I found this in Fizzy

accessories:
  load-balancer:
    image: basecamp/kamal-proxy:lb
    hosts:
      - fizzy-lb-101.df-iad-int.37signals.com
      - fizzy-lb-102.df-iad-int.37signals.com
      - fizzy-lb-01.sc-chi-int.37signals.com
      - fizzy-lb-02.sc-chi-int.37signals.com
      - fizzy-lb-401.df-ams-int.37signals.com
      - fizzy-lb-402.df-ams-int.37signals.com
    labels:
      otel_role: load-balancer
      otel_service: fizzy-load-balancer
      otel_scrape_enabled: true
    options:
      publish:
        - 80:80
        - 443:443
      # NFS mount for certificates
      # See https://3.basecamp.com/2914079/buckets/37331921/todos/9180260061
      mount: type=volume,src=certificates,dst=/certificates,volume-driver=local,volume-opt=type=nfs,volume-opt=device=:/fizzy-production-certificates,"volume-opt=o=addr=purestorage.sc-chi-int.37signals.com,nfsvers=3,rw,noatime,nconnect=8,soft,timeo=30,retrans=2"
    volumes:
      - load-balancer:/home/kamal-proxy/.config/kamal-proxy

@kevinmcconnell or anybody, can us tell more, please?

[kevinmcconnell]

Sure @zdehasek, I'd be happy to!

We've been working on extending Kamal Proxy's load balancing abilities. Typically when doing a Kamal deployment to multiple app servers, you need something in front of it all that balances traffic between them. By bringing this inside Kamal, it means you'll be able to do an entire multi-server deployment with having to set up load balancing with other tools. This is the approach we've taken with Fizzy, and it's working well so far.

The load balancing options are not yet fully integrated into Kamal, though. By building this out manually first, we've been able to experiment more with how it works before things start to ossify into the configuration. But ultimately (soon, hopefully) this will be fully baked-in, so the entire deployment (including load balancer config) can be specified in your deploy.yml.

In Fizzy's specific case, our setup looks basically like this:

• We run Fizzy out of 3 datacenters: Chicago, Virginia and Amsterdam
• In each DC we run a pair of Kamal Proxy instances as load balancers (they are paired for redundancy)
• We use Cloudflare to route traffic into the nearest DC automatically. So for example, if you're in Europe, your traffic will be served out of the Amsterdam DC.
• Virginia is the primary DC, where the writeable database is; the other 2 contain read replicas. Because you don't want app servers to use databases in a remote DC (because the per-query latency would make that very slow), the load balancers split the traffic into reads and writes. Reads go to local app servers, and writes all go to the primary DC's app servers.
• These load balancers also terminate TLS on the traffic that comes from Cloudflare. We use Let's Encrypt to set that up automatically. The NFS mount you see mentioned above is a bit of shared storage that allows the Let's Encrypt provisioning/renewals to work seamlessly across the multiple load balancer instances.

This setup means that everyone should get the fastest reads, because they use the nearest DC, while writes still land at the primary which is the fastest way to process those. The bulk of requests in an app like Fizzy are reads, so this works out well.

We run our load balancers as Kamal Accessories (that's the snippet you showed above). The settings those load balancers use are managed with a simple script.

This setup probably sounds complicated, but in practice it's been very simple to set up and manage.

And as I say, one of my goals with all of this is that anybody should be able to set up a multi-server, multi-region, load-balanced deployment for their app using just Kamal and a deploy.yml. With a bit of luck we'll have this ready for the next major Kamal release, so that it's super easy to use.

[zdehasek]

Hello @kevinmcconnell, thanks a lot for the detailed explanation.

I've been wondering for a while if you might eventually hit the limits of a single-server deployment, so it's really nice to see you testing and validating this approach with Fizzy.

The idea of enabling a multi-server, multi-region, load-balanced deployment using just Kamal is super exciting. I can't wait to try it out when it's ready.

And also, thanks for Fizzy. It's an awesome product. You all rock!