INFINITY mode

[mhdsbq]

[pshenok]

@mhdsbq Sorry for the late reply — only catching this now while going through old discussions.

Honestly, this is both impressive and useful for us. You're clearly reading the failure modes right (100% rep for 80 minutes is no joke). But it also exposes a real problem on our side — once an architecture stabilizes past the ×4 RPS milestone, the simulation just doesn't have anything left to throw at it. "INFINITY mode" is the right name because that's literally what happens.

Quick question if you remember — at roughly what minute did you stop adding/changing services and the architecture just started farming on its own?

I'm working on late-game balance, but it's still on the to-do list. Thanks for sharing — and for being patient with the reply.

[mhdsbq]

Hi @pshenok, Happy to know that you are still working on improvements.

Answering your question - Most of the load was handled by single 10 compute cluster. But during surge/outage it was crashing. 2 clusters was more than enough and the 3rd one was just fancy.

I have a few more observations

1. API GW and Queue was a downgrade on availability
2. Anything more to this setup did't really add any value. Replica and Search was not even getting hit

Since i am noob with cloud architecture this helped me a lot in visualizing an actual environment.

One thing i would really like to see is product scenarios. Right now we can just create a compute connect it to some db and it works. Game does't care about where the actual Data is. I kind of roleplayed by assuming each of these cluster is a microservice. It would be amazing if we can think like we are bulding some platform may be netflix or whatsapp or something. Then the request types will be custom may be we need to label each service ourself idk.. But that would really help me to learn proper system design.

[pshenok]

Thanks for the detail — really useful.

The API GW + Queue being a downgrade is the most important signal here. They're supposed to soften failures, not hurt availability. Same with Replica/Search not getting hit — auto-routing probably prefers cheaper paths over specialized ones. Both turned into tracked issues just now: #166 and #167.

The product scenarios idea is great. Campaign Mode is moving in that direction, but yours is more ambitious — custom request types and naming your own services ("User Service", "Recommendation Engine") would make for an amazing next chapter. Worth opening as a separate discussion if you'd like to flesh it out — happy to brainstorm.

[pshenok]

@mhdsbq

[pshenok]

Update: both issues shipped in #169.

• API Gateway tier ceilings scaled up so T3 caps at 200 RPS instead of 80 — should stay useful into the ×4 milestone instead of throttling you off the cliff.
• Compute now prefers Search Engine over Cache for SEARCH (cache hit rate on SEARCH is only 15% — it was mostly wasted latency), and prefers Read Replica when Cache load goes above 60% instead of piling everything through a saturated Cache.

Your INFINITY-mode run drove both changes. Thanks for the writeup.