What kind of networking should X game use?

[erlend-sh]

Once again I've found myself linking to this excellent networking explainer when someone making a multiplayer game showed up with questions about best practice. Having it stored on an unindexable Discord server within a thread does it a disservice, so I'm gonna repost it here in hopes that it can soon make its way into a documentation page.

From Joy @maniwani:

What kind of networking should X game use?

IMO answering these three questions can decide for you.

1. Deterministic or authoritative?

Think about:

• How complex is the simulation? How many entities need to be replicated?
• Is there hidden information? Is it important that nobody can access it before they're supposed to?
• Do any of the libraries you want to use introduce non-determinism?

With deterministic replication, clients exchange inputs (directly or indirectly) and independently simulate the world.

With authoritative replication, each element of the game state has an owner who is its source of updates. Usually, the server owns everything ("server auth") and clients only send inputs to it, but ownership can be distributed among clients ("client auth"), and then they can send state as well.

Let's call 'em DR and AR.

If you're making an RTS game, consider DR. Players might control hundreds of units at once. AR would need lots of packets to exchange information about all of them, but bandwidth is often limited. DR only has to exchange inputs, no matter the number of entities, so DR is generally better for games with lots of networked entities. (There are RTS games that use AR though.)

Some exceptions to that heuristic would be games with a massive number of players (in the same session) and games with massive, open-world environments. With DR, each device generally has to simulate the entire world. I don't think anyone can run all of Teyvat at 60 FPS. AR more easily allows for cherry-picking what to send to and simulate on each client, making it the better choice in those scenarios.

For the same reasons, AR is generally better for games that require hidden information. If you're making an online poker game, it'd be really bad if you could use a program to leak another player's hand before they reveal it. DR is probably the wrong choice here, since every player will have the entire game state in memory. That said, DR could work if you are careful to send hidden information as input only, and only when it's ready to be revealed.

Determinism is hard to guarantee

In practice, before you compare DR and AR, you should probably ask yourself if DR is even a feasible option.

Dependencies

To be deterministic, the functions you use in your simulation should always return the same results given the same arguments. Ideally, those functions are also pure/stateless, i.e. all of their arguments are explict. Being stateful is OK if you can access, modify, and clone that state.

Unfortunately, oftentimes you can't.

Many physics libraries, for example, have internal data structures to speed things up (alongside the colliders, bodies, and joints supplied by the user), like BVHs for the collision detection and islands for the constrained dynamics. PhysX, Bullet, and Box2D are some of the most popular libraries, yet none of them (or at least the game engines that integrate them) seem to expose their internal state, which makes them incompatible with DR when you also want prediction since you can't rollback properly. (This is inconvenient for AR as well.)

Math

Floating-point math is a notorious source of non-determinism.

• Floating-point math operations round their results (because precision is finite), so floating-point math is non-associative. (a + b) + c and a + (b + c) are not equivalent.
• IEEE-754 exactly defines the binary representation of floating-point numbers and how addition, subtraction, multiplication, and division should be calculated, so they (should) match on all platforms. However, it does not constrain a bunch of other operations (e.g. sqrt, sin, cos, tan, exp, log, etc.), so hardware implementations of those operations can differ.
• Compilers basically don't comply at all with IEEE-754 on NaN values.

But good news! It is possible (apparently) to address those issues. If you can:

• Make sure the compiler can't mess with your floating-point expressions.
• Make sure the rounding mode is consistent.
• Mostly stick to add, subtract, multiply, and divide.
• Source any transcendental functions (e.g. sqrt, sin, log, etc.) from a library like libm (piecewise polynomial approximations) instead of std (hardware intrinsics) to get consistent results.
• Never rely on the exact bits in a NaN value.

Then you should get results that are consistent cross-platform.

2. To predict or not? And how much?

Think about:

• Does the game run in real-time?
• Is visual weirdness from mispredictions OK?
• Is visual weirdness from lag compensation OK?
• What can clients afford to simulate?

Nobody likes lag, so if your game runs in real-time, your game clients should try to give players input feedback as quickly as possible. Prediction (rollback) is the most common way to hide the network round-trip time. Not the only way though.

On one hand, fighting game players love rollback and want it in every game. On the other hand, mispredictions can look glitchy. RTS games often run in lockstep (no prediction) but add artificial input delay to give all players just enough time to exchange inputs without stalling (hence "delay-based netcode"), then they try to hide that delay with animations.

DR clients mostly predict everything or predict nothing. AR clients more commonly predict some things and not others, with the server doing "lag compensation" to resolve interactions between predicted and non-predicted entities. Very common in FPS games but causes those "How did that hit connect?!" moments.

Predicting everything really helps in fighting games and physics-heavy games like Rocket League because players can react to what's happening and interact with objects (i.e. slam into a giant, flying soccer ball) at the same time without any weirdness, but it's more expensive.

3. Client-server or peer-to-peer?

Think about:

• Just how vague networking terminology is.

If you're asking in the sense of "star network or mesh network?", you really can't go wrong with the star network topology. Mesh networks:

• Consume more bandwidth (more connections, more data). Some consumer-grade internet plans may not have enough.
• Encounter more NAT/firewall issues (more connections, more problems).
• Tend to make replication more complicated by introducing edge cases that inspire complex consensus models to resolve.

If you're asking in the sense of "dedicated server or client host?", that's more of a money question. The server is just a role. Whether that role is performed by a player's machine or a datacenter rack shouldn't affect any code. You would need something to do NAT traversal (e.g. ICE, STUN, TURN) to support clients hosting (until IPv6 becomes the dominant protocol), but other than that, it's all the same.

Also: Netcode Terminology

[lokimckay]

Super clear, fantastic resource thanks for posting.

RE mesh network bandwidth: "Some consumer-grade internet plans may not have enough"
Does this also apply to DR (only inputs sent)?

I suppose it would depend on the frequency and volume of inputs, but I'm not sure what a good rule of thumb would be to consider transitioning from mesh -> star.