Extended ElementData Sync

[Pirulax]

Trying a lot of different techniques to sync data between server and clients i'm always facing bandwidth and cpu drops on server-side or fps-drops on clientside.

Summarizing my experience, i would like to suggest a feature, something like

setElementDataEx(element, key, value, syncType=normal, syncTargets = root, forceSync=true)
forceElementDataSync()

This function should work in conjunction with the classic elementData, but the difference is how things will be synchronized.

syncType:

1. only players listed in syncTargets will receive the update. if new player joins or reconnects the server, the date is hidden for him, but if same key and same value will be set targeting to this player, he should receive the update.

2. normal - direct update will receive only syncTargets, sets data as public, if new player join the server, he should receive this data as well.

3. stream - this data should be synchronized only when player streams the element or element already streamed in. syncTargets is ignored here, but can be used to filter players who should receive data when element streamed. When element streams out, the data should stay in memory, so next time only delta will be synced. (it's a little bit tricky, we need to store some timestamp of last sync, but it will save a bandwidth and cpu.)

forceSync:
if true - data should be directly sent like setElementData, if false - the MTA should push it into buffer and collect all changes until current scope ends or method forceElementDataSync() triggered. This means the data should be synchronized not like many single packets, but one batch operation, or can be splitted in to few buckets for better shipping and client-side processing. The order of operations should be keeped. If newly set value overlaps previously set one in pending buffer, it should be overwritten.

forceElementDataSync() - forces operations buffer to be synced.

p.s. i know that most of it can be done on lua side, but there is a few limitations with setElementData without sync, which forces you to keep a duplicated element data tree. Actually i already did most of it in lua, but some of operations causing FPS drops due to lua quickness, so i think it should be implemented closely to core :)

From mantis.

[Einheit-101]

It would be more important to fix incredible CPU usage of setElementData() and include a feature that prevents multiple calls of setElementData with the same data. You can use 20x setElementData(root, "test", true) in 1 frame and the server (And all clients) will receive the same "true" element data 20 times while it would only be necessary to send 1 (final) element data with 1 "true" once the current Lua frame is finished.

[Pirulax]

Element datas should be stored in a buffer and synced by a specified interval(100ms or so, I use 1500ms in my Lua integration of this thing), that'd solve the problem you described.

[ccw808]

@Einheit-101 setElementData only sends the data if it has changed

[haron4igg]

@Pirulax oh thanks mate for moving it from mantis 👍

[Pirulax]

Alright, so.
What I'll do is I'll simply add syncTo, adding buffer isn't necessary, since I don't think I'd make any sense(it only'd if you're stupid, and use elementData for a 3rd thing which it isn't intended to)

[qaisjp]

3. stream - this data should be synchronized only when player streams the element or element already streamed in. syncTargets is ignored here, but can be used to filter players who should receive data when element streamed. When element streams out, the data should stay in memory, so next time only delta will be synced. (it's a little bit tricky, we need to store some timestamp of last sync, but it will save a bandwidth and cpu.)

In my original comment I suggested the type and targets arguments should be merged. I don't think it should be merged anymore/

Also, this should just be added to the existing setElementData. The current synchronize argument (in setElementData) can be replaced with an argument type that accepts both booleans and enum strings.

I think triggerClientEvent would work fine to send data to just certain users, but I agree that triggering an event to just do setElementData is a little pointless. Is there a measurable difference in performance? If there isn't a measurable difference, is there any point doing exactly this for scripters?

addEvent("setClientElementData", true)
addEventHandler("setClientElementData", root, function(key, value)
    setElementData(source, key, value, false)
end)

What I do think, however, is that there might be some value in type="stream".

If there's value in targets alongside type="stream", we may as well implement targets in the general case. Conversely, if targets is implemented in the general case, it must work with type="stream".

---

I also mentioned that the forceSync arg should be removed, but that doesn't make sense as it implies that data is batched by default.

I think batching (forceSync) should be done as part of separate issue/feature/pull request. The new issue can include:

• the forceSync=true argument should be renamed to batch=false
• forceElementDataSync() renamed to sendBatchedData()
• discussion of whether or not batch=true should auto send batched data (maybe you want certain data to send immediately, and other data to be sent in one go?)
• discussion of feasibility, since you may as well just use a table instead of multiple keys/values
• discussion of replacing batch=bool with batch=batchObject, so that you can have multiple batches being composed in parallel? You don't really want to keep track of whether certain function calls will send of batches you are currently composing, and this would fix that problem.

[WADmitry]

I often need the following cases:

1. Synced data between server and only one client (one-side direction, from server to client)
2. Same, but two-side directions, from server to client and from client to server)
3. Single time set value from server to an one client (or group of targets) (unsynced)

So, something like

setElementDataExt( player, "key", value, sync=false, clientChanges=false )
setElementDataExt( { player, vehicle }, "key", value, sync=false, clientChanges=false )

where sync is "keep data synced" (of course it should send data only when it has been changed) and clientChanges is "accept chages made on client side".

I will be happy if this is supported.

[Pirulax]

Implementing several batches is a very good idea, kudos for you.

However, how would we implement target sync type?
Maybe by just sending out a dictionary, looping thru that dictionary and setting a bHasDataToBeSyncedInOnStream variable to true on every player in the packet?

Then onClientElementStream in check if the player has data to be synced in and then sync it it.
But I think it'd cause errors in scripts that try to get the value immediately after the element has streamed in, so we'd need to delay the event and only trigger it after we've retrieved the data.

Replying to you WADmtry:

1. Can be implemented
2. Can be implemented as well, but i find it pointless, theres triggerServerEvent for that.
3. Same as with 2.

[WADmitry]

@Pirulax Can be implemented as well, but i find it pointless, theres triggerServerEvent for that.
Sure. All of these things can be implemented using events and some code (including all from the your issue). So I'm talking about simplier life for developers.

There are some points why I would prefer element data instead of event. For an example:

• It should automatically wait when the element on client-side will be ready and set the value after it. For example, I created an element (vehicle, player, etc) on the server side and exactly after it I define some element data. But for some reason (lags, etc) it is created on client side a few later than event was received. So if I use events then I have to complicate architecture (create element, then wait event from client about it ready, then set data on server side, trigger event and only then set data on client side) (actually this is exacly what I do now). Would be better if the engine do it (because MTA already do this for synced data, as I understand).
• Consistency. It's simpler to use just one function to set element data instead of setElementData on server + event + setElementData on client.

[ccw808]

For example, I created an element (vehicle, player, etc) on the server side and exactly after it I define some element data. But for some reason (lags, etc) it is created on client side a few later than event was received

@WADmitry Please add a new issue with an example on how to reproduce this problem