Diagnosing slow sends

[edasaki]

Hi! I was wondering if I could get some guidance as to the reason behind slow sends.

            long start = System.currentTimeMillis();
            socket.send(message);
            if (System.currentTimeMillis() - start > 15) {
                log.db("Slow send: " + (System.currentTimeMillis() - start));
            }

With the code above, I'm usually having no problems, but occasionally the sends will take 100+ms (as opposed to usually taking 2-5ms.

What are possible causes of this? Is it more likely to be the messages being too big? Or is it because of the actual server that this code is running on having network latency problems?

And would it be safe to send using the same socket on separate threads?

Thank you for any help!

[marci4]

Hello @edasaki,

WebSocket.send() is actually not sending the message right away. It is putting it into a queue (converting it to a ByteBuffer etc already).
So the possible cause for this increased delay could just be a gc call or a heap increase.

Regarding if it is thread safe to send a message to the websocket, it is not right now with version 1.3.4. But it will be with 1.3.5 (see #491)

Hope this clears up your questions!

Greetings
marci4

[edasaki]

Okay, thank you!
I was also wondering, is it possible that a client with bad connection could cause the entire queue to lag? For example if one client has latency of 500ms but all others have latency of 30ms, will the 500ms client cause issues for everyone else? Or does it not make a difference?

[marci4]

Hello @edasaki,

that could be theoretically happening. The write operation of a socket channel is returning when all the bytes have been written.
Another thing you can do is to deactivate the Nagle's algorithm ( see https://github.com/TooTallNate/Java-WebSocket/wiki/Enable-TCP_NODELAY)

Gretings
marci4

[edasaki]

Oh, thanks for linking that, I didn't see it before! Since I am using websockets for a game it seems like a good option to use. Hopefully it will improve things a little bit. Thank you!

[marci4]

Hello @edasaki,

could you keep me posted if slow endpoints are still an issue for you then? If this is the case, we should think about fixing this!

Greetings
marci4

[edasaki]

Sure thing, will do!

[edasaki]

It's been some time but recently I've started having this issue again. I didn't have many users lately so the problem never occurred, but now that usage has gone up a bit I've noticed the problem appearing again. It's strange in that sometimes I can handle high loads with very low latency, but other times even with much smaller loads the latency is very high.

Could it possibly be beneficial to use more than the default number of WebSocketWorkers? I see that by default the number of threads is equal to the number of processors. But are there any blocking operations during the decoding where having more threads than processors could be time-efficient? As I understand it, if one thread is waiting on something then another thread can do some work, so maybe it'd be better to raise the number of threads. I'm not familiar enough with async stuff to be sure. Thanks!

Edit: It looks like each connection is assigned one worker, so I suppose it's not actually a good solution?

[marci4]

Hello @edasaki,

thank you very much for your feedback.
I will look into some possible solutions and will get back to you ok?

This of course will need some refactoring on my part!

Greetings
marci4

[edasaki]

Thank you!

Worth noting that I have since moved the stuff I was doing on my socket server in onMessage() to a separate thread pool, and performance seems to have improved dramatically. So the onMessage() just creates a runnable and submits it to the pool rather than actually doing anything. Doing this seems to have prevented one slow message from slowing down other ones for me at least.

[marci4]

Hello @edasaki,

of course this will speed up everything.
The thread which is calling onMessage is responsible for reading from each connection as well as writing to the connection. If you add additional load by performing any business logic, this will right now slow down everything.
So I will probably add a thread pool for every received message as well as message to write to reduce load of this thread.

Greetings
marci4

[marci4]

Hello @edasaki,
just so you know. I am now working on the threadpool impelementation.

Greetings
marci4