Windows support

[daurnimator]

I've been mulling over how hard adding windows support to cqueues would be.
Lower level windows programming is not something I'm familar with, so please correct any incorrect assumptions I've made.

I'd love to have cqueues be a library I can depend on everywhere: if I write a network client in cqueues, even windows developers should still be able to use it (though maybe not in a performant way, which is the approach many applications take, e.g. nginx).

Comments @wahern has made before:

The concept is not portable to Windows because Windows lacks an analog to the pollable event queues of modern Unix systems. Libraries which try to abstract the two approaches—event readiness signaling versus event completion—invariably produce a pallid least common denominator library suffering all the weaknesses and enjoying none of the strengths of each approach.

Microsoft Windows support is basically out of the question¹, for far too many reasons to put here.
Aside from the more technical reasons, Windows I/O and networking programming interfaces have
a fundamentally different character than on Unix. Unix historically relies on readiness polling, while
Windows uses event completion callbacks. There are strengths and weaknesses to each approach.
Trying to paper over the chasm between the two approaches invariably results in a framework with
the strengths of neither and the weaknesses of both. The purpose of cqueues is to leverage the
strengths of polling as well as address the weaknesses.

¹ I have been toying with the idea of using an fd set in-place of a pollable descriptor on Windows, and taking the union of all fd sets when polling.

---

Solution 1. Add :pollset

One possible route is catering to the lowest common denominator (which helps with other platforms, such as AIX) and use select(). This can be accomplished by iterating over the fileno LLRB and building a pollset; this could be done on demand in a :pollset() function, or inside of fileno_ctl.

Issues

select does not expose a file handle we can wait on itself, and makes things much harder to nest/stack, :pollsets will need to be propogated up to the parent controller.

On windows, select boils down to a WaitForMultipleObjectsEx call, this has a maximum of 64 handles; which could easily be exceeded by a regular application. To get around this, you can spawn extra threads that each wait for events, and wait for them, and now we've recreated stackable event waiting.

---

Solution 2. Use an IOCP and RegisterWaitForSingleObject

• Create an IOCP port for each cqueues object.
• To 'add' a file descriptor to it, use RegisterWaitForSingleObject where the callback uses PostQueuedCompletionStatus
• To cancel a file descriptor, use UnregisterWaitEx
• kpoll_wait uses GetQueuedCompletionStatusEx
• :pollfd() returns the IOCP handle

Issues

Windows potentially has more than just readable and writable events; you need to use WSACreateEvent + WSAEventSelect to manage a HANDLE that encapsulates the events being waited for.

It is not possible to specify different event objects for different network events. The following code will not work; the second call will cancel the effects of the first, and only the FD_WRITE network event will be associated with hEventObject2:

rc = WSAEventSelect(s, hEventObject1, FD_READ);
rc = WSAEventSelect(s, hEventObject2, FD_WRITE); //bad

[daurnimator]

Solution 2.5:

@piscisaureus alerted me to the undocumented IOCTL_AFD_POLL fast polling interface.
This is what select() on windows uses internally, it's not limited to 64 descriptors.
libuv uses it.
This is apparently incompatible with some LSPs, but they're deprecated, so we can probably just not support that case.

[wahern]

On Thu, Mar 26, 2015 at 05:34:46PM -0700, daurnimator wrote:

Solution 2.5:

@piscisaureus alerted me to the undocumented IOCTL_AFD_POLL fast polling interface.
This is what select() on windows uses internally, it's not limited to 64 descriptors.
libuv uses it.
This is apparently incompatible with some LSPs, but they're deprecated, so we can probably just not support that case.

Very cool. Looks like the work originated here

https://github.com/piscisaureus/epoll_windows

according to this thread

http://comments.gmane.org/gmane.comp.lang.javascript.nodejs.libuv/1049

[wahern]

More information about what AFD is: http://mista.nu/blog/2012/02/17/cve-2012-0148-a-deep-dive-into-afd/

[daurnimator]

I had a play around with AFD today, check out this example: https://gist.github.com/daurnimator/63d2970aedc952f0beb3

[daurnimator]

One of the roadblocks I learnt about was that you can't have an IOCP listen for events on an IOCP.
Or at least, I haven't figured it out yet.

• You can't add an IOCP handle to another IOCP
• You can't register waits on IOCP handles

Maybe we can have some arrangement where a worker thread polls GetQueuedCompletionStatus and posts across? I'm not sure...

[daurnimator]

For the pollfd code style to work, it needs to be able to return any type of HANDLE, I imagine we have users yield HANDLEs as lightuserdata?
Once you have a handle you may need to know what to do with it. To find out the type of a HANDLE, you need to use NtQueryObject() with PUBLIC_OBJECT_TYPE_INFORMATION, and dispatch based on TypeName.

[daurnimator]

My current opinion has tended back to Solution 1.
It seems that an array of HANDLEs will be needed no matter what: the windows primitive of WaitForMultipleObjectsEx and the AFD stuff I found both need it.
If we implement :pollset now, we can then special case socket descriptors later to go via AFD.

[daurnimator]

Solution 1. Add :pollset

I played around with adding a :pollset function; + a pair'd function :pollresult over on a branch: https://github.com/daurnimator/cqueues/tree/pollset

[daurnimator]

@katlogic suggested using https://github.com/richardhundt/upoll
But it looks like upoll doesn't implement "stacking"; however it might not be hard to add.

[wahern]

On Mon, Mar 07, 2016 at 06:31:51PM -0800, daurnimator wrote:

@katlogic suggested using https://github.com/richardhundt/upoll
But it looks like upoll doesn't implement "stacking"; however it might not be hard to add.

It's not doing anything that the existing code isn't already doing except
for a dozen or so lines for select. It'd be easier to add select to the
existing code than to add kqueue, ports, etc support to upoll.c. Likewise
for adding select stacking.

I had an idea for supporting Windows overlapped I/O in a performant manner,
but it first involves properly breaking out the buffering code.

But simply supporting select on Windows is probably the best interim
measure. Slightly easier (stacking notwithstanding) would be just supporting
WSAPoll, available since Windows 2008.

[daurnimator]

In midipix there's an interesting approach: for sockets, use AFD; but for other handle types, do a 0 byte read/write using the NT API in a worker thread. These block until there is data available for read/write. At which point you can signal an event. This works out at scale because on NT, canceling io operations is cheap.

[wahern]

On Thu, May 19, 2016 at 09:46:19PM -0700, daurnimator wrote:

In midipix there's an interesting approach: for sockets, use AFD; but for
other handle types, do a 0 byte read/write using the NT API in a worker
thread. These block until there is data available for read/write. At which
point you can signal an event. This works out at scale because on NT,
canceling io operations is cheap.

Can you post a link?

[hanxi]

I suggest use select on windows. Generally in windows are use for client. like https://github.com/daurnimator/lua-http , I want use http client in windows, But now can't.

Here has a epoll in windows by select.

https://github.com/dpull/skynet-mingw/blob/master/platform/epoll.c

[andrewstarks]

Somewhat tangential, but I was reading up on Linux's io_uring API and had the thought that adding support for it in cqueues might make it simpler to also support IOCP, given that there seem to be similarities.

I'm oversimplifying and understand that this would likely be a lot of work. I suggest this because cqueues and Lua-HTTP are my favorite libraries for their purpose and a lack of Windows is a frequent issue.

[andrewstarks]

Looking in the comments, I see that @wahern isn't huge fan of this, so maybe it's not a great fit. :)