root_agent: system_config: Handle maxstdio parameter to increase limit of opening files at once

[cosmo0920]

Not sure this feature is wanted in Fluentd but Fluent Bit side already implements this type of operation on Windows version of that package.
This is parity patch for supporting increasing the number of limit of opening files at once.
ref: https://github.com/fluent/fluent-bit/pull/9707

This should be used for rescuing Fluentd process on Windows, just like under too many open files situations.
Once entering those kind of situations, there is no way to recover to normal processing situation.

Note that this PR is just proofing a concept of implementing increasing limit of opening files on Windows.

Which issue(s) this PR fixes:
Fixes #

What this PR does / why we need it:

Docs Changes:

Release Note:

[daipom]

Thanks for this improvement!

Rebased to the latest master to include Windows CI fix.

[daipom]

9fd9c0f

I have added some tests to check this behavior. (They are incomplete, at present.)
It appears that maxstdio does not work, at least for File.open of Ruby.
It appears to be determined by the C-runtime, not maxstdio.

This means 8189 is the max number independent of maxstdio for Ruby 3.2.2 (x64-mingw-ucrt).

$ ruby --version
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x64-mingw-ucrt]
$ bundle exec rake test TESTOPTS="-t'SupervisorTest' -n'test_maxstdio'"
(...)
Started
E
==============================================================================================================================================================================================================================================
Error: test_maxstdio[Default](SupervisorTest): TypeError: no implicit conversion from nil to integer
C:/Users/reang/Documents/work/fluentd/fluentd/lib/fluent/win32api.rb:38:in `call'
C:/Users/reang/Documents/work/fluentd/fluentd/lib/fluent/win32api.rb:38:in `_setmaxstdio'
C:/Users/reang/Documents/work/fluentd/fluentd/lib/fluent/supervisor.rb:762:in `setup_maxstdio'
C:/Users/reang/Documents/work/fluentd/fluentd/test/test_supervisor.rb:1173:in `test_maxstdio'
     1170:     omit "maxstdio is only for Windows" unless Fluent.windows?
     1171:
     1172:     supervisor = Fluent::Supervisor.new(cl_opt)
  => 1173:     supervisor.setup_maxstdio
     1174:
     1175:     files = []
     1176:     begin
==============================================================================================================================================================================================================================================
8189
E
==============================================================================================================================================================================================================================================
Error: test_maxstdio[Small](SupervisorTest): Errno::EMFILE: Too many open files @ rb_sysopen - C:/Users/reang/Documents/work/fluentd/fluentd/test/tmp/supervisor/1cf5e97860efefb5ad17/8189.txt
C:/Users/reang/Documents/work/fluentd/fluentd/test/test_supervisor.rb:1178:in `initialize'
C:/Users/reang/Documents/work/fluentd/fluentd/test/test_supervisor.rb:1178:in `open'
C:/Users/reang/Documents/work/fluentd/fluentd/test/test_supervisor.rb:1178:in `block in test_maxstdio'
     1175:     files = []
     1176:     begin
     1177:       10000.times do |i|
  => 1178:         file = File.open(File.join(@tmp_dir, "#{i}.txt"), "w")
     1179:         files.append(file)
     1180:       end
     1181:     ensure
C:/Users/reang/Documents/work/fluentd/fluentd/test/test_supervisor.rb:1177:in `times'
C:/Users/reang/Documents/work/fluentd/fluentd/test/test_supervisor.rb:1177:in `test_maxstdio'
==============================================================================================================================================================================================================================================
8189
E
==============================================================================================================================================================================================================================================
Error: test_maxstdio[Large](SupervisorTest): Errno::EMFILE: Too many open files @ rb_sysopen - C:/Users/reang/Documents/work/fluentd/fluentd/test/tmp/supervisor/ce86d182a864a4517d09/8189.txt
C:/Users/reang/Documents/work/fluentd/fluentd/test/test_supervisor.rb:1178:in `initialize'
C:/Users/reang/Documents/work/fluentd/fluentd/test/test_supervisor.rb:1178:in `open'
C:/Users/reang/Documents/work/fluentd/fluentd/test/test_supervisor.rb:1178:in `block in test_maxstdio'
     1175:     files = []
     1176:     begin
     1177:       10000.times do |i|
  => 1178:         file = File.open(File.join(@tmp_dir, "#{i}.txt"), "w")
     1179:         files.append(file)
     1180:       end
     1181:     ensure
C:/Users/reang/Documents/work/fluentd/fluentd/test/test_supervisor.rb:1177:in `times'
C:/Users/reang/Documents/work/fluentd/fluentd/test/test_supervisor.rb:1177:in `test_maxstdio'
==============================================================================================================================================================================================================================================
8189
E
==============================================================================================================================================================================================================================================
Error: test_maxstdio[Very large](SupervisorTest): Errno::EMFILE: Too many open files @ rb_sysopen - C:/Users/reang/Documents/work/fluentd/fluentd/test/tmp/supervisor/1fc0ef8898e008ce1a9a/8189.txt
C:/Users/reang/Documents/work/fluentd/fluentd/test/test_supervisor.rb:1178:in `initialize'
C:/Users/reang/Documents/work/fluentd/fluentd/test/test_supervisor.rb:1178:in `open'
C:/Users/reang/Documents/work/fluentd/fluentd/test/test_supervisor.rb:1178:in `block in test_maxstdio'
     1175:     files = []
     1176:     begin
     1177:       10000.times do |i|
  => 1178:         file = File.open(File.join(@tmp_dir, "#{i}.txt"), "w")
     1179:         files.append(file)
     1180:       end
     1181:     ensure
C:/Users/reang/Documents/work/fluentd/fluentd/test/test_supervisor.rb:1177:in `times'
C:/Users/reang/Documents/work/fluentd/fluentd/test/test_supervisor.rb:1177:in `test_maxstdio'
==============================================================================================================================================================================================================================================
Finished in 13.5831727 seconds.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
4 tests, 0 assertions, 0 failures, 4 errors, 0 pendings, 0 omissions, 0 notifications
0% passed
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0.29 tests/s, 0.00 assertions/s
(...)

[cosmo0920]

I have added some tests to check this behavior. (They are incomplete, at present.) It appears that maxstdio does not work, at least for File.open of Ruby. It appears to be determined by the C-runtime, not maxstdio.

This means 8189 is the max number independent of maxstdio for Ruby 3.2.2 (x64-mingw-ucrt).

This is because the 8192 number is came from low-level C runtime limitation.
Windows C runtime has two level of I/O which are stream I/O and low level I/O.

• https://learn.microsoft.com/en-us/cpp/c-runtime-library/stream-i-o?view=msvc-170
• https://learn.microsoft.com/en-us/cpp/c-runtime-library/low-level-i-o?view=msvc-170

Obviously, _open and _close functions are belonging to low-level I/O. So, it won't be effective.

See the remarks of _setstdio:

The _setmaxstdio function changes the maximum value for the number of files that may be open simultaneously at the stream I/O level.

C run-time I/O now supports up to 8,192 files open simultaneously at the low I/O level. This level includes files opened and accessed using the _open, _read, and _write family of I/O functions. By default, up to 512 files can be open simultaneously at the stream I/O level. This level includes files opened and accessed using the fopen, fgetc, and fputc family of functions. The limit of 512 open files at the stream I/O level can be increased to a maximum of 8,192 by use of the _setmaxstdio function.

Because stream I/O-level functions, such as fopen, are built on top of the low I/O-level functions, the maximum of 8,192 is a hard upper limit for the number of simultaneously open files accessed through the C run-time library.

low-level I/O functions have hard limit of opening files and it cannot be changed via this API.

[daipom]

Thanks!

Obviously, _open and _close functions are belonging to low-level I/O. So, it won't be effective.

low-level I/O functions have hard limit of opening files and it cannot be changed via this API.

So, the key point would be whether there is any part of Fluentd that handles files at the stream I/O level, right?

[cosmo0920]

Thanks!

Obviously, _open and _close functions are belonging to low-level I/O. So, it won't be effective.

low-level I/O functions have hard limit of opening files and it cannot be changed via this API.

So, the key point would be whether there is any part of Fluentd that handles files at the stream I/O level, right?

Yes, the key point of this PR is increasing the limit of opening file descriptors for stream I/O at the same time.