http_server: Ready to support Async 2.0 gem by ashie · Pull Request #3842 · fluent/fluentd

ashie · 2022-08-03T04:32:01Z

Which issue(s) this PR fixes:
Fixes #

What this PR does / why we need it:
http_server plugin helper doesn't work with Async 2.x gem due to using obsolete usage.
This PR updates it to follow current documented way:
https://github.com/socketry/async-http/blob/0a65acd7cf7486e1877f0da86580e1692cd8207b/readme.md#usage

It's applicable both Async 2.x & Async 1.x.

But this PR still stay on Async 1.x because io-event gem which is required by Async 2.x can't build on Windows.

Docs Changes:
None

Release Note:
Same with the title

ashie · 2022-08-03T05:08:22Z

Hmm, can't build io-event on Windows: https://github.com/fluent/fluentd/runs/7645045887?check_suite_focus=true

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
C:/hostedtoolcache/windows/Ruby/3.1.2/x64/lib/ruby/gems/3.1.0/gems/io-event-1.0.9/ext
C:/hostedtoolcache/windows/Ruby/3.1.2/x64/bin/ruby.exe -I
C:/hostedtoolcache/windows/Ruby/3.1.2/x64/lib/ruby/3.1.0 -r
./siteconf20220803-6988-hlcmr.rb extconf.rb
checking for rb_ext_ractor_safe()... yes
checking for &rb_fiber_transfer()... yes
checking for -luring... no
checking for sys/epoll.h... no
checking for sys/event.h... no
checking for sys/eventfd.h... no
checking for rb_io_descriptor()... yes
checking for &rb_process_status_wait()... no
checking for rb_fiber_current()... yes
checking for &rb_fiber_raise()... yes
checking for ruby/io/buffer.h... yes
creating extconf.h
creating Makefile

current directory:
C:/hostedtoolcache/windows/Ruby/3.1.2/x64/lib/ruby/gems/3.1.0/gems/io-event-1.0.9/ext
make.exe DESTDIR\= clean

current directory:
C:/hostedtoolcache/windows/Ruby/3.1.2/x64/lib/ruby/gems/3.1.0/gems/io-event-1.0.9/ext
make.exe DESTDIR\=
generating IO_Event-x64-mingw-ucrt.def
compiling ./io/event/event.c
compiling ./io/event/selector/selector.c
./io/event/selector/selector.c:26:10: fatal error: sys/wait.h: No such file or
directory
   26 | #include <sys/wait.h>
      |          ^~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:246: selector.o] Error 1

make failed, exit code 2

lib/fluent/plugin_helper/http_server/server.rb

daipom · 2023-02-14T04:09:39Z

I'm seeing..

daipom

LGTM.

test/plugin_helper/test_http_server_helper.rb

daipom · 2023-02-14T09:44:35Z

Hmm, this log is not outputted. I don't know why...

[debug]: #0 fluent/log.rb:309:debug:   0.0s: Async::IO::Socket
      | Binding to #<Addrinfo: 0.0.0.0:24220 TCP>

In master, when launching Fluentd with this config, the log is outputted using ConsoleAdapter.

<source>
 @type monitor_agent
</source>

daipom · 2023-02-14T10:07:21Z

https://github.com/socketry/async-io/blob/v1.34.3/lib/async/io/socket.rb#L157

At this point, somehow Console.logger is Console::Terminal::Logger, not Fluent::Log::ConsoleAdapter.

I'm thinking about the possibility that the logger is not passed to the child task correctly.

daipom · 2023-02-14T10:19:48Z

It seems that Console.logger = Fluent::Log::ConsoleAdapter.wrap(@logger) is not applied to this thread.

fluentd/lib/fluent/plugin_helper/http_server.rb

Lines 74 to 76 in bf7498d

    
           thread_create(title) do 
        
             @_http_server.start(notify) 
        
           end

daipom · 2023-02-14T10:31:45Z

Console.logger= sets the value as Fiber::local::instance.
This is why Console.logger = Fluent::Log::ConsoleAdapter.wrap(@logger) is not applied to other threads.

https://github.com/socketry/console/blob/main/lib/console.rb#L13-L19
https://github.com/socketry/console/blob/main/lib/console/logger.rb#L19-L20
https://github.com/socketry/fiber-local/blob/main/lib/fiber/local.rb#L34-L53

daipom · 2023-02-14T10:44:11Z

We may have to do something like this...
Maybe there is a more correct way...

--- a/lib/fluent/plugin_helper/http_server/server.rb
+++ b/lib/fluent/plugin_helper/http_server/server.rb
@@ -40,7 +40,6 @@ module Fluent
           @uri = URI("#{scheme}://#{@addr}:#{@port}").to_s
           @router = Router.new(default_app)
           @server_task = nil
-          Console.logger = Fluent::Log::ConsoleAdapter.wrap(@logger)
 
           opts = if tls_context
                    { ssl_context: tls_context }
@@ -55,10 +54,12 @@ module Fluent
         end
 
         def start(notify = nil)
+          Console.logger = Fluent::Log::ConsoleAdapter.wrap(@logger)
           @logger.debug("Start async HTTP server listening #{@uri}")
 
           Async do |task|
             @server_task = task.async do
+              Console.logger = Fluent::Log::ConsoleAdapter.wrap(@logger)
               @server.run
             end
             if notify

ashie · 2023-02-15T00:33:44Z

We may have to do something like this... Maybe there is a more correct way...

Thanks for checking it.
It seems there is no other way to replace them, so I applied the fix.
But we don't need to rush merging this.
I've reverted the status of this PR to draft. We'll check it later.

daipom · 2023-02-15T03:05:28Z

Note: We have to consider the possibility of Async::HTTP::Server::run() creating child tasks inside.
(If so, how should we apply our logger to the child tasks' threads?)

Signed-off-by: Takuro Ashie <ashie@clear-code.com>

Because dependent io-event (v1.0.9) can't build on Windows. Signed-off-by: Takuro Ashie <ashie@clear-code.com>

Signed-off-by: Takuro Ashie <ashie@clear-code.com>

kenhys · 2024-04-04T07:32:14Z

Hmm, GitHub "update branch" feature is not what I want. rebased manually.

kenhys · 2024-04-04T07:34:19Z

Hmm, GitHub "update branch" feature is not what I want. rebased manually.

It should be "Update with rebase".

stevehipwell · 2024-04-29T11:40:14Z

It looks like as of v0.65.0 the async-http Gem required async to be >= 2.10.2.

ashie · 2024-04-30T07:01:29Z

It looks like as of v0.65.0 the async-http Gem required async to be >= 2.10.2.

Thanks notifying it, we'll check it.

ashie · 2024-04-30T07:02:09Z

This pull request isn't ready for v1.17, we postpone merging this.

github-actions · 2024-05-30T10:01:51Z

This PR has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this PR will be closed in 7 days

Watson1978 · 2024-08-21T02:56:28Z

Seems that currently io-event gem supports windows platform.
socketry/io-event#46

PS C:\Users\watson> ruby -v
ruby 3.3.3 (2024-06-12 revision f1c7b6f435) [x64-mingw-ucrt]
PS C:\Users\watson> gem install async
Fetching fiber-storage-1.0.0.gem
Fetching io-event-1.6.5.gem
Fetching fiber-annotation-0.2.0.gem
Fetching async-2.15.3.gem
Fetching fiber-local-1.1.0.gem
Fetching console-1.27.0.gem
Temporarily enhancing PATH for MSYS/MINGW...
Building native extensions. This could take a while...
Successfully installed io-event-1.6.5
Successfully installed fiber-annotation-0.2.0
Successfully installed fiber-storage-1.0.0
Successfully installed fiber-local-1.1.0
Successfully installed console-1.27.0
Successfully installed async-2.15.3
Parsing documentation for io-event-1.6.5
Installing ri documentation for io-event-1.6.5
Parsing documentation for fiber-annotation-0.2.0
Installing ri documentation for fiber-annotation-0.2.0
Parsing documentation for fiber-storage-1.0.0
Installing ri documentation for fiber-storage-1.0.0
Parsing documentation for fiber-local-1.1.0
Installing ri documentation for fiber-local-1.1.0
Parsing documentation for console-1.27.0
Installing ri documentation for console-1.27.0
Parsing documentation for async-2.15.3
Installing ri documentation for async-2.15.3
Done installing documentation for io-event, fiber-annotation, fiber-storage, fiber-local, console, async after 1 seconds
6 gems installed

Watson1978 · 2024-08-21T04:29:13Z

Async 2.x supports Ruby 3.1 or later only.
We have to still support Async 1.x for Ruby 2.7 and 3.0...

Watson1978 · 2024-08-21T04:52:02Z

Hmm, io-event gem still have a problem on windows platform.
If we use Ruby 3.1 on Windows, it fails to install io-event gem.

PS C:\Users\watson> ruby -v
ruby 3.1.6p260 (2024-05-29 revision a777087be6) [x64-mingw-ucrt]
PS C:\Users\watson> gem install io-event
Temporarily enhancing PATH for MSYS/MINGW...
Building native extensions. This could take a while...
ERROR:  Error installing io-event:
        ERROR: Failed to build gem native extension.

    current directory: C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/io-event-1.6.5/ext
C:/Ruby31-x64/bin/ruby.exe -I C:/Ruby31-x64/lib/ruby/3.1.0 extconf.rb
checking for rb_ext_ractor_safe()... yes
checking for &rb_fiber_transfer()... yes
checking for -luring... no
checking for sys/epoll.h... no
checking for sys/event.h... no
checking for sys/wait.h... no
checking for sys/eventfd.h... no
checking for rb_io_descriptor()... yes
checking for &rb_process_status_wait()... no
checking for rb_fiber_current()... yes
checking for &rb_fiber_raise()... yes
checking for epoll_pwait2()... no
checking for ruby/io/buffer.h... yes
creating extconf.h
creating Makefile

current directory: C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/io-event-1.6.5/ext
make DESTDIR\= sitearchdir\=./.gem.20240821-200492-eo54tq sitelibdir\=./.gem.20240821-200492-eo54tq clean

current directory: C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/io-event-1.6.5/ext
make DESTDIR\= sitearchdir\=./.gem.20240821-200492-eo54tq sitelibdir\=./.gem.20240821-200492-eo54tq
generating IO_Event-x64-mingw-ucrt.def
compiling ./io/event/event.c
compiling ./io/event/selector/selector.c
compiling ./io/event/interrupt.c
./io/event/interrupt.c: In function 'IO_Event_Interrupt_open':
./io/event/interrupt.c:71:9: error: implicit declaration of function 'pipe'; did you mean '_pipe'? [-Wimplicit-function-declaration]
   71 |         pipe(interrupt->descriptor);
      |         ^~~~
      |         _pipe
make: *** [Makefile:247: interrupt.o] エラー 1

make failed, exit code 2

Gem files will remain installed in C:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/io-event-1.6.5 for inspection.
Results logged to C:/Ruby31-x64/lib/ruby/gems/3.1.0/extensions/x64-mingw-ucrt/3.1.0/io-event-1.6.5/gem_make.out

https://github.com/Watson1978/fluentd/actions/runs/10483331006/job/29035871921

ashie · 2024-08-28T06:57:11Z

We'll continue this work in #4619

ashie force-pushed the async-2.0 branch from 3de3071 to 061f39c Compare August 3, 2022 04:33

ashie mentioned this pull request Aug 3, 2022

Remaining tasks to ready for Ruby 3.1 or later fluent/fluent-package-builder#380

Closed

4 tasks

ashie changed the title ~~http_server: Support Async 2.0 gem~~ http_server: Ready to support Async 2.0 gem Aug 9, 2022

ashie force-pushed the async-2.0 branch from 8045e30 to 2cc193e Compare August 9, 2022 05:49

ashie commented Aug 10, 2022

View reviewed changes

lib/fluent/plugin_helper/http_server/server.rb Outdated Show resolved Hide resolved

ashie mentioned this pull request Oct 27, 2022

Add initial version fluent/fluentd-aggregator-docker-image#1

Merged

ashie force-pushed the async-2.0 branch from 2cc193e to 4399e42 Compare November 23, 2022 13:56

ashie mentioned this pull request Nov 24, 2022

CI: Fail on Ruby 3.2 preview2 on Windows #3869

Closed

ashie mentioned this pull request Jan 3, 2023

Adapt Console::Logger to Fluent::Log #3987

Merged

ashie force-pushed the async-2.0 branch from 4399e42 to 01198b4 Compare January 27, 2023 07:05

ashie marked this pull request as ready for review February 10, 2023 08:23

ashie requested a review from daipom February 14, 2023 02:17

daipom previously approved these changes Feb 14, 2023

View reviewed changes

test/plugin_helper/test_http_server_helper.rb Outdated Show resolved Hide resolved

ashie marked this pull request as draft February 15, 2023 00:30

daipom mentioned this pull request Feb 27, 2023

Proposing a new Fluentd maintainer #4069

Merged

ashie added this to the v1.17.0 milestone Mar 28, 2023

ashie added 4 commits April 4, 2024 16:30

http_server: Support Async 2.0 gem

42163b3

Signed-off-by: Takuro Ashie <ashie@clear-code.com>

Still stay on async-1.x

388a8cf

Because dependent io-event (v1.0.9) can't build on Windows. Signed-off-by: Takuro Ashie <ashie@clear-code.com>

test_http_server_helper: Remove a needless argument

e285c76

Signed-off-by: Takuro Ashie <ashie@clear-code.com>

http_server: Replace Console.logger of all related threads

e9886fc

Signed-off-by: Takuro Ashie <ashie@clear-code.com>

kenhys force-pushed the async-2.0 branch from 79560bd to e9886fc Compare April 4, 2024 07:31

ashie removed this from the v1.17.0 milestone Apr 30, 2024

github-actions bot added the stale label May 30, 2024

daipom added enhancement Feature request or improve operations and removed stale labels May 30, 2024

ashie dismissed daipom’s stale review via e9886fc August 6, 2024 09:42

ashie assigned Watson1978 Aug 21, 2024

This was referenced Aug 21, 2024

Build error with Ruby 3.1 on Windows socketry/io-event#111

Closed

Async HTTP Server does not respond on Windows if cool.io gem is loaded socketry/async-http#175

Closed

ashie closed this Aug 28, 2024

kenhys deleted the async-2.0 branch December 26, 2024 05:12

Conversation

ashie commented Aug 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashie commented Aug 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

daipom commented Feb 14, 2023

Uh oh!

daipom left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

daipom commented Feb 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daipom commented Feb 14, 2023

Uh oh!

daipom commented Feb 14, 2023

Uh oh!

daipom commented Feb 14, 2023

Uh oh!

daipom commented Feb 14, 2023

Uh oh!

ashie commented Feb 15, 2023

Uh oh!

daipom commented Feb 15, 2023

Uh oh!

kenhys commented Apr 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kenhys commented Apr 4, 2024

Uh oh!

stevehipwell commented Apr 29, 2024

Uh oh!

ashie commented Apr 30, 2024

Uh oh!

ashie commented Apr 30, 2024

Uh oh!

github-actions bot commented May 30, 2024

Uh oh!

Watson1978 commented Aug 21, 2024

Uh oh!

Watson1978 commented Aug 21, 2024

Uh oh!

Watson1978 commented Aug 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashie commented Aug 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ashie commented Aug 3, 2022 •

edited

Loading

ashie commented Aug 3, 2022 •

edited

Loading

daipom commented Feb 14, 2023 •

edited

Loading

kenhys commented Apr 4, 2024 •

edited

Loading

Watson1978 commented Aug 21, 2024 •

edited

Loading