Skip to content

S3 refactor timer coros awspr multi worker#30

Open
PettitWesley wants to merge 36 commits into2.31.12-all-cherrypicksfrom
s3-refactor-timer-coros-awspr-multi-worker
Open

S3 refactor timer coros awspr multi worker#30
PettitWesley wants to merge 36 commits into2.31.12-all-cherrypicksfrom
s3-refactor-timer-coros-awspr-multi-worker

Conversation

@PettitWesley
Copy link
Copy Markdown
Owner


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
This commit combines many changes designed to
improve S3 stability:
- code clean up and refactoring for readability
- re-implement preserve_data_ordering code to
  fix several bugs and simplify logic.
- all uploads now happen outside of cb_flush
- uploads performed with new scheduled timer
  jobs with coroutines
- S3 always uses async IO, with mutex lock
  to protect it from concurrency issues
- remove trailing slash in store_dir to
  prevent double // in final path

Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
…t async timers on the sched

Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
@PettitWesley
Copy link
Copy Markdown
Owner Author

I still don't know why the build is failing. I'm gonna try rebasing these changes on top of master and debug there instead.

[ 40%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_config.c.o
In file included from /home/ec2-user/fluent-bit/include/fluent-bit/flb_io.h:27:0,
                 from /home/ec2-user/fluent-bit/include/fluent-bit/flb_upstream.h:29,
                 from /home/ec2-user/fluent-bit/include/fluent-bit/flb_output_thread.h:24,
                 from /home/ec2-user/fluent-bit/src/flb_async_timer.c:22:
/home/ec2-user/fluent-bit/include/fluent-bit/flb_output.h: In function ‘flb_output_flush_create’:
/home/ec2-user/fluent-bit/include/fluent-bit/flb_output.h:591:35: error: dereferencing pointer to incomplete type ‘struct flb_out_thread_instance’
         pthread_mutex_lock(&th_ins->flush_mutex);
                                   ^~
[ 40%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_socket.c.o
make[2]: *** [src/CMakeFiles/fluent-bit-static.dir/flb_async_timer.c.o] Error 1
make[2]: *** Waiting for unfinished jobs....
[ 40%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_network.c.o
make[1]: *** [src/CMakeFiles/fluent-bit-static.dir/all] Error 2
make: *** [all] Error 2

Signed-off-by: Wesley Pettit <wppttt@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant