From 039fb24fb4f7a53ef85139e51982acaa881830dc Mon Sep 17 00:00:00 2001 From: Maciej Mensfeld Date: Thu, 11 Dec 2025 15:38:50 +0100 Subject: [PATCH] Update wiki documentation for Shoryuken v7.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Major documentation overhaul to match Karafka-level quality: New pages (12): - Upgrade-Guide-7.0.md - Breaking changes and migration steps - ActiveJob-Continuations.md - Rails 8.1+ stopping? support - CurrentAttributes.md - Request context persistence - Bulk-Enqueuing.md - perform_all_later (Rails 7.1+) - Health-Checks.md - healthy? API, K8s/ECS probes - Container-Deployment.md - Docker, Kubernetes, ECS/Fargate - Security-Best-Practices.md - IAM, encryption, VPC endpoints - Performance-Tuning.md - Concurrency, pooling, optimization - Monitoring-and-Metrics.md - Prometheus, CloudWatch, alerting - Troubleshooting.md - Common issues and solutions - Scaling-Patterns.md - Auto-scaling, sharding patterns - Architecture-Internals.md - Internal component documentation Updated pages (17): - Home.md - Karafka-style 13-section navigation - Getting-Started.md - Prerequisites, Zeitwerk notes - Rails-Integration-Active-Job.md - All v7 features - Lifecycle-Events.md - utilization_update event - Deployment.md - Heroku, systemd, Capistrano - AWS-Beanstalk-Config.md - AL2/AL2023, removed AL1 - Signals.md - Container orchestration guidance - Middleware.md - 5 built-in middleware documented - Shoryuken-options.md - CLI/YAML/programmatic config - Amazon-SQS-IAM-Policy.md - Role-specific policies - Testing.md - RSpec/Minitest, CI configuration - Using-a-local-mock-SQS-server.md - LocalStack/ElasticMQ/Moto - Configure-the-AWS-Client.md - IRSA, SSO, assume role - Worker-options.md - Complete options reference - Sending-a-message.md - Bulk sending, FIFO, attributes - From-Sidekiq-to-Shoryuken.md - Feature comparison, checklist Renamed: - Complete-Shoryuken-Setup-for-Rails-5-with-ActiveJob.md → Complete-Shoryuken-Setup-for-Rails.md Focus on v7.0.0+ (Ruby 3.2+, Rails 7.2+) with complete examples. --- AWS-Beanstalk-Config.md | 221 +++++-- ActiveJob-Continuations.md | 343 +++++++++++ Amazon-SQS-IAM-Policy-for-Shoryuken.md | 405 ++++++++++++- Architecture-Internals.md | 466 +++++++++++++++ Bulk-Enqueuing.md | 259 +++++++++ ...ryuken-Setup-for-Rails-5-with-ActiveJob.md | 76 --- Complete-Shoryuken-Setup-for-Rails.md | 309 ++++++++++ Configure-the-AWS-Client.md | 327 ++++++++++- Container-Deployment.md | 543 ++++++++++++++++++ CurrentAttributes.md | 302 ++++++++++ Deployment.md | 299 +++++++++- From-Sidekiq-to-Shoryuken.md | 300 ++++++++-- Getting-Started.md | 138 ++++- Health-Checks.md | 396 +++++++++++++ Home.md | 98 +++- Lifecycle-Events.md | 310 +++++++++- Middleware.md | 374 +++++++++++- Monitoring-and-Metrics.md | 393 +++++++++++++ Performance-Tuning.md | 377 ++++++++++++ Rails-Integration-Active-Job.md | 347 +++++++++-- Scaling-Patterns.md | 358 ++++++++++++ Security-Best-Practices.md | 446 ++++++++++++++ Sending-a-message.md | 274 +++++++-- Shoryuken-options.md | 340 +++++++++-- Signals.md | 326 ++++++++++- Testing.md | 408 +++++++++++-- ...eceive_message(...)\"-in-the-log-file?.md" | 15 + Troubleshooting.md | 393 +++++++++++++ Upgrade-Guide-7.0.md | 155 +++++ Using-a-local-mock-SQS-server.md | 354 +++++++++++- Worker-options.md | 339 +++++++++-- 31 files changed, 9156 insertions(+), 535 deletions(-) create mode 100644 ActiveJob-Continuations.md create mode 100644 Architecture-Internals.md create mode 100644 Bulk-Enqueuing.md delete mode 100644 Complete-Shoryuken-Setup-for-Rails-5-with-ActiveJob.md create mode 100644 Complete-Shoryuken-Setup-for-Rails.md create mode 100644 Container-Deployment.md create mode 100644 CurrentAttributes.md create mode 100644 Health-Checks.md create mode 100644 Monitoring-and-Metrics.md create mode 100644 Performance-Tuning.md create mode 100644 Scaling-Patterns.md create mode 100644 Security-Best-Practices.md create mode 100644 "Too-many-\"receive_message(...)\"-in-the-log-file?.md" create mode 100644 Troubleshooting.md create mode 100644 Upgrade-Guide-7.0.md diff --git a/AWS-Beanstalk-Config.md b/AWS-Beanstalk-Config.md index 6e3065c..6fb98e0 100644 --- a/AWS-Beanstalk-Config.md +++ b/AWS-Beanstalk-Config.md @@ -1,25 +1,112 @@ -### Web server environment -To restart (or just start) Shoryuken on each deployment to Beanstalk, prepare the configuration files as described below. -Make sure the selected EC2 instance type has enough memory to ensure the functioning of both the web server and Shoryuken. +# AWS Elastic Beanstalk Configuration -#### Amazon Linux 2 and 2023 with Procfile +This guide covers running Shoryuken workers on AWS Elastic Beanstalk. -Generally, it is recommended to run more complex projects on AWS EB with your own Procfile, it gives greater control over the version of the web server and other parameters. -The easiest way to add a shoryuken worker in Elastic Beanstalk is to simply add the command to run it to the Procfile as `worker:` +## Recommended: Procfile Method + +The simplest and most reliable approach is using a Procfile. This works with Amazon Linux 2 and Amazon Linux 2023. + +### Setup + +Create a `Procfile` in your application root: ``` web: bundle exec puma -C /opt/elasticbeanstalk/config/private/pumaconf.rb --pidfile /var/app/current/tmp/pids/server.pid -worker: bundle exec shoryuken -R -P /var/app/current/tmp/pids/shoryuken.pid -L /var/app/current/log/shoryuken.log +worker: bundle exec shoryuken -R -C config/shoryuken.yml -L /var/app/current/log/shoryuken.log +``` + +### With Health Checks + +``` +web: bundle exec puma -C /opt/elasticbeanstalk/config/private/pumaconf.rb +worker: bundle exec shoryuken -R -C config/shoryuken.yml +``` + +Configure health check via Shoryuken initializer (see [[Health Checks]]). + +### Configuration File + +```yaml +# config/shoryuken.yml +concurrency: 25 +timeout: 25 +queues: + - [default, 1] +``` + +### IAM Role + +Ensure your Elastic Beanstalk instance role has SQS permissions: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "sqs:ReceiveMessage", + "sqs:DeleteMessage", + "sqs:DeleteMessageBatch", + "sqs:GetQueueAttributes", + "sqs:GetQueueUrl", + "sqs:SendMessage", + "sqs:SendMessageBatch", + "sqs:ChangeMessageVisibility", + "sqs:ChangeMessageVisibilityBatch" + ], + "Resource": "arn:aws:sqs:*:*:myapp-*" + } + ] +} +``` + +--- + +## Instance Sizing + +When running both web and worker on the same instance, ensure adequate resources: + +| Instance Type | Web + Worker | Notes | +|---------------|--------------|-------| +| t3.small | Light workloads | 2 GB RAM | +| t3.medium | Medium workloads | 4 GB RAM | +| t3.large | Heavy workloads | 8 GB RAM | + +### Memory Allocation + +```yaml +# config/shoryuken.yml +concurrency: 10 # Reduce if sharing with web server +``` + +--- + +## Environment Variables + +Set via Elastic Beanstalk console or `.ebextensions`: + +```yaml +# .ebextensions/env.config +option_settings: + aws:elasticbeanstalk:application:environment: + RAILS_ENV: production + AWS_REGION: us-east-1 ``` -#### Amazon Linux 2 with hooks (deprecated) +--- + +## Alternative: Platform Hooks (Deprecated) + +For more control, you can use platform hooks. This approach is more complex and is deprecated in favor of the Procfile method. + +### Amazon Linux 2/2023 Hooks -`.platform/hooks/postdeploy/restart_shoryuken.sh` +Create `.platform/hooks/postdeploy/restart_shoryuken.sh`: -```sh +```bash #!/bin/bash -# NOTE: get-config platformconfig returns paths WITH the trailing slash .../ APP_DEPLOY_DIR=$(/opt/elasticbeanstalk/bin/get-config platformconfig -k AppDeployDir) LOG_DIR="${APP_DEPLOY_DIR}log" PID_DIR="${APP_DEPLOY_DIR}tmp/pids" @@ -27,9 +114,8 @@ EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config platformconfig -k AppUser) mkdir -m 777 -p $PID_DIR -if [ -f $PID_DIR/shoryuken.pid ] -then - kill -TERM `cat $PID_DIR/shoryuken.pid` || echo "The shoryuken process was not running." +if [ -f $PID_DIR/shoryuken.pid ]; then + kill -TERM $(cat $PID_DIR/shoryuken.pid) || echo "Shoryuken was not running" rm -rf $PID_DIR/shoryuken.pid fi @@ -37,84 +123,95 @@ sleep 10 cd $APP_DEPLOY_DIR -# NOTE: in local development environment, run `bundle exec dotenv shoryuken \` su -s /bin/bash -c "bundle exec shoryuken \ -R \ -P $PID_DIR/shoryuken.pid \ -C ${APP_DEPLOY_DIR}config/shoryuken.yml \ - -r ${APP_DEPLOY_DIR}app/jobs \ -L $LOG_DIR/shoryuken.log \ -d" $EB_APP_USER exit 0 ``` ----- +Create `.platform/hooks/predeploy/mute_shoryuken.sh`: -`.platform/hooks/predeploy/mute_shoryuken.sh` - -```sh +```bash #!/bin/bash APP_DEPLOY_DIR=$(/opt/elasticbeanstalk/bin/get-config platformconfig -k AppDeployDir) PID_DIR="${APP_DEPLOY_DIR}tmp/pids" EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config platformconfig -k AppUser) -if [ -f $PID_DIR/shoryuken.pid ] -then - su -s /bin/bash -c "kill -USR1 `cat $PID_DIR/shoryuken.pid`" $EB_APP_USER || echo "The shoryuken process was not running." +if [ -f $PID_DIR/shoryuken.pid ]; then + su -s /bin/bash -c "kill -USR1 $(cat $PID_DIR/shoryuken.pid)" $EB_APP_USER || echo "Shoryuken was not running" fi exit 0 ``` -#### Amazon Linux 1 (deprecated platform) -`.ebextensions/shoryuken.config` inside your repo. +Make scripts executable: + +```bash +chmod +x .platform/hooks/postdeploy/restart_shoryuken.sh +chmod +x .platform/hooks/predeploy/mute_shoryuken.sh +``` + +--- + +## Worker Environment -``` yaml +For dedicated worker environments (no web tier), use a Worker Environment: -# .ebextensions/shoryuken.config -# Based on the conversation in https://github.com/phstc/shoryuken/issues/48 +1. Create a Worker environment in Elastic Beanstalk +2. Configure the worker queue in the console +3. Use a simple Procfile: -files: - "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_shoryuken": - mode: "000777" - content: | - APP_DEPLOY_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir) - LOG_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_log_dir) - PID_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir) +``` +worker: bundle exec shoryuken -R -C config/shoryuken.yml +``` + +Note: Worker environments use SQS daemon, which may conflict with Shoryuken. Consider using a Web environment with `web=0` scaled down instead. - EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir) - EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir) - . $EB_SUPPORT_DIR/envvars - . $EB_SCRIPT_DIR/use-app-ruby.sh +--- - if [ -f $PID_DIR/shoryuken.pid ] - then - kill -TERM `cat $PID_DIR/shoryuken.pid` || echo "The shoryuken process was not running." - rm -rf $PID_DIR/shoryuken.pid - fi +## Troubleshooting - sleep 10 +### Logs Location - cd $APP_DEPLOY_DIR +``` +/var/app/current/log/shoryuken.log +/var/log/eb-engine.log +``` - bundle exec shoryuken \ - -R \ - -P $PID_DIR/shoryuken.pid \ - -C $APP_DEPLOY_DIR/config/shoryuken.yml \ - -L $LOG_DIR/shoryuken.log \ - -d +### View Logs - "/opt/elasticbeanstalk/hooks/appdeploy/pre/03_mute_shoryuken": - mode: "000777" - content: | - PID_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir) - if [ -f $PID_DIR/shoryuken.pid ] - then - kill -USR1 `cat $PID_DIR/shoryuken.pid` || echo "The shoryuken process was not running." - fi +```bash +eb logs +# or +eb ssh +tail -f /var/app/current/log/shoryuken.log ``` -### Worker environment -For information on deploying to a worker environment see the thread in https://github.com/phstc/shoryuken/issues/48 \ No newline at end of file +### Common Issues + +**Worker not starting:** +- Check IAM permissions +- Verify config file path +- Check log file for errors + +**Out of memory:** +- Reduce concurrency +- Upgrade instance type +- Check for memory leaks in jobs + +**Permission errors:** +- Ensure hooks are executable +- Check file ownership + +--- + +## Related + +- [[Deployment]] - General deployment guide +- [[Amazon SQS IAM Policy for Shoryuken]] - IAM permissions +- [[Health Checks]] - Health monitoring diff --git a/ActiveJob-Continuations.md b/ActiveJob-Continuations.md new file mode 100644 index 0000000..275b2e6 --- /dev/null +++ b/ActiveJob-Continuations.md @@ -0,0 +1,343 @@ +# ActiveJob Continuations + +ActiveJob Continuations allow long-running jobs to gracefully checkpoint their progress during shutdown and resume after restart. + +**Requires:** Rails 8.1+ + +## Overview + +When Shoryuken receives a shutdown signal (SIGTERM, SIGINT), it sets a `stopping?` flag. Jobs can check this flag and: + +1. Save their current progress +2. Re-enqueue themselves to continue later +3. Exit gracefully + +This prevents jobs from being interrupted mid-process and losing work. + +## How It Works + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Job Execution Flow │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ Job starts → Process items → Check stopping? → Continue │ +│ │ │ │ +│ ▼ ▼ │ +│ Process more Save progress │ +│ │ Re-enqueue job │ +│ ▼ Exit gracefully │ +│ Complete │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Basic Usage + +```ruby +class DataExportJob < ApplicationJob + queue_as :exports + + def perform(export) + export.records_to_process.find_each do |record| + # Check if Shoryuken is shutting down + if stopping? + # Checkpoint progress + export.update!(last_processed_id: record.id) + + # Re-enqueue to continue from checkpoint + self.class.perform_later(export) + return + end + + process_record(record) + end + + export.mark_completed! + end + + private + + def process_record(record) + # Your processing logic + end +end +``` + +## The `stopping?` Method + +The `stopping?` method returns `true` when: + +1. Shoryuken receives `SIGTERM` (graceful shutdown) +2. Shoryuken receives `SIGINT` (Ctrl+C) +3. The `stop` or `stop!` method is called on the launcher + +### Implementation Details + +Internally, Shoryuken tracks the stopping state in the `Launcher`: + +```ruby +# In the adapter +def stopping? + launcher = Shoryuken::Runner.instance.launcher + launcher&.stopping? || false +end +``` + +This state is propagated to all ActiveJob jobs via Rails' Continuations API. + +## Patterns + +### Batch Processing with Checkpoints + +```ruby +class BulkImportJob < ApplicationJob + BATCH_SIZE = 100 + + def perform(import, offset: 0) + records = import.source_records.offset(offset).limit(BATCH_SIZE) + + records.each_with_index do |record, index| + if stopping? + # Save checkpoint and re-enqueue + self.class.perform_later(import, offset: offset + index) + return + end + + import_record(record) + end + + if records.size == BATCH_SIZE + # More records to process + self.class.perform_later(import, offset: offset + BATCH_SIZE) + else + import.mark_completed! + end + end +end +``` + +### Progress Tracking + +```ruby +class VideoTranscodeJob < ApplicationJob + def perform(video, start_frame: 0) + total_frames = video.frame_count + current_frame = start_frame + + while current_frame < total_frames + if stopping? + video.update!(transcoding_progress: current_frame) + self.class.perform_later(video, start_frame: current_frame) + return + end + + transcode_frame(video, current_frame) + current_frame += 1 + + # Update progress periodically + if current_frame % 1000 == 0 + video.update!(transcoding_progress: current_frame) + end + end + + video.finalize_transcode! + end +end +``` + +### External API Pagination + +```ruby +class SyncUsersJob < ApplicationJob + def perform(page: 1) + response = ExternalApi.fetch_users(page: page) + + response.users.each_with_index do |user_data, index| + if stopping? + # Can't checkpoint mid-page, re-enqueue current page + self.class.perform_later(page: page) + return + end + + User.upsert(user_data) + end + + if response.has_more_pages? + self.class.perform_later(page: page + 1) + end + end +end +``` + +## Best Practices + +### 1. Check `stopping?` at Natural Boundaries + +Check at logical breakpoints in your processing: + +```ruby +# Good: Check between records +records.each do |record| + return if stopping? + process(record) +end + +# Avoid: Checking too frequently adds overhead +# Bad +data.each_char do |char| + return if stopping? # Too granular + process(char) +end +``` + +### 2. Keep Checkpoint Data Small + +Store only essential progress information: + +```ruby +# Good: Store IDs and offsets +export.update!(last_processed_id: record.id) + +# Avoid: Storing large data structures +export.update!(remaining_records: unprocessed_records.to_json) # Bad +``` + +### 3. Make Jobs Resumable + +Design jobs to resume from checkpoints: + +```ruby +class ReportJob < ApplicationJob + def perform(report, last_id: nil) + scope = report.data_source + scope = scope.where('id > ?', last_id) if last_id + + scope.find_each do |record| + if stopping? + self.class.perform_later(report, last_id: record.id) + return + end + + process(record) + end + end +end +``` + +### 4. Handle Idempotency + +Ensure processing is idempotent in case a job restarts: + +```ruby +class ProcessOrderJob < ApplicationJob + def perform(order, processed_items: []) + order.line_items.each do |item| + next if processed_items.include?(item.id) + + if stopping? + self.class.perform_later(order, processed_items: processed_items) + return + end + + fulfill_item(item) + processed_items << item.id + end + + order.mark_fulfilled! + end +end +``` + +## Testing Continuations + +### RSpec Example + +```ruby +RSpec.describe DataExportJob do + describe '#perform' do + let(:export) { create(:export, :with_records) } + + context 'when stopping? is false' do + it 'processes all records' do + expect { described_class.perform_now(export) } + .to change { export.reload.status }.to('completed') + end + end + + context 'when stopping? becomes true' do + before do + allow_any_instance_of(described_class) + .to receive(:stopping?).and_return(false, false, true) + end + + it 'checkpoints progress and re-enqueues' do + expect(described_class).to receive(:perform_later) + .with(export) + + described_class.perform_now(export) + + expect(export.reload.last_processed_id).to be_present + end + end + end +end +``` + +### Minitest Example + +```ruby +class DataExportJobTest < ActiveJob::TestCase + test 'checkpoints and re-enqueues on shutdown' do + export = exports(:large_export) + + # Simulate stopping after 2 records + job = DataExportJob.new(export) + call_count = 0 + job.define_singleton_method(:stopping?) do + call_count += 1 + call_count > 2 + end + + job.perform_now + + assert_enqueued_jobs 1, only: DataExportJob + assert_not_nil export.reload.last_processed_id + end +end +``` + +## Limitations + +### SQS Delay Limit + +When re-enqueueing, be aware of SQS's 15-minute delay limit: + +```ruby +# This works +self.class.set(wait: 5.minutes).perform_later(args) + +# This raises an error +self.class.set(wait: 20.minutes).perform_later(args) # > 15 minutes +``` + +For longer delays, use EventBridge Scheduler or a similar service. + +### Visibility Timeout + +Ensure your visibility timeout is long enough for checkpoint + re-enqueue operations: + +```yaml +# config/shoryuken.yml +queues: + - [exports, 1] + +# Set visibility_timeout on the SQS queue to accommodate processing time +``` + +## Related + +- [[Rails Integration Active Job]] - Basic ActiveJob setup +- [[Signals]] - Understanding shutdown signals +- [[Lifecycle Events]] - Hooks for shutdown events +- [Rails PR #55127](https://github.com/rails/rails/pull/55127) - Original Rails implementation diff --git a/Amazon-SQS-IAM-Policy-for-Shoryuken.md b/Amazon-SQS-IAM-Policy-for-Shoryuken.md index 3707f34..bd5b4b7 100644 --- a/Amazon-SQS-IAM-Policy-for-Shoryuken.md +++ b/Amazon-SQS-IAM-Policy-for-Shoryuken.md @@ -1,27 +1,392 @@ -Below is a minimum IAM Policy required for Shoryuken to function. +# Amazon SQS IAM Policy for Shoryuken + +This guide provides IAM policy examples for different Shoryuken use cases. + +## Minimum Required Policy + +Basic policy for consuming and producing messages: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "ShoryukenAccess", + "Effect": "Allow", + "Action": [ + "sqs:ReceiveMessage", + "sqs:DeleteMessage", + "sqs:DeleteMessageBatch", + "sqs:GetQueueAttributes", + "sqs:GetQueueUrl", + "sqs:SendMessage", + "sqs:SendMessageBatch", + "sqs:ChangeMessageVisibility", + "sqs:ChangeMessageVisibilityBatch" + ], + "Resource": "arn:aws:sqs:us-east-1:123456789:myapp-*" + } + ] +} +``` + +--- + +## Role-Specific Policies + +### Consumer Only (Worker) + +For workers that only process messages: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "ShoryukenConsumer", + "Effect": "Allow", + "Action": [ + "sqs:ReceiveMessage", + "sqs:DeleteMessage", + "sqs:DeleteMessageBatch", + "sqs:GetQueueAttributes", + "sqs:GetQueueUrl", + "sqs:ChangeMessageVisibility", + "sqs:ChangeMessageVisibilityBatch" + ], + "Resource": "arn:aws:sqs:us-east-1:123456789:myapp-*" + } + ] +} +``` + +### Producer Only (Web App) + +For applications that only enqueue messages: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "ShoryukenProducer", + "Effect": "Allow", + "Action": [ + "sqs:SendMessage", + "sqs:SendMessageBatch", + "sqs:GetQueueUrl" + ], + "Resource": "arn:aws:sqs:us-east-1:123456789:myapp-*" + } + ] +} +``` + +### Admin (CLI Tools) + +For using `shoryuken sqs` commands: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "ShoryukenAdmin", + "Effect": "Allow", + "Action": [ + "sqs:ListQueues", + "sqs:CreateQueue", + "sqs:DeleteQueue", + "sqs:PurgeQueue", + "sqs:GetQueueAttributes", + "sqs:GetQueueUrl", + "sqs:ReceiveMessage", + "sqs:SendMessage", + "sqs:SendMessageBatch", + "sqs:DeleteMessage", + "sqs:DeleteMessageBatch" + ], + "Resource": "*" + } + ] +} +``` + +--- + +## Feature-Specific Permissions + +### FIFO Queues + +FIFO queues use the same permissions as standard queues. No additional permissions required. + +### Dead Letter Queues + +Add permissions for the DLQ: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "MainQueue", + "Effect": "Allow", + "Action": [ + "sqs:ReceiveMessage", + "sqs:DeleteMessage", + "sqs:GetQueueAttributes", + "sqs:GetQueueUrl" + ], + "Resource": "arn:aws:sqs:us-east-1:123456789:myapp-jobs" + }, + { + "Sid": "DeadLetterQueue", + "Effect": "Allow", + "Action": [ + "sqs:SendMessage", + "sqs:GetQueueUrl" + ], + "Resource": "arn:aws:sqs:us-east-1:123456789:myapp-jobs-dlq" + } + ] +} +``` + +### KMS Encrypted Queues + +For queues using SSE-KMS: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "SQSAccess", + "Effect": "Allow", + "Action": [ + "sqs:ReceiveMessage", + "sqs:DeleteMessage", + "sqs:SendMessage", + "sqs:GetQueueAttributes", + "sqs:GetQueueUrl" + ], + "Resource": "arn:aws:sqs:us-east-1:123456789:myapp-*" + }, + { + "Sid": "KMSAccess", + "Effect": "Allow", + "Action": [ + "kms:Decrypt", + "kms:GenerateDataKey" + ], + "Resource": "arn:aws:kms:us-east-1:123456789:key/12345678-1234-1234-1234-123456789012", + "Condition": { + "StringEquals": { + "kms:ViaService": "sqs.us-east-1.amazonaws.com" + } + } + } + ] +} +``` + +--- + +## Resource Restrictions + +### By Queue Name Pattern + +```json +{ + "Resource": "arn:aws:sqs:us-east-1:123456789:myapp-*" +} +``` + +### By Environment + +```json +{ + "Resource": [ + "arn:aws:sqs:us-east-1:123456789:production-*", + "arn:aws:sqs:us-east-1:123456789:staging-*" + ] +} +``` + +### Single Queue + +```json +{ + "Resource": "arn:aws:sqs:us-east-1:123456789:myapp-jobs" +} +``` + +--- + +## Cross-Account Access + +### Producer Account Policy + +Allow sending to queues in another account: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "sqs:SendMessage", + "sqs:GetQueueUrl" + ], + "Resource": "arn:aws:sqs:us-east-1:OTHER_ACCOUNT:shared-queue" + } + ] +} +``` + +### Queue Policy (Owner Account) + +Allow access from another account: ```json { - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "ShoryukenAccess", - "Effect": "Allow", - "Action": [ - "sqs:ChangeMessageVisibility", - "sqs:DeleteMessage", - "sqs:GetQueueAttributes", - "sqs:GetQueueUrl", - "sqs:ReceiveMessage", - "sqs:SendMessage", - "sqs:ListQueues" - ], - "Resource": "arn:aws:sqs:REGION:ACCOUNT:*" - } - ] + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "AWS": "arn:aws:iam::PRODUCER_ACCOUNT:root" + }, + "Action": "sqs:SendMessage", + "Resource": "arn:aws:sqs:us-east-1:OWNER_ACCOUNT:shared-queue" + } + ] } ``` -*Note:* `sqs:ListQueues` is only needed for running `shoryuken sqs ls` (for listing queues, see `shoryuken help sqs`). You don't need it for sending and receiving messages. +--- + +## IAM Role Examples + +### ECS Task Role + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "sqs:ReceiveMessage", + "sqs:DeleteMessage", + "sqs:DeleteMessageBatch", + "sqs:GetQueueAttributes", + "sqs:GetQueueUrl", + "sqs:ChangeMessageVisibility" + ], + "Resource": "arn:aws:sqs:*:*:myapp-*" + } + ] +} +``` + +Trust policy: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Service": "ecs-tasks.amazonaws.com" + }, + "Action": "sts:AssumeRole" + } + ] +} +``` + +### EKS Service Account (IRSA) + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "sqs:ReceiveMessage", + "sqs:DeleteMessage", + "sqs:DeleteMessageBatch", + "sqs:GetQueueAttributes", + "sqs:GetQueueUrl", + "sqs:ChangeMessageVisibility" + ], + "Resource": "arn:aws:sqs:*:*:myapp-*" + } + ] +} +``` + +Trust policy: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Federated": "arn:aws:iam::123456789:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE" + }, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringEquals": { + "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:default:shoryuken-worker" + } + } + } + ] +} +``` + +--- + +## Troubleshooting + +### Common Errors + +**AccessDenied on ReceiveMessage:** +- Check queue ARN in policy matches actual queue +- Verify IAM role is attached to instance/task +- Check for condition keys that might restrict access + +**AccessDenied on SendMessage:** +- Verify producer has `sqs:SendMessage` permission +- Check queue policy allows the principal + +**KMS Access Denied:** +- Add `kms:Decrypt` and `kms:GenerateDataKey` permissions +- Ensure KMS key policy allows the IAM role + +### Testing Permissions + +Use AWS CLI to verify: + +```bash +# Test receive +aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789/myqueue + +# Test send +aws sqs send-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789/myqueue --message-body "test" +``` + +--- + +## Related -See [IAM Policies](http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html) for the AWS User Guide. \ No newline at end of file +- [[Security Best Practices]] - Comprehensive security guide +- [[Configure the AWS Client]] - Credential configuration +- [AWS SQS IAM Documentation](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-authentication-and-access-control.html) diff --git a/Architecture-Internals.md b/Architecture-Internals.md new file mode 100644 index 0000000..c64c94b --- /dev/null +++ b/Architecture-Internals.md @@ -0,0 +1,466 @@ +# Architecture Internals + +This guide explains Shoryuken's internal architecture for contributors and advanced users. + +## Overview + +``` +┌─────────────────────────────────────────────────────────┐ +│ Runner │ +│ (Entry point, signal handling, Rails initialization) │ +└──────────────────────────┬──────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────┐ +│ Launcher │ +│ (Lifecycle management, health checks, managers) │ +└──────────────────────────┬──────────────────────────────┘ + │ + ┌────────────┴────────────┐ + ▼ ▼ +┌─────────────────────────┐ ┌─────────────────────────┐ +│ Manager (Group 1) │ │ Manager (Group 2) │ +│ ┌─────────────────┐ │ │ ┌─────────────────┐ │ +│ │ Fetcher │ │ │ │ Fetcher │ │ +│ └────────┬────────┘ │ │ └────────┬────────┘ │ +│ │ │ │ │ │ +│ ┌────────▼────────┐ │ │ ┌────────▼────────┐ │ +│ │ Polling Strategy│ │ │ │ Polling Strategy│ │ +│ └────────┬────────┘ │ │ └────────┬────────┘ │ +│ │ │ │ │ │ +│ ┌────────▼────────┐ │ │ ┌────────▼────────┐ │ +│ │ Processor(s) │ │ │ │ Processor(s) │ │ +│ └─────────────────┘ │ │ └─────────────────┘ │ +└─────────────────────────┘ └─────────────────────────┘ +``` + +--- + +## Core Components + +### Runner + +Entry point for the Shoryuken process. + +**Location:** `lib/shoryuken/runner.rb` + +**Responsibilities:** +- Parse CLI options +- Load Rails/application +- Initialize Launcher +- Handle Unix signals (TERM, INT, USR1, TTIN, TSTP) + +```ruby +# Signal handling +Signal.trap('TERM') { launcher.stop! } +Signal.trap('USR1') { launcher.stop } +Signal.trap('TTIN') { print_debug_info } +``` + +### Launcher + +Coordinates the lifecycle of all processing managers. + +**Location:** `lib/shoryuken/launcher.rb` + +**Responsibilities:** +- Create managers for each processing group +- Start/stop managers +- Health check coordination +- Fire lifecycle events + +**Key Methods:** +```ruby +def start # Starts all managers +def stop # Graceful shutdown (waits for jobs) +def stop! # Hard shutdown (with timeout) +def stopping? # Check if shutting down +def healthy? # All managers running? +``` + +### Manager + +Manages message dispatching for a single processing group. + +**Location:** `lib/shoryuken/manager.rb` + +**Responsibilities:** +- Dispatch loop (fetch → assign → process) +- Track busy/ready processor count +- Coordinate with polling strategy +- Fire utilization events + +**Key Attributes:** +```ruby +@group # Processing group name +@fetcher # Fetcher instance +@polling_strategy # Queue selection strategy +@max_processors # Concurrency limit +@busy_processors # AtomicCounter +``` + +**Dispatch Flow:** +``` +dispatch_loop + │ + ├── Check: stop requested? not running? → exit + │ + ├── Check: ready processors > 0? + │ └── No → sleep(0.1) + │ + ├── Get next queue from polling strategy + │ └── None available → sleep(0.1) + │ + ├── Fetch messages from queue + │ + └── Assign to processor(s) + │ + └── Loop back +``` + +### Fetcher + +Retrieves messages from SQS. + +**Location:** `lib/shoryuken/fetcher.rb` + +**Responsibilities:** +- Call SQS receive_message API +- Handle long polling +- Apply message limits + +### Processor + +Executes worker logic on messages. + +**Location:** `lib/shoryuken/processor.rb` + +**Responsibilities:** +- Invoke middleware chain +- Instantiate worker +- Call `worker.perform(sqs_msg, body)` +- Handle exceptions + +--- + +## Polling Strategies + +### BaseStrategy + +**Location:** `lib/shoryuken/polling/base_strategy.rb` + +Abstract base for queue selection. + +### WeightedRoundRobin (Default) + +**Location:** `lib/shoryuken/polling/weighted_round_robin.rb` + +Selects queues based on configured weights. + +```ruby +queues: + - [critical, 3] # Selected 3× more often + - [default, 1] +``` + +### StrictPriority + +**Location:** `lib/shoryuken/polling/strict_priority.rb` + +Always processes higher-priority queues first. + +```ruby +polling: :strict_priority +queues: + - critical # Always first + - default # Only when critical is empty +``` + +--- + +## Thread Safety + +### Atomic Helpers + +Shoryuken v7 uses custom thread-safe primitives (no concurrent-ruby dependency). + +**Location:** `lib/shoryuken/helpers/` + +```ruby +# AtomicCounter - thread-safe integer +counter = Shoryuken::Helpers::AtomicCounter.new(0) +counter.increment +counter.decrement +counter.value + +# AtomicBoolean - thread-safe boolean +flag = Shoryuken::Helpers::AtomicBoolean.new(false) +flag.make_true +flag.make_false +flag.true? + +# AtomicHash - thread-safe hash +hash = Shoryuken::Helpers::AtomicHash.new +hash[:key] = value +``` + +### Executor + +Uses `Concurrent::Promise` for async processing: + +```ruby +Concurrent::Promise + .execute(executor: @executor) { Processor.process(queue, msg) } + .then { processor_done(queue) } + .rescue { processor_done(queue) } +``` + +--- + +## Middleware Chain + +**Location:** `lib/shoryuken/middleware/chain.rb` + +```ruby +chain.entries # Array of middleware classes +chain.add(Middleware) +chain.insert_before(Existing, New) +chain.insert_after(Existing, New) +chain.remove(Middleware) +chain.clear +``` + +**Execution:** +```ruby +def invoke(worker, queue, sqs_msg, body) + chain = @entries.dup + traverse_chain = lambda do + if chain.empty? + yield # Worker.perform + else + middleware = chain.shift + middleware.new.call(worker, queue, sqs_msg, body, &traverse_chain) + end + end + traverse_chain.call +end +``` + +--- + +## Lifecycle Events + +**Location:** `lib/shoryuken/options.rb` + +Events fired during operation: + +| Event | When | Parameters | +|-------|------|------------| +| `startup` | Process starts | None | +| `dispatch` | Before fetching | `queue_name` | +| `utilization_update` | Processor assigned/freed | `group`, `max_processors`, `busy_processors` | +| `quiet` | Stop requested | None | +| `shutdown` | Shutting down | None | +| `stopped` | Fully stopped | None | + +**Registration:** +```ruby +Shoryuken.configure_server do |config| + config.on(:startup) { puts "Starting" } + config.on(:utilization_update) do |group, utilization| + # utilization = busy / max (0.0 - 1.0) + end +end +``` + +--- + +## Worker Registry + +**Location:** `lib/shoryuken/worker_registry.rb` + +Maps queues to worker classes. + +```ruby +Shoryuken.worker_registry.register('my_queue', MyWorker) +Shoryuken.worker_registry.fetch('my_queue') # => MyWorker +Shoryuken.worker_registry.workers # => { 'my_queue' => MyWorker } +``` + +--- + +## ActiveJob Integration + +### Adapter + +**Location:** `lib/active_job/queue_adapters/shoryuken_adapter.rb` + +```ruby +def enqueue(job, options = {}) + # Build SQS message + # Send to queue +end + +def enqueue_at(job, timestamp) + # Calculate delay_seconds + # Call enqueue with delay +end + +def enqueue_all(jobs) + # Batch enqueue (Rails 7.1+) +end + +def stopping? + # Check launcher state +end +``` + +### JobWrapper + +**Location:** `lib/shoryuken/active_job/job_wrapper.rb` + +Worker that wraps ActiveJob execution: + +```ruby +class JobWrapper + include Shoryuken::Worker + + def perform(sqs_msg, body) + ActiveJob::Base.execute(body) + end +end +``` + +### CurrentAttributes + +**Location:** `lib/shoryuken/active_job/current_attributes.rb` + +Persists Rails CurrentAttributes across job boundaries: + +```ruby +module Persistence + # Prepended to adapter + def enqueue(job, options = {}) + # Serialize current attributes into job + super + end +end + +module Loading + # Prepended to JobWrapper + def perform(sqs_msg, body) + # Restore current attributes + super + ensure + # Reset current attributes + end +end +``` + +--- + +## Directory Structure + +``` +lib/shoryuken/ +├── active_job/ +│ ├── current_attributes.rb +│ └── job_wrapper.rb +├── helpers/ +│ ├── atomic_boolean.rb +│ ├── atomic_counter.rb +│ ├── atomic_hash.rb +│ ├── hash_utils.rb +│ └── string_utils.rb +├── middleware/ +│ ├── chain.rb +│ └── server/ +│ ├── active_record.rb +│ ├── auto_delete.rb +│ ├── auto_extend_visibility.rb +│ ├── exponential_backoff_retry.rb +│ └── timing.rb +├── polling/ +│ ├── base_strategy.rb +│ ├── strict_priority.rb +│ └── weighted_round_robin.rb +├── client.rb +├── extensions/ +├── fetcher.rb +├── launcher.rb +├── logging.rb +├── manager.rb +├── message/ +├── options.rb +├── processor.rb +├── queue.rb +├── runner.rb +├── util.rb +├── worker.rb +└── worker_registry.rb +``` + +--- + +## Zeitwerk Autoloading + +Shoryuken v7 uses Zeitwerk for autoloading: + +```ruby +# lib/shoryuken.rb +loader = Zeitwerk::Loader.for_gem +loader.setup +``` + +File/class naming follows Rails conventions: +- `lib/shoryuken/manager.rb` → `Shoryuken::Manager` +- `lib/shoryuken/helpers/atomic_counter.rb` → `Shoryuken::Helpers::AtomicCounter` + +--- + +## Extension Points + +### Custom Polling Strategy + +```ruby +class MyStrategy < Shoryuken::Polling::BaseStrategy + def initialize(queues, delay) + @queues = queues + @delay = delay + end + + def next_queue + # Return queue to poll or nil + end + + def messages_found(queue, count) + # Called after fetch + end + + def active_queues + # Return currently active queues + end +end + +# Register +Shoryuken.polling_strategy = MyStrategy +``` + +### Custom Executor + +```ruby +Shoryuken.launcher_executor = Concurrent::ThreadPoolExecutor.new( + min_threads: 5, + max_threads: 50, + max_queue: 100 +) +``` + +--- + +## Related + +- [[Performance Tuning]] - Optimization +- [[Middleware]] - Middleware system +- [[Lifecycle Events]] - Event hooks +- [[Processing Groups]] - Group configuration diff --git a/Bulk-Enqueuing.md b/Bulk-Enqueuing.md new file mode 100644 index 0000000..705b4fa --- /dev/null +++ b/Bulk-Enqueuing.md @@ -0,0 +1,259 @@ +# Bulk Enqueuing + +Shoryuken provides efficient bulk enqueuing of ActiveJob jobs using the SQS `SendMessageBatch` API. + +**Requires:** Rails 7.1+ + +## Overview + +Instead of sending individual `SendMessage` calls for each job, bulk enqueuing batches jobs into groups of 10 and sends them with a single API call. This is significantly more efficient for enqueuing many jobs at once. + +## Basic Usage + +```ruby +# Create job instances +jobs = users.map { |user| NotifyUserJob.new(user) } + +# Enqueue all jobs efficiently +ActiveJob.perform_all_later(jobs) +``` + +## How It Works + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Bulk Enqueue Flow │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ 100 jobs → Group by queue → Batch into 10s → SendBatch │ +│ │ +│ Queue A (60 jobs) → 6 × SendMessageBatch calls │ +│ Queue B (40 jobs) → 4 × SendMessageBatch calls │ +│ │ +│ Total: 10 API calls instead of 100 │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Processing Logic + +1. Jobs are grouped by queue name +2. Each queue's jobs are batched into groups of 10 (SQS limit) +3. Each batch is sent with a single `send_message_batch` call +4. Success/failure is tracked per job + +## Batch Limits + +| Limit | Value | +|-------|-------| +| Messages per batch | 10 | +| Total batch size | 1 MB | +| Individual message size | 256 KB | + +If a batch exceeds 1 MB, you may need to reduce the payload size of individual jobs. + +## Examples + +### Basic Bulk Enqueue + +```ruby +# Send welcome emails to new users +new_users = User.where(created_at: 24.hours.ago..) +jobs = new_users.map { |user| WelcomeEmailJob.new(user) } + +ActiveJob.perform_all_later(jobs) +``` + +### Multi-Queue Bulk Enqueue + +Jobs targeting different queues are automatically grouped: + +```ruby +jobs = [] + +# High priority notifications +urgent_users.each do |user| + jobs << NotifyJob.new(user).tap { |j| j.queue_name = 'high_priority' } +end + +# Normal priority notifications +regular_users.each do |user| + jobs << NotifyJob.new(user).tap { |j| j.queue_name = 'default' } +end + +# Both queues are batched efficiently +ActiveJob.perform_all_later(jobs) +``` + +### With Job Options + +```ruby +jobs = orders.map do |order| + ProcessOrderJob.new(order).tap do |job| + # Set FIFO queue options + job.message_group_id = order.customer_id.to_s + job.message_deduplication_id = "order-#{order.id}" + end +end + +ActiveJob.perform_all_later(jobs) +``` + +### Checking Results + +After bulk enqueuing, check which jobs were successfully enqueued: + +```ruby +jobs = users.map { |user| NotifyJob.new(user) } +ActiveJob.perform_all_later(jobs) + +# Check results +successful = jobs.select(&:successfully_enqueued?) +failed = jobs.reject(&:successfully_enqueued?) + +if failed.any? + Rails.logger.warn "Failed to enqueue #{failed.count} jobs" + # Handle failures (retry, alert, etc.) +end +``` + +## Performance Comparison + +| Method | 1000 Jobs | API Calls | Typical Time | +|--------|-----------|-----------|--------------| +| Individual `perform_later` | 1000 | 1000 | ~30-60s | +| Bulk `perform_all_later` | 1000 | 100 | ~3-6s | + +**~10x faster** for bulk operations. + +## Error Handling + +### Partial Failures + +SQS batch operations can partially succeed. Some jobs may be enqueued while others fail: + +```ruby +jobs = users.map { |user| ProcessJob.new(user) } +enqueued_count = ActiveJob.perform_all_later(jobs) + +if enqueued_count < jobs.count + failed_jobs = jobs.reject(&:successfully_enqueued?) + + failed_jobs.each do |job| + # Retry individually or log for investigation + Rails.logger.error "Failed to enqueue job for #{job.arguments}" + end +end +``` + +### Handling Large Payloads + +If jobs have large payloads, batches may exceed the 1 MB limit: + +```ruby +# For large payloads, consider storing data externally +class LargeDataJob < ApplicationJob + def perform(data_key) + data = Rails.cache.fetch(data_key) + process(data) + end +end + +# Store data in cache, pass only the key +jobs = large_datasets.map do |data| + key = "job_data:#{SecureRandom.uuid}" + Rails.cache.write(key, data, expires_in: 1.hour) + LargeDataJob.new(key) +end + +ActiveJob.perform_all_later(jobs) +``` + +## Best Practices + +### 1. Use for Batch Operations + +Bulk enqueuing is ideal for: +- Batch notifications +- Data migrations +- Report generation +- Scheduled bulk operations + +```ruby +# Good use case: Nightly cleanup +class NightlyCleanupJob < ApplicationJob + def self.schedule_all + jobs = Tenant.active.map { |t| new(t) } + ActiveJob.perform_all_later(jobs) + end +end +``` + +### 2. Handle Memory Efficiently + +For very large batches, process in chunks to avoid memory issues: + +```ruby +User.where(needs_notification: true).find_in_batches(batch_size: 500) do |users| + jobs = users.map { |user| NotifyJob.new(user) } + ActiveJob.perform_all_later(jobs) +end +``` + +### 3. Consider Queue Isolation + +For critical jobs, consider separate queues: + +```ruby +# Critical payment jobs get their own queue +payment_jobs = pending_payments.map do |p| + ProcessPaymentJob.new(p).tap { |j| j.queue_name = 'payments' } +end + +# Non-critical jobs on default queue +notification_jobs = users.map do |u| + NotifyJob.new(u).tap { |j| j.queue_name = 'default' } +end + +ActiveJob.perform_all_later(payment_jobs + notification_jobs) +``` + +## Implementation Details + +The `enqueue_all` method in Shoryuken's adapter: + +1. Groups jobs by `queue_name` +2. For each queue, processes jobs in batches of 10 +3. Calls `send_message_batch` for each batch +4. Tracks success/failure per job via `successfully_enqueued?` +5. Returns total count of successfully enqueued jobs + +```ruby +# Simplified implementation +def enqueue_all(jobs) + jobs.group_by(&:queue_name).each do |queue_name, queue_jobs| + queue = Shoryuken::Client.queues(queue_name) + + queue_jobs.each_slice(10) do |batch| + entries = batch.map.with_index do |job, idx| + { id: idx.to_s }.merge(message(queue, job)) + end + + response = queue.send_messages(entries: entries) + + # Mark successful jobs + response.successful.each do |r| + batch[r.id.to_i].successfully_enqueued = true + end + end + end + + jobs.count(&:successfully_enqueued?) +end +``` + +## Related + +- [[Rails Integration Active Job]] - Basic ActiveJob setup +- [[Sending a message]] - Individual message sending +- [AWS SQS SendMessageBatch](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessageBatch.html) diff --git a/Complete-Shoryuken-Setup-for-Rails-5-with-ActiveJob.md b/Complete-Shoryuken-Setup-for-Rails-5-with-ActiveJob.md deleted file mode 100644 index 9422921..0000000 --- a/Complete-Shoryuken-Setup-for-Rails-5-with-ActiveJob.md +++ /dev/null @@ -1,76 +0,0 @@ -## Create SQS queue - -Login to your Amazon AWS Console and go to SQS to create the queue or use the Shoryuken CLI. - -```shell -shoryuken sqs create QUEUE-NAME -``` - -See `shoryuken sqs help create`. - -## Install gem - -```ruby -gem 'shoryuken' -gem 'aws-sdk-sqs' -``` - -**Note:** `aws-sdk` gem is not necessary. - -## Configure your rails application to use Shoryuken - -```ruby -# config/application.rb -module YourRailsApp - class Application < Rails::Application - config.active_job.queue_adapter = :shoryuken - config.autoload_paths << "#{Rails.root}/app/jobs" - - # For jobs not otherwise autoloaded add the following: - config.eager_load_paths << "#{Rails.root}/app/jobs" - end -end -``` - -## Create your job - -```ruby -# app/jobs/your_job.rb -class YourJob < ApplicationJob - queue_as 'name_of_your_sqs_queue' - - def perform(resource_id) - resource = Resource.find(resource_id) - # perform your task on resource ... - end -end -``` - -Trigger your job where required: - -```ruby -YourJob.perform_later(resource.id) -``` - - -## Configure shoryuken.yml - -Make sure it's located in `config/shoryuken.yml`. This will be used by the worker to consume and process the messages sent to the queue in SQS. - -```yaml -concurrency: 15 -queues: - - [name_of_your_queue_in_sqs, 1] - - [<%= ENV['OTHER_QUEUE'] %>, 1] -``` - -You can use [dotenv](https://github.com/bkeepers/dotenv) to setup your environment variables. - -The number `1` above after the queue name is the queue weight, please refer to [Polling strategies -](https://github.com/phstc/shoryuken/wiki/Polling-strategies) for more information. - -## Run the following command to start consuming the SQS queue - -```shell -bundle exec shoryuken -R -C config/shoryuken.yml -``` \ No newline at end of file diff --git a/Complete-Shoryuken-Setup-for-Rails.md b/Complete-Shoryuken-Setup-for-Rails.md new file mode 100644 index 0000000..cbdde1a --- /dev/null +++ b/Complete-Shoryuken-Setup-for-Rails.md @@ -0,0 +1,309 @@ +# Complete Shoryuken Setup for Rails + +This guide provides a step-by-step walkthrough for setting up Shoryuken with Rails and ActiveJob. + +## Requirements + +- Ruby 3.2+ +- Rails 7.2+ +- AWS account with SQS access + +## 1. Create SQS Queue + +### Using AWS Console + +1. Log in to AWS Console +2. Navigate to SQS +3. Click "Create queue" +4. Choose Standard or FIFO +5. Configure settings (visibility timeout, retention) +6. Create queue + +### Using Shoryuken CLI + +```bash +shoryuken sqs create myapp-default +shoryuken sqs create myapp-critical +``` + +See `shoryuken sqs help create` for options. + +--- + +## 2. Install Gems + +```ruby +# Gemfile +gem 'shoryuken' +gem 'aws-sdk-sqs', '>= 1.66' +``` + +```bash +bundle install +``` + +--- + +## 3. Configure Rails + +```ruby +# config/application.rb +module MyApp + class Application < Rails::Application + config.active_job.queue_adapter = :shoryuken + end +end +``` + +**Note:** With Zeitwerk (Rails 7+), jobs in `app/jobs` are autoloaded automatically. + +--- + +## 4. Configure AWS Credentials + +Choose one method: + +### Environment Variables (Development) + +```bash +export AWS_ACCESS_KEY_ID=your_key +export AWS_SECRET_ACCESS_KEY=your_secret +export AWS_REGION=us-east-1 +``` + +### AWS Credentials File + +```ini +# ~/.aws/credentials +[default] +aws_access_key_id = your_key +aws_secret_access_key = your_secret +``` + +### IAM Role (Production) + +Use Instance Profiles (EC2), Task Roles (ECS), or IRSA (EKS). See [[Configure the AWS Client]]. + +--- + +## 5. Create Configuration File + +```yaml +# config/shoryuken.yml +concurrency: 25 +timeout: 25 +queues: + - [myapp-critical, 3] + - [myapp-default, 1] +``` + +--- + +## 6. Create Initializer + +```ruby +# config/initializers/shoryuken.rb +Shoryuken.configure_server do |config| + # Use Rails logger + Rails.logger = Shoryuken::Logging.logger + Rails.logger.level = Rails.application.config.log_level + + # Lifecycle events + config.on(:startup) do + Rails.logger.info "Shoryuken starting..." + end + + config.on(:shutdown) do + Rails.logger.info "Shoryuken shutting down..." + end +end + +# Enable ActiveJob queue name prefixing (optional) +Shoryuken.active_job_queue_name_prefixing = true +``` + +--- + +## 7. Create Jobs + +```ruby +# app/jobs/application_job.rb +class ApplicationJob < ActiveJob::Base + # Retry on transient failures + retry_on StandardError, wait: :polynomially_longer, attempts: 5 + + # Discard jobs with invalid records + discard_on ActiveJob::DeserializationError +end +``` + +```ruby +# app/jobs/process_order_job.rb +class ProcessOrderJob < ApplicationJob + queue_as :myapp_default + + def perform(order_id) + order = Order.find(order_id) + order.process! + end +end +``` + +--- + +## 8. Enqueue Jobs + +```ruby +# Enqueue for later processing +ProcessOrderJob.perform_later(order.id) + +# Enqueue with delay +ProcessOrderJob.set(wait: 5.minutes).perform_later(order.id) + +# Enqueue to specific queue +ProcessOrderJob.set(queue: :myapp_critical).perform_later(order.id) +``` + +--- + +## 9. Run Workers + +### Development + +```bash +bundle exec shoryuken -R -C config/shoryuken.yml +``` + +### With Foreman + +``` +# Procfile.dev +web: bin/rails server +worker: bundle exec shoryuken -R -C config/shoryuken.yml +``` + +```bash +foreman start -f Procfile.dev +``` + +--- + +## 10. Configure Database Pool + +Ensure your database connection pool is at least equal to Shoryuken concurrency: + +```yaml +# config/database.yml +production: + pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 25 } %> +``` + +--- + +## Complete Example + +### Gemfile + +```ruby +gem 'rails', '~> 8.0' +gem 'shoryuken' +gem 'aws-sdk-sqs', '>= 1.66' +``` + +### config/shoryuken.yml + +```yaml +concurrency: <%= ENV.fetch('SHORYUKEN_CONCURRENCY', 25) %> +timeout: 25 +queues: + - [<%= Rails.env %>_critical, 3] + - [<%= Rails.env %>_default, 1] +``` + +### config/initializers/shoryuken.rb + +```ruby +Shoryuken.configure_server do |config| + Rails.logger = Shoryuken::Logging.logger + Rails.logger.level = Rails.application.config.log_level + + config.on(:startup) do + Rails.logger.info "[Shoryuken] Starting with #{Shoryuken.options[:concurrency]} threads" + end +end + +# Use environment-based queue prefixes +Shoryuken.active_job_queue_name_prefixing = true +``` + +### config/environments/production.rb + +```ruby +# Enable ActiveJob queue prefix +config.active_job.queue_name_prefix = Rails.env +``` + +### app/jobs/application_job.rb + +```ruby +class ApplicationJob < ActiveJob::Base + retry_on StandardError, wait: :polynomially_longer, attempts: 5 + discard_on ActiveJob::DeserializationError +end +``` + +### app/jobs/send_notification_job.rb + +```ruby +class SendNotificationJob < ApplicationJob + queue_as :default + + def perform(user_id, message) + user = User.find(user_id) + NotificationService.send(user, message) + end +end +``` + +### Usage + +```ruby +# In a controller or service +SendNotificationJob.perform_later(current_user.id, "Welcome!") +``` + +--- + +## Next Steps + +- [[Rails Integration Active Job]] - Advanced ActiveJob features +- [[ActiveJob Continuations]] - Handle worker shutdowns +- [[CurrentAttributes]] - Pass request context to jobs +- [[Deployment]] - Production deployment +- [[Health Checks]] - Health monitoring + +--- + +## Troubleshooting + +### "No worker found for queue" + +Ensure queue names in `shoryuken.yml` match your job's `queue_as` setting. + +### Connection Pool Exhaustion + +Increase database pool to match concurrency: + +```yaml +pool: <%= ENV.fetch("SHORYUKEN_CONCURRENCY", 25) %> +``` + +### AWS Credentials Not Found + +Verify credentials are set: + +```bash +aws sts get-caller-identity +``` + +See [[Configure the AWS Client]] for credential options. diff --git a/Configure-the-AWS-Client.md b/Configure-the-AWS-Client.md index 68b103a..9999298 100644 --- a/Configure-the-AWS-Client.md +++ b/Configure-the-AWS-Client.md @@ -1,8 +1,323 @@ -There are a few ways to configure the AWS client: +# Configure the AWS Client -* Ensure the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` env vars are set. -* Create a `~/.aws/credentials` file. -* Set `Aws.config[:credentials]` or `Shoryuken.sqs_client = Aws::SQS::Client.new(...)` from Ruby code (e.g. in a Rails initializer) -* Use the Instance Profiles feature. The IAM role of the targeted machine must have an adequate SQS Policy. +This guide covers configuring AWS credentials for Shoryuken. -You can read about these in more detail [here](http://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Client.html). +## Credential Resolution Order + +The AWS SDK resolves credentials in this order: + +1. Explicit credentials in code +2. Environment variables +3. Shared credentials file (`~/.aws/credentials`) +4. IAM instance profile (EC2) / Task role (ECS) / IRSA (EKS) + +--- + +## Environment Variables + +```bash +export AWS_ACCESS_KEY_ID=your_access_key +export AWS_SECRET_ACCESS_KEY=your_secret_key +export AWS_REGION=us-east-1 + +# Optional: session token for temporary credentials +export AWS_SESSION_TOKEN=your_session_token +``` + +--- + +## Shared Credentials File + +```ini +# ~/.aws/credentials +[default] +aws_access_key_id = your_access_key +aws_secret_access_key = your_secret_key + +[production] +aws_access_key_id = prod_access_key +aws_secret_access_key = prod_secret_key +``` + +```ini +# ~/.aws/config +[default] +region = us-east-1 + +[profile production] +region = eu-west-1 +``` + +Use a named profile: + +```bash +export AWS_PROFILE=production +``` + +--- + +## Programmatic Configuration + +### Configure SQS Client + +```ruby +# config/initializers/shoryuken.rb +sqs_client = Aws::SQS::Client.new( + region: 'us-east-1', + access_key_id: ENV['AWS_ACCESS_KEY_ID'], + secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'] +) + +Shoryuken.configure_client do |config| + config.sqs_client = sqs_client +end + +Shoryuken.configure_server do |config| + config.sqs_client = sqs_client +end +``` + +### With AWS Configuration + +```ruby +# Set global AWS config +Aws.config.update( + region: 'us-east-1', + credentials: Aws::Credentials.new( + ENV['AWS_ACCESS_KEY_ID'], + ENV['AWS_SECRET_ACCESS_KEY'] + ) +) +``` + +--- + +## IAM Roles (Recommended for Production) + +### EC2 Instance Profile + +1. Create an IAM role with SQS permissions +2. Attach the role to your EC2 instance +3. No code changes needed - SDK auto-discovers credentials + +```ruby +# Just works - no explicit credentials +sqs_client = Aws::SQS::Client.new(region: 'us-east-1') +``` + +### ECS Task Role + +1. Create an IAM role with SQS permissions +2. Configure the role in your task definition: + +```json +{ + "family": "shoryuken-worker", + "taskRoleArn": "arn:aws:iam::123456789:role/shoryuken-task-role", + "containerDefinitions": [...] +} +``` + +### EKS IRSA (IAM Roles for Service Accounts) + +1. Create an IAM role with SQS permissions +2. Configure the OIDC trust relationship +3. Annotate the Kubernetes service account: + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: shoryuken-worker + annotations: + eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/shoryuken-worker-role +``` + +4. Use the service account in your deployment: + +```yaml +spec: + serviceAccountName: shoryuken-worker +``` + +--- + +## AWS SSO + +For development with AWS SSO: + +```bash +# Configure SSO +aws configure sso + +# Login +aws sso login --profile your-profile + +# Export profile +export AWS_PROFILE=your-profile +``` + +--- + +## Assume Role + +```ruby +# config/initializers/shoryuken.rb +sts_client = Aws::STS::Client.new(region: 'us-east-1') + +credentials = Aws::AssumeRoleCredentials.new( + client: sts_client, + role_arn: 'arn:aws:iam::123456789:role/ShoryukenRole', + role_session_name: 'shoryuken-session' +) + +sqs_client = Aws::SQS::Client.new( + region: 'us-east-1', + credentials: credentials +) + +Shoryuken.configure_server do |config| + config.sqs_client = sqs_client +end +``` + +--- + +## Cross-Account Access + +Access queues in another AWS account: + +```ruby +# Assume role in target account +credentials = Aws::AssumeRoleCredentials.new( + role_arn: 'arn:aws:iam::TARGET_ACCOUNT:role/CrossAccountSQS', + role_session_name: 'shoryuken' +) + +sqs_client = Aws::SQS::Client.new( + region: 'us-east-1', + credentials: credentials +) + +Shoryuken.configure_server do |config| + config.sqs_client = sqs_client +end +``` + +The target account role needs: +1. Trust policy allowing your account to assume the role +2. SQS permissions for the queues + +--- + +## Custom Endpoint (LocalStack/ElasticMQ) + +```ruby +# For local development +sqs_client = Aws::SQS::Client.new( + endpoint: ENV.fetch('SQS_ENDPOINT', 'http://localhost:4566'), + region: 'us-east-1', + access_key_id: 'test', + secret_access_key: 'test' +) + +Shoryuken.configure_client do |config| + config.sqs_client = sqs_client +end +``` + +--- + +## Configuration File (YAML) + +```yaml +# config/shoryuken.yml +aws: + access_key_id: <%= ENV['AWS_ACCESS_KEY_ID'] %> + secret_access_key: <%= ENV['AWS_SECRET_ACCESS_KEY'] %> + region: us-east-1 +``` + +**Note:** Prefer IAM roles over access keys in the config file. + +--- + +## Verifying Configuration + +```bash +# Verify AWS credentials +aws sts get-caller-identity + +# List SQS queues +aws sqs list-queues + +# Or via Shoryuken CLI +shoryuken sqs ls +``` + +--- + +## Troubleshooting + +### Credentials Not Found + +``` +Aws::Errors::MissingCredentialsError +``` + +**Solutions:** +1. Set environment variables +2. Configure `~/.aws/credentials` +3. Attach IAM role (EC2/ECS/EKS) + +### Invalid Credentials + +``` +Aws::SQS::Errors::InvalidClientTokenId +``` + +**Solutions:** +1. Verify credentials are correct +2. Check credentials haven't expired +3. Ensure IAM user/role is active + +### Region Not Specified + +``` +Aws::Errors::MissingRegionError +``` + +**Solutions:** +1. Set `AWS_REGION` environment variable +2. Configure region in `~/.aws/config` +3. Specify region in code + +### Access Denied + +``` +Aws::SQS::Errors::AccessDenied +``` + +**Solutions:** +1. Verify IAM permissions (see [[Amazon SQS IAM Policy for Shoryuken]]) +2. Check queue policy allows access +3. Verify assume role permissions + +--- + +## Best Practices + +1. **Use IAM roles** in production instead of access keys +2. **Rotate credentials** regularly if using access keys +3. **Use least privilege** - only grant necessary permissions +4. **Separate environments** - use different credentials/roles per environment +5. **Never commit credentials** - use environment variables or secrets management + +--- + +## Related + +- [[Amazon SQS IAM Policy for Shoryuken]] - IAM permissions +- [[Security Best Practices]] - Security guidelines +- [[Using a local mock SQS server]] - Local development +- [AWS SDK for Ruby Documentation](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/) diff --git a/Container-Deployment.md b/Container-Deployment.md new file mode 100644 index 0000000..1812ff4 --- /dev/null +++ b/Container-Deployment.md @@ -0,0 +1,543 @@ +# Container Deployment + +This guide covers deploying Shoryuken workers in containerized environments including Docker, Kubernetes, and AWS ECS/Fargate. + +## Docker + +### Basic Dockerfile + +```dockerfile +FROM ruby:3.3-slim + +WORKDIR /app + +# Install dependencies +RUN apt-get update -qq && \ + apt-get install -y --no-install-recommends \ + build-essential \ + libpq-dev \ + && rm -rf /var/lib/apt/lists/* + +# Install gems +COPY Gemfile Gemfile.lock ./ +RUN bundle config set --local deployment 'true' && \ + bundle config set --local without 'development test' && \ + bundle install + +# Copy application +COPY . . + +# Create non-root user +RUN useradd -m appuser && chown -R appuser:appuser /app +USER appuser + +# Health check port +EXPOSE 3001 + +# Default command +CMD ["bundle", "exec", "shoryuken", "-R", "-C", "config/shoryuken.yml"] +``` + +### Production Dockerfile (Multi-Stage) + +```dockerfile +# Build stage +FROM ruby:3.3-slim AS builder + +WORKDIR /app + +RUN apt-get update -qq && \ + apt-get install -y --no-install-recommends \ + build-essential \ + libpq-dev \ + git + +COPY Gemfile Gemfile.lock ./ +RUN bundle config set --local deployment 'true' && \ + bundle config set --local without 'development test' && \ + bundle install && \ + rm -rf ~/.bundle/cache + +COPY . . + +# Runtime stage +FROM ruby:3.3-slim + +WORKDIR /app + +RUN apt-get update -qq && \ + apt-get install -y --no-install-recommends \ + libpq5 \ + curl \ + && rm -rf /var/lib/apt/lists/* + +# Copy gems and app from builder +COPY --from=builder /app /app +COPY --from=builder /usr/local/bundle /usr/local/bundle + +# Create non-root user +RUN useradd -m -u 1000 appuser && chown -R appuser:appuser /app +USER appuser + +ENV RAILS_ENV=production +ENV HEALTH_PORT=3001 + +EXPOSE 3001 + +HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \ + CMD curl -f http://localhost:3001/health || exit 1 + +CMD ["bundle", "exec", "shoryuken", "-R", "-C", "config/shoryuken.yml"] +``` + +### Docker Compose + +```yaml +# docker-compose.yml +version: '3.8' + +services: + worker: + build: . + command: bundle exec shoryuken -R -C config/shoryuken.yml + environment: + - RAILS_ENV=production + - DATABASE_URL=postgres://db:5432/myapp + - AWS_REGION=us-east-1 + - AWS_ACCESS_KEY_ID + - AWS_SECRET_ACCESS_KEY + - HEALTH_PORT=3001 + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:3001/health"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 30s + depends_on: + - db + deploy: + replicas: 2 + resources: + limits: + memory: 512M + cpus: '1' + reservations: + memory: 256M + cpus: '0.25' + + db: + image: postgres:15 + volumes: + - postgres_data:/var/lib/postgresql/data + environment: + - POSTGRES_DB=myapp + - POSTGRES_PASSWORD=secret + +volumes: + postgres_data: +``` + +--- + +## Kubernetes + +### Deployment + +```yaml +# k8s/worker-deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: shoryuken-worker + labels: + app: shoryuken-worker +spec: + replicas: 3 + selector: + matchLabels: + app: shoryuken-worker + template: + metadata: + labels: + app: shoryuken-worker + spec: + serviceAccountName: shoryuken-worker + terminationGracePeriodSeconds: 300 # 5 minutes for graceful shutdown + containers: + - name: worker + image: myapp:latest + command: ["bundle", "exec", "shoryuken", "-R", "-C", "config/shoryuken.yml"] + env: + - name: RAILS_ENV + value: "production" + - name: HEALTH_PORT + value: "3001" + - name: DATABASE_URL + valueFrom: + secretKeyRef: + name: myapp-secrets + key: database-url + envFrom: + - configMapRef: + name: myapp-config + ports: + - containerPort: 3001 + name: health + resources: + requests: + memory: "256Mi" + cpu: "250m" + limits: + memory: "512Mi" + cpu: "1000m" + livenessProbe: + httpGet: + path: /health + port: health + initialDelaySeconds: 30 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 + readinessProbe: + httpGet: + path: /ready + port: health + initialDelaySeconds: 5 + periodSeconds: 5 + timeoutSeconds: 3 + failureThreshold: 2 + lifecycle: + preStop: + exec: + # Give time for load balancer to deregister + command: ["sleep", "5"] +``` + +### Service Account with IRSA (AWS) + +```yaml +# k8s/service-account.yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: shoryuken-worker + annotations: + # IAM Roles for Service Accounts + eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/shoryuken-worker-role +``` + +### ConfigMap + +```yaml +# k8s/configmap.yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: myapp-config +data: + AWS_REGION: "us-east-1" + RAILS_LOG_TO_STDOUT: "true" +``` + +### Horizontal Pod Autoscaler + +```yaml +# k8s/hpa.yaml +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: shoryuken-worker-hpa +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: shoryuken-worker + minReplicas: 2 + maxReplicas: 10 + metrics: + - type: External + external: + metric: + name: sqs_approximate_number_of_messages_visible + selector: + matchLabels: + queue_name: myapp-default + target: + type: AverageValue + averageValue: "100" # Scale up when > 100 messages per pod +``` + +### Pod Disruption Budget + +```yaml +# k8s/pdb.yaml +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: shoryuken-worker-pdb +spec: + minAvailable: 1 + selector: + matchLabels: + app: shoryuken-worker +``` + +--- + +## AWS ECS + +### Task Definition + +```json +{ + "family": "shoryuken-worker", + "networkMode": "awsvpc", + "requiresCompatibilities": ["FARGATE"], + "cpu": "512", + "memory": "1024", + "executionRoleArn": "arn:aws:iam::123456789:role/ecsTaskExecutionRole", + "taskRoleArn": "arn:aws:iam::123456789:role/shoryuken-worker-task-role", + "containerDefinitions": [ + { + "name": "worker", + "image": "123456789.dkr.ecr.us-east-1.amazonaws.com/myapp:latest", + "command": ["bundle", "exec", "shoryuken", "-R", "-C", "config/shoryuken.yml"], + "essential": true, + "environment": [ + {"name": "RAILS_ENV", "value": "production"}, + {"name": "HEALTH_PORT", "value": "3001"} + ], + "secrets": [ + { + "name": "DATABASE_URL", + "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789:secret:myapp/database-url" + } + ], + "portMappings": [ + { + "containerPort": 3001, + "protocol": "tcp" + } + ], + "healthCheck": { + "command": ["CMD-SHELL", "curl -f http://localhost:3001/health || exit 1"], + "interval": 30, + "timeout": 5, + "retries": 3, + "startPeriod": 60 + }, + "logConfiguration": { + "logDriver": "awslogs", + "options": { + "awslogs-group": "/ecs/shoryuken-worker", + "awslogs-region": "us-east-1", + "awslogs-stream-prefix": "worker" + } + }, + "stopTimeout": 300 + } + ] +} +``` + +### ECS Service + +```json +{ + "serviceName": "shoryuken-worker", + "cluster": "myapp-cluster", + "taskDefinition": "shoryuken-worker", + "desiredCount": 3, + "launchType": "FARGATE", + "networkConfiguration": { + "awsvpcConfiguration": { + "subnets": ["subnet-xxx", "subnet-yyy"], + "securityGroups": ["sg-xxx"], + "assignPublicIp": "DISABLED" + } + }, + "deploymentConfiguration": { + "maximumPercent": 200, + "minimumHealthyPercent": 100 + } +} +``` + +### ECS Auto Scaling + +```json +{ + "ScalableTarget": { + "ServiceNamespace": "ecs", + "ResourceId": "service/myapp-cluster/shoryuken-worker", + "ScalableDimension": "ecs:service:DesiredCount", + "MinCapacity": 2, + "MaxCapacity": 10 + }, + "ScalingPolicy": { + "PolicyName": "sqs-queue-depth", + "PolicyType": "TargetTrackingScaling", + "TargetTrackingScalingPolicyConfiguration": { + "TargetValue": 100, + "CustomizedMetricSpecification": { + "MetricName": "ApproximateNumberOfMessagesVisible", + "Namespace": "AWS/SQS", + "Dimensions": [ + {"Name": "QueueName", "Value": "myapp-default"} + ], + "Statistic": "Average" + }, + "ScaleInCooldown": 300, + "ScaleOutCooldown": 60 + } + } +} +``` + +--- + +## Graceful Shutdown + +### Understanding Container Signals + +When a container is terminated: + +1. **SIGTERM** is sent to the main process +2. Container has `terminationGracePeriodSeconds` to shut down +3. If still running, **SIGKILL** is sent + +### Shoryuken Shutdown Behavior + +``` +SIGTERM received + │ + ▼ +stopping? = true ────► Jobs can checkpoint via Continuations + │ + ▼ +Stop fetching new messages + │ + ▼ +Wait for in-flight messages to complete + │ + ▼ +Fire shutdown events + │ + ▼ +Exit cleanly +``` + +### Recommended Grace Period + +Set `terminationGracePeriodSeconds` based on your longest job: + +```yaml +# Kubernetes +spec: + terminationGracePeriodSeconds: 300 # 5 minutes + +# ECS Task Definition +"stopTimeout": 300 +``` + +### Shoryuken Timeout Configuration + +```yaml +# config/shoryuken.yml +timeout: 25 # Seconds to wait for workers during shutdown +``` + +--- + +## Resource Sizing + +### Memory + +| Concurrency | Recommended Memory | +|-------------|-------------------| +| 5 | 256 MB | +| 25 (default) | 512 MB | +| 50 | 1 GB | +| 100 | 2 GB | + +Add more for Rails applications with large codebases. + +### CPU + +| Concurrency | Recommended CPU | +|-------------|-----------------| +| 5 | 0.25 vCPU | +| 25 | 0.5-1 vCPU | +| 50 | 1-2 vCPU | + +### Database Connections + +Set your connection pool size ≥ concurrency: + +```yaml +# config/database.yml +production: + pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 25 } %> +``` + +--- + +## Environment Variables + +| Variable | Description | Example | +|----------|-------------|---------| +| `RAILS_ENV` | Rails environment | `production` | +| `AWS_REGION` | AWS region | `us-east-1` | +| `DATABASE_URL` | Database connection | `postgres://...` | +| `HEALTH_PORT` | Health check port | `3001` | +| `RAILS_LOG_TO_STDOUT` | Log to stdout | `true` | + +### AWS Credentials + +For containers, prefer IAM roles over access keys: + +- **EKS**: Use IRSA (IAM Roles for Service Accounts) +- **ECS**: Use Task IAM Role +- **EC2**: Use Instance Profile + +--- + +## Logging + +Configure Rails to log to stdout for container log aggregation: + +```ruby +# config/environments/production.rb +if ENV["RAILS_LOG_TO_STDOUT"].present? + logger = ActiveSupport::Logger.new(STDOUT) + logger.formatter = config.log_formatter + config.logger = ActiveSupport::TaggedLogging.new(logger) +end +``` + +### Structured Logging + +```ruby +# config/initializers/shoryuken.rb +Shoryuken.configure_server do |config| + config.on(:startup) do + Shoryuken.logger.formatter = proc do |severity, datetime, progname, msg| + { + timestamp: datetime.iso8601, + level: severity, + message: msg, + service: 'shoryuken-worker' + }.to_json + "\n" + end + end +end +``` + +--- + +## Related + +- [[Deployment]] - General deployment options +- [[Health Checks]] - Health check configuration +- [[Signals]] - Signal handling +- [[Lifecycle Events]] - Startup and shutdown hooks diff --git a/CurrentAttributes.md b/CurrentAttributes.md new file mode 100644 index 0000000..ea6dcd2 --- /dev/null +++ b/CurrentAttributes.md @@ -0,0 +1,302 @@ +# CurrentAttributes + +Shoryuken can automatically persist Rails `ActiveSupport::CurrentAttributes` from the code that enqueues a job to the job's execution. This is useful for maintaining request context like current user, tenant, or locale across background job processing. + +## Overview + +When you enqueue a job, Shoryuken serializes your CurrentAttributes into the message payload. When the job executes, these attributes are restored before `perform` runs and reset afterward. + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Request Context Flow │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ HTTP Request → Enqueue Job → Execute Job │ +│ Current.user = X (serialize X) Current.user = X │ +│ Current.tenant = Y (serialize Y) Current.tenant= Y│ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Setup + +### 1. Define Your CurrentAttributes + +```ruby +# app/models/current.rb +class Current < ActiveSupport::CurrentAttributes + attribute :user + attribute :tenant + attribute :request_id + attribute :locale +end +``` + +### 2. Configure Shoryuken to Persist Attributes + +```ruby +# config/initializers/shoryuken.rb +require 'shoryuken/active_job/current_attributes' +Shoryuken::ActiveJob::CurrentAttributes.persist('Current') +``` + +### 3. Set Attributes in Your Controller + +```ruby +# app/controllers/application_controller.rb +class ApplicationController < ActionController::Base + before_action :set_current_attributes + + private + + def set_current_attributes + Current.user = current_user + Current.tenant = current_tenant + Current.request_id = request.request_id + Current.locale = I18n.locale + end +end +``` + +### 4. Access Attributes in Your Jobs + +```ruby +# app/jobs/audit_job.rb +class AuditJob < ApplicationJob + queue_as :audit + + def perform(action:, resource:) + AuditLog.create!( + user: Current.user, + tenant: Current.tenant, + action: action, + resource: resource, + request_id: Current.request_id + ) + end +end +``` + +## Multiple CurrentAttributes Classes + +You can persist multiple CurrentAttributes classes: + +```ruby +# app/models/current.rb +class Current < ActiveSupport::CurrentAttributes + attribute :user, :request_id +end + +# app/models/request_context.rb +class RequestContext < ActiveSupport::CurrentAttributes + attribute :ip_address, :user_agent +end + +# config/initializers/shoryuken.rb +require 'shoryuken/active_job/current_attributes' +Shoryuken::ActiveJob::CurrentAttributes.persist('Current', 'RequestContext') +``` + +Both classes will be serialized and restored. + +## Serialization + +CurrentAttributes are serialized using `ActiveJob::Arguments`, which supports: + +- Primitive types (String, Integer, Float, etc.) +- Symbols +- Date, Time, DateTime +- Hash, Array +- GlobalID-compatible objects (ActiveRecord models) + +### Example with ActiveRecord + +```ruby +class Current < ActiveSupport::CurrentAttributes + attribute :user # ActiveRecord model +end + +# In your controller +Current.user = User.find(123) + +# In your job - the user is restored via GlobalID +def perform + Current.user # => # +end +``` + +### Handling Complex Objects + +For objects that don't support GlobalID, store identifiers instead: + +```ruby +class Current < ActiveSupport::CurrentAttributes + attribute :user_id, :tenant_id + + def user + @user ||= User.find(user_id) if user_id + end + + def tenant + @tenant ||= Tenant.find(tenant_id) if tenant_id + end +end +``` + +## Error Handling + +If an attribute fails to deserialize (e.g., the object was deleted), Shoryuken logs a warning but continues with the job: + +``` +WARN: Failed to restore CurrentAttributes Current: Couldn't find User with 'id'=123 +``` + +The job still executes, but that attribute will be `nil`. + +## Testing + +### RSpec + +```ruby +RSpec.describe AuditJob do + let(:user) { create(:user) } + let(:tenant) { create(:tenant) } + + before do + Current.user = user + Current.tenant = tenant + end + + after do + Current.reset + end + + it 'creates audit log with current user' do + expect { + described_class.perform_now(action: 'login', resource: user) + }.to change(AuditLog, :count).by(1) + + log = AuditLog.last + expect(log.user).to eq(user) + expect(log.tenant).to eq(tenant) + end +end +``` + +### Minitest + +```ruby +class AuditJobTest < ActiveJob::TestCase + setup do + @user = users(:one) + @tenant = tenants(:one) + Current.user = @user + Current.tenant = @tenant + end + + teardown do + Current.reset + end + + test 'creates audit log with current attributes' do + AuditJob.perform_now(action: 'update', resource: @user) + + log = AuditLog.last + assert_equal @user, log.user + assert_equal @tenant, log.tenant + end +end +``` + +## Common Use Cases + +### Multi-Tenant Applications + +```ruby +class Current < ActiveSupport::CurrentAttributes + attribute :tenant + + def tenant=(value) + super + # Optionally set tenant on connection when restored + ActiveRecord::Base.connection.execute("SET app.current_tenant = '#{value.id}'") if value + end +end +``` + +### Request Tracing + +```ruby +class Current < ActiveSupport::CurrentAttributes + attribute :request_id, :correlation_id + + def request_id + super || SecureRandom.uuid + end +end + +# In your job +class ProcessOrderJob < ApplicationJob + def perform(order) + Rails.logger.tagged(Current.request_id) do + # All logs include the request_id + order.process! + end + end +end +``` + +### Localization + +```ruby +class Current < ActiveSupport::CurrentAttributes + attribute :locale + + def locale=(value) + super + I18n.locale = value if value + end +end + +# Emails sent from jobs use the original request's locale +class WelcomeEmailJob < ApplicationJob + def perform(user) + I18n.with_locale(Current.locale || user.preferred_locale) do + UserMailer.welcome(user).deliver_now + end + end +end +``` + +## Implementation Details + +### How It Works + +1. **On Enqueue**: The `Persistence` module prepends to `ShoryukenAdapter#message` and adds current attributes to the message body. + +2. **On Execute**: The `Loading` module prepends to `JobWrapper#perform` and restores attributes before calling the original perform method. + +3. **After Execute**: Attributes are reset in an `ensure` block to prevent leaking between jobs. + +### Message Format + +```json +{ + "job_class": "AuditJob", + "arguments": ["action", "resource"], + "cattr": { + "user": { "_aj_globalid": "gid://app/User/123" }, + "tenant": { "_aj_globalid": "gid://app/Tenant/456" }, + "request_id": "abc-123" + } +} +``` + +## Comparison with Sidekiq + +This implementation is based on [Sidekiq's CurrentAttributes middleware](https://github.com/sidekiq/sidekiq/blob/main/lib/sidekiq/middleware/current_attributes.rb), using the same `ActiveJob::Arguments` serialization approach. + +## Related + +- [[Rails Integration Active Job]] - Basic ActiveJob setup +- [Rails CurrentAttributes Guide](https://api.rubyonrails.org/classes/ActiveSupport/CurrentAttributes.html) diff --git a/Deployment.md b/Deployment.md index 1e6632f..9747410 100644 --- a/Deployment.md +++ b/Deployment.md @@ -1,35 +1,308 @@ +# Deployment + +This guide covers deploying Shoryuken to various platforms. For containerized deployments (Docker, Kubernetes, ECS), see [[Container Deployment]]. + ## Heroku -Add `worker` in your [Procfile](https://devcenter.heroku.com/articles/procfile): +### Procfile Setup + +Add a `worker` process to your [Procfile](https://devcenter.heroku.com/articles/procfile): -```shell -worker: bundle exec shoryuken ... +``` +web: bundle exec puma -C config/puma.rb +worker: bundle exec shoryuken -R -C config/shoryuken.yml ``` -Then: +### Scale Workers -```shel +```shell +# Start worker dyno heroku ps:scale worker=1 + +# Scale to multiple workers +heroku ps:scale worker=3 + +# Disable web if not needed +heroku ps:scale web=0 ``` -Shoryuken does not require a `web` process, so you can disable `web`, in case you don't need it. +### Configuration + +Set AWS credentials via config vars: ```shell -heroku ps:scale web=0 +heroku config:set AWS_ACCESS_KEY_ID=xxx +heroku config:set AWS_SECRET_ACCESS_KEY=xxx +heroku config:set AWS_REGION=us-east-1 +``` + +### Graceful Shutdown + +Heroku sends SIGTERM and allows 30 seconds for shutdown. Configure timeout accordingly: + +```yaml +# config/shoryuken.yml +timeout: 25 # Leave headroom for cleanup +``` + +--- + +## systemd + +### Service File + +Create `/etc/systemd/system/shoryuken.service`: + +```ini +[Unit] +Description=Shoryuken background job processor +After=network.target + +[Service] +Type=simple +User=deploy +Group=deploy +WorkingDirectory=/var/www/myapp/current + +# Environment +Environment=RAILS_ENV=production +EnvironmentFile=/var/www/myapp/shared/.env + +# Command +ExecStart=/usr/local/bin/bundle exec shoryuken -R -C config/shoryuken.yml + +# Graceful shutdown +ExecReload=/bin/kill -USR1 $MAINPID +TimeoutStopSec=300 +KillSignal=SIGTERM + +# Restart on failure +Restart=on-failure +RestartSec=5 + +# Logging +StandardOutput=append:/var/log/shoryuken/shoryuken.log +StandardError=append:/var/log/shoryuken/shoryuken.log + +[Install] +WantedBy=multi-user.target +``` + +### Commands + +```shell +# Enable service +sudo systemctl enable shoryuken + +# Start +sudo systemctl start shoryuken + +# Stop (graceful) +sudo systemctl stop shoryuken + +# Reload (soft restart) +sudo systemctl reload shoryuken + +# Check status +sudo systemctl status shoryuken + +# View logs +sudo journalctl -u shoryuken -f +``` + +### Multiple Workers + +For multiple worker instances, use a template: + +```ini +# /etc/systemd/system/shoryuken@.service +[Unit] +Description=Shoryuken worker %i +After=network.target + +[Service] +Type=simple +User=deploy +WorkingDirectory=/var/www/myapp/current +Environment=RAILS_ENV=production +ExecStart=/usr/local/bin/bundle exec shoryuken -R -C config/shoryuken.yml +TimeoutStopSec=300 + +[Install] +WantedBy=multi-user.target ``` +Start multiple instances: + +```shell +sudo systemctl enable shoryuken@{1..3} +sudo systemctl start shoryuken@{1..3} +``` + +--- + ## Capistrano -Use [capistrano-shoryuken](https://github.com/joekhoobyar/capistrano-shoryuken). +Use [capistrano-shoryuken](https://github.com/joekhoobyar/capistrano-shoryuken) for deployment integration. + +### Gemfile -## OpsWorks +```ruby +group :development do + gem 'capistrano-shoryuken', require: false +end +``` + +### Capfile -See [OpsWorks Recipe](https://github.com/carpet/opsworks-shoryuken). +```ruby +require 'capistrano/shoryuken' +``` + +### Deploy Configuration + +```ruby +# config/deploy.rb +set :shoryuken_config, -> { "#{current_path}/config/shoryuken.yml" } +set :shoryuken_role, :worker +set :shoryuken_processes, 2 +``` + +--- ## AWS Elastic Beanstalk -See [.ebextension file here](https://github.com/phstc/shoryuken/wiki/AWS-Beanstalk-Config). +See [[AWS Beanstalk Config]] for detailed configuration. + +### Quick Setup (Procfile Method) + +``` +# Procfile +web: bundle exec puma -C /opt/elasticbeanstalk/config/private/pumaconf.rb +worker: bundle exec shoryuken -R -C config/shoryuken.yml +``` + +--- + +## AWS OpsWorks + +Use the [OpsWorks Recipe](https://github.com/carpet/opsworks-shoryuken) for Chef-based deployments. + +--- + +## Production Checklist + +### Before Deploying + +- [ ] AWS credentials configured (prefer IAM roles) +- [ ] SQS queues created +- [ ] IAM permissions set (see [[Amazon SQS IAM Policy for Shoryuken]]) +- [ ] Database connection pool sized ≥ concurrency +- [ ] Shoryuken configuration file ready + +### Configuration + +```yaml +# config/shoryuken.yml +concurrency: 25 +timeout: 25 +queues: + - [critical, 3] + - [default, 2] + - [low, 1] +``` + +### Database Pool + +```yaml +# config/database.yml +production: + pool: <%= ENV.fetch("DB_POOL") { 25 } %> +``` + +### Health Checks + +See [[Health Checks]] for setting up health monitoring. + +### Logging + +```ruby +# config/initializers/shoryuken.rb +Shoryuken.configure_server do |config| + Rails.logger = Shoryuken::Logging.logger + Rails.logger.level = Rails.application.config.log_level +end +``` + +--- + +## Monitoring + +### Signals for Debugging + +```shell +# Print thread backtraces and stats +kill -TTIN $(cat tmp/pids/shoryuken.pid) +``` + +See [[Signals]] for all available signals. + +### CloudWatch Metrics + +Monitor these SQS metrics: +- `ApproximateNumberOfMessagesVisible` - Queue depth +- `ApproximateAgeOfOldestMessage` - Processing delay +- `NumberOfMessagesReceived` - Throughput +- `NumberOfMessagesDeleted` - Completion rate + +### Error Tracking + +See [[Sentry.io Integration]] and [[Honeybadger Integration]]. + +--- + +## Scaling Strategies + +### Horizontal Scaling + +Add more worker processes/containers. Each worker: +- Has its own connection pool +- Polls independently +- Handles `concurrency` simultaneous jobs + +### Queue-Based Auto Scaling + +Scale based on queue depth: + +``` +Messages per worker = ApproximateNumberOfMessagesVisible / desired_workers +Target: 50-100 messages per worker +``` + +### Processing Groups + +Isolate critical queues: + +```yaml +# config/shoryuken.yml +groups: + critical: + concurrency: 10 + queues: + - [payments, 1] + default: + concurrency: 25 + queues: + - [emails, 2] + - [reports, 1] +``` + +--- -## systemd integration +## Related -See [systemd integration](https://github.com/phstc/shoryuken/issues/320#issuecomment-282290069). \ No newline at end of file +- [[Container Deployment]] - Docker, Kubernetes, ECS +- [[AWS Beanstalk Config]] - Elastic Beanstalk specifics +- [[Health Checks]] - Health monitoring +- [[Signals]] - Process control +- [[Processing Groups]] - Queue isolation diff --git a/From-Sidekiq-to-Shoryuken.md b/From-Sidekiq-to-Shoryuken.md index 02a5aea..618d48b 100644 --- a/From-Sidekiq-to-Shoryuken.md +++ b/From-Sidekiq-to-Shoryuken.md @@ -1,60 +1,140 @@ -Shoryuken is a drop-in replacement for Sidekiq, the code changes should be minor `s/sidekiq/shoryuken`. But as Shoryuken "reads" messages from SQS, instead of Redis, you will probably need a three steps migration: +# From Sidekiq to Shoryuken -* Stop sending jobs to Sidekiq -* Start using Shoryuken -* Keep Sidekiq running until it consumes all pending jobs. +This guide helps you migrate from Sidekiq to Shoryuken. The transition is mostly straightforward since Shoryuken's API is similar to Sidekiq's. +## Migration Strategy -## Worker +1. **Stop sending jobs to Sidekiq** - Update code to use Shoryuken +2. **Start Shoryuken workers** - Begin processing new jobs +3. **Keep Sidekiq running** - Until it drains pending jobs from Redis -### Sidekiq +--- + +## Feature Comparison + +| Feature | Sidekiq | Shoryuken | +|---------|---------|-----------| +| Backend | Redis | AWS SQS | +| Web UI | Built-in | Not included | +| Retry mechanism | Built-in | Via `retry_intervals` or DLQ | +| Scheduled jobs | Built-in (Redis) | SQS delay (15 min max) | +| Unique jobs | Sidekiq Enterprise | Not built-in | +| Batches | Sidekiq Pro | Not built-in | +| Rate limiting | Sidekiq Enterprise | Not built-in | +| CurrentAttributes | Sidekiq 7+ | Shoryuken v7+ | + +--- + +## Worker Migration + +### Sidekiq Worker ```ruby class MyWorker include Sidekiq::Worker - sidekiq_options queue: 'my_queue' + sidekiq_options queue: 'my_queue', retry: 5 - def perform(arg) - # ... + def perform(user_id, action) + user = User.find(user_id) + user.send(action) end end ``` -### Shoryuken +### Shoryuken Worker ```ruby class MyWorker include Shoryuken::Worker - shoryuken_options queue: 'my_queue', auto_delete: true + shoryuken_options queue: 'my_queue', + auto_delete: true, + body_parser: :json, + retry_intervals: [60, 300, 900, 3600] - def perform(sqs_msg, arg) - # ... + def perform(sqs_msg, body) + user = User.find(body['user_id']) + user.send(body['action']) end end ``` -Note that if your Sidekiq worker has more than 1 argument you will need to configure message body parser for Shoryuken worker and define only one argument in a `#perform` method: +### Key Differences + +| Aspect | Sidekiq | Shoryuken | +|--------|---------|-----------| +| Module | `Sidekiq::Worker` | `Shoryuken::Worker` | +| Options | `sidekiq_options` | `shoryuken_options` | +| Perform args | Multiple args | `sqs_msg, body` | +| Auto delete | Automatic | `auto_delete: true` | +| Retries | `retry: N` | `retry_intervals: [...]` | + +--- + +## Multiple Arguments + +Sidekiq passes multiple arguments directly. Shoryuken passes the message and parsed body. + +### Sidekiq ```ruby -class YourWorker - include Shoryuken::Worker +def perform(user_id, post_id, action) + # Direct access to args +end - shoryuken_options queue: "queue", auto_delete: true, body_parser: JSON +MyWorker.perform_async(123, 456, 'publish') +``` - def perform(_sqs_message, args) - args.fetch("user_id") - args.fetch("post_id") - # ... - end +### Shoryuken + +```ruby +shoryuken_options body_parser: :json + +def perform(sqs_msg, body) + user_id = body['user_id'] + post_id = body['post_id'] + action = body['action'] end -YourWorker.perform_async(user_id: "XXX", post_id: "YYY") +MyWorker.perform_async(user_id: 123, post_id: 456, action: 'publish') ``` +--- + +## ActiveJob (Recommended) -## Configuration file +Using ActiveJob makes migration simpler - no worker changes needed: + +### Sidekiq Configuration + +```ruby +# config/application.rb +config.active_job.queue_adapter = :sidekiq +``` + +### Shoryuken Configuration + +```ruby +# config/application.rb +config.active_job.queue_adapter = :shoryuken +``` + +### Job Class (No Changes) + +```ruby +class MyJob < ApplicationJob + queue_as :default + + def perform(user_id, action) + # Same code works with both adapters + end +end +``` + +--- + +## Configuration File ### sidekiq.yml @@ -63,43 +143,197 @@ concurrency: 25 pidfile: tmp/pids/sidekiq.pid queues: - default - - [myqueue, 2] + - [critical, 3] + - [low, 1] ``` ### shoryuken.yml ```yaml concurrency: 25 +timeout: 25 pidfile: tmp/pids/shoryuken.pid queues: - default - - [myqueue, 2] + - [critical, 3] + - [low, 1] ``` -## Sending messages +**Note:** Add `timeout` for graceful shutdown behavior. + +--- + +## Sending Jobs ### Sidekiq ```ruby MyWorker.perform_async('test') +MyWorker.perform_in(5.minutes, 'test') +MyWorker.perform_at(1.hour.from_now, 'test') ``` ### Shoryuken ```ruby MyWorker.perform_async('test') +MyWorker.perform_in(5.minutes, 'test') # Max 15 minutes (SQS limit) +MyWorker.perform_at(1.hour.from_now, 'test') # Max 15 minutes delay +``` + +**Note:** SQS has a maximum delay of 15 minutes. For longer delays, use scheduled jobs with a scheduler or re-enqueue pattern. + +--- + +## Running Workers + +### Sidekiq + +```bash +bundle exec sidekiq -C config/sidekiq.yml +``` + +### Shoryuken + +```bash +bundle exec shoryuken -R -C config/shoryuken.yml ``` -## To get it running +**Note:** Use `-R` flag for Rails applications. + +--- + +## Retry Handling ### Sidekiq -```shell -bundle exec sidekiq -r ./my_worker.rb -q my_queue +```ruby +sidekiq_options retry: 5 # Automatic exponential backoff ``` ### Shoryuken -```shell -bundle exec shoryuken -r ./my_worker.rb -q my_queue -``` \ No newline at end of file +```ruby +# Option 1: Explicit intervals +shoryuken_options retry_intervals: [60, 300, 900, 3600, 7200] + +# Option 2: Dynamic calculation +shoryuken_options retry_intervals: ->(attempts) { (attempts ** 2) * 60 } + +# Option 3: Use Dead Letter Queue (recommended for production) +# Configure in SQS console +``` + +--- + +## Middleware + +### Sidekiq + +```ruby +Sidekiq.configure_server do |config| + config.server_middleware do |chain| + chain.add MyMiddleware + end +end +``` + +### Shoryuken + +```ruby +Shoryuken.configure_server do |config| + config.server_middleware do |chain| + chain.add MyMiddleware + end +end +``` + +Middleware signature differs slightly: + +```ruby +# Sidekiq +def call(worker, job, queue) + yield +end + +# Shoryuken +def call(worker_instance, queue, sqs_msg, body) + yield +end +``` + +--- + +## CurrentAttributes (Sidekiq 7+ / Shoryuken 7+) + +### Sidekiq + +```ruby +# Automatically persisted in Sidekiq 7+ +class Current < ActiveSupport::CurrentAttributes + attribute :user +end +``` + +### Shoryuken + +```ruby +# config/initializers/shoryuken.rb +Shoryuken::ActiveJob::CurrentAttributes.persist('Current') +``` + +--- + +## Things to Consider + +### No Web UI + +Shoryuken doesn't include a web UI. Monitor via: +- AWS CloudWatch (SQS metrics) +- Custom dashboards +- [[Monitoring and Metrics]] + +### No Built-in Scheduled Jobs + +SQS delays max 15 minutes. For scheduled jobs: +- Use AWS EventBridge Scheduler +- Use a separate scheduler service +- Re-enqueue pattern for longer delays + +### Queue Creation + +Create SQS queues before deployment: + +```bash +shoryuken sqs create my_queue +shoryuken sqs create critical +``` + +### IAM Permissions + +Ensure your AWS credentials have SQS access. See [[Amazon SQS IAM Policy for Shoryuken]]. + +--- + +## Migration Checklist + +- [ ] Create SQS queues for each Sidekiq queue +- [ ] Configure AWS credentials +- [ ] Update Gemfile (`sidekiq` → `shoryuken`, add `aws-sdk-sqs`) +- [ ] Convert workers or switch ActiveJob adapter +- [ ] Update configuration file +- [ ] Add `auto_delete: true` to workers +- [ ] Configure retry strategy (intervals or DLQ) +- [ ] Update deployment scripts +- [ ] Set up monitoring (CloudWatch, etc.) +- [ ] Test thoroughly in staging + +--- + +## Related + +- [[Getting Started]] - Initial setup +- [[Worker options]] - Worker configuration +- [[Shoryuken options]] - Global configuration +- [[Rails Integration Active Job]] - ActiveJob integration +- [[CurrentAttributes]] - Request context persistence diff --git a/Getting-Started.md b/Getting-Started.md index 156df88..c048fe6 100644 --- a/Getting-Started.md +++ b/Getting-Started.md @@ -1,8 +1,60 @@ +# Getting Started + +## Prerequisites + +Shoryuken requires: + +- **Ruby 3.2+** +- **Rails 7.2+** (for Rails/ActiveJob integration) +- **aws-sdk-sqs >= 1.66** + +## Installation + +Add Shoryuken to your Gemfile: + +```ruby +gem 'shoryuken' +``` + +Run bundler: + +```shell +bundle install +``` + +## AWS Configuration + +Shoryuken uses the AWS SDK for Ruby. Configure your AWS credentials using one of these methods: + +1. **Environment variables** (recommended for development): + ```shell + export AWS_ACCESS_KEY_ID=your_access_key + export AWS_SECRET_ACCESS_KEY=your_secret_key + export AWS_REGION=us-east-1 + ``` + +2. **AWS credentials file** (`~/.aws/credentials`): + ```ini + [default] + aws_access_key_id = your_access_key + aws_secret_access_key = your_secret_key + ``` + +3. **IAM role** (recommended for production on AWS): + - EC2 instance profile + - ECS task role + - EKS service account + +See [[Configure the AWS Client]] for more details. + +--- + ## Plain Ruby -### Create a worker +### 1. Create a Worker ```ruby +# hello_worker.rb class HelloWorker include Shoryuken::Worker @@ -14,31 +66,44 @@ class HelloWorker end ``` -### Create a queue +### 2. Create a Queue ```shell bundle exec shoryuken sqs create hello ``` -### Start Shoryuken +### 3. Start Shoryuken ```shell bundle exec shoryuken -q hello -r ./hello_worker.rb ``` -### Enqueue a message +### 4. Enqueue a Message ```ruby +require_relative 'hello_worker' + HelloWorker.perform_async('Ken') ``` +--- + ## Rails -### Create a job +### 1. Create an ApplicationJob + +```ruby +# app/jobs/application_job.rb +class ApplicationJob < ActiveJob::Base + # Common job configuration can go here +end +``` + +### 2. Create a Job ```ruby # app/jobs/hello_job.rb -class HelloJob < ActiveJob::Base +class HelloJob < ApplicationJob queue_as 'hello' def perform(name) @@ -47,13 +112,13 @@ class HelloJob < ActiveJob::Base end ``` -### Create a queue +### 3. Create a Queue ```shell bundle exec shoryuken sqs create hello ``` -### Set the queue backend +### 4. Configure the Queue Adapter ```ruby # config/application.rb @@ -64,15 +129,68 @@ module YourApp end ``` -### Start Shoryuken +### 5. Create a Shoryuken Initializer (Optional) + +```ruby +# config/initializers/shoryuken.rb +Shoryuken.configure_server do |config| + # Replace Rails logger so messages are logged to Shoryuken's log + Rails.logger = Shoryuken::Logging.logger + Rails.logger.level = Rails.application.config.log_level + + # Add server middleware + # config.server_middleware do |chain| + # chain.add MyMiddleware + # end +end + +Shoryuken.configure_client do |config| + # Client-side configuration (used when enqueuing jobs) +end +``` + +### 6. Start Shoryuken ```shell bundle exec shoryuken -q hello -R ``` -### Enqueue a message +The `-R` flag loads your Rails application. Workers in `app/jobs` are auto-loaded via Zeitwerk. + +### 7. Enqueue a Message ```ruby HelloJob.perform_later('Ken') ``` +--- + +## Configuration File + +For more complex setups, create a configuration file: + +```yaml +# config/shoryuken.yml +concurrency: 25 +delay: 0 +queues: + - [default, 1] + - [high_priority, 2] +``` + +Start Shoryuken with the configuration file: + +```shell +bundle exec shoryuken -R -C config/shoryuken.yml +``` + +See [[Shoryuken options]] for all available configuration options. + +--- + +## Next Steps + +- [[Rails Integration Active Job]] - Full ActiveJob integration guide +- [[Worker options]] - Configure worker behavior +- [[Middleware]] - Add cross-cutting concerns +- [[Deployment]] - Deploy to production diff --git a/Health-Checks.md b/Health-Checks.md new file mode 100644 index 0000000..c882e06 --- /dev/null +++ b/Health-Checks.md @@ -0,0 +1,396 @@ +# Health Checks + +Shoryuken provides a health check API to monitor the status of processing groups. This is useful for load balancer health checks, Kubernetes probes, and monitoring systems. + +## The `healthy?` Method + +The `Launcher` class provides a `healthy?` method that returns `true` when all processing groups are running normally: + +```ruby +launcher = Shoryuken::Runner.instance.launcher +launcher.healthy? # => true or false +``` + +### What It Checks + +The `healthy?` method verifies that: +- All configured processing groups have a running manager +- Each manager's `running?` method returns `true` + +```ruby +def healthy? + Shoryuken.groups.keys.all? do |group| + manager = @managers.find { |m| m.group == group } + manager && manager.running? + end +end +``` + +## HTTP Health Endpoint + +Shoryuken doesn't include a built-in HTTP server, but you can easily add one using a lifecycle event: + +### Using WEBrick + +```ruby +# config/initializers/shoryuken.rb +require 'webrick' + +Shoryuken.configure_server do |config| + config.on(:startup) do + @health_server = WEBrick::HTTPServer.new(Port: 3001, Logger: WEBrick::Log.new("/dev/null")) + + @health_server.mount_proc '/health' do |req, res| + launcher = Shoryuken::Runner.instance.launcher + + if launcher&.healthy? + res.status = 200 + res.body = 'OK' + else + res.status = 503 + res.body = 'Unhealthy' + end + end + + Thread.new { @health_server.start } + end + + config.on(:shutdown) do + @health_server&.shutdown + end +end +``` + +### Using Rack + +```ruby +# config/initializers/shoryuken.rb +require 'rack' +require 'rack/handler/webrick' + +Shoryuken.configure_server do |config| + config.on(:startup) do + health_app = lambda do |env| + launcher = Shoryuken::Runner.instance.launcher + + if launcher&.healthy? + [200, { 'Content-Type' => 'text/plain' }, ['OK']] + else + [503, { 'Content-Type' => 'text/plain' }, ['Unhealthy']] + end + end + + Thread.new do + Rack::Handler::WEBrick.run(health_app, Port: 3001, Logger: WEBrick::Log.new("/dev/null")) + end + end +end +``` + +### Using Puma (Production Recommended) + +For production, consider using a minimal Puma server: + +```ruby +# lib/shoryuken_health_server.rb +require 'puma' +require 'rack' + +class ShoryukenHealthServer + def initialize(port: 3001) + @port = port + end + + def start + app = self + + @server = Puma::Server.new(app) + @server.add_tcp_listener('0.0.0.0', @port) + @server.run + end + + def stop + @server&.stop(true) + end + + def call(env) + case env['PATH_INFO'] + when '/health', '/healthz' + health_check + when '/ready', '/readiness' + readiness_check + else + [404, {}, ['Not Found']] + end + end + + private + + def health_check + launcher = Shoryuken::Runner.instance.launcher + + if launcher&.healthy? + [200, { 'Content-Type' => 'application/json' }, [{ status: 'healthy' }.to_json]] + else + [503, { 'Content-Type' => 'application/json' }, [{ status: 'unhealthy' }.to_json]] + end + end + + def readiness_check + launcher = Shoryuken::Runner.instance.launcher + + if launcher && !launcher.stopping? + [200, { 'Content-Type' => 'application/json' }, [{ status: 'ready' }.to_json]] + else + [503, { 'Content-Type' => 'application/json' }, [{ status: 'not_ready' }.to_json]] + end + end +end + +# config/initializers/shoryuken.rb +require_relative '../../lib/shoryuken_health_server' + +Shoryuken.configure_server do |config| + config.on(:startup) do + @health_server = ShoryukenHealthServer.new(port: ENV.fetch('HEALTH_PORT', 3001)) + Thread.new { @health_server.start } + end + + config.on(:shutdown) do + @health_server&.stop + end +end +``` + +## Kubernetes Integration + +### Liveness Probe + +Use for determining if the container should be restarted: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: shoryuken-worker +spec: + template: + spec: + containers: + - name: worker + image: myapp:latest + command: ["bundle", "exec", "shoryuken", "-R", "-C", "config/shoryuken.yml"] + ports: + - containerPort: 3001 + name: health + livenessProbe: + httpGet: + path: /health + port: health + initialDelaySeconds: 30 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 +``` + +### Readiness Probe + +Use for determining if the container should receive traffic (useful if you have HTTP endpoints): + +```yaml + readinessProbe: + httpGet: + path: /ready + port: health + initialDelaySeconds: 5 + periodSeconds: 5 + timeoutSeconds: 3 + failureThreshold: 2 +``` + +### Complete Example + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: shoryuken-worker +spec: + replicas: 3 + selector: + matchLabels: + app: shoryuken-worker + template: + metadata: + labels: + app: shoryuken-worker + spec: + terminationGracePeriodSeconds: 300 # 5 minutes for graceful shutdown + containers: + - name: worker + image: myapp:latest + command: ["bundle", "exec", "shoryuken", "-R", "-C", "config/shoryuken.yml"] + env: + - name: HEALTH_PORT + value: "3001" + ports: + - containerPort: 3001 + name: health + resources: + requests: + memory: "256Mi" + cpu: "250m" + limits: + memory: "512Mi" + cpu: "1000m" + livenessProbe: + httpGet: + path: /health + port: health + initialDelaySeconds: 30 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 + readinessProbe: + httpGet: + path: /ready + port: health + initialDelaySeconds: 5 + periodSeconds: 5 +``` + +## AWS ECS Integration + +### Task Definition Health Check + +```json +{ + "containerDefinitions": [ + { + "name": "shoryuken-worker", + "image": "myapp:latest", + "command": ["bundle", "exec", "shoryuken", "-R", "-C", "config/shoryuken.yml"], + "healthCheck": { + "command": ["CMD-SHELL", "curl -f http://localhost:3001/health || exit 1"], + "interval": 30, + "timeout": 5, + "retries": 3, + "startPeriod": 60 + }, + "portMappings": [ + { + "containerPort": 3001, + "protocol": "tcp" + } + ] + } + ] +} +``` + +## Load Balancer Health Checks + +### AWS ALB/NLB + +```yaml +# CloudFormation +TargetGroup: + Type: AWS::ElasticLoadBalancingV2::TargetGroup + Properties: + HealthCheckPath: /health + HealthCheckPort: "3001" + HealthCheckProtocol: HTTP + HealthCheckIntervalSeconds: 30 + HealthCheckTimeoutSeconds: 10 + HealthyThresholdCount: 2 + UnhealthyThresholdCount: 3 +``` + +## Health Check During Shutdown + +During graceful shutdown: + +1. `stopping?` becomes `true` immediately +2. `healthy?` remains `true` while managers are running +3. `healthy?` becomes `false` when managers stop + +This allows load balancers to drain connections while the worker processes remaining messages. + +### Recommended Shutdown Flow + +``` +SIGTERM received + │ + ▼ +stopping? = true ─────► Readiness probe fails (stop accepting new work) + │ + ▼ +Process in-flight messages + │ + ▼ +Managers stop + │ + ▼ +healthy? = false ─────► Liveness probe fails (container can be removed) +``` + +## Monitoring Integration + +### Prometheus Metrics + +```ruby +# config/initializers/shoryuken.rb +require 'prometheus/client' + +prometheus = Prometheus::Client.registry + +shoryuken_healthy = Prometheus::Client::Gauge.new( + :shoryuken_healthy, + docstring: 'Whether Shoryuken is healthy', + labels: [:group] +) +prometheus.register(shoryuken_healthy) + +Shoryuken.configure_server do |config| + config.on(:startup) do + Thread.new do + loop do + launcher = Shoryuken::Runner.instance.launcher + + Shoryuken.groups.keys.each do |group| + manager = launcher&.instance_variable_get(:@managers)&.find { |m| m.group == group } + healthy = manager&.running? ? 1 : 0 + shoryuken_healthy.set(healthy, labels: { group: group }) + end + + sleep 10 + end + end + end +end +``` + +### DataDog + +```ruby +# config/initializers/shoryuken.rb +Shoryuken.configure_server do |config| + config.on(:startup) do + Thread.new do + loop do + launcher = Shoryuken::Runner.instance.launcher + healthy = launcher&.healthy? ? 1 : 0 + + Datadog::Statsd.instance.gauge('shoryuken.healthy', healthy) + sleep 10 + end + end + end +end +``` + +## Related + +- [[Lifecycle Events]] - Startup and shutdown hooks +- [[Signals]] - Understanding shutdown signals +- [[Deployment]] - Production deployment guides diff --git a/Home.md b/Home.md index 7d458b4..c6d72ff 100644 --- a/Home.md +++ b/Home.md @@ -1,14 +1,84 @@ -Welcome to the Shoryuken wiki! - -* [[Getting Started]] -* [[Rails Integration Active Job]] -* [[Worker options]] -* [[Shoryuken options]] -* [[From Sidekiq to Shoryuken]] -* [[Long Polling]] -* [[Processing Groups]] -* [[Middleware]] -* [[Retrying a message]] -* [[Sending a message]] -* [[Deployment]] -* [[Using a local mock SQS server]] +# Shoryuken Wiki + +Shoryuken is a super-efficient AWS SQS thread-based message processor for Ruby. + +**Requirements:** Ruby 3.2+ | Rails 7.2+ | aws-sdk-sqs >= 1.66 + +--- + +## Getting Started + +- [[Getting Started]] - Installation and basic setup +- [[Configure the AWS Client]] - AWS credentials configuration +- [[Complete Shoryuken Setup for Rails]] - Full Rails integration walkthrough + +## Rails & ActiveJob + +- [[Rails Integration Active Job]] - ActiveJob adapter setup and usage +- [[ActiveJob Continuations]] - Graceful job interruption for long-running jobs (Rails 8.1+) +- [[CurrentAttributes]] - Persist Rails CurrentAttributes across job execution +- [[Bulk Enqueuing]] - Efficient batch enqueuing with `perform_all_later` + +## Workers & Queues + +- [[Worker options]] - Worker configuration (`shoryuken_options`) +- [[Shoryuken options]] - Global configuration and CLI options +- [[Processing Groups]] - Group queues with separate concurrency +- [[FIFO Queues]] - FIFO queue support and configuration +- [[Queue URL]] - Multi-region and cross-account queue access + +## Message Handling + +- [[Sending a message]] - Enqueuing messages and jobs +- [[Retrying a message]] - Retry strategies and exponential backoff +- [[Middleware]] - Server middleware for cross-cutting concerns +- [[Lifecycle Events]] - Hooks for startup, shutdown, and processing events + +## Polling & Performance + +- [[Polling strategies]] - WeightedRoundRobin and StrictPriority +- [[Long Polling]] - Reduce SQS costs with long polling +- [[Receive Message options]] - Fine-tune SQS receive parameters + +## Operations + +- [[Deployment]] - Deploy to Heroku, ECS, Kubernetes, and more +- [[Signals]] - Process control signals (USR1, TSTP, TTIN) +- [[Health Checks]] - Health check API for load balancers and orchestrators + +## Monitoring & Logging + +- [[Logging]] - Configure logging and log levels +- [[Sentry.io Integration]] - Error tracking with Sentry +- [[Honeybadger Integration]] - Error tracking with Honeybadger + +## Testing & Development + +- [[Testing]] - Test strategies for Shoryuken workers +- [[Shoryuken Inline adapter]] - Synchronous execution for testing +- [[Using a local mock SQS server]] - LocalStack and moto for local development +- [[Enable hot reload for rails dev environment]] - Development auto-reload + +## Security & IAM + +- [[Amazon SQS IAM Policy for Shoryuken]] - Minimum required IAM permissions + +## Migration & Upgrade + +- [[Upgrade Guide 7.0]] - Upgrading to Shoryuken 7.0 +- [[From Sidekiq to Shoryuken]] - Migration guide from Sidekiq +- [[Best Practices]] - Idempotency and reliability patterns + +## Advanced Topics + +- [[Worker registry]] - Custom worker registration +- [[Multiple ways of configuring queues in Shoryuken]] - Queue configuration options +- [[Scheduled Jobs]] - Running jobs on a schedule with EventBridge/Lambda + +--- + +## Quick Links + +- [GitHub Repository](https://github.com/ruby-shoryuken/shoryuken) +- [RubyGems](https://rubygems.org/gems/shoryuken) +- [CHANGELOG](https://github.com/ruby-shoryuken/shoryuken/blob/master/CHANGELOG.md) diff --git a/Lifecycle-Events.md b/Lifecycle-Events.md index b18efd4..fcce9d8 100644 --- a/Lifecycle-Events.md +++ b/Lifecycle-Events.md @@ -1,25 +1,321 @@ -Shoryuken fires process lifecycle events when starting up and shutting down: +# Lifecycle Events + +Shoryuken fires lifecycle events at key points during processing. Use these events for initialization, cleanup, monitoring, and custom integrations. + +## Available Events + +| Event | When Fired | Use Cases | +|-------|------------|-----------| +| `startup` | After queues validated, before fetching | Initialize connections, start health server | +| `dispatch` | Each time messages are fetched from SQS | Metrics, heartbeats | +| `utilization_update` | When worker pool utilization changes | Monitoring, auto-scaling signals | +| `quiet` | On soft shutdown (USR1 signal) | Pre-shutdown cleanup | +| `shutdown` | During shutdown sequence | Close connections, flush buffers | +| `stopped` | After executor fully stopped | Final cleanup | + +## Event Flow + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Lifecycle Flow │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ Start │ +│ │ │ +│ ▼ │ +│ startup ──────────► Processing Loop │ +│ │ │ +│ ┌────┴────┐ │ +│ ▼ ▼ │ +│ dispatch utilization_update │ +│ │ │ │ +│ └────┬────┘ │ +│ │ │ +│ (repeat) │ +│ │ │ +│ SIGTERM/USR1 ───────────►│ │ +│ ▼ │ +│ quiet │ +│ │ │ +│ ▼ │ +│ shutdown │ +│ │ │ +│ ▼ │ +│ stopped │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Basic Usage ```ruby +# config/initializers/shoryuken.rb Shoryuken.configure_server do |config| config.on(:startup) do - # called when it's about to start fetching messages, after validating queues, params etc + Rails.logger.info "Shoryuken starting up" + # Initialize resources + end + + config.on(:dispatch) do + # Called every time messages are fetched + end + + config.on(:utilization_update) do |group, utilization| + # Called when worker pool utilization changes + Rails.logger.debug "Group #{group} utilization: #{utilization}%" end config.on(:quiet) do - # called for soft shutdown kill -USR1, before calling shutdown + Rails.logger.info "Shoryuken entering quiet mode" + # Prepare for shutdown + end + + config.on(:shutdown) do + Rails.logger.info "Shoryuken shutting down" + # Close connections, flush buffers + end + + config.on(:stopped) do + Rails.logger.info "Shoryuken fully stopped" + # Final cleanup + end +end +``` + +## Event Details + +### startup + +Fired after Shoryuken validates queues and configuration, just before starting to fetch messages. + +**Use cases:** +- Initialize database connection pools +- Start health check HTTP server +- Register with service discovery +- Initialize monitoring/metrics + +```ruby +config.on(:startup) do + # Start health check server + @health_server = WEBrick::HTTPServer.new(Port: 3001) + Thread.new { @health_server.start } + + # Warm up connections + ActiveRecord::Base.connection_pool.checkout + ActiveRecord::Base.connection_pool.checkin + + # Register with service discovery + ServiceDiscovery.register(name: 'shoryuken-worker', port: 3001) +end +``` + +### dispatch + +Fired each time Shoryuken attempts to fetch messages from SQS. This happens in a continuous loop during normal operation. + +**Use cases:** +- Heartbeat to monitoring systems +- Track dispatch frequency +- Log activity + +```ruby +config.on(:dispatch) do + # Send heartbeat + Heartbeat.ping('shoryuken-worker') + + # Track dispatch metrics + StatsD.increment('shoryuken.dispatch') +end +``` + +**Note:** This event fires frequently. Keep handlers lightweight to avoid impacting performance. + +### utilization_update + +Fired when the worker pool utilization changes. The callback receives the processing group name and current utilization percentage. + +**Use cases:** +- Monitor worker saturation +- Trigger auto-scaling +- Performance dashboards + +```ruby +config.on(:utilization_update) do |group, utilization| + # Send to monitoring + StatsD.gauge("shoryuken.utilization.#{group}", utilization) + + # Log high utilization + if utilization > 90 + Rails.logger.warn "High utilization in group #{group}: #{utilization}%" + end + + # Signal auto-scaler + if utilization > 80 + AutoScaler.request_scale_up(group: group) end +end +``` + +**Utilization calculation:** +- `0%` = No workers busy +- `100%` = All workers busy +- Value changes as workers start/finish processing messages + +### quiet +Fired when Shoryuken receives a soft shutdown signal (USR1) before the shutdown sequence begins. + +**Use cases:** +- Stop accepting new work +- Deregister from load balancer +- Notify external systems + +```ruby +config.on(:quiet) do + # Deregister from service discovery + ServiceDiscovery.deregister('shoryuken-worker') + + # Notify monitoring + PagerDuty.notify('Shoryuken worker entering quiet mode') +end +``` + +### shutdown + +Fired during the shutdown sequence, before the executor is stopped. + +**Use cases:** +- Close external connections +- Flush buffers and caches +- Send final metrics + +```ruby +config.on(:shutdown) do + # Flush logs + Rails.logger.flush + + # Close Redis connections + Redis.current.quit + + # Send final metrics + StatsD.flush + + # Stop health server + @health_server&.shutdown +end +``` + +### stopped + +Fired after the executor has fully stopped and all workers have completed. + +**Use cases:** +- Final cleanup +- Remove lock files +- Exit notification + +```ruby +config.on(:stopped) do + # Remove PID file + File.delete('tmp/pids/shoryuken.pid') if File.exist?('tmp/pids/shoryuken.pid') + + # Final notification + Rails.logger.info "Shoryuken worker #{Process.pid} has stopped" +end +``` + +## Complete Example + +```ruby +# config/initializers/shoryuken.rb +require 'webrick' + +Shoryuken.configure_server do |config| + # ── Startup ── + config.on(:startup) do + Rails.logger.info "[Shoryuken] Starting worker #{Process.pid}" + + # Start health check server + @health_server = WEBrick::HTTPServer.new( + Port: ENV.fetch('HEALTH_PORT', 3001), + Logger: WEBrick::Log.new("/dev/null") + ) + + @health_server.mount_proc '/health' do |req, res| + launcher = Shoryuken::Runner.instance.launcher + res.status = launcher&.healthy? ? 200 : 503 + res.body = launcher&.healthy? ? 'OK' : 'Unhealthy' + end + + Thread.new { @health_server.start } + + # Initialize metrics + StatsD.increment('shoryuken.startup') + end + + # ── Dispatch ── config.on(:dispatch) do - # called every time it tries to fetch messages from SQS + StatsD.increment('shoryuken.dispatch') + end + + # ── Utilization ── + config.on(:utilization_update) do |group, utilization| + StatsD.gauge("shoryuken.utilization.#{group}", utilization) + end + + # ── Quiet ── + config.on(:quiet) do + Rails.logger.info "[Shoryuken] Entering quiet mode" end + # ── Shutdown ── config.on(:shutdown) do - # called when shut-downing + Rails.logger.info "[Shoryuken] Shutting down" + @health_server&.shutdown + StatsD.flush end + # ── Stopped ── config.on(:stopped) do - # called after executor stopped + Rails.logger.info "[Shoryuken] Worker #{Process.pid} stopped" + StatsD.increment('shoryuken.stopped') end end -``` \ No newline at end of file +``` + +## Multiple Handlers + +You can register multiple handlers for the same event: + +```ruby +config.on(:startup) do + initialize_database_pool +end + +config.on(:startup) do + start_health_server +end + +config.on(:startup) do + register_with_service_discovery +end +``` + +Handlers are called in the order they were registered. + +## Event Timing Considerations + +| Event | Blocking? | Notes | +|-------|-----------|-------| +| `startup` | Yes | Runs before processing starts. Keep initialization quick. | +| `dispatch` | No | Called frequently. Keep handlers very lightweight. | +| `utilization_update` | No | Called on utilization changes. Keep handlers lightweight. | +| `quiet` | Yes | Can block soft shutdown. Keep quick. | +| `shutdown` | Yes | Blocks shutdown. Has timeout limit. | +| `stopped` | Yes | Called after executor stopped. | + +## Related + +- [[Signals]] - Process control signals +- [[Health Checks]] - Health check API +- [[Middleware]] - Per-message processing hooks diff --git a/Middleware.md b/Middleware.md index 63602a1..6276e09 100644 --- a/Middleware.md +++ b/Middleware.md @@ -1,68 +1,394 @@ -## Basic middleware +# Middleware + +Middleware provides a way to add cross-cutting concerns to message processing. Each message passes through the middleware chain before reaching the worker. + +## Built-in Middleware + +Shoryuken includes several built-in server middleware: + +| Middleware | Purpose | Auto-enabled | +|------------|---------|--------------| +| `ActiveRecord` | Clears database connections after each job | No | +| `AutoDelete` | Deletes messages after successful processing | Via `auto_delete` option | +| `AutoExtendVisibility` | Extends visibility timeout during processing | Via `auto_visibility_timeout` option | +| `ExponentialBackoffRetry` | Implements retry with exponential backoff | Via `retry_intervals` option | +| `Timing` | Logs processing duration | No | + +--- + +## Creating Middleware + +### Basic Structure ```ruby class MyMiddleware def call(worker_instance, queue, sqs_msg, body) - puts 'Before work' + # Before processing + puts "Processing message #{sqs_msg.message_id}" + + yield # Continue to next middleware/worker + + # After processing (only on success) + puts "Completed message #{sqs_msg.message_id}" + rescue => e + # Handle errors + puts "Failed: #{e.message}" + raise + end +end +``` + +### With Initialization Parameters + +```ruby +class LoggingMiddleware + def initialize(logger:) + @logger = logger + end + + def call(worker_instance, queue, sqs_msg, body) + @logger.info "Starting #{worker_instance.class.name}" yield - puts 'After work' + @logger.info "Completed #{worker_instance.class.name}" + end +end + +# Registration +Shoryuken.configure_server do |config| + config.server_middleware do |chain| + chain.add LoggingMiddleware, logger: Rails.logger end end ``` -### Registering a global middleware +--- + +## Registering Middleware + +### Global Middleware ```ruby +# config/initializers/shoryuken.rb Shoryuken.configure_server do |config| config.server_middleware do |chain| chain.add MyMiddleware - # chain.remove MyMiddleware - # chain.add MyMiddleware, foo: 1, bar: 2 - # chain.insert_before MyMiddleware, MyMiddlewareNew - # chain.insert_after MyMiddleware, MyMiddlewareNew end end ``` -### Registering per worker middleware +### Chain Operations + +```ruby +config.server_middleware do |chain| + # Add to end of chain + chain.add MyMiddleware + + # Add with parameters + chain.add MyMiddleware, foo: 1, bar: 2 + + # Insert before another middleware + chain.insert_before ExistingMiddleware, NewMiddleware + + # Insert after another middleware + chain.insert_after ExistingMiddleware, NewMiddleware + + # Remove middleware + chain.remove UnwantedMiddleware + + # Clear all middleware + chain.clear +end +``` + +### Per-Worker Middleware ```ruby class MyWorker include Shoryuken::Worker - def perform(sqs_mg, body) + shoryuken_options queue: 'default' + + # Add worker-specific middleware + server_middleware do |chain| + chain.add WorkerSpecificMiddleware + end + + def perform(sqs_msg, body) # ... end +end +``` + +### Replacing Global Middleware for a Worker + +```ruby +class MyWorker + include Shoryuken::Worker server_middleware do |chain| - # This will join all "global" middleware with `MyWorkerSpecificMiddleware` - # if you want to run only `MyWorkerSpecificMiddleware` for this worker - # you can `chain.clear` - # or to remove specific middleware for this worker you can `chain.remove OtherMiddleware` - chain.add MyWorkerSpecificMiddleware + # Clear all global middleware + chain.clear + + # Add only what this worker needs + chain.add MinimalMiddleware + end + + def perform(sqs_msg, body) + # ... + end +end +``` + +--- + +## Built-in Middleware Details + +### ActiveRecord Middleware + +Ensures database connections are returned to the pool after each job: + +```ruby +# Enable manually if needed +Shoryuken.configure_server do |config| + config.server_middleware do |chain| + chain.add Shoryuken::Middleware::Server::ActiveRecord + end +end +``` + +**Note:** For Rails applications, ActiveRecord connections are typically managed automatically. Add this middleware if you encounter connection pool exhaustion. + +### AutoDelete Middleware + +Automatically deletes messages after successful processing. Enabled via worker options: + +```ruby +class MyWorker + include Shoryuken::Worker + + shoryuken_options queue: 'default', auto_delete: true + + def perform(sqs_msg, body) + # Message is deleted after this returns successfully + end +end +``` + +### AutoExtendVisibility Middleware + +Extends the message visibility timeout during long-running jobs: + +```ruby +class MyWorker + include Shoryuken::Worker + + shoryuken_options queue: 'default', auto_visibility_timeout: true + + def perform(sqs_msg, body) + # Visibility timeout is extended automatically during processing + long_running_operation end end ``` -## Rejecting messages with a middleware +### ExponentialBackoffRetry Middleware -If you don't `yield` in a Middleware, no other middleware or worker will process it. +Implements retry with exponential backoff: ```ruby -class RejectInvalidMessagesMiddleware +class MyWorker + include Shoryuken::Worker + + shoryuken_options queue: 'default', + retry_intervals: [60, 300, 3600] # 1min, 5min, 1hr + + def perform(sqs_msg, body) + # On failure, message visibility timeout is set based on retry count + end +end +``` + +### Timing Middleware + +Logs the duration of message processing: + +```ruby +Shoryuken.configure_server do |config| + config.server_middleware do |chain| + chain.add Shoryuken::Middleware::Server::Timing + end +end +``` + +--- + +## Common Middleware Examples + +### Error Tracking (Sentry) + +```ruby +class SentryMiddleware + def call(worker_instance, queue, sqs_msg, body) + yield + rescue => e + Sentry.capture_exception(e, extra: { + queue: queue, + message_id: sqs_msg.message_id, + worker: worker_instance.class.name + }) + raise + end +end +``` + +### Request ID Tracking + +```ruby +class RequestIdMiddleware + def call(worker_instance, queue, sqs_msg, body) + request_id = sqs_msg.message_attributes.dig('request_id', 'string_value') || + SecureRandom.uuid + + Thread.current[:request_id] = request_id + + Rails.logger.tagged(request_id) do + yield + end + ensure + Thread.current[:request_id] = nil + end +end +``` + +### Metrics Collection + +```ruby +class MetricsMiddleware + def call(worker_instance, queue, sqs_msg, body) + start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC) + success = false + + yield + success = true + rescue => e + StatsD.increment("shoryuken.job.failed", tags: ["queue:#{queue}"]) + raise + ensure + duration = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_time + + StatsD.timing("shoryuken.job.duration", duration * 1000, tags: [ + "queue:#{queue}", + "worker:#{worker_instance.class.name}", + "success:#{success}" + ]) + + StatsD.increment("shoryuken.job.processed", tags: ["queue:#{queue}"]) if success + end +end +``` + +### Rate Limiting + +```ruby +class RateLimitMiddleware + def initialize(redis:, limit:, period:) + @redis = redis + @limit = limit + @period = period + end + + def call(worker_instance, queue, sqs_msg, body) + key = "rate_limit:#{worker_instance.class.name}" + + if @redis.incr(key) > @limit + # Re-queue for later processing + raise "Rate limit exceeded" + end + + @redis.expire(key, @period) + yield + end +end +``` + +--- + +## Rejecting Messages + +To reject a message without processing, don't call `yield`: + +```ruby +class ValidationMiddleware def call(worker_instance, queue, sqs_msg, body) - if valid?(sqs_msg) - # will consume the message + if valid?(body) yield else - # will not consume the message - Shoryuken.logger.info "sqs_msg '#{sqs_msg.id}' is invalid and was rejected" - sqs_msg.delete + Shoryuken.logger.warn "Rejecting invalid message #{sqs_msg.message_id}" + sqs_msg.delete # Remove from queue + # Don't yield - message won't be processed + end + end + + private + + def valid?(body) + body.is_a?(Hash) && body['type'].present? + end +end +``` + +--- + +## Batch Processing + +When batch processing is enabled, `sqs_msg` and `body` are arrays: + +```ruby +class BatchMiddleware + def call(worker_instance, queue, sqs_msg, body) + if sqs_msg.is_a?(Array) + # Batch mode + Shoryuken.logger.info "Processing batch of #{sqs_msg.size} messages" + else + # Single message mode + Shoryuken.logger.info "Processing message #{sqs_msg.message_id}" end + + yield end end ``` +--- + +## Middleware Order + +Middleware executes in the order registered: + +```ruby +config.server_middleware do |chain| + chain.add FirstMiddleware # Runs first (outer) + chain.add SecondMiddleware # Runs second + chain.add ThirdMiddleware # Runs last (inner) +end +``` + +Execution flow: + +``` +FirstMiddleware.before + SecondMiddleware.before + ThirdMiddleware.before + Worker.perform + ThirdMiddleware.after + SecondMiddleware.after +FirstMiddleware.after +``` + +--- -*Note:* When [batch is enabled](https://github.com/phstc/shoryuken/wiki/Worker-options#batch), the `sqs_msg` and `body` arguments are arrays. +## Related +- [[Worker options]] - Worker-level configuration +- [[Sentry.io Integration]] - Error tracking example +- [[Lifecycle Events]] - Process-level hooks diff --git a/Monitoring-and-Metrics.md b/Monitoring-and-Metrics.md new file mode 100644 index 0000000..28045c5 --- /dev/null +++ b/Monitoring-and-Metrics.md @@ -0,0 +1,393 @@ +# Monitoring and Metrics + +This guide covers monitoring Shoryuken workers and collecting metrics for observability. + +## Lifecycle Events for Metrics + +Use lifecycle events to emit metrics: + +```ruby +# config/initializers/shoryuken.rb +Shoryuken.configure_server do |config| + config.on(:startup) do + Rails.logger.info "Shoryuken started" + StatsD.increment('shoryuken.started') + end + + config.on(:shutdown) do + Rails.logger.info "Shoryuken stopped" + StatsD.increment('shoryuken.stopped') + end + + config.on(:utilization_update) do |group, utilization| + StatsD.gauge("shoryuken.utilization.#{group}", utilization) + end +end +``` + +--- + +## Key Metrics + +### Worker Metrics + +| Metric | Description | Alert Threshold | +|--------|-------------|-----------------| +| `jobs.processed` | Jobs completed | N/A (track rate) | +| `jobs.failed` | Jobs that raised errors | > 1% of processed | +| `jobs.duration` | Processing time | p99 > SLA | +| `utilization` | Thread usage (0-1) | Sustained > 0.9 | +| `memory_mb` | Process memory | > 80% of limit | + +### Queue Metrics (SQS) + +| Metric | Description | Alert Threshold | +|--------|-------------|-----------------| +| `ApproximateNumberOfMessagesVisible` | Queue depth | Growing trend | +| `ApproximateAgeOfOldestMessage` | Processing delay | > acceptable latency | +| `NumberOfMessagesReceived` | Throughput | Sudden drop | +| `NumberOfMessagesDeleted` | Completion rate | Diverging from received | + +--- + +## Metrics Middleware + +### StatsD / Datadog + +```ruby +class MetricsMiddleware + def call(worker_instance, queue, sqs_msg, body) + start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC) + success = false + + yield + success = true + rescue => e + StatsD.increment('shoryuken.jobs.failed', tags: [ + "queue:#{queue}", + "worker:#{worker_instance.class.name}", + "error:#{e.class.name}" + ]) + raise + ensure + duration = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_time + + StatsD.timing('shoryuken.jobs.duration', duration * 1000, tags: [ + "queue:#{queue}", + "worker:#{worker_instance.class.name}", + "success:#{success}" + ]) + + StatsD.increment('shoryuken.jobs.processed', tags: [ + "queue:#{queue}", + "success:#{success}" + ]) + end +end + +# Register +Shoryuken.configure_server do |config| + config.server_middleware do |chain| + chain.add MetricsMiddleware + end +end +``` + +### Prometheus + +```ruby +require 'prometheus/client' + +class PrometheusMiddleware + def initialize + @registry = Prometheus::Client.registry + + @jobs_processed = @registry.counter( + :shoryuken_jobs_processed_total, + docstring: 'Total jobs processed', + labels: [:queue, :worker, :status] + ) + + @job_duration = @registry.histogram( + :shoryuken_job_duration_seconds, + docstring: 'Job processing duration', + labels: [:queue, :worker], + buckets: [0.1, 0.5, 1, 5, 10, 30, 60] + ) + end + + def call(worker_instance, queue, sqs_msg, body) + start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC) + status = 'success' + + yield + rescue => e + status = 'failed' + raise + ensure + duration = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_time + worker = worker_instance.class.name + + @jobs_processed.increment(labels: { queue: queue, worker: worker, status: status }) + @job_duration.observe(duration, labels: { queue: queue, worker: worker }) + end +end +``` + +--- + +## CloudWatch Integration + +### Custom Metrics + +```ruby +require 'aws-sdk-cloudwatch' + +class CloudWatchMetrics + def initialize + @cloudwatch = Aws::CloudWatch::Client.new + @namespace = 'Shoryuken' + end + + def record_job(queue:, duration:, success:) + @cloudwatch.put_metric_data( + namespace: @namespace, + metric_data: [ + { + metric_name: 'JobDuration', + dimensions: [{ name: 'Queue', value: queue }], + value: duration, + unit: 'Seconds' + }, + { + metric_name: success ? 'JobsSucceeded' : 'JobsFailed', + dimensions: [{ name: 'Queue', value: queue }], + value: 1, + unit: 'Count' + } + ] + ) + end +end +``` + +### SQS Queue Metrics + +SQS automatically publishes metrics to CloudWatch: + +- `ApproximateNumberOfMessagesVisible` +- `ApproximateNumberOfMessagesNotVisible` +- `ApproximateAgeOfOldestMessage` +- `NumberOfMessagesReceived` +- `NumberOfMessagesDeleted` +- `NumberOfMessagesSent` + +Access via CloudWatch console or API. + +--- + +## Logging for Observability + +### Structured Logging + +```ruby +# config/initializers/shoryuken.rb +Shoryuken.configure_server do |config| + config.on(:startup) do + Shoryuken.logger.formatter = proc do |severity, datetime, progname, msg| + { + timestamp: datetime.iso8601(3), + level: severity, + message: msg, + service: 'shoryuken', + environment: Rails.env + }.to_json + "\n" + end + end +end +``` + +### Job Context Logging + +```ruby +class LoggingMiddleware + def call(worker_instance, queue, sqs_msg, body) + job_context = { + queue: queue, + worker: worker_instance.class.name, + message_id: sqs_msg.message_id + } + + Rails.logger.tagged(job_context.to_json) do + Rails.logger.info "Job started" + yield + Rails.logger.info "Job completed" + end + rescue => e + Rails.logger.error "Job failed: #{e.message}" + raise + end +end +``` + +--- + +## Health Monitoring + +### Health Check Endpoint + +```ruby +# config/initializers/shoryuken.rb +Shoryuken.configure_server do |config| + config.on(:startup) do + Thread.new do + require 'webrick' + server = WEBrick::HTTPServer.new(Port: 3001, Logger: WEBrick::Log.new("/dev/null")) + + server.mount_proc '/health' do |req, res| + launcher = Shoryuken::Runner.instance.launcher + if launcher&.healthy? + res.status = 200 + res.body = JSON.generate(status: 'healthy') + else + res.status = 503 + res.body = JSON.generate(status: 'unhealthy') + end + res['Content-Type'] = 'application/json' + end + + server.mount_proc '/metrics' do |req, res| + # Expose Prometheus metrics + res.status = 200 + res.body = Prometheus::Client::Formats::Text.marshal(Prometheus::Client.registry) + res['Content-Type'] = 'text/plain' + end + + server.start + end + end +end +``` + +See [[Health Checks]] for more details. + +--- + +## Alerting + +### Recommended Alerts + +| Alert | Condition | Severity | +|-------|-----------|----------| +| High error rate | Failed jobs > 5% in 5 min | Critical | +| Queue backlog | Queue depth > 10,000 | Warning | +| Old messages | Oldest message > 1 hour | Warning | +| Worker down | No heartbeat in 5 min | Critical | +| High latency | p99 duration > 60s | Warning | +| Memory pressure | Memory > 90% limit | Warning | + +### Example: Datadog Alert + +```yaml +# Datadog monitor definition +name: "Shoryuken High Error Rate" +type: "metric alert" +query: "sum(last_5m):sum:shoryuken.jobs.failed{*} / sum:shoryuken.jobs.processed{*} > 0.05" +message: "Shoryuken error rate is above 5%" +thresholds: + critical: 0.05 + warning: 0.02 +``` + +### Example: CloudWatch Alarm + +```json +{ + "AlarmName": "SQS-Queue-Depth-High", + "MetricName": "ApproximateNumberOfMessagesVisible", + "Namespace": "AWS/SQS", + "Dimensions": [ + {"Name": "QueueName", "Value": "myapp-default"} + ], + "Statistic": "Average", + "Period": 300, + "EvaluationPeriods": 2, + "Threshold": 10000, + "ComparisonOperator": "GreaterThanThreshold" +} +``` + +--- + +## Dashboards + +### Key Panels + +1. **Throughput** - Jobs processed per minute +2. **Error Rate** - Failed / Total percentage +3. **Latency** - p50, p95, p99 duration +4. **Queue Depth** - Messages waiting per queue +5. **Worker Utilization** - Thread usage +6. **Memory/CPU** - Resource consumption + +### Grafana Dashboard Example + +```json +{ + "panels": [ + { + "title": "Jobs Processed", + "type": "graph", + "targets": [ + { + "expr": "rate(shoryuken_jobs_processed_total[5m])", + "legendFormat": "{{queue}} - {{status}}" + } + ] + }, + { + "title": "Job Duration (p99)", + "type": "graph", + "targets": [ + { + "expr": "histogram_quantile(0.99, rate(shoryuken_job_duration_seconds_bucket[5m]))", + "legendFormat": "{{queue}}" + } + ] + } + ] +} +``` + +--- + +## Distributed Tracing + +### OpenTelemetry + +```ruby +require 'opentelemetry-sdk' + +class TracingMiddleware + def call(worker_instance, queue, sqs_msg, body) + tracer = OpenTelemetry.tracer_provider.tracer('shoryuken') + + tracer.in_span("#{worker_instance.class.name}#perform") do |span| + span.set_attribute('messaging.system', 'aws_sqs') + span.set_attribute('messaging.destination', queue) + span.set_attribute('messaging.message_id', sqs_msg.message_id) + + yield + end + end +end +``` + +--- + +## Related + +- [[Lifecycle Events]] - Event hooks +- [[Health Checks]] - Health endpoints +- [[Performance Tuning]] - Optimization +- [[Logging]] - Log configuration diff --git a/Performance-Tuning.md b/Performance-Tuning.md new file mode 100644 index 0000000..a29e41c --- /dev/null +++ b/Performance-Tuning.md @@ -0,0 +1,377 @@ +# Performance Tuning + +This guide covers optimizing Shoryuken for throughput, latency, and cost efficiency. + +## Concurrency + +### Setting Concurrency + +```yaml +# config/shoryuken.yml +concurrency: 25 # Default +``` + +### Guidelines + +| Workload Type | Recommended Concurrency | Notes | +|--------------|------------------------|-------| +| CPU-bound | 1-2× CPU cores | Limited by CPU | +| I/O-bound (API calls) | 25-50 | Higher due to waiting | +| Database-heavy | 10-25 | Limited by connection pool | +| Mixed | 15-25 | Balance based on bottleneck | + +### Memory Impact + +Each thread consumes memory. Monitor and adjust: + +``` +Memory per worker ≈ Base Rails memory + (concurrency × per-thread overhead) +``` + +Typical per-thread overhead: 5-20 MB depending on job complexity. + +--- + +## Database Connections + +### Connection Pool Sizing + +**Rule:** Pool size ≥ Shoryuken concurrency + +```yaml +# config/database.yml +production: + pool: <%= ENV.fetch("DB_POOL") { 25 } %> +``` + +### Connection Exhaustion Signs + +- `ActiveRecord::ConnectionTimeoutError` +- Slow job processing +- Database connection spikes + +### Solutions + +```ruby +# config/initializers/shoryuken.rb +Shoryuken.configure_server do |config| + # Add ActiveRecord middleware to return connections after each job + config.server_middleware do |chain| + chain.add Shoryuken::Middleware::Server::ActiveRecord + end +end +``` + +--- + +## Polling Configuration + +### Delay Setting + +```yaml +# config/shoryuken.yml +delay: 0 # Seconds to wait before re-polling empty queue +``` + +| Delay | Behavior | Cost Impact | +|-------|----------|-------------| +| 0 | Continuous polling | Higher (more API calls) | +| 5-10 | Balanced | Medium | +| 25+ | Cost-optimized | Lower latency | + +### Long Polling + +SQS long polling (up to 20 seconds) reduces empty responses: + +```ruby +# Set via receive_message options +Shoryuken.sqs_client_receive_message_opts = { + wait_time_seconds: 20 # Long polling +} +``` + +**Benefits:** +- Fewer API calls (cost savings) +- Reduced empty responses +- Lower latency for new messages + +--- + +## Queue Weights + +Prioritize critical queues: + +```yaml +# config/shoryuken.yml +queues: + - [critical, 6] # 6× more polling + - [default, 3] # 3× more polling + - [low, 1] # Baseline +``` + +### Strict Priority + +For absolute priority (process all critical before default): + +```yaml +# config/shoryuken.yml +polling: :strict_priority +queues: + - critical + - default + - low +``` + +See [[Polling strategies]] for details. + +--- + +## Processing Groups + +Isolate workloads with different characteristics: + +```yaml +# config/shoryuken.yml +groups: + fast: + concurrency: 50 + delay: 0 + queues: + - [webhooks, 1] + slow: + concurrency: 5 + delay: 5 + queues: + - [reports, 1] +``` + +**Use cases:** +- Separate fast/slow jobs +- Isolate resource-intensive jobs +- Different SLA requirements + +--- + +## Batch Processing + +### Enable Batch Mode + +```ruby +class BatchWorker + include Shoryuken::Worker + + shoryuken_options queue: 'batch_queue', batch: true + + def perform(sqs_msgs, bodies) + # Process up to 10 messages at once + bodies.each do |body| + process(body) + end + end +end +``` + +### Benefits + +- Reduced per-message overhead +- Better for high-volume, fast jobs +- Fewer database round-trips possible + +### Considerations + +- All-or-nothing visibility handling +- More complex error handling +- Not suitable for long-running jobs + +--- + +## Message Size Optimization + +### SQS Limits + +| Limit | Value | +|-------|-------| +| Max message size | 256 KB | +| Max batch size | 10 messages | +| Max batch payload | 1 MB | + +### Strategies + +**Pass IDs, not data:** + +```ruby +# BAD - Large payload +ProcessDataJob.perform_later(large_data_hash) + +# GOOD - Reference by ID +ProcessDataJob.perform_later(record_id: 123) +``` + +**Use S3 for large payloads:** + +```ruby +class LargeDataJob < ApplicationJob + def perform(s3_key) + data = S3Client.get_object(bucket: 'jobs', key: s3_key) + process(data.body.read) + end +end +``` + +--- + +## Visibility Timeout + +### Setting + +Configure in SQS console or via API. Default: 30 seconds. + +### Guidelines + +``` +Visibility timeout > Max job execution time + buffer +``` + +**Example:** If jobs take up to 60 seconds, set visibility to 90-120 seconds. + +### Auto-Extend + +For variable-length jobs: + +```ruby +class LongRunningWorker + include Shoryuken::Worker + + shoryuken_options queue: 'long_jobs', auto_visibility_timeout: true + + def perform(sqs_msg, body) + # Visibility automatically extended during processing + long_operation + end +end +``` + +--- + +## Memory Optimization + +### Monitor Memory + +```ruby +# config/initializers/shoryuken.rb +Shoryuken.configure_server do |config| + config.on(:utilization_update) do |group, utilization| + memory_mb = `ps -o rss= -p #{Process.pid}`.to_i / 1024 + Rails.logger.info "Memory: #{memory_mb} MB, Utilization: #{utilization}" + end +end +``` + +### Reduce Memory Usage + +1. **Lower concurrency** - Fewer threads = less memory +2. **Process in batches** - Load data incrementally +3. **Clear caches** - Reset memoized data between jobs +4. **Avoid loading large datasets** - Stream or paginate + +### Memory Leaks + +Watch for growing memory over time: + +```ruby +class MemoryCheckMiddleware + def call(worker, queue, sqs_msg, body) + yield + ensure + GC.start if rand < 0.01 # Occasional GC + end +end +``` + +--- + +## Cost Optimization + +### SQS Pricing Factors + +- Number of requests (polling, sending, deleting) +- Data transfer (cross-region, internet) + +### Reduce Costs + +1. **Increase delay** for low-priority queues +2. **Use long polling** (fewer empty responses) +3. **Batch operations** where possible +4. **Same-region deployment** (no cross-region transfer) +5. **Cache visibility timeout** to reduce API calls: + +```ruby +Shoryuken.cache_visibility_timeout = true +``` + +--- + +## Benchmarking + +### Measure Throughput + +```ruby +class MetricsMiddleware + def initialize + @processed = 0 + @start_time = Time.now + end + + def call(worker, queue, sqs_msg, body) + yield + @processed += 1 + + if @processed % 1000 == 0 + elapsed = Time.now - @start_time + rate = @processed / elapsed + Rails.logger.info "Throughput: #{rate.round(1)} jobs/sec" + end + end +end +``` + +### Key Metrics + +- **Throughput**: Jobs processed per second +- **Latency**: Time from enqueue to completion +- **Queue depth**: Messages waiting +- **Error rate**: Failed jobs percentage +- **Utilization**: Worker thread usage + +--- + +## Common Bottlenecks + +| Symptom | Likely Cause | Solution | +|---------|--------------|----------| +| Low throughput, low CPU | I/O bound | Increase concurrency | +| High CPU, moderate throughput | CPU bound | Add workers, optimize code | +| Connection errors | Pool exhausted | Increase pool, add middleware | +| Growing queue depth | Under-provisioned | Add workers | +| Memory growth | Leak or large objects | Profile, reduce concurrency | + +--- + +## Production Checklist + +- [ ] Concurrency matches workload type +- [ ] Database pool ≥ concurrency +- [ ] Visibility timeout > max job time +- [ ] Long polling enabled +- [ ] Monitoring in place +- [ ] Memory limits set (containers) +- [ ] Auto-scaling configured + +--- + +## Related + +- [[Shoryuken options]] - Configuration reference +- [[Processing Groups]] - Workload isolation +- [[Polling strategies]] - Queue prioritization +- [[Monitoring and Metrics]] - Observability diff --git a/Rails-Integration-Active-Job.md b/Rails-Integration-Active-Job.md index 92e9264..99d8a69 100644 --- a/Rails-Integration-Active-Job.md +++ b/Rails-Integration-Active-Job.md @@ -1,106 +1,347 @@ -You can tell Shoryuken to load your Rails application by passing the `-R` or `--rails` flag to the `shoryuken` command. +# Rails Integration - Active Job -If you load Rails, assuming your workers are located in the `app/jobs` directory, they will be auto-loaded. This means you don't need to require them explicitly with `-r`. +Shoryuken provides full support for Rails [Active Job](https://guides.rubyonrails.org/active_job_basics.html), enabling you to write processor-agnostic jobs that work with any queue backend. -For middleware and other configurations, you might want to create an initializer: +## Requirements + +- Rails 7.2+ +- Ruby 3.2+ + +## Basic Setup + +### 1. Configure the Queue Adapter ```ruby -# config/initializers/shoryuken.rb +# config/application.rb +module YourApp + class Application < Rails::Application + config.active_job.queue_adapter = :shoryuken + end +end +``` -Shoryuken.configure_server do |config| - # Replace Rails logger so messages are logged wherever Shoryuken is logging - # Note: this entire block is only run by the processor, so we don't overwrite - # the logger when the app is running as usual. +### 2. Create an ApplicationJob +```ruby +# app/jobs/application_job.rb +class ApplicationJob < ActiveJob::Base + # Retry on common transient errors + retry_on StandardError, wait: :polynomially_longer, attempts: 5 + + # Discard jobs that reference deleted records + discard_on ActiveJob::DeserializationError +end +``` + +### 3. Create a Job + +```ruby +# app/jobs/process_photo_job.rb +class ProcessPhotoJob < ApplicationJob + queue_as :default + + def perform(photo) + photo.process_image! + end +end +``` + +### 4. Create a Shoryuken Initializer + +```ruby +# config/initializers/shoryuken.rb +Shoryuken.configure_server do |config| + # Replace Rails logger so messages are logged to Shoryuken's log Rails.logger = Shoryuken::Logging.logger Rails.logger.level = Rails.application.config.log_level + # Add server middleware # config.server_middleware do |chain| - # chain.add Shoryuken::MyMiddleware - # end - - # For dynamically adding queues prefixed by Rails.env - # Shoryuken.add_group('default', 25) - # %w(queue1 queue2).each do |name| - # Shoryuken.add_queue("#{Rails.env}_#{name}", 1, 'default') + # chain.add MyMiddleware # end end ``` -*Note:* In the above case, since we are replacing the Rails logger, it's desired that this initializer runs before other initializers (in case they themselves use the logger). Since, by Rails conventions, initializers are executed in alphabetical order, this can be achieved by prepending the initializer filename with `00_` (assuming no other initializers alphabetically precede this one). +### 5. Start Shoryuken -### Active Job Support +```shell +bundle exec shoryuken -q default -R +``` + +--- -Yes, Shoryuken supports [Active Job](http://edgeguides.rubyonrails.org/active_job_basics.html)! This means that you can put your jobs in processor-agnostic `ActiveJob::Base` subclasses, and change processors whenever you want (or better yet, switch to Shoryuken from another processor easily!). +## Queue Adapters -It works as expected once your [queuing backend is set](http://edgeguides.rubyonrails.org/active_job_basics.html#setting-the-backend) to `:shoryuken`. Just put your job in `app/jobs`. Here's an example: +### Standard Adapter (Synchronous) + +The default adapter sends messages synchronously: ```ruby -# app/jobs/process_photo_job.rb -class ProcessPhotoJob < ActiveJob::Base - queue_as :default +# config/application.rb +config.active_job.queue_adapter = :shoryuken +``` + +### Concurrent Send Adapter (Asynchronous) + +For high-throughput scenarios, use the async adapter with optional success/error handlers: + +```ruby +# config/initializers/shoryuken.rb +success_handler = ->(response, job, options) { + Rails.logger.info "Enqueued #{job.class.name} successfully" +} - rescue_from ActiveJob::DeserializationError do |ex| - Shoryuken.logger.error ex - Shoryuken.logger.error ex.backtrace.join("\n") +error_handler = ->(error, job, options) { + Rails.logger.error "Failed to enqueue #{job.class.name}: #{error.message}" + # Report to error tracking service + Sentry.capture_exception(error) +} + +Rails.application.config.active_job.queue_adapter = + ActiveJob::QueueAdapters::ShoryukenConcurrentSendAdapter.new(success_handler, error_handler) +``` + +--- + +## Queue Configuration + +### Queue Name Prefixing + +Active Job supports [queue name prefixing](https://guides.rubyonrails.org/active_job_basics.html#queues). To have Shoryuken honor these prefixes: + +```ruby +# config/initializers/shoryuken.rb +Shoryuken.active_job_queue_name_prefixing = true +``` + +Example with prefixing enabled: + +```ruby +# config/environments/production.rb +config.active_job.queue_name_prefix = "production" + +# This job will use queue "production_reports" +class ReportJob < ApplicationJob + queue_as :reports +end +``` + +### Dynamic Queue Names + +```ruby +class TenantJob < ApplicationJob + queue_as { "tenant_#{Current.tenant_id}" } + + def perform(data) + # Process tenant-specific work end +end +``` - def perform(photo) - photo.process_image! +--- + +## SQS Message Parameters + +Use `.set` to customize SQS message parameters: + +```ruby +MyJob.set( + message_group_id: 'group-123', # For FIFO queues + message_deduplication_id: 'dedupe-456', # For FIFO queues + message_attributes: { + 'correlation_id' => { + string_value: SecureRandom.uuid, + data_type: 'String' + } + } +).perform_later(data) +``` + +--- + +## Scheduling and Delays + +### Delayed Execution + +```ruby +# Process in 5 minutes +MyJob.set(wait: 5.minutes).perform_later(data) + +# Process at a specific time +MyJob.set(wait_until: Date.tomorrow.noon).perform_later(data) +``` + +**Note:** SQS limits delays to 15 minutes maximum. For longer delays, use a scheduling solution like EventBridge Scheduler. + +--- + +## Bulk Enqueuing (Rails 7.1+) + +Enqueue multiple jobs efficiently using the SQS batch API: + +```ruby +# Enqueue many jobs in batches of 10 +jobs = users.map { |user| NotifyUserJob.new(user) } +ActiveJob.perform_all_later(jobs) +``` + +This uses `send_message_batch` internally, which is more efficient than individual `send_message` calls. + +**Batch Limits:** +- Maximum 10 messages per batch +- Maximum 1MB total batch size + +See [[Bulk Enqueuing]] for more details. + +--- + +## ActiveJob Continuations (Rails 8.1+) + +Shoryuken supports ActiveJob Continuations for graceful interruption of long-running jobs: + +```ruby +class DataExportJob < ApplicationJob + def perform(export) + export.records.find_each do |record| + # Check if Shoryuken is shutting down + if stopping? + # Save progress and re-enqueue + export.update!(last_processed_id: record.id) + self.class.perform_later(export) + return + end + + process_record(record) + end + + export.mark_completed! end end ``` -*Note:* When queueing jobs to be performed in the future (e.g. when setting the `wait` or `wait_until` Active Job options), SQS limits the amount of time to 15 minutes (see [`delay_seconds`](https://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Client.html#send_message-instance_method)). Shoryuken will raise an exception if you attempt to schedule a job further into the future than this limit. +The `stopping?` method returns `true` when Shoryuken receives a shutdown signal, allowing jobs to checkpoint their progress. + +See [[ActiveJob Continuations]] for more details. +--- -#### Active Job Queue Name Support +## CurrentAttributes Persistence -*Note:* Active Job allows you to [prefix the queue names](http://edgeguides.rubyonrails.org/active_job_basics.html#queues) of all jobs. Shoryuken supports this behavior natively. By default, though, queue names defined in the config file (or passed to the CLI), are not prefixed similarly. To have Shoryuken honor Active Job prefixes, you must enable that option explicitly. A good place to do that in Rails is in an initializer: +Automatically flow Rails `ActiveSupport::CurrentAttributes` from enqueue to job execution: ```ruby # config/initializers/shoryuken.rb -Shoryuken.active_job_queue_name_prefixing = true +require 'shoryuken/active_job/current_attributes' +Shoryuken::ActiveJob::CurrentAttributes.persist('Current') ``` +```ruby +# app/models/current.rb +class Current < ActiveSupport::CurrentAttributes + attribute :user, :request_id, :tenant +end +``` -#### Active Job Queue Adapters +```ruby +# In your controller +Current.user = current_user +Current.tenant = current_tenant -Shoryuken has two queue adapters, the default uses a synchronous producer, that is set up normally in your rails config as: +# The job will have access to Current.user and Current.tenant +AuditJob.perform_later(action: 'updated_profile') +``` + +See [[CurrentAttributes]] for more details. + +--- + +## Transaction Safety (Rails 7.2+) + +Jobs are automatically enqueued after the database transaction commits: ```ruby -# config/application.rb -config.active_job.queue_adapter = :shoryuken +User.transaction do + user = User.create!(name: 'Ken') + WelcomeEmailJob.perform_later(user) # Enqueued AFTER transaction commits +end ``` -There is also a second option to use an async producer: +This prevents jobs from processing before the record exists. Shoryuken enables this by implementing `enqueue_after_transaction_commit?`. + +--- + +## Retrying Failed Jobs + +### Using ActiveJob retry_on ```ruby -# config/application.rb -config.active_job.queue_adapter = ActiveJob::QueueAdapters::ShoryukenConcurrentSendAdapter.new +class ProcessPaymentJob < ApplicationJob + retry_on PaymentGatewayError, wait: :polynomially_longer, attempts: 5 + discard_on InvalidPaymentError + + def perform(payment) + PaymentGateway.process(payment) + end +end +``` + +### Using Shoryuken's Exponential Backoff + +For more control, use Shoryuken's `retry_intervals`: + +```ruby +class MyWorker + include Shoryuken::Worker + + shoryuken_options queue: 'default', + auto_delete: true, + retry_intervals: [60, 300, 3600] # 1min, 5min, 1hr + + def perform(sqs_msg, body) + # Process message + end +end ``` -If you have a lot of jobs to produce, consider using the `ShoryukenConcurrentSendAdapter`. Another hint for quick job enqueuing for `ActiveRecord` objects is to use a query that only selects the `id`: +See [[Retrying a message]] for more details. + +--- + +## Error Handling + +### Deserialization Errors ```ruby -Rails.application.config.active_job.queue_adapter = ActiveJob::QueueAdapters::ShoryukenConcurrentSendAdapter.new -MyModel.expensive_scope.select(:id).load.each { |obj| MyJob.perform_later(obj) } +class MyJob < ApplicationJob + # Discard jobs with serialization errors (e.g., deleted records) + discard_on ActiveJob::DeserializationError + + def perform(record) + record.process! + end +end ``` -#### Set SQS SendMessage parameters with `.set` +### Custom Exception Handlers ```ruby -MyJob.set(message_group_id: 'abc123', - message_deduplication_id: 'dedupe', - message_attributes: { - 'key' => { - string_value: 'myvalue', - data_type: 'String' - } - }) - .perform_later(123, 'abc') +# config/initializers/shoryuken.rb +Shoryuken.configure_server do |config| + config.on_exception do |ex, queue, sqs_msg| + Sentry.capture_exception(ex, extra: { + queue: queue, + message_id: sqs_msg.message_id + }) + end +end ``` -### How to use [`retry_on`](https://edgeguides.rubyonrails.org/active_job_basics.html#retrying-or-discarding-failed-jobs) +--- + +## Next Steps -Please follow the steps detailed [here](https://github.com/phstc/shoryuken/issues/553#issuecomment-465813111) to make `retry_on` work with Shoryuken. \ No newline at end of file +- [[ActiveJob Continuations]] - Handle graceful shutdowns +- [[CurrentAttributes]] - Persist request context +- [[Bulk Enqueuing]] - Efficient batch operations +- [[Middleware]] - Add cross-cutting concerns +- [[Testing]] - Test your jobs diff --git a/Scaling-Patterns.md b/Scaling-Patterns.md new file mode 100644 index 0000000..d37c61e --- /dev/null +++ b/Scaling-Patterns.md @@ -0,0 +1,358 @@ +# Scaling Patterns + +This guide covers strategies for scaling Shoryuken workers to handle varying workloads. + +## Horizontal Scaling + +### Adding More Workers + +Each worker process: +- Has its own database connection pool +- Polls SQS independently +- Handles `concurrency` threads + +``` +Total capacity = Number of workers × Concurrency per worker +``` + +### Example: 3 Workers × 25 Concurrency = 75 Concurrent Jobs + +```yaml +# config/shoryuken.yml +concurrency: 25 +``` + +Deploy multiple instances of your worker process. + +--- + +## Vertical Scaling + +### Increase Concurrency + +```yaml +# config/shoryuken.yml +concurrency: 50 # Up from default 25 +``` + +### Considerations + +| Concurrency | Memory | Database Pool | Best For | +|-------------|--------|---------------|----------| +| 10-25 | Low | 25 | I/O-bound jobs | +| 25-50 | Medium | 50 | Mixed workloads | +| 50-100 | High | 100 | CPU-light, I/O-heavy | + +**Limits:** +- Database connection pool must match +- Memory increases with concurrency +- CPU-bound jobs don't benefit beyond core count + +--- + +## Queue-Based Scaling + +### Separate Queues by Priority + +```yaml +# config/shoryuken.yml +queues: + - [critical, 10] + - [default, 3] + - [low, 1] +``` + +### Dedicated Workers per Queue + +```yaml +# Worker 1: critical-worker.yml +queues: + - critical + +# Worker 2: default-worker.yml +queues: + - default + - low +``` + +Deploy different numbers of each: +- 3× critical workers +- 2× default workers + +--- + +## Processing Groups + +Isolate workloads within a single process: + +```yaml +# config/shoryuken.yml +groups: + realtime: + concurrency: 10 + delay: 0 + queues: + - [webhooks, 1] + - [notifications, 1] + batch: + concurrency: 5 + delay: 5 + queues: + - [reports, 1] + - [exports, 1] +``` + +See [[Processing Groups]] for details. + +--- + +## Auto-Scaling + +### By Queue Depth + +Scale workers based on `ApproximateNumberOfMessagesVisible`: + +``` +Target: 50-100 messages per worker +``` + +#### Kubernetes HPA with KEDA + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: shoryuken-worker +spec: + scaleTargetRef: + name: shoryuken-worker + minReplicaCount: 2 + maxReplicaCount: 20 + triggers: + - type: aws-sqs-queue + metadata: + queueURL: https://sqs.us-east-1.amazonaws.com/123456789/myqueue + queueLength: "100" # Target messages per pod + awsRegion: us-east-1 +``` + +#### AWS ECS Auto Scaling + +```json +{ + "ScalingPolicy": { + "PolicyName": "sqs-queue-depth", + "PolicyType": "TargetTrackingScaling", + "TargetTrackingScalingPolicyConfiguration": { + "TargetValue": 100, + "CustomizedMetricSpecification": { + "MetricName": "ApproximateNumberOfMessagesVisible", + "Namespace": "AWS/SQS", + "Dimensions": [ + {"Name": "QueueName", "Value": "myapp-default"} + ], + "Statistic": "Average" + }, + "ScaleInCooldown": 300, + "ScaleOutCooldown": 60 + } + } +} +``` + +### By Message Age + +Scale when messages wait too long: + +```yaml +# KEDA trigger +triggers: + - type: aws-cloudwatch + metadata: + namespace: AWS/SQS + dimensionName: QueueName + dimensionValue: myqueue + metricName: ApproximateAgeOfOldestMessage + targetMetricValue: "300" # 5 minutes +``` + +--- + +## Queue Sharding + +### By Customer/Tenant + +```ruby +class TenantJob < ApplicationJob + queue_as -> { "tenant_#{arguments.first[:tenant_id] % 10}" } + + def perform(tenant_id:, data:) + # Process for specific tenant + end +end +``` + +Queues: `tenant_0`, `tenant_1`, ... `tenant_9` + +### By Data Type + +```ruby +class ProcessDataJob < ApplicationJob + queue_as -> { + case arguments.first[:type] + when 'urgent' then 'data_urgent' + when 'large' then 'data_large' + else 'data_default' + end + } +end +``` + +--- + +## Worker Specialization + +### Heavy vs Light Workers + +```yaml +# light-worker.yml - Many small jobs +concurrency: 50 +queues: + - notifications + - emails + +# heavy-worker.yml - Few large jobs +concurrency: 5 +queues: + - reports + - exports +``` + +### Resource Allocation + +| Worker Type | CPU | Memory | Concurrency | +|------------|-----|--------|-------------| +| Light | 0.5 | 512 MB | 50 | +| Medium | 1 | 1 GB | 25 | +| Heavy | 2 | 2 GB | 5 | + +--- + +## Scaling Strategies by Workload + +### Bursty Traffic + +- Use auto-scaling with aggressive scale-out +- Keep minimum workers running +- Scale in slowly (cooldown period) + +```yaml +# KEDA +minReplicaCount: 2 +maxReplicaCount: 50 +cooldownPeriod: 300 +``` + +### Steady Traffic + +- Fixed worker count +- Size based on average throughput +- Manual scaling for growth + +### Time-Based + +Schedule workers for predictable patterns: + +```yaml +# Kubernetes CronJob to scale +apiVersion: batch/v1 +kind: CronJob +metadata: + name: scale-workers-morning +spec: + schedule: "0 8 * * 1-5" # 8 AM weekdays + jobTemplate: + spec: + template: + spec: + containers: + - name: kubectl + command: + - kubectl + - scale + - deployment/shoryuken-worker + - --replicas=10 +``` + +--- + +## Cost Optimization + +### Right-Sizing + +Monitor utilization and adjust: + +```ruby +Shoryuken.configure_server do |config| + config.on(:utilization_update) do |group, utilization| + # Log for analysis + Rails.logger.info "Utilization #{group}: #{(utilization * 100).round}%" + end +end +``` + +Target: 60-80% utilization + +### Spot/Preemptible Instances + +Workers are stateless - safe for spot instances: + +```yaml +# Kubernetes +spec: + nodeSelector: + node-type: spot + tolerations: + - key: "spot" + operator: "Equal" + value: "true" +``` + +### Scale to Zero + +For dev/staging with KEDA: + +```yaml +minReplicaCount: 0 +idleReplicaCount: 0 +``` + +--- + +## Monitoring Scaling + +### Key Metrics + +| Metric | Healthy Range | Action if Outside | +|--------|---------------|-------------------| +| Queue depth | < 1000 | Scale out | +| Message age | < 5 min | Scale out | +| Utilization | 60-80% | Adjust concurrency | +| Error rate | < 1% | Investigate | + +### Dashboard + +Track: +- Workers running +- Messages processed/minute +- Queue depth over time +- Scaling events + +--- + +## Related + +- [[Processing Groups]] - Workload isolation +- [[Performance Tuning]] - Optimization +- [[Container Deployment]] - Kubernetes/ECS deployment +- [[Monitoring and Metrics]] - Observability diff --git a/Security-Best-Practices.md b/Security-Best-Practices.md new file mode 100644 index 0000000..b9ffa4f --- /dev/null +++ b/Security-Best-Practices.md @@ -0,0 +1,446 @@ +# Security Best Practices + +This guide covers security considerations for Shoryuken deployments with AWS SQS. + +## IAM Permissions + +### Principle of Least Privilege + +Grant only the permissions required for your use case: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "sqs:ReceiveMessage", + "sqs:DeleteMessage", + "sqs:DeleteMessageBatch", + "sqs:GetQueueAttributes", + "sqs:GetQueueUrl", + "sqs:ChangeMessageVisibility", + "sqs:ChangeMessageVisibilityBatch" + ], + "Resource": "arn:aws:sqs:us-east-1:123456789:myapp-*" + } + ] +} +``` + +### Separate Producer and Consumer Roles + +**Consumer (Worker) Role:** +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "sqs:ReceiveMessage", + "sqs:DeleteMessage", + "sqs:DeleteMessageBatch", + "sqs:GetQueueAttributes", + "sqs:GetQueueUrl", + "sqs:ChangeMessageVisibility" + ], + "Resource": "arn:aws:sqs:*:*:myapp-*" + } + ] +} +``` + +**Producer (Web App) Role:** +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "sqs:SendMessage", + "sqs:SendMessageBatch", + "sqs:GetQueueUrl" + ], + "Resource": "arn:aws:sqs:*:*:myapp-*" + } + ] +} +``` + +See [[Amazon SQS IAM Policy for Shoryuken]] for detailed policy examples. + +--- + +## Credential Management + +### Prefer IAM Roles Over Access Keys + +| Environment | Recommended Method | +|-------------|-------------------| +| EC2 | Instance Profile | +| ECS | Task Role | +| EKS | IRSA (IAM Roles for Service Accounts) | +| Lambda | Execution Role | +| Local Dev | AWS SSO or temporary credentials | + +### Never Hardcode Credentials + +```ruby +# BAD - Never do this +Aws.config.update( + access_key_id: 'AKIAXXXXXXXX', + secret_access_key: 'xxxxx' +) + +# GOOD - Use environment or IAM roles +# Credentials are automatically loaded from: +# 1. Environment variables +# 2. Shared credentials file (~/.aws/credentials) +# 3. IAM instance profile/task role +``` + +### Rotate Access Keys Regularly + +If you must use access keys: +1. Create new key pair +2. Update application configuration +3. Deploy and verify +4. Deactivate old key +5. Delete old key after grace period + +--- + +## Queue Security + +### Server-Side Encryption (SSE) + +Enable SSE-SQS or SSE-KMS for encryption at rest: + +```json +{ + "QueueName": "myapp-jobs", + "Attributes": { + "SqsManagedSseEnabled": "true" + } +} +``` + +For SSE-KMS (customer-managed key): + +```json +{ + "Attributes": { + "KmsMasterKeyId": "alias/myapp-sqs-key" + } +} +``` + +**IAM permissions for KMS:** +```json +{ + "Effect": "Allow", + "Action": [ + "kms:Decrypt", + "kms:GenerateDataKey" + ], + "Resource": "arn:aws:kms:us-east-1:123456789:key/xxx" +} +``` + +### VPC Endpoints + +Use VPC endpoints to keep traffic within AWS: + +```hcl +# Terraform example +resource "aws_vpc_endpoint" "sqs" { + vpc_id = aws_vpc.main.id + service_name = "com.amazonaws.us-east-1.sqs" + vpc_endpoint_type = "Interface" + + security_group_ids = [aws_security_group.sqs_endpoint.id] + subnet_ids = aws_subnet.private[*].id + + private_dns_enabled = true +} +``` + +### Queue Policies + +Restrict queue access by account/role: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "AWS": "arn:aws:iam::123456789:role/myapp-worker" + }, + "Action": [ + "sqs:ReceiveMessage", + "sqs:DeleteMessage" + ], + "Resource": "arn:aws:sqs:us-east-1:123456789:myapp-jobs" + } + ] +} +``` + +--- + +## Message Security + +### Sensitive Data in Messages + +**Avoid storing sensitive data directly in messages:** + +```ruby +# BAD - Sensitive data in message +UserNotificationJob.perform_later( + email: user.email, + ssn: user.ssn # Never include PII +) + +# GOOD - Reference by ID +UserNotificationJob.perform_later(user_id: user.id) +``` + +### Encryption for Sensitive Payloads + +If you must include sensitive data, encrypt it: + +```ruby +class SecureJob < ApplicationJob + def perform(encrypted_data) + data = decrypt(encrypted_data) + # Process data + end + + private + + def decrypt(encrypted) + cipher = OpenSSL::Cipher.new('AES-256-GCM') + cipher.decrypt + cipher.key = Rails.application.credentials.encryption_key + # ... decryption logic + end +end + +# When enqueuing +encrypted = encrypt(sensitive_data) +SecureJob.perform_later(encrypted) +``` + +### Message Validation + +Validate messages before processing: + +```ruby +class ValidationMiddleware + def call(worker_instance, queue, sqs_msg, body) + unless valid_signature?(sqs_msg) + Shoryuken.logger.warn "Invalid message signature" + sqs_msg.delete + return + end + + yield + end + + private + + def valid_signature?(sqs_msg) + # Verify message integrity + end +end +``` + +--- + +## Network Security + +### Security Groups + +Restrict outbound traffic to only required AWS services: + +```hcl +resource "aws_security_group" "worker" { + name = "shoryuken-worker" + + # SQS via VPC endpoint + egress { + from_port = 443 + to_port = 443 + protocol = "tcp" + prefix_list_ids = [aws_vpc_endpoint.sqs.prefix_list_id] + } + + # Database + egress { + from_port = 5432 + to_port = 5432 + protocol = "tcp" + security_groups = [aws_security_group.database.id] + } +} +``` + +### Private Subnets + +Deploy workers in private subnets without public IPs: + +```yaml +# ECS Task Definition +networkMode: awsvpc +networkConfiguration: + awsvpcConfiguration: + subnets: + - subnet-private-1 + - subnet-private-2 + assignPublicIp: DISABLED +``` + +--- + +## Secrets Management + +### AWS Secrets Manager + +```ruby +# config/initializers/shoryuken.rb +require 'aws-sdk-secretsmanager' + +Shoryuken.configure_server do |config| + config.on(:startup) do + client = Aws::SecretsManager::Client.new + secret = client.get_secret_value(secret_id: 'myapp/database') + + ENV['DATABASE_URL'] = JSON.parse(secret.secret_string)['url'] + end +end +``` + +### Environment Variables + +Use encrypted environment variables in container platforms: + +```yaml +# ECS Task Definition +secrets: + - name: DATABASE_URL + valueFrom: arn:aws:secretsmanager:us-east-1:123456789:secret:myapp/database +``` + +--- + +## Audit Logging + +### CloudTrail + +Enable CloudTrail for SQS API calls: + +```json +{ + "eventSource": "sqs.amazonaws.com", + "eventName": "ReceiveMessage", + "sourceIPAddress": "10.0.1.50", + "userAgent": "aws-sdk-ruby3/3.0.0" +} +``` + +### Application Logging + +Log job execution for audit trails: + +```ruby +class AuditMiddleware + def call(worker_instance, queue, sqs_msg, body) + Rails.logger.info({ + event: 'job_started', + job: worker_instance.class.name, + queue: queue, + message_id: sqs_msg.message_id, + timestamp: Time.current.iso8601 + }.to_json) + + yield + + Rails.logger.info({ + event: 'job_completed', + job: worker_instance.class.name, + message_id: sqs_msg.message_id, + timestamp: Time.current.iso8601 + }.to_json) + rescue => e + Rails.logger.error({ + event: 'job_failed', + job: worker_instance.class.name, + message_id: sqs_msg.message_id, + error: e.message, + timestamp: Time.current.iso8601 + }.to_json) + raise + end +end +``` + +--- + +## Dead Letter Queue Security + +### Protect DLQ Contents + +Dead letter queues may contain sensitive data from failed jobs: + +1. Apply same encryption as main queue +2. Restrict access to operators only +3. Set appropriate retention period +4. Monitor DLQ depth for anomalies + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "AWS": "arn:aws:iam::123456789:role/myapp-operator" + }, + "Action": [ + "sqs:ReceiveMessage", + "sqs:DeleteMessage", + "sqs:GetQueueAttributes" + ], + "Resource": "arn:aws:sqs:us-east-1:123456789:myapp-jobs-dlq" + } + ] +} +``` + +--- + +## Security Checklist + +- [ ] IAM roles use least privilege +- [ ] No hardcoded credentials +- [ ] Server-side encryption enabled +- [ ] VPC endpoints configured (if applicable) +- [ ] Workers in private subnets +- [ ] Sensitive data not stored in messages +- [ ] CloudTrail enabled for SQS +- [ ] DLQ access restricted +- [ ] Security groups properly configured +- [ ] Secrets managed securely + +--- + +## Related + +- [[Amazon SQS IAM Policy for Shoryuken]] - IAM policy examples +- [[Configure the AWS Client]] - Credential configuration +- [[Container Deployment]] - Secure container deployments diff --git a/Sending-a-message.md b/Sending-a-message.md index 835a52b..f747eab 100644 --- a/Sending-a-message.md +++ b/Sending-a-message.md @@ -1,69 +1,158 @@ -There are a couple ways to send a message using Shoryuken. +# Sending a Message -### Using `perform_async`: +This guide covers different ways to send messages to SQS queues with Shoryuken. + +## Shoryuken Workers + +### perform_async + +Send a message for immediate processing: ```ruby MyWorker.perform_async('Pablo') -``` +``` + +With a Hash (automatically converted to JSON): + +```ruby +MyWorker.perform_async(user_id: 123, action: 'notify') +``` + +Override the queue: + +```ruby +MyWorker.perform_async('Pablo', queue: 'important') +``` + +### perform_in / perform_at + +Delay message delivery (up to 15 minutes - SQS limit): + +```ruby +# Delay by seconds +MyWorker.perform_in(60, 'Pablo') # 60 seconds + +# Delay until specific time +MyWorker.perform_at(5.minutes.from_now, 'Pablo') +``` + +--- + +## ActiveJob + +### perform_later + +```ruby +MyJob.perform_later(user_id: 123) +``` + +### Delayed Jobs + +```ruby +# Delay by duration +MyJob.set(wait: 5.minutes).perform_later(user_id: 123) + +# Delay until specific time +MyJob.set(wait_until: 1.hour.from_now).perform_later(user_id: 123) +``` + +### Bulk Enqueuing (Rails 7.1+) + +Enqueue multiple jobs efficiently: + +```ruby +jobs = users.map { |user| NotifyUserJob.new(user.id) } +ActiveJob.perform_all_later(jobs) +``` + +See [[Bulk Enqueuing]] for details. + +--- -it also accepts a Hash as a parameter, which is automatically converted into JSON: +## Direct SQS Client + +Use the SQS client directly for more control: + +### Single Message ```ruby -MyWorker.perform_async(field: 'test', other_field: 'other') +Shoryuken::Client.queues('default').send_message('message body') ``` -it's also possible to override where the job is queued to with an options hash: +With options: ```ruby -MyWorker.perform_async('Pablo') # will queue to the default queue -MyWorker.perform_async('Pablo', queue: 'important') # will queue to the 'important' queue +Shoryuken::Client.queues('default').send_message( + message_body: { user_id: 123 }.to_json, + message_attributes: { + 'type' => { + string_value: 'notification', + data_type: 'String' + } + } +) ``` -### Or using the queue object directly , which wraps [Aws::SQS::Client](https://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Client.html): +### Multiple Messages ```ruby -# To send a single message -Shoryuken::Client.queues('default').send_message('msg 1') +Shoryuken::Client.queues('default').send_messages([ + 'message 1', + 'message 2', + 'message 3' +]) +``` -# To send a single message as a Hash you need to put it in a hash with `message_body` as the key. -Shoryuken::Client.queues('default').send_message(message_body: { example: "data" }) +With full options: -# To send multiple messages -Shoryuken::Client.queues('default').send_messages(['msg 1', 'msg 2']) +```ruby +Shoryuken::Client.queues('default').send_messages( + entries: [ + { id: '1', message_body: 'msg 1' }, + { id: '2', message_body: 'msg 2' }, + { id: '3', message_body: 'msg 3', delay_seconds: 60 } + ] +) ``` -#### Sending a message directly for ActiveJob workers -In the case that the job message is generated outside of your Rails Shoryuken via ActiveJob environment (like from a serverless function), you can still send a message to be processed by your workers. + +### Batch Limits + +| Limit | Value | +|-------|-------| +| Max messages per batch | 10 | +| Max total batch size | 1 MB (256 KB × 4) | +| Max single message | 256 KB | + +--- + +## Sending ActiveJob Messages Externally + +Send messages from outside Rails (e.g., Lambda, external service): + ```ruby queue_name = 'my-queue' job_args = { - "job_class": "MyActiveJob", - "job_id": SecureRandom.uuid, - "provider_job_id": nil, - "queue_name": queue_name, - "priority": nil, - "arguments": [ + job_class: 'NotifyUserJob', + job_id: SecureRandom.uuid, + queue_name: queue_name, + arguments: [ { - "arg1": 'arg1 value', - "arg2": 'arg2 value', - "arg3": 'arg3 value', - "_aj_symbol_keys": [ - 'arg1', - 'arg2', - 'arg3', - ] + user_id: 123, + message: 'Hello', + _aj_symbol_keys: ['user_id', 'message'] } ], - "executions": 0, - "locale": "en" + executions: 0, + locale: 'en' } -message = { - message_body: job_args, +message = { + message_body: job_args.to_json, message_attributes: { shoryuken_class: { string_value: 'ActiveJob::QueueAdapters::ShoryukenAdapter::JobWrapper', - data_type: "String" + data_type: 'String' } } } @@ -71,15 +160,116 @@ message = { Shoryuken::Client.queues(queue_name).send_message(message) ``` -## Delaying a message +--- + +## FIFO Queues + +FIFO queues require message group ID and optionally deduplication ID: + +```ruby +Shoryuken::Client.queues('my-queue.fifo').send_message( + message_body: 'order processing', + message_group_id: 'order-123', + message_deduplication_id: SecureRandom.uuid +) +``` + +With content-based deduplication enabled on the queue, you can omit `message_deduplication_id`. + +See [[FIFO Queues]] for more details. + +--- + +## Message Attributes + +Add metadata to messages: + +```ruby +Shoryuken::Client.queues('default').send_message( + message_body: 'test', + message_attributes: { + 'correlation_id' => { + string_value: SecureRandom.uuid, + data_type: 'String' + }, + 'priority' => { + string_value: '1', + data_type: 'Number' + }, + 'binary_data' => { + binary_value: Base64.encode64('raw data'), + data_type: 'Binary' + } + } +) +``` + +Access in worker: -You can delay a message up to 15 minutes. +```ruby +def perform(sqs_msg, body) + correlation_id = sqs_msg.message_attributes.dig('correlation_id', 'string_value') +end +``` + +--- + +## Concurrent Sending (Async) + +Use the concurrent adapter for non-blocking sends: + +```ruby +# config/application.rb +config.active_job.queue_adapter = ActiveJob::QueueAdapters::ShoryukenConcurrentSendAdapter.new( + ->(job, response) { Rails.logger.info "Enqueued: #{job.job_id}" }, + ->(job, error) { Rails.logger.error "Failed: #{error.message}" } +) +``` + +See [[Rails Integration Active Job]] for details. + +--- + +## Error Handling + +### Rescue Send Errors ```ruby -MyWorker.perform_in(60, 'Pablo') # 60 seconds -MyWorker.perform_at(Time.now, 'Pablo') +begin + MyWorker.perform_async('test') +rescue Aws::SQS::Errors::ServiceError => e + Rails.logger.error "SQS error: #{e.message}" + # Handle error (retry, fallback, etc.) +end ``` -## AWS credentials +### Batch Failures + +When sending batches, check for partial failures: + +```ruby +response = Shoryuken::Client.queues('default').send_messages( + entries: messages +) + +if response.failed.any? + response.failed.each do |failure| + Rails.logger.error "Failed to send #{failure.id}: #{failure.message}" + end +end +``` + +--- + +## AWS Credentials + +Ensure AWS credentials are configured before sending messages. See [[Configure the AWS Client]]. + +--- + +## Related -Check [AWS credentials](https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-config.html) to make sure you have your aws-sdk before sending messages. \ No newline at end of file +- [[Bulk Enqueuing]] - Efficient batch enqueuing +- [[FIFO Queues]] - Ordered message processing +- [[Rails Integration Active Job]] - ActiveJob integration +- [[Worker options]] - Worker configuration diff --git a/Shoryuken-options.md b/Shoryuken-options.md index 631e730..bc7c068 100644 --- a/Shoryuken-options.md +++ b/Shoryuken-options.md @@ -1,106 +1,346 @@ -Shoryuken options can be set via CLI: +# Shoryuken Options + +Shoryuken can be configured via CLI options, a YAML configuration file, or programmatically. + +## Configuration Methods + +### CLI Options ```shell -shoryuken -q queue1 -L ./shoryuken.log -P ./shoryuken.pid +shoryuken -q queue1 -q queue2 -c 25 -R -C config/shoryuken.yml ``` -or via configuration file: + +### Configuration File ```yaml -# shoryuken.yml -logfile: ./shoryuken.log -pidfile: ./shoryuken.pid -queues: - - queue1 +# config/shoryuken.yml +concurrency: 25 +timeout: 25 +queues: + - [default, 2] + - [low_priority, 1] +``` + +Start with the config file: + +```shell +shoryuken -C config/shoryuken.yml -R +``` + +### Programmatic Configuration + +```ruby +# config/initializers/shoryuken.rb +Shoryuken.configure_server do |config| + # Server-side configuration +end + +Shoryuken.configure_client do |config| + # Client-side configuration (when enqueuing jobs) +end +``` + +--- + +## CLI Options + +| Option | Description | +|--------|-------------| +| `-C, --config FILE` | Path to YAML configuration file | +| `-R, --rails` | Load Rails application | +| `-r, --require FILE` | Require a file or directory | +| `-q, --queue QUEUE[,WEIGHT]` | Queue to process (can specify multiple) | +| `-c, --concurrency INT` | Number of worker threads | +| `-t, --timeout INT` | Shutdown timeout in seconds | +| `-L, --logfile FILE` | Log file path | +| `-P, --pidfile FILE` | PID file path | +| `-d, --daemon` | Daemonize process | + +### Examples + +```shell +# Basic usage with Rails +shoryuken -R -C config/shoryuken.yml + +# Multiple queues with weights +shoryuken -R -q critical,3 -q default,2 -q low,1 + +# With logging and PID file +shoryuken -R -C config/shoryuken.yml -L log/shoryuken.log -P tmp/pids/shoryuken.pid + +# Daemonize +shoryuken -R -C config/shoryuken.yml -d +``` + +For all CLI options: + +```shell +shoryuken help start ``` -When using a configuration file, you must set via CLI `shoryuken -C ./shoryuken.yml`, otherwise Shoryuken won't use it. +--- + +## Configuration File Options -Some options available in the configuration file, are not available in the CLI. For checking all options available in the CLI: `shoryuken help start`. +### concurrency -## delay +Number of worker threads. Default: `25` + +```yaml +concurrency: 25 +``` -Default: `0` +**Note:** Set your database connection pool size to at least this value. -Delay is the number of seconds to pause fetching from when an empty queue. +### timeout -Given this configuration: +Shutdown timeout in seconds. Default: `8` ```yaml -delay: 25 -queues: - - queue1 - - queue2 +timeout: 25 ``` -If Shoryuken tries to fetch messages from `queue1` and it has no messages, Shoryuken will pause fetching from `queue1` for 25 seconds. +Workers that don't finish within this time during SIGTERM are forcefully terminated. -Usually having a delay is more cost-efficient, but if you want to consume messages as soon as they get in the queue, I would recommend setting `delay: 0` (default value) to stop pausing empty queues. +### delay -[Check the AWS SQS pricing page for more information](https://aws.amazon.com/sqs/pricing/). +Seconds to pause before re-polling an empty queue. Default: `0` -## queues +```yaml +delay: 25 +``` + +Setting `delay: 0` polls continuously (faster but more expensive). + +### queues -A single Shoryuken process can consume messages from multiple queues. You can define your queues as follows: +Queues to process with optional weights: ```yaml +# Simple list queues: - queue1 - queue2 -``` -You can also configure the queue URLs instead of names. +# With weights (higher = more priority) +queues: + - [critical, 3] + - [default, 2] + - [low, 1] -```yaml +# Using URLs (for cross-account/cross-region) queues: - - https://sqs.eu-west-1.amazonaws.com:000/1111111111/queue1 - - https://sqs.eu-east-1.amazonaws.com:000/2222222222/queue2 + - https://sqs.us-east-1.amazonaws.com/123456789/myqueue + +# Using ARNs +queues: + - arn:aws:sqs:us-east-1:123456789:myqueue ``` -Or ARN. +### logfile + +Path to log file: ```yaml -queues: - - arn:aws:sqs:us-east-1:123456789012:queue1 - - arn:aws:sqs:us-east-1:123456789012:queue2 +logfile: log/shoryuken.log ``` -### Load balancing +### pidfile -Supposing you have `queue1`, which you would like to fetch messages twice as much as `queue2`, you can configure that as follows: +Path to PID file: ```yaml -queues: - - [queue1, 8] - - [queue2, 4] - - [queue3, 1] +pidfile: tmp/pids/shoryuken.pid ``` -The setup above will cause Shoryuken to fetch messages in cycles of `queue1` 8 times, then `queue2` 4 times, then `queue3` once, then repeat. +--- -*Note:* Each fetch can fetch up to 10 messages at a time (SQS limitation). The fetch size will also depend on the available workers at the time of the fetch. +## Processing Groups -See [Polling strategies](https://github.com/phstc/shoryuken/wiki/Polling-strategies#weightedroundrobin). +Isolate queues with separate concurrency settings: -## concurrency +```yaml +groups: + critical: + concurrency: 10 + delay: 0 + queues: + - [payments, 1] + default: + concurrency: 25 + delay: 5 + queues: + - [emails, 2] + - [reports, 1] +``` + +See [[Processing Groups]] for more details. -Default: `25` +--- -Concurrency is the number of threads available for processing messages at a time. Be careful with changing this value, otherwise, you could crush your machine with I/O. +## AWS Configuration -*Note:* When accessing databases (`ActiveRecord`) or other resources with `pool` support in your workers, make sure to set `pool` size with the same value or higher than the concurrency set in Shoryuken. +Configure AWS credentials in the config file: ```yaml -concurrency: <%= ENV.fetch('YOUR_CONCURRENCY', 25) %> +aws: + access_key_id: <%= ENV['AWS_ACCESS_KEY_ID'] %> + secret_access_key: <%= ENV['AWS_SECRET_ACCESS_KEY'] %> + region: us-east-1 ``` -## cache_visibility_timeout +**Recommended:** Use IAM roles instead of access keys. See [[Configure the AWS Client]]. + +--- + +## Programmatic Options -By default, Shoryuken will make requests against SQS for getting the updated queue visibility timeout, so if you change the visibility timeout in SQS, Shoryuken will automatically update it. +### Server Configuration + +```ruby +Shoryuken.configure_server do |config| + # Logger + config.logger = Rails.logger + config.logger.level = Logger::INFO + + # Middleware + config.server_middleware do |chain| + chain.add MyMiddleware + end + + # Lifecycle events + config.on(:startup) { puts "Starting" } + config.on(:shutdown) { puts "Stopping" } + + # Exception handler + config.on_exception do |ex, queue, sqs_msg| + Sentry.capture_exception(ex) + end +end +``` -If you want to reduce the number of requests against SQS (therefore reducing your quota usage), you can disable this behavior by: +### Client Configuration + +```ruby +Shoryuken.configure_client do |config| + # SQS client configuration + config.sqs_client = Aws::SQS::Client.new(region: 'us-east-1') +end +``` + +### Dynamic Queue Registration + +```ruby +# Add a queue at runtime +Shoryuken.add_queue('new_queue', 1, 'default') + +# Add a group +Shoryuken.add_group('critical', 10) +Shoryuken.add_queue('payments', 1, 'critical') +``` + +--- + +## Other Options + +### active_job_queue_name_prefixing + +Honor ActiveJob queue name prefixes: + +```ruby +# config/initializers/shoryuken.rb +Shoryuken.active_job_queue_name_prefixing = true +``` + +### cache_visibility_timeout + +Cache queue visibility timeout to reduce SQS API calls: ```ruby Shoryuken.cache_visibility_timeout = true ``` -With `cache_visibility_timeout` enabled, every time you change the visibility timeout in SQS, you will need to restart Shoryuken, otherwise, it won't get updated. \ No newline at end of file +**Note:** With caching enabled, restart Shoryuken after changing visibility timeout in SQS. + +--- + +## ERB in Configuration Files + +Configuration files support ERB: + +```yaml +# config/shoryuken.yml +concurrency: <%= ENV.fetch('SHORYUKEN_CONCURRENCY', 25) %> +timeout: <%= ENV.fetch('SHORYUKEN_TIMEOUT', 25) %> +queues: + - <%= ENV.fetch('QUEUE_NAME', 'default') %> +``` + +--- + +## Environment-Specific Configuration + +```yaml +# config/shoryuken.yml +<% if Rails.env.production? %> +concurrency: 50 +<% else %> +concurrency: 5 +<% end %> + +queues: + - <%= "#{Rails.env}_default" %> +``` + +--- + +## Complete Example + +```yaml +# config/shoryuken.yml +concurrency: <%= ENV.fetch('SHORYUKEN_CONCURRENCY', 25) %> +timeout: 25 +delay: 5 +logfile: <%= Rails.root.join('log', 'shoryuken.log') %> +pidfile: <%= Rails.root.join('tmp', 'pids', 'shoryuken.pid') %> + +groups: + critical: + concurrency: 10 + delay: 0 + queues: + - [payments, 1] + - [webhooks, 1] + default: + concurrency: 25 + delay: 5 + queues: + - [emails, 3] + - [reports, 2] + - [cleanup, 1] +``` + +```ruby +# config/initializers/shoryuken.rb +Shoryuken.configure_server do |config| + Rails.logger = Shoryuken::Logging.logger + Rails.logger.level = Rails.application.config.log_level + + config.on(:startup) do + Rails.logger.info "Shoryuken starting with concurrency #{Shoryuken.options[:concurrency]}" + end + + config.on(:shutdown) do + Rails.logger.info "Shoryuken shutting down" + end +end +``` + +--- + +## Related + +- [[Worker options]] - Per-worker configuration +- [[Processing Groups]] - Queue isolation +- [[Configure the AWS Client]] - AWS credentials +- [[Polling strategies]] - Queue polling behavior diff --git a/Signals.md b/Signals.md index eef8a5f..4f99db5 100644 --- a/Signals.md +++ b/Signals.md @@ -1,37 +1,331 @@ -## TTIN +# Signals -TTIN tells Shoryuken to print backtraces for all threads in the process along with the current number of busy and ready processors and the actual weights of the queues. +Shoryuken responds to Unix signals for process control. Understanding these signals is essential for graceful deployments and debugging. + +## Signal Summary + +| Signal | Effect | Use Case | +|--------|--------|----------| +| `TTIN` | Print debug info | Debugging, monitoring | +| `TSTP` | Quiet mode (stop fetching) | Pre-shutdown, deployments | +| `USR1` | Graceful shutdown | Deployments, scaling down | +| `TERM` | Hard shutdown with timeout | Container termination | +| `INT` | Same as TERM | Ctrl+C in terminal | + +--- + +## TTIN - Debug Information + +Prints thread backtraces, processor stats, and queue weights to the log. ```bash -kill -TTIN [shoryuken_pid] +kill -TTIN $(cat tmp/pids/shoryuken.pid) ``` -## TSTP +### Output Includes + +- Backtraces for all threads +- Number of busy/ready processors +- Current queue weights +- Processing group status + +### Use Cases -TSTP tells Shoryuken to shut down gracefully. It will stop sending any new requests to fetch from SQS, but will ensure that all the requests already sent will be processed (guaranteeing that no message stayed as non-visible on SQS). +- Debugging hung workers +- Identifying slow jobs +- Monitoring thread utilization -This signal does not exit Shoryuken, you still need to TERM for exiting the process. +--- + +## TSTP - Quiet Mode + +Stops fetching new messages but continues processing in-flight messages. The process remains running. ```bash -kill -TSTP [shoryuken_pid] +kill -TSTP $(cat tmp/pids/shoryuken.pid) ``` -## USR1 +### Behavior + +1. Stops sending requests to SQS +2. Processes all messages already fetched +3. Messages remain visible (not lost) +4. Process stays alive + +### Use Cases + +- Pre-deployment pause +- Manual traffic drain +- Debugging without termination + +### Notes + +- Does **not** exit the process +- Requires `TERM` or `USR1` to fully stop +- Fires the `quiet` lifecycle event -USR1 tells Shoryuken to shut down gracefully. It will stop sending any new requests to fetch from SQS, but will ensure that all the requests already sent will be processed (guaranteeing that no message stayed as non-visible on SQS). +--- -Shoryuken exits once everything is processed. +## USR1 - Graceful Shutdown + +Initiates graceful shutdown: stops fetching and exits after processing all in-flight messages. ```bash -kill -USR1 [shoryuken_pid] +kill -USR1 $(cat tmp/pids/shoryuken.pid) ``` -## TERM +### Behavior + +1. Stops fetching new messages +2. Waits for all in-flight messages to complete +3. Fires lifecycle events (`quiet`, `shutdown`, `stopped`) +4. Exits cleanly + +### Use Cases + +- Deployments +- Scaling down +- Graceful restarts -TERM tells Shoryuken to hard shutdown within a `timeout`. Any processors that do not finish within the timeout are hard terminated and their messages are pushed back to AWS SQS. The timeout defaults to 8 seconds since all Heroku processes must exit within 10 seconds. +### Lifecycle Events + +``` +USR1 received + │ + ▼ +quiet event ──────► Stop accepting new work + │ + ▼ +Process in-flight messages + │ + ▼ +shutdown event ───► Close connections + │ + ▼ +stopped event ────► Final cleanup + │ + ▼ +Exit(0) +``` -The `timeout` can be defined through the `-t` option or in the `-C shoryuken.yml` config by setting the `timeout: N` value. +--- + +## TERM - Hard Shutdown + +Initiates shutdown with a timeout. Workers that don't finish within the timeout are terminated. ```bash -kill -TERM [shoryuken_pid] -``` \ No newline at end of file +kill -TERM $(cat tmp/pids/shoryuken.pid) +``` + +### Behavior + +1. Stops fetching new messages +2. Waits up to `timeout` seconds for workers +3. Forcefully terminates remaining workers +4. Unfinished messages return to SQS (visibility timeout) +5. Exits + +### Timeout Configuration + +```yaml +# config/shoryuken.yml +timeout: 25 # seconds +``` + +Or via CLI: + +```bash +shoryuken -t 25 ... +``` + +### Default Timeout + +The default timeout is 8 seconds (designed for Heroku's 10-second limit). + +### Use Cases + +- Container termination (Docker, Kubernetes) +- Heroku dyno restarts +- Emergency shutdown + +--- + +## INT - Interrupt + +Same behavior as TERM. Typically sent when pressing Ctrl+C in a terminal. + +```bash +# Ctrl+C in terminal +# or +kill -INT $(cat tmp/pids/shoryuken.pid) +``` + +--- + +## Container Orchestration + +### Kubernetes + +Kubernetes sends SIGTERM on pod termination: + +```yaml +spec: + terminationGracePeriodSeconds: 300 # 5 minutes + containers: + - name: worker + # ... +``` + +Set Shoryuken timeout lower than grace period: + +```yaml +# config/shoryuken.yml +timeout: 280 # Leave 20s buffer +``` + +### Docker / ECS + +Configure stop timeout in task definition: + +```json +{ + "stopTimeout": 300 +} +``` + +### Heroku + +Heroku allows 30 seconds for SIGTERM shutdown: + +```yaml +# config/shoryuken.yml +timeout: 25 # Leave 5s buffer +``` + +--- + +## ActiveJob Continuations + +With Rails 8.1+, jobs can check `stopping?` to checkpoint progress: + +```ruby +class LongRunningJob < ApplicationJob + def perform(data) + data.each do |item| + if stopping? + # Save progress and re-enqueue + self.class.perform_later(remaining_data) + return + end + process(item) + end + end +end +``` + +The `stopping?` flag is set when: +- `USR1` is received +- `TERM` is received +- `stop` or `stop!` is called on the launcher + +See [[ActiveJob Continuations]] for more details. + +--- + +## Health Check Integration + +During shutdown: + +| Phase | `stopping?` | `healthy?` | +|-------|-------------|------------| +| Normal operation | `false` | `true` | +| After TERM/USR1 | `true` | `true` | +| Workers stopping | `true` | `true` | +| Workers stopped | `true` | `false` | + +Use `stopping?` for readiness probes (stop accepting new connections): + +```ruby +# Readiness check +if launcher&.stopping? + [503, {}, ['Shutting down']] +else + [200, {}, ['Ready']] +end +``` + +See [[Health Checks]] for more details. + +--- + +## Signal Handling in Scripts + +### Deployment Script + +```bash +#!/bin/bash + +PID_FILE="tmp/pids/shoryuken.pid" + +if [ -f "$PID_FILE" ]; then + echo "Sending USR1 to Shoryuken..." + kill -USR1 $(cat $PID_FILE) + + # Wait for graceful shutdown + while [ -f "$PID_FILE" ]; do + sleep 1 + done + + echo "Shoryuken stopped" +fi + +# Start new worker +bundle exec shoryuken -R -C config/shoryuken.yml -P $PID_FILE -d +echo "Shoryuken started" +``` + +### systemd Integration + +```ini +[Service] +ExecReload=/bin/kill -USR1 $MAINPID +TimeoutStopSec=300 +KillSignal=SIGTERM +``` + +--- + +## Debugging with Signals + +### Finding Stuck Jobs + +```bash +# Send TTIN to get thread dump +kill -TTIN $(cat tmp/pids/shoryuken.pid) + +# Check logs for backtraces +tail -100 log/shoryuken.log +``` + +### Manual Drain + +```bash +# Enter quiet mode +kill -TSTP $(cat tmp/pids/shoryuken.pid) + +# Monitor queue processing +watch "cat tmp/pids/shoryuken.pid | xargs ps -o pid,stat,time" + +# When ready, terminate +kill -USR1 $(cat tmp/pids/shoryuken.pid) +``` + +--- + +## Related + +- [[Lifecycle Events]] - Event hooks during shutdown +- [[ActiveJob Continuations]] - Job checkpointing +- [[Health Checks]] - Health monitoring during shutdown +- [[Deployment]] - Deployment strategies diff --git a/Testing.md b/Testing.md index 0f6951f..48731d2 100644 --- a/Testing.md +++ b/Testing.md @@ -1,58 +1,402 @@ -There are few options for testing Shoryuken workers, there are some examples below with RSpec, which probably works very similar with most of the testing frameworks. +# Testing -Another alternative is to enable the [Shoryuken Inline adapter](https://github.com/phstc/shoryuken/wiki/Shoryuken-Inline-adapter), or if you are using Active Job, refer to [Active Job Inline adapter -](http://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/InlineAdapter.html). +This guide covers testing strategies for Shoryuken workers and ActiveJob jobs. -## Testing workers with RSpec +## Testing Strategies -For testing with RSpec you can stub `perform_async` and `perform_in`: +| Strategy | Best For | Pros | Cons | +|----------|----------|------|------| +| Unit tests | Worker logic | Fast, isolated | Doesn't test middleware | +| Inline adapter | Full integration | Tests middleware chain | Synchronous only | +| LocalStack | End-to-end | Real SQS behavior | Requires Docker | + +--- + +## Unit Testing Workers + +### RSpec ```ruby -# expecting the worker to be called called -expect(MyJob).to receive(:perform_async) +# spec/workers/my_worker_spec.rb +require 'rails_helper' + +RSpec.describe MyWorker do + let(:sqs_msg) do + double( + message_id: 'fc754df7-9cc2-4c41-96ca-5996a44b771e', + body: 'test', + delete: nil + ) + end + let(:body) { { 'user_id' => 123 } } + + describe '#perform' do + subject(:worker) { described_class.new } + + it 'processes the message' do + user = create(:user, id: 123) + + worker.perform(sqs_msg, body) -# calling the original worker -allow(MyJob).to receive(:perform_async) { |sqs_msg, body| MyJob.new.perform(sqs_msg, body) } + expect(user.reload.processed).to be true + end + + it 'deletes the message after processing' do + expect(sqs_msg).to receive(:delete) + + worker.perform(sqs_msg, body) + end + end +end ``` -### Full testing example +### Minitest ```ruby -require 'spec_helper' -# OR -# require 'rails_helper' # if using Rails 4.x +# test/workers/my_worker_test.rb +require 'test_helper' -RSpec.describe MyWorker do - let(:sqs_msg) { double message_id: 'fc754df7-9cc2-4c41-96ca-5996a44b771e', - body: 'test', - delete: nil } +class MyWorkerTest < ActiveSupport::TestCase + def setup + @sqs_msg = Minitest::Mock.new + @sqs_msg.expect :message_id, 'test-123' + @sqs_msg.expect :body, 'test' + @body = { 'user_id' => 123 } + end - let(:body) { 'test' } + test 'processes message correctly' do + user = users(:one) + MyWorker.new.perform(@sqs_msg, @body) + + assert user.reload.processed + end +end +``` + +--- + +## Testing ActiveJob Jobs + +### RSpec + +```ruby +# spec/jobs/process_order_job_spec.rb +require 'rails_helper' + +RSpec.describe ProcessOrderJob, type: :job do describe '#perform' do - subject { MyWorker.new } + let(:order) { create(:order) } + + it 'processes the order' do + described_class.perform_now(order) - it 'prints the body message' do - expect { subject.perform(sqs_msg, body) }.to output('test').to_stdout + expect(order.reload.status).to eq('processed') end - it 'deletes the message' do - expect(sqs_msg).to receive(:delete) + it 'enqueues to the correct queue' do + expect { + described_class.perform_later(order) + }.to have_enqueued_job.on_queue('orders') + end + end +end +``` + +### ActiveJob Test Helpers + +```ruby +# spec/rails_helper.rb +RSpec.configure do |config| + config.include ActiveJob::TestHelper +end + +# In your spec +RSpec.describe OrdersController, type: :controller do + describe 'POST #create' do + it 'enqueues a processing job' do + expect { + post :create, params: { order: valid_attributes } + }.to have_enqueued_job(ProcessOrderJob) + end + end +end +``` + +### Testing with perform_enqueued_jobs + +```ruby +RSpec.describe 'Order processing', type: :feature do + it 'processes orders end-to-end' do + order = create(:order) + + perform_enqueued_jobs do + ProcessOrderJob.perform_later(order) + end + + expect(order.reload.status).to eq('completed') + end +end +``` + +--- + +## Testing CurrentAttributes + +```ruby +# spec/jobs/audit_job_spec.rb +require 'rails_helper' + +RSpec.describe AuditJob, type: :job do + let(:user) { create(:user) } + + before do + Current.user = user + end + + after do + Current.reset + end + + it 'uses current attributes' do + expect { + described_class.perform_now(action: 'test') + }.to change(AuditLog, :count).by(1) + + expect(AuditLog.last.user).to eq(user) + end +end +``` + +--- + +## Testing ActiveJob Continuations + +```ruby +# spec/jobs/data_export_job_spec.rb +require 'rails_helper' + +RSpec.describe DataExportJob, type: :job do + let(:export) { create(:export, :with_records) } + + describe 'when stopping? is false' do + it 'processes all records' do + described_class.perform_now(export) + + expect(export.reload.status).to eq('completed') + end + end + + describe 'when stopping? becomes true' do + it 'checkpoints and re-enqueues' do + job = described_class.new(export) + + # Simulate shutdown after processing 2 records + call_count = 0 + allow(job).to receive(:stopping?) do + call_count += 1 + call_count > 2 + end + + expect { + job.perform_now + }.to have_enqueued_job(described_class) + + expect(export.reload.last_processed_id).to be_present + end + end +end +``` + +--- + +## Inline Adapter + +The inline adapter processes messages synchronously, useful for testing: + +### For Shoryuken Workers + +```ruby +# spec/rails_helper.rb or test environment +Shoryuken.configure_server do |config| + config.sqs_client = Shoryuken::InlineExecutor +end + +# Or per-test +RSpec.describe 'Integration', type: :feature do + around do |example| + original = Shoryuken.sqs_client + Shoryuken.sqs_client = Shoryuken::InlineExecutor + example.run + Shoryuken.sqs_client = original + end + + it 'processes jobs inline' do + MyWorker.perform_async('test') + # Job runs immediately + end +end +``` + +### For ActiveJob + +```ruby +# config/environments/test.rb +config.active_job.queue_adapter = :inline +``` + +Or use ActiveJob's test adapter: + +```ruby +# config/environments/test.rb +config.active_job.queue_adapter = :test +``` + +--- + +## Mocking SQS Client + +```ruby +# spec/workers/notification_worker_spec.rb +RSpec.describe NotificationWorker do + let(:queue) { instance_double(Shoryuken::Queue) } + + before do + allow(Shoryuken::Client).to receive(:queues) + .with('notifications') + .and_return(queue) + end - subject.perform(sqs_msg, body) + it 'sends notification to queue' do + expect(queue).to receive(:send_message).with( + hash_including(message_body: hash_including('type' => 'alert')) + ) + + described_class.new.perform(sqs_msg, body) + end +end +``` + +--- + +## Testing with LocalStack + +For full integration tests with real SQS behavior, use LocalStack: + +```ruby +# spec/support/localstack.rb +RSpec.configure do |config| + config.before(:suite) do + Shoryuken.configure_client do |c| + c.sqs_client = Aws::SQS::Client.new( + endpoint: 'http://localhost:4566', + region: 'us-east-1', + access_key_id: 'test', + secret_access_key: 'test' + ) end - it 'pushes a new message' do - sqs_queue = double 'other queue' + # Create test queues + client = Shoryuken.sqs_client + client.create_queue(queue_name: 'test-queue') + end +end +``` + +See [[Using a local mock SQS server]] for LocalStack setup. - allow(Shoryuken::Client).to receive(:queues). - with('other_queue').and_return(sqs_queue) +--- - expect(sqs_queue).to receive(:send_message). - with('new test') +## Testing Middleware + +```ruby +# spec/middleware/logging_middleware_spec.rb +RSpec.describe LoggingMiddleware do + let(:worker) { MyWorker.new } + let(:queue) { 'default' } + let(:sqs_msg) { double(message_id: 'test-123') } + let(:body) { { 'data' => 'test' } } - subject.perform(sqs_msg, body) + it 'logs before and after processing' do + expect(Rails.logger).to receive(:info).with(/Starting/) + expect(Rails.logger).to receive(:info).with(/Completed/) + + described_class.new.call(worker, queue, sqs_msg, body) do + # Worker perform would run here end end + + it 'logs errors' do + expect(Rails.logger).to receive(:error).with(/Failed/) + + expect { + described_class.new.call(worker, queue, sqs_msg, body) do + raise 'Test error' + end + }.to raise_error('Test error') + end end -``` \ No newline at end of file +``` + +--- + +## CI Configuration + +### GitHub Actions + +```yaml +# .github/workflows/test.yml +name: Tests + +on: [push, pull_request] + +jobs: + test: + runs-on: ubuntu-latest + + services: + localstack: + image: localstack/localstack + ports: + - 4566:4566 + env: + SERVICES: sqs + + steps: + - uses: actions/checkout@v4 + - uses: ruby/setup-ruby@v1 + with: + bundler-cache: true + + - name: Create SQS queues + run: | + aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name test-queue + env: + AWS_ACCESS_KEY_ID: test + AWS_SECRET_ACCESS_KEY: test + AWS_DEFAULT_REGION: us-east-1 + + - name: Run tests + run: bundle exec rspec + env: + SQS_ENDPOINT: http://localhost:4566 +``` + +--- + +## Best Practices + +1. **Isolate worker logic** - Keep perform methods simple, extract complex logic to services +2. **Use factories** - Create realistic test data with FactoryBot or fixtures +3. **Test edge cases** - Empty bodies, invalid data, network errors +4. **Mock external services** - Use VCR or WebMock for HTTP calls +5. **Test idempotency** - Ensure workers handle duplicate messages correctly + +--- + +## Related + +- [[Shoryuken Inline adapter]] - Synchronous execution +- [[Using a local mock SQS server]] - LocalStack setup +- [[CurrentAttributes]] - Testing with CurrentAttributes +- [[ActiveJob Continuations]] - Testing job interruption diff --git "a/Too-many-\"receive_message(...)\"-in-the-log-file?.md" "b/Too-many-\"receive_message(...)\"-in-the-log-file?.md" new file mode 100644 index 0000000..501f84a --- /dev/null +++ "b/Too-many-\"receive_message(...)\"-in-the-log-file?.md" @@ -0,0 +1,15 @@ +``` +[Aws::SQS::Client 200 0.452184 0 retries] receive_message(max_number_of_messages:1,message_attribute_names:["All"],attribute_names:["All"],queue_url:"https://sqs.us-east-1.amazonaws.com/000011112222/my-queue-Queue-K1AB9G02OU8S") +``` + +Those messages are not because of Shoryuken, it's most likely you or another gem (see #353) enabled `AWS.config(log_level: :debug)`. + + +To get rid of them, try to set a specific SQS Client for Shoryuken: + +```ruby +Shoryuken.configure_server do |config| + # https://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Client.html#initialize-instance_method + config.sqs_client = Aws::SQS::Client.new(log_level: :info) +end +``` \ No newline at end of file diff --git a/Troubleshooting.md b/Troubleshooting.md new file mode 100644 index 0000000..705a32b --- /dev/null +++ b/Troubleshooting.md @@ -0,0 +1,393 @@ +# Troubleshooting + +Common issues and solutions when running Shoryuken. + +## Worker Issues + +### "No worker found for queue" + +**Cause:** Shoryuken can't find a worker class registered for the queue. + +**Solutions:** + +1. **For Rails with ActiveJob:** Use the `-R` flag: + +```bash +bundle exec shoryuken -R -C config/shoryuken.yml +``` + +2. **For plain Ruby:** Use `-r` to require your workers: + +```bash +bundle exec shoryuken -r ./app/workers -C config/shoryuken.yml +``` + +3. **Ensure queue names match:** The queue in `shoryuken.yml` must match the worker's queue: + +```yaml +# config/shoryuken.yml +queues: + - my_queue +``` + +```ruby +class MyWorker + include Shoryuken::Worker + shoryuken_options queue: 'my_queue' # Must match +end +``` + +4. **Check Zeitwerk autoloading:** With Rails 7+, ensure workers are in `app/workers` or `app/jobs`. + +--- + +### Workers Not Processing Messages + +**Check these in order:** + +1. **Is Shoryuken running?** +```bash +ps aux | grep shoryuken +``` + +2. **Are there messages in the queue?** +```bash +aws sqs get-queue-attributes \ + --queue-url https://sqs.us-east-1.amazonaws.com/123456789/myqueue \ + --attribute-names ApproximateNumberOfMessagesVisible +``` + +3. **Is the worker registered?** +```bash +bundle exec shoryuken -R -C config/shoryuken.yml +# Look for "Registered workers" in startup output +``` + +4. **Check logs:** +```bash +tail -f log/shoryuken.log +``` + +--- + +### Jobs Running Multiple Times + +**Causes:** +- Visibility timeout too short +- Job taking longer than visibility timeout +- Not deleting messages + +**Solutions:** + +1. **Increase visibility timeout** on the queue (AWS Console or CLI) + +2. **Enable auto_visibility_timeout** for long jobs: +```ruby +shoryuken_options auto_visibility_timeout: true +``` + +3. **Ensure auto_delete is enabled:** +```ruby +shoryuken_options auto_delete: true +``` + +4. **Make jobs idempotent** - design jobs to handle duplicate execution safely + +--- + +## AWS/SQS Issues + +### AWS Credentials Not Found + +``` +Aws::Errors::MissingCredentialsError +``` + +**Solutions:** + +1. Set environment variables: +```bash +export AWS_ACCESS_KEY_ID=xxx +export AWS_SECRET_ACCESS_KEY=xxx +export AWS_REGION=us-east-1 +``` + +2. Configure `~/.aws/credentials` + +3. Use IAM role (EC2/ECS/EKS) + +See [[Configure the AWS Client]] for details. + +--- + +### Access Denied + +``` +Aws::SQS::Errors::AccessDenied +``` + +**Solutions:** + +1. Check IAM policy includes required permissions: +```json +{ + "Action": [ + "sqs:ReceiveMessage", + "sqs:DeleteMessage", + "sqs:GetQueueAttributes", + "sqs:GetQueueUrl" + ] +} +``` + +2. Verify queue policy allows access + +3. Check resource ARN matches your queue + +See [[Amazon SQS IAM Policy for Shoryuken]]. + +--- + +### Queue Not Found + +``` +Aws::SQS::Errors::NonExistentQueue +``` + +**Solutions:** + +1. Verify queue exists: +```bash +aws sqs list-queues +``` + +2. Check queue name spelling in `shoryuken.yml` + +3. Verify region is correct + +4. Create the queue: +```bash +shoryuken sqs create my_queue +``` + +--- + +### Region Not Specified + +``` +Aws::Errors::MissingRegionError +``` + +**Solutions:** + +1. Set environment variable: +```bash +export AWS_REGION=us-east-1 +``` + +2. Configure in `~/.aws/config` + +3. Set in code: +```ruby +Aws.config[:region] = 'us-east-1' +``` + +--- + +## Logging Issues + +### Too Many "receive_message(...)" Log Messages + +``` +[Aws::SQS::Client 200 0.452184 0 retries] receive_message(...) +``` + +**Cause:** AWS SDK logging is set to debug level. + +**Solution:** Set a specific log level for the SQS client: + +```ruby +Shoryuken.configure_server do |config| + config.sqs_client = Aws::SQS::Client.new(log_level: :info) +end +``` + +--- + +### No Log Output + +**Solutions:** + +1. Check log file path: +```bash +bundle exec shoryuken -R -C config/shoryuken.yml -L log/shoryuken.log +``` + +2. Configure logger level: +```ruby +Shoryuken.configure_server do |config| + config.logger.level = Logger::DEBUG +end +``` + +3. For Rails, sync logger with Rails: +```ruby +Shoryuken.configure_server do |config| + Rails.logger = Shoryuken::Logging.logger +end +``` + +--- + +## Connection Issues + +### Database Connection Pool Exhausted + +``` +ActiveRecord::ConnectionTimeoutError: could not obtain a database connection +``` + +**Solution:** Increase pool size to match concurrency: + +```yaml +# config/database.yml +production: + pool: <%= ENV.fetch("SHORYUKEN_CONCURRENCY", 25) %> +``` + +Or reduce Shoryuken concurrency: + +```yaml +# config/shoryuken.yml +concurrency: 10 +``` + +--- + +### Redis Connection Issues (if using Redis) + +**Solution:** Configure connection pool: + +```ruby +Shoryuken.configure_server do |config| + config.on(:startup) do + Redis.current = Redis.new(url: ENV['REDIS_URL']) + end +end +``` + +--- + +## Shutdown Issues + +### Jobs Not Completing on Shutdown + +**Cause:** Timeout too short. + +**Solution:** Increase shutdown timeout: + +```yaml +# config/shoryuken.yml +timeout: 60 # seconds +``` + +Or for Kubernetes: + +```yaml +spec: + terminationGracePeriodSeconds: 300 +``` + +--- + +### Stuck on Shutdown + +**Diagnosis:** Send TTIN signal to get thread dump: + +```bash +kill -TTIN $(cat tmp/pids/shoryuken.pid) +``` + +Check logs for stuck threads. + +**Solution:** Use graceful shutdown signal: + +```bash +kill -USR1 $(cat tmp/pids/shoryuken.pid) +``` + +--- + +## LocalStack/Development Issues + +### Connection Refused to LocalStack + +``` +Aws::SQS::Errors::InternalServerError: Connection refused +``` + +**Solutions:** + +1. Verify LocalStack is running: +```bash +docker-compose ps +``` + +2. Check endpoint URL: +```ruby +Aws::SQS::Client.new(endpoint: 'http://localhost:4566') +``` + +3. Wait for LocalStack to be ready before starting Shoryuken + +--- + +### MD5 Checksum Errors with Mock SQS + +**Solution:** Disable checksum verification: + +```ruby +Aws::SQS::Client.new( + endpoint: 'http://localhost:4566', + verify_checksums: false +) +``` + +--- + +## Debug Mode + +Enable verbose logging: + +```ruby +Shoryuken.configure_server do |config| + config.logger.level = Logger::DEBUG +end +``` + +Or via environment: + +```bash +SHORYUKEN_LOG_LEVEL=debug bundle exec shoryuken -R -C config/shoryuken.yml +``` + +--- + +## Getting Help + +1. Check [GitHub Issues](https://github.com/phstc/shoryuken/issues) +2. Search existing issues before creating new ones +3. Include: + - Shoryuken version + - Ruby/Rails version + - Full error message and backtrace + - Relevant configuration + +--- + +## Related + +- [[Configure the AWS Client]] - AWS credential setup +- [[Amazon SQS IAM Policy for Shoryuken]] - IAM permissions +- [[Signals]] - Process control +- [[Logging]] - Log configuration diff --git a/Upgrade-Guide-7.0.md b/Upgrade-Guide-7.0.md new file mode 100644 index 0000000..feb2bfc --- /dev/null +++ b/Upgrade-Guide-7.0.md @@ -0,0 +1,155 @@ +# Upgrading to Shoryuken 7.0 + +This guide covers the changes required when upgrading from Shoryuken 6.x to 7.0. + +## Requirements + +Shoryuken 7.0 requires: + +- **Ruby 3.2+** (dropped support for Ruby 3.1 and earlier) +- **Rails 7.2+** (dropped support for Rails 7.0 and 7.1) +- **aws-sdk-sqs >= 1.66** + +If you cannot upgrade to these versions, remain on Shoryuken 6.x. + +## Breaking Changes + +### 1. Ruby and Rails Version Requirements + +Shoryuken 7.0 drops support for older Ruby and Rails versions: + +| Shoryuken | Ruby | Rails | +|-----------|------|-------| +| 7.0+ | 3.2, 3.3, 3.4 | 7.2, 8.0, 8.1 | +| 6.x | 2.7+ | 6.0+ | + +### 2. Removed concurrent-ruby Dependency + +Shoryuken 7.0 removes the `concurrent-ruby` dependency for core atomic operations. The gem now uses pure Ruby implementations: + +- `Concurrent::AtomicFixnum` → `Shoryuken::Helpers::AtomicCounter` +- `Concurrent::AtomicBoolean` → `Shoryuken::Helpers::AtomicBoolean` +- `Concurrent::Hash` → `Shoryuken::Helpers::AtomicHash` + +**Impact**: This change is transparent for most users. Your code should work without modification unless you were directly using these concurrent-ruby classes through Shoryuken's internals. + +**Note**: The `concurrent-ruby` gem is still used by `ShoryukenConcurrentSendAdapter` for `Concurrent::Promises`. + +### 3. Removed Core Extensions + +Shoryuken 7.0 removes monkey-patching of Ruby core classes. If your code relied on these extensions being globally available, you'll need to update it: + +**Before (6.x):** +```ruby +# Hash and String were monkey-patched with these methods +some_hash.deep_symbolize_keys +"SomeClass".constantize +``` + +**After (7.0):** +```ruby +# Use the helper modules directly +Shoryuken::Helpers::HashUtils.deep_symbolize_keys(some_hash) +Shoryuken::Helpers::StringUtils.constantize("SomeClass") +``` + +### 4. Zeitwerk Autoloading + +Shoryuken 7.0 uses Zeitwerk for autoloading. This change is mostly transparent but affects how polling strategy files are organized: + +- `Shoryuken::Polling::BaseStrategy` (was nested in polling.rb) +- `Shoryuken::Polling::QueueConfiguration` (was nested in polling.rb) + +**Impact**: If you have custom polling strategies that inherit from Shoryuken classes, ensure your `require` statements are updated. + +### 5. aws-sdk-sqs Version Requirement + +The minimum required version of `aws-sdk-sqs` is now 1.66. Update your Gemfile: + +```ruby +gem 'aws-sdk-sqs', '>= 1.66' +``` + +### 6. SendMessageBatch Size Limit + +The `SendMessageBatch` size limit has been increased from 256KB to 1MB, aligning with AWS SQS limits. This allows for larger batch operations. + +## New Features + +Shoryuken 7.0 introduces several new features. See the individual wiki pages for details: + +- [[ActiveJob Continuations]] - Graceful job interruption (Rails 8.1+) +- [[CurrentAttributes]] - Persist Rails CurrentAttributes across job execution +- [[Bulk Enqueuing]] - Efficient batch enqueuing with `perform_all_later` +- [[Health Checks]] - Health check API via `launcher.healthy?` +- New `utilization_update` lifecycle event + +## Upgrade Checklist + +1. **Update Ruby version** to 3.2 or later +2. **Update Rails version** to 7.2 or later +3. **Update Gemfile dependencies:** + ```ruby + gem 'shoryuken', '~> 7.0' + gem 'aws-sdk-sqs', '>= 1.66' + ``` +4. **Run `bundle update shoryuken aws-sdk-sqs`** +5. **Search for core extension usage** in your codebase: + - `deep_symbolize_keys` calls on strings/hashes that might rely on Shoryuken's patches + - `constantize` calls on strings +6. **Test your workers** in development/staging before deploying +7. **Review custom polling strategies** for compatibility with Zeitwerk autoloading + +## Step-by-Step Migration + +### For Rails Applications + +1. Update your `Gemfile`: + ```ruby + # Gemfile + gem 'shoryuken', '~> 7.0' + ``` + +2. Update Rails queue adapter configuration (if needed): + ```ruby + # config/application.rb + config.active_job.queue_adapter = :shoryuken + ``` + +3. Workers in `app/jobs` are auto-loaded via Zeitwerk - no changes needed. + +4. If using `ActiveJob::Base` directly, update to `ApplicationJob`: + ```ruby + # Before + class MyJob < ActiveJob::Base + end + + # After + class MyJob < ApplicationJob + end + ``` + +### For Standalone Shoryuken (Non-Rails) + +1. Update your `Gemfile`: + ```ruby + gem 'shoryuken', '~> 7.0' + ``` + +2. Ensure workers are required correctly. With Zeitwerk, files must match class names: + ``` + lib/ + my_worker.rb # defines MyWorker + ``` + +3. Start Shoryuken with your configuration: + ```bash + bundle exec shoryuken -r ./lib/my_worker.rb -C config/shoryuken.yml + ``` + +## Getting Help + +If you encounter issues during the upgrade: + +1. Check the [GitHub Issues](https://github.com/ruby-shoryuken/shoryuken/issues) +2. Review the full [CHANGELOG](https://github.com/ruby-shoryuken/shoryuken/blob/master/CHANGELOG.md) diff --git a/Using-a-local-mock-SQS-server.md b/Using-a-local-mock-SQS-server.md index 1c0b6c2..3b41cd0 100644 --- a/Using-a-local-mock-SQS-server.md +++ b/Using-a-local-mock-SQS-server.md @@ -1,67 +1,361 @@ -It is possible to configure Shoryuken to use a local mock SQS server implementation in place of Amazon's real SQS. This is useful, for example, when doing local development without an Internet connection. +# Using a Local Mock SQS Server -This guide will help you to configure one such mock SQS service implementation, [`moto`](https://github.com/spulec/moto). We'll set up `aws-sdk-ruby` and `shoryuken` to use `moto` instead of the real SQS provided by AWS. +Configure Shoryuken to use a local SQS mock for development and testing without connecting to AWS. -# Install and run `moto` stand alone server mode +## Options -[Install moto](https://github.com/spulec/moto#stand-alone-server-mode) with: +| Service | Type | Best For | +|---------|------|----------| +| **LocalStack** | Full AWS mock | CI/CD, comprehensive testing | +| **ElasticMQ** | SQS-only | Lightweight, fast | +| **Moto** | Python AWS mock | Python projects, simple setup | +--- + +## LocalStack (Recommended) + +LocalStack is the recommended option and is used in Shoryuken's own CI. + +### Docker Compose + +```yaml +# docker-compose.yml +version: '3.8' + +services: + localstack: + image: localstack/localstack:latest + ports: + - "4566:4566" + environment: + - SERVICES=sqs + - DEBUG=0 + - DOCKER_HOST=unix:///var/run/docker.sock + volumes: + - "./localstack-data:/var/lib/localstack" + - "/var/run/docker.sock:/var/run/docker.sock" ``` -pip install moto[server] + +Start LocalStack: + +```bash +docker-compose up -d localstack ``` -To run it: +### Create Queues + +```bash +# Using AWS CLI +aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name myapp-default +aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name myapp-critical +# Or using awslocal (LocalStack CLI) +awslocal sqs create-queue --queue-name myapp-default ``` -moto_server sqs -p 4576 -H localhost + +### Configure Shoryuken + +```ruby +# config/initializers/shoryuken.rb +if Rails.env.development? || Rails.env.test? + sqs_client = Aws::SQS::Client.new( + endpoint: ENV.fetch('SQS_ENDPOINT', 'http://localhost:4566'), + region: 'us-east-1', + access_key_id: 'test', + secret_access_key: 'test' + ) + + Shoryuken.configure_client do |config| + config.sqs_client = sqs_client + end + + Shoryuken.configure_server do |config| + config.sqs_client = sqs_client + end +end ``` -# Upgrade `aws-sdk` +### Startup Script -To use a local SQS server, you must be running `aws-sdk` version 2.2.15 or later. If your version of `aws-sdk` is older, update it. +```bash +#!/bin/bash +# scripts/setup_localstack.sh +echo "Waiting for LocalStack..." +until aws --endpoint-url=http://localhost:4566 sqs list-queues 2>/dev/null; do + sleep 1 +done + +echo "Creating queues..." +aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name myapp-default +aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name myapp-critical + +echo "Queues ready!" ``` -bundle update aws-sdk + +--- + +## ElasticMQ + +ElasticMQ is a lightweight, in-memory SQS-compatible server. + +### Docker Compose + +```yaml +# docker-compose.yml +version: '3.8' + +services: + elasticmq: + image: softwaremill/elasticmq-native:latest + ports: + - "9324:9324" + - "9325:9325" # UI + volumes: + - "./elasticmq.conf:/opt/elasticmq.conf" ``` -# Configure Shoryuken to use `moto` +### Configuration File -We'll need to change the `:sqs_endpoint` key of Shoryuken's AWS config to point to your local `moto` instance. We'll also have to disable MD5 checks from `send_message*` calls in the AWS SDK. +```hocon +# elasticmq.conf +include classpath("application.conf") -## Configure Client +node-address { + protocol = http + host = localhost + port = 9324 + context-path = "" +} + +rest-sqs { + enabled = true + bind-port = 9324 + bind-hostname = "0.0.0.0" +} + +queues { + myapp-default { + defaultVisibilityTimeout = 30 seconds + delay = 0 seconds + receiveMessageWait = 0 seconds + } + myapp-critical { + defaultVisibilityTimeout = 30 seconds + delay = 0 seconds + receiveMessageWait = 0 seconds + } +} +``` + +### Configure Shoryuken ```ruby -Shoryuken.configure_client do |config| - config.sqs_client = Aws::SQS::Client.new( - region: ENV["AWS_REGION"], - access_key_id: ENV["AWS_ACCESS_KEY_ID"], - secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"], - endpoint: 'http://localhost:4576', - verify_checksums: false +# config/initializers/shoryuken.rb +if Rails.env.development? + sqs_client = Aws::SQS::Client.new( + endpoint: 'http://localhost:9324', + region: 'us-east-1', + access_key_id: 'test', + secret_access_key: 'test' ) + + Shoryuken.configure_client do |config| + config.sqs_client = sqs_client + end + + Shoryuken.configure_server do |config| + config.sqs_client = sqs_client + end end ``` -## Configure Server +--- + +## Moto + +Moto is a Python-based AWS mock library with standalone server mode. + +### Install and Run + +```bash +# Install +pip install moto[server] + +# Run +moto_server sqs -p 4576 -H localhost +``` + +### Docker + +```yaml +# docker-compose.yml +services: + moto: + image: motoserver/moto:latest + ports: + - "4576:4576" + command: ["-p", "4576"] +``` + +### Configure Shoryuken ```ruby -Shoryuken.configure_server do |config| - config.sqs_client = Aws::SQS::Client.new( - region: ENV["AWS_REGION"], - access_key_id: ENV["AWS_ACCESS_KEY_ID"], - secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"], +# config/initializers/shoryuken.rb +if Rails.env.development? + sqs_client = Aws::SQS::Client.new( endpoint: 'http://localhost:4576', + region: 'us-east-1', + access_key_id: 'test', + secret_access_key: 'test', verify_checksums: false ) + + Shoryuken.configure_client do |config| + config.sqs_client = sqs_client + end + + Shoryuken.configure_server do |config| + config.sqs_client = sqs_client + end end ``` -## Running SQS commands locally +--- -If you want to use `shoryuken sqs` command with a different endpoint you can pass an `--endpoint` option to the command, example: +## Using Shoryuken CLI + +### With Environment Variable + +```bash +export SHORYUKEN_SQS_ENDPOINT=http://localhost:4566 +shoryuken sqs ls +shoryuken sqs create myapp-default +``` + +### With CLI Option + +```bash +shoryuken sqs ls --endpoint=http://localhost:4566 +shoryuken sqs create myapp-default --endpoint=http://localhost:4566 +``` + +--- + +## Development Workflow + +### Procfile.dev + +``` +web: bundle exec rails server +worker: bundle exec shoryuken -R -C config/shoryuken.yml +localstack: docker-compose up localstack +``` + +### Start Everything + +```bash +# Using Foreman +foreman start -f Procfile.dev + +# Or separately +docker-compose up -d localstack +bundle exec shoryuken -R -C config/shoryuken.yml +``` + +--- + +## Troubleshooting + +### Connection Refused + +``` +Aws::SQS::Errors::InternalServerError: Connection refused +``` + +**Solution:** Ensure LocalStack/ElasticMQ is running: ```bash -shoryuken sqs ls --endpoint=http://localhost:4576 +docker-compose ps +docker-compose logs localstack ``` -if you don't want to pass the `--endpoint` option to all `sqs` commands you can set an env var for it in the application `SHORYUKEN_SQS_ENDPOINT=http://localhost:4576`. \ No newline at end of file +### Queue Not Found + +``` +Aws::SQS::Errors::NonExistentQueue +``` + +**Solution:** Create queues before starting workers: + +```bash +aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name myapp-default +``` + +### MD5 Checksum Errors + +Some mock servers don't compute MD5 correctly. Disable verification: + +```ruby +Aws::SQS::Client.new( + endpoint: 'http://localhost:4566', + verify_checksums: false +) +``` + +--- + +## CI Integration + +### GitHub Actions + +```yaml +# .github/workflows/test.yml +services: + localstack: + image: localstack/localstack + ports: + - 4566:4566 + env: + SERVICES: sqs + +steps: + - name: Setup queues + run: | + aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name test-queue + env: + AWS_ACCESS_KEY_ID: test + AWS_SECRET_ACCESS_KEY: test + AWS_DEFAULT_REGION: us-east-1 + + - name: Run tests + run: bundle exec rspec + env: + SQS_ENDPOINT: http://localhost:4566 +``` + +### GitLab CI + +```yaml +# .gitlab-ci.yml +test: + services: + - name: localstack/localstack + alias: localstack + variables: + AWS_ACCESS_KEY_ID: test + AWS_SECRET_ACCESS_KEY: test + AWS_DEFAULT_REGION: us-east-1 + SQS_ENDPOINT: http://localstack:4566 + script: + - aws --endpoint-url=$SQS_ENDPOINT sqs create-queue --queue-name test-queue + - bundle exec rspec +``` + +--- + +## Related + +- [[Testing]] - Testing strategies +- [[Configure the AWS Client]] - AWS client configuration +- [[Getting Started]] - Initial setup diff --git a/Worker-options.md b/Worker-options.md index 37f4bdb..8030e2d 100644 --- a/Worker-options.md +++ b/Worker-options.md @@ -1,112 +1,357 @@ -*Note:* `shoryuken_options` is only available in Shoryuken workers. It isn't supported in [Active Job jobs](https://github.com/phstc/shoryuken/wiki/Rails-Integration-Active-Job). +# Worker Options -### queue +Configure Shoryuken workers using `shoryuken_options`. These options are only available for native Shoryuken workers, not ActiveJob jobs (see [[Rails Integration Active Job]]). -Default: `default` +## Options Summary -The `queue` option associates a queue with a worker. +| Option | Default | Description | +|--------|---------|-------------| +| `queue` | `'default'` | Queue name(s) to process | +| `auto_delete` | `false` | Delete message after successful processing | +| `batch` | `false` | Receive messages in batches | +| `body_parser` | `:text` | Parse message body before `perform` | +| `auto_visibility_timeout` | `false` | Auto-extend visibility timeout | +| `retry_intervals` | `nil` | Exponential backoff intervals | + +--- + +## queue + +Associates a worker with one or more queues. + +### Single Queue ```ruby class HelloWorker include Shoryuken::Worker - + shoryuken_options queue: 'hello' + + def perform(sqs_msg, body) + puts body + end end ``` -You can also pass a block to define the queue: +### Dynamic Queue Name + +Use a lambda for environment-specific queues: ```ruby -shoryuken_options queue: ->{ "#{ENV['RAILS_ENV']}-hello" } -shoryuken_options queue: ->{ "#{Socket.gethostname}-hello" } +class HelloWorker + include Shoryuken::Worker + + shoryuken_options queue: -> { "#{Rails.env}_hello" } + + def perform(sqs_msg, body) + # ... + end +end ``` -Or an array to associate multiple queues to a single worker: +### Multiple Queues + +One worker can process messages from multiple queues: ```ruby -shoryuken_options queue: %w[queue1 queue2 queue3] +class MultiQueueWorker + include Shoryuken::Worker + + shoryuken_options queue: %w[queue1 queue2 queue3] + + def perform(sqs_msg, body) + # ... + end +end ``` -### auto_delete +--- -Default: `false` +## auto_delete -When it's enabled, Shoryuken auto deletes messages after their consumption, but only in case, the worker doesn't raise any exception during the consumption. +**Default:** `false` -As `auto_delete` is `false` by default, remember to set it to `true` or call `sqs_msg.delete` before ending the `perform` method, otherwise, the messages will get back to the queue, becoming available to be consumed again. +Automatically deletes messages after successful processing. -*Note:* `auto_delete` is `true` by default when using Active Job. +```ruby +class MyWorker + include Shoryuken::Worker -### batch + shoryuken_options queue: 'default', auto_delete: true -Default: `false` + def perform(sqs_msg, body) + # Message is deleted after this method returns without error + process(body) + end +end +``` + +### Manual Deletion -When `batch` is `true`, Shoryuken sends the messages in batches instead of individually. +When `auto_delete: false`, you must delete messages manually: -One of the advantages of `batch` is when you use APIs that accept batch, for example [Keen IO](http://keen.io). +```ruby +class MyWorker + include Shoryuken::Worker + shoryuken_options queue: 'default', auto_delete: false + + def perform(sqs_msg, body) + if process(body) + sqs_msg.delete # Explicit deletion + end + # If not deleted, message returns to queue after visibility timeout + end +end +``` + +**Note:** ActiveJob jobs have `auto_delete: true` by default. + +--- + +## batch + +**Default:** `false` + +Receive up to 10 messages at once for batch processing. ```ruby -class KeenIOWorker +class BatchWorker include Shoryuken::Worker - shoryuken_options queue: 'keen-io', auto_delete: true, batch: true, body_parser: :json + shoryuken_options queue: 'events', batch: true, auto_delete: true - def perform(sqs_msgs, events) - Keen.publish_batch('stats' => events) + def perform(sqs_msgs, bodies) + # sqs_msgs and bodies are arrays + bodies.each do |body| + process(body) + end end end ``` -If you are using a custom middleware, check [this article](https://github.com/phstc/shoryuken/wiki/Middleware#be-careful-with-batchable-workers), the `sqs_msg` and `body` are arrays when `batch=true`. +### Use Cases + +- Bulk database inserts +- Batch API calls +- High-throughput, low-latency jobs + +### Considerations -Another important observation regarding batchable workers is if one of your messages causes an exception, consequently `auto_delete` won't be executed for any message. +- If any message raises an exception, `auto_delete` won't run for any message +- Middleware receives arrays for `sqs_msg` and `body` parameters +- Not compatible with `auto_visibility_timeout` or `retry_intervals` -### body_parser +--- -Default: `:text` +## body_parser -The `body_parser` allows you to parse the body before calling the `perform` method. It accepts: `:json`, `:text`, a block or a class that responds to `.parse`. +**Default:** `:text` + +Parse message body before calling `perform`. + +### Built-in Parsers ```ruby +# JSON parsing shoryuken_options body_parser: :json -shoryuken_options body_parser: ->(sqs_msg){ REXML::Document.new(sqs_msg.body) } +# Raw text (default) +shoryuken_options body_parser: :text +``` + +### Custom Parser (Lambda) + +```ruby +shoryuken_options body_parser: ->(sqs_msg) { + REXML::Document.new(sqs_msg.body) +} +``` + +### Custom Parser (Class) + +Any class that responds to `.parse`: +```ruby shoryuken_options body_parser: JSON +shoryuken_options body_parser: Oj +shoryuken_options body_parser: MyCustomParser ``` -### auto_visibility_timeout +### Example -Default: `false` +```ruby +class JsonWorker + include Shoryuken::Worker -Although I strongly recommend setting the [visibility timeout](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html) to a nice long value, sometimes we can't control it. So, for these cases, you can enable the `auto_visibility_timeout` by setting its value to `true`. + shoryuken_options queue: 'json_queue', body_parser: :json -When it's enabled, 5 seconds before the default visibility timeout expires, Shoryuken will reset it to its original value again. + def perform(sqs_msg, body) + # body is already a Hash/Array + puts body['name'] + end +end +``` -> Be generous while configuring the default visibility_timeout for a queue. If your worker in the worst case takes 2 minutes to consume a message, set the visibility_timeout to at least 4 minutes. It doesn't hurt and it will be better than having the same message being consumed more than the expected. -> http://www.pablocantero.com/blog/2014/11/29/sqs-to-the-rescue/ +--- -*Note:* This feature isn't supported when using [`batch=true`](https://github.com/phstc/shoryuken/wiki/Worker-options#batch). +## auto_visibility_timeout +**Default:** `false` -### retry_intervals +Automatically extends visibility timeout during long-running jobs. -Default: `nil` +```ruby +class LongRunningWorker + include Shoryuken::Worker + + shoryuken_options queue: 'long_jobs', auto_visibility_timeout: true + + def perform(sqs_msg, body) + # Visibility timeout is extended automatically + long_running_operation # Can take longer than queue's visibility timeout + end +end +``` -If a worker raises an exception while consuming a message, and does not delete the message, [this message will be available to be consumed again after its visibility timeout expiration](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html). +### How It Works -But if you want to increase or decrease the next time a failing message will be available to be consumed again, you can use `retry_intervals` to implement an [exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff). +5 seconds before the visibility timeout expires, Shoryuken resets it to the original value. This repeats until the job completes. -*Note:* This feature isn't supported when using [`batch=true`](https://github.com/phstc/shoryuken/wiki/Worker-options#batch). +### Best Practices + +Prefer setting a generous visibility timeout on the queue instead: + +> If your worker in the worst case takes 2 minutes to consume a message, set the visibility_timeout to at least 4 minutes. + +**Not supported** with `batch: true`. + +--- + +## retry_intervals + +**Default:** `nil` + +Implement exponential backoff for failed jobs. + +### Array of Intervals + +```ruby +class RetryWorker + include Shoryuken::Worker + + shoryuken_options queue: 'default', + retry_intervals: [60, 300, 1800, 3600] + # 1 min, 5 min, 30 min, 1 hour + + def perform(sqs_msg, body) + # On failure, next retry after interval based on attempt count + unreliable_operation + end +end +``` + +### Dynamic Intervals (Lambda) + +```ruby +shoryuken_options retry_intervals: ->(attempts) { + (attempts ** 2) * 60 # Exponential: 60, 240, 540, 960... +} +``` + +### How It Works + +When a job fails: +1. Shoryuken changes the message's visibility timeout +2. Message becomes visible after the interval +3. Retry count is tracked via message attributes + +### Limits + +- Maximum visibility timeout: **12 hours** (SQS limit) +- Intervals exceeding 12 hours are capped automatically + +**Not supported** with `batch: true`. + +--- + +## Per-Worker Middleware + +Add middleware specific to a worker: + +```ruby +class MyWorker + include Shoryuken::Worker + + shoryuken_options queue: 'default' + + server_middleware do |chain| + chain.add WorkerSpecificMiddleware + end + + def perform(sqs_msg, body) + # ... + end +end +``` + +See [[Middleware]] for more details. + +--- + +## Complete Example + +```ruby +class OrderProcessor + include Shoryuken::Worker + + shoryuken_options( + queue: -> { "#{Rails.env}_orders" }, + auto_delete: true, + body_parser: :json, + retry_intervals: [60, 300, 900, 3600] + ) + + server_middleware do |chain| + chain.add MetricsMiddleware + end + + def perform(sqs_msg, body) + order = Order.find(body['order_id']) + order.process! + rescue ActiveRecord::RecordNotFound + # Don't retry - order doesn't exist + Rails.logger.warn "Order #{body['order_id']} not found" + end +end +``` + +--- + +## ActiveJob Equivalent + +For ActiveJob jobs, configure via job class: ```ruby -shoryuken_options retry_intervals: [300, 1200, 3600] # 5.minutes, 20.minutes and 1.hour -shoryuken_options retry_intervals: ->(attempts) { calculate_next_attempt_interval(attempts) } +class MyJob < ApplicationJob + queue_as :default + + # Retry with exponential backoff (built into ActiveJob) + retry_on StandardError, wait: :polynomially_longer, attempts: 5 + + def perform(args) + # ... + end +end ``` -Keep in mind that Amazon SQS does not officially support exponential backoff, it's something implemented in Shoryuken using the [visibility Timeout](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html), which can be extended to up to 12 hours. If your interval exceeds the 12 hour maximum, Shoryuken will automatically cap it to the maximum allowed. +See [[Rails Integration Active Job]] for ActiveJob-specific options. + +--- + +## Related -> https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ChangeMessageVisibility.html -> -> You can continue to call ChangeMessageVisibility to extend the visibility timeout to a maximum of 12 hours. If you try to extend beyond 12 hours, the request will be rejected. \ No newline at end of file +- [[Shoryuken options]] - Global configuration +- [[Middleware]] - Middleware chain +- [[Rails Integration Active Job]] - ActiveJob integration +- [[Retrying a message]] - Retry strategies