32 errors returns a success error code

[schneems]

Expected

If my test suite says it had 32 errors, it will fail

Actual

$ bundle exec rspec-queue --max-requeues=3 --timeout 180 --queue redis://localhost:6379/0 --format documentation --build lol --worker lol; echo Done $?

...

While loading ./spec/unit/bash_functions_spec.rb a `raise SyntaxError` occurred, RSpec will now quit.
Failure/Error: require 'language_pack'

SyntaxError:
  --> /Users/rschneeman/Documents/projects/work/heroku-buildpack-ruby/lib/language_pack.rb
  expected a newline or semicolon after the statementcannot parse the expressionexpected an expression after `,`unexpected write target
     8  module LanguagePack
    22    def self.detect(app_path:, cache_path:, gemfile_lock: )
    27      if pack_klass
    28        pack_klass.new(
  > 29          app_path: app_path,
  > 30          cache_path: cache_path
  > 31          gemfile_lock: gemfile_lock
    32        )
    33      else
    36    end
    37  end

  /Users/rschneeman/Documents/projects/work/heroku-buildpack-ruby/lib/language_pack.rb:31: syntax error, unexpected local variable or method, expecting ')' (SyntaxError)
          gemfile_lock: gemfile_lock
          ^~~~~~~~~~~~
# ./spec/spec_helper.rb:7:in `<top (required)>'
# ./spec/unit/bash_functions_spec.rb:1:in `<top (required)>'
# /Users/rschneeman/.gem/ruby/3.3.8/gems/ci-queue-0.67.0/lib/rspec/queue.rb:361:in `setup'

Finished in 0.00064 seconds (files took 0.28631 seconds to load)
0 examples, 0 failures, 32 errors occurred outside of examples

Done 0

Repro

I pushed my code here https://github.com/heroku/heroku-buildpack-ruby/tree/schneems/ci-queue-repro-2025-may-22. There's a syntax error in it so all tests should fail:

$ git clone https://github.com/heroku/heroku-buildpack-ruby
$ cd heroku-buildpack-ruby
$ git checkout ci-queue-repro-2025-may-22
$ bundle exec rspec-queue --max-requeues=3 --timeout 180 --queue redis://localhost:6379/0 --format documentation --build lol --worker lol; echo Done $?

[schneems]

Not exactly related (root cause) but the failure mode is similar in that and status returns success on errors: #292.

I'm wondering of there's some sort of a more robust "assume something failed unless we get overwhelming positive confirmation from rspec" design.

[casperisfine]

As mentioned last year in #292

This behavior is actually by design. ci-queue is designed so that all workers are expendable and soft failing except for actual test failures. This include not being able to reach Redis.

As such, the reporter step isn't optional, it's 100% required as it's the only thing that ensure all queued tests were ran.

We'd probably take a PR to add a flag to change that behavior, but since you are using the rspec integration that's no longer supported, you might have better luck with one of the alternatives like https://github.com/skroutz/rspecq

[schneems]

Heard. I'm moving to parallel_split_test as my tests have a ton of IO and don't really need all the CPU. Also, the way Heroku CI works, the redis instance is provisioned on-demand and sometimes takes longer to boot than the tests take to run. Net I'm looking at faster times. More opening for visibility for others facing something similar and to raise that this could still possibly happen to non-rspec users (I think).

I understand the desire to have partition tolerance (CAP), I also feel that consistency of result in a CI is really important. It might be that the entire shopify suite has never failed all at once and reported green, but if it happens that code is likely going to be shipped. For not-rspec, I'm wondering if it makes sense to have some sort of sentinal value or double-book-keping in Redis where a test that knows it completed successfully waits to assert that tests were run and that not all nodes ran zero tests.