bolt crashes with SystemStackError

[vchepkov]

Describe the Bug

This problem exists in puppetlabs/bolt since version 4, so presumably openbolt has inherited the issue.
bolt v3 works as expected.

I execute patching plan from github - https://github.com/EncoreTechnologies/puppet-patching

plan home::patching (
  TargetSpec $targets = 'all',
) {
  run_plan('patching',
    targets                => $targets,
    filter_offline_targets => true,
  )
}

bolt crashes once it determines that reboot is required, never performing the reboot:

$ bolt plan run home::patching -t virtual
Starting: plan home::patching
Starting: plan patching
Starting: plan patching::check_puppet
Starting: task puppet_agent::version on rhel10.chepkov.lan, puppet9.chepkov.lan, fedora.chepkov.lan, foreman.chepkov.lan, holder.chepkov.lan
Finished: task puppet_agent::version with 0 failures in 3.41 sec
Removing the following offline targets: []
Starting: plan patching::puppet_facts
Starting: task patching::puppet_facts on rhel10.chepkov.lan, puppet9.chepkov.lan, fedora.chepkov.lan, foreman.chepkov.lan, holder.chepkov.lan
Finished: task patching::puppet_facts with 0 failures in 17.14 sec
Finished: plan patching::puppet_facts in 17.17 sec
Finished: plan patching::check_puppet in 20.61 sec
Starting: plan patching::ordered_groups
Groups = [1]
Group '1' targets = [rhel10.chepkov.lan, puppet9.chepkov.lan, fedora.chepkov.lan, foreman.chepkov.lan, holder.chepkov.lan]
Finished: plan patching::ordered_groups in 0.02 sec
Starting: task patching::cache_update on rhel10.chepkov.lan, puppet9.chepkov.lan, fedora.chepkov.lan, foreman.chepkov.lan, holder.chepkov.lan
Finished: task patching::cache_update with 0 failures in 3.97 sec
Starting: plan patching::available_updates
Starting: task patching::available_updates on rhel10.chepkov.lan, puppet9.chepkov.lan, fedora.chepkov.lan, foreman.chepkov.lan, holder.chepkov.lan
Finished: task patching::available_updates with 0 failures in 8.54 sec
Host update status: ('+' has available update; '-' no update) [num updates]
 - rhel10.chepkov.lan [0]
 + puppet9.chepkov.lan [17]
 + fedora.chepkov.lan [16]
 + foreman.chepkov.lan [24]
 + holder.chepkov.lan [24]
Finished: plan patching::available_updates in 8.57 sec
Starting: plan patching::pre_update
Starting: plan patching::get_targets
Finished: plan patching::get_targets in 0.01 sec
Starting: plan patching::pre_post_update
pre_post_update - noop = false, update_phase = pre, task = patching::pre_update
Starting: plan patching::get_targets
Finished: plan patching::get_targets in 0.01 sec
Starting: task patching::pre_update on puppet9.chepkov.lan, fedora.chepkov.lan, foreman.chepkov.lan, holder.chepkov.lan
Finished: task patching::pre_update with 0 failures in 22.17 sec
Finished: plan patching::pre_post_update in 22.26 sec
Finished: plan patching::pre_update in 22.29 sec
Starting: task patching::update on puppet9.chepkov.lan, fedora.chepkov.lan, foreman.chepkov.lan, holder.chepkov.lan
Finished: task patching::update with 0 failures in 98.22 sec
Starting: plan patching::post_update
Starting: plan patching::get_targets
Finished: plan patching::get_targets in 0.01 sec
Starting: plan patching::pre_post_update
pre_post_update - noop = false, update_phase = post, task = patching::post_update
Starting: plan patching::get_targets
Finished: plan patching::get_targets in 0.01 sec
Starting: task patching::post_update on puppet9.chepkov.lan, fedora.chepkov.lan, foreman.chepkov.lan, holder.chepkov.lan
Finished: task patching::post_update with 0 failures in 206.3 sec
Finished: plan patching::pre_post_update in 3 min, 26 sec
Finished: plan patching::post_update in 3 min, 26 sec
Starting: plan patching::reboot_required
Starting: plan patching::get_targets
Finished: plan patching::get_targets in 0.01 sec
Starting: task patching::reboot_required on puppet9.chepkov.lan, fedora.chepkov.lan, foreman.chepkov.lan, holder.chepkov.lan
Finished: task patching::reboot_required with 0 failures in 4.92 sec
Reboot strategy: only_required
Host reboot required status: ('+' reboot required; '-' reboot NOT required)
 + puppet9.chepkov.lan
 + fedora.chepkov.lan
 + foreman.chepkov.lan
 + holder.chepkov.lan
Finished: plan patching::reboot_required in 4.96 sec
Finished: plan patching in 6 min, 5 sec
Finished: plan home::patching in 6 min, 5 sec
/opt/puppetlabs/bolt/lib/ruby/gems/3.2.0/gems/openvox-8.24.2/lib/puppet/pops/model/factory.rb:78:in `each_pair': stack level too deep (SystemStackError)
	from /opt/puppetlabs/bolt/lib/ruby/gems/3.2.0/gems/openvox-8.24.2/lib/puppet/pops/model/factory.rb:78:in `create_model'
	from /opt/puppetlabs/bolt/lib/ruby/gems/3.2.0/gems/openvox-8.24.2/lib/puppet/pops/model/factory.rb:1151:in `contained_current'
	from /opt/puppetlabs/bolt/lib/ruby/gems/3.2.0/gems/openvox-8.24.2/lib/puppet/pops/model/factory.rb:1138:in `block in factory_to_model'
	from /opt/puppetlabs/bolt/lib/ruby/gems/3.2.0/gems/openvox-8.24.2/lib/puppet/pops/model/factory.rb:1138:in `each'
	from /opt/puppetlabs/bolt/lib/ruby/gems/3.2.0/gems/openvox-8.24.2/lib/puppet/pops/model/factory.rb:1138:in `each_with_index'
	from /opt/puppetlabs/bolt/lib/ruby/gems/3.2.0/gems/openvox-8.24.2/lib/puppet/pops/model/factory.rb:1138:in `factory_to_model'
	from /opt/puppetlabs/bolt/lib/ruby/gems/3.2.0/gems/openvox-8.24.2/lib/puppet/pops/model/factory.rb:78:in `block in create_model'
	from /opt/puppetlabs/bolt/lib/ruby/gems/3.2.0/gems/openvox-8.24.2/lib/puppet/pops/model/factory.rb:78:in `each_pair'
	 ... 369 levels...
	from /opt/puppetlabs/bolt/lib/ruby/gems/3.2.0/gems/openvox-8.24.2/lib/puppet/pops/evaluator/runtime3_support.rb:286:in `external_call_function'
	from /opt/puppetlabs/bolt/lib/ruby/gems/3.2.0/gems/openvox-8.24.2/lib/puppet/pal/compiler.rb:30:in `call_function'
	from /opt/puppetlabs/bolt/lib/ruby/gems/3.2.0/gems/openbolt-5.3.0/lib/bolt/pal.rb:766:in `block (2 levels) in run_plan'
	from /opt/puppetlabs/bolt/lib/ruby/gems/3.2.0/gems/openbolt-5.3.0/lib/bolt/fiber_executor.rb:46:in `block in create_future'


$ bolt --version
5.3.0

[jay7x]

I suspect it might be somehow related to puppetlabs/bolt#3373 (but had no real code check).

P.S. After closer look at the trace output, I don't think those issues are related though.. Just consequences are the same.

[nmburgan]

Sorry, I haven't had time to look at this yet, but just wanted you to know it's on our radar! I started looking at it and then promptly forgot to write down my findings 😅 But yeah, seems like a regression in Bolt 4 that carried over.