Support nested iteration

[bdewater]

Alternate take on #295 - unlike the suggested fix in that PR, we retain the existing "start after" cursor logic this gem uses (see eg array and CSVEnumerator):

• cursor: nil yields the first item
• cursor: 0 yields the second item
• cursor: 1 yields the third item
• etcetera

The main difference in code:

     def each(&block)
       return to_enum unless block_given?
 
-      iterate([], [], 0, &block)
+      iterate([], 0, &block)
     end
 
     private
 
-    def iterate(current_items, current_cursor, index, &block)
+    def iterate(current_items, index, &block)
       cursor = @cursor[index]
       enum = @enums[index].call(*current_items, cursor)
 
       enum.each do |item, cursor_value|
         if index == @cursor.size - 1
-          yield item, current_cursor + [cursor_value]
+          yield item, @cursor
         else
-          iterate(current_items + [item], current_cursor + [cursor_value], index + 1, &block)
+          iterate(current_items + [item], index + 1, &block)
+          @cursor[index + 1] = nil
         end
+        @cursor[index] = cursor_value
       end
     end
   end

cc @fatkodima
closes #295

[fatkodima]

Thanks for working on this! I tested locally all the edge cases, seems like it is working correctly 👍.

I haven't grasped your suggestion initially, because with these changes current cursor != [parent_id, child_id] (so "need to do the mapping in the head") and it still can reprocess previous records (I initially wanted to avoid that).

[bdewater]

@Mangara anything I can do to help get this over the finish line?

[Mangara]

The code LGTM. Just one question and a test failure.

Thank you @fatkodima for starting this, and @bdewater for finishing it ❤️

[Mangara]

Why is this necessary?

[bdewater]

To be honest I'm not entirely sure why this is happening, but the added test in test/unit/iteration_test.rb fails otherwise: the notification payload is 12 times {:job_class=>"JobIterationTest::JobWithNestedEnumerator", :cursor_position=>[2, nil]} instead of the expected increasing nested cursor 🤔

[bdewater]

It seems to hang on to @cursors yielded from the nested enumerator.

irb(#<JobIterationTest:0x00000001066631c0>):003:0> events.map { |e| e[:cursor_position].object_id }
=> [243980, 243980, 243980, 243980, 243980, 243980, 243980, 243980, 243980, 243980, 243980, 243980]

cursor_from_enumerator.dup instead of the tags object works too, I'll push that up in a sec.