changes feed goes in infinite loop

[benoitc]

The changes feed goes from time to time in an infinite loop using the since parameter with the last sequence when one node is missing.

How to reproduce:

Using the 3 dev modes, get changes using this code:

DbName = <<"testdb">>,
Since = <<"last sequence of the feed">>,

fabric:changes(DbName, fun
    ({change, Change}, Acc) ->
        twig:log(info, "got change ~p", [Change]),
        {ok, Acc};
    ({stop, EndSeq}, _Acc) ->
        twig:log(info, "got last seq ~p", [EndSeq]),
        {ok, Acc};

    (timeout, Acc) ->
        twig:log(info, "got timeout", []),
        {ok, Acc};
    (Other, Acc) ->
        twig:log(info, "other msg", [Other]),
        {ok, Acc} 
    end,
    undefined,
    #changes_args{
        include_docs = true,
        feed = "normal",
        since = Since,
        filter = main_only,
        timeout = infinity,
        db_open_options = []
        }
    )

Then retry from last seq with the first node closed. It will goes in an infinite loop getting a timeout.

[bdionne]

@benoitc, this example doesn't actually work with Acc undefined like that, but anyway I think I see the problem is here[1]. receive results it apears goes into an infinite loop

[1] https://github.com/cloudant/fabric/blob/master/src/fabric_view_changes.erl#L134

[benoitc]

yes, i dived until that, not sure how to go further though :)I tried to only use the online shards but without success.

Also the code in mem3_rep_manager is quite similar, so I assume it is subject to the same issue. https://github.com/cloudant/mem3/blob/master/src/mem3_rep_manager.erl#L230

[kocolosk]

I believe the infinite loop bug will be addressed by @bdionne's work in #7. rexi includes a feature that allows for out-of-band monitoring of remote processes, but we don't leverage it as often as we ought to in fabric. Monitoring the rexi_server on the down remote node will cause a rexi_DOWN message to be sent to the coordinator. In this case I believe the system will error out when it encounters that message for the last active copy of a shard (and requests with a since parameter only activate one copy of each shard).