Unreliable retrievability of partial datasets

[benbierens]

Testing has revealed some unreliability where it shouldn't be, with regards to partial datasets and erasure-coding.

Test:

• Start 5 host nodes each with availability to fill a single slot.
• Start 1 client node.
• DO NOT start a validator. (We don't want SlotFreed events or repair for this test.)
• Create a storage contract. Wait until it has started.
• Stop the client node.

The storage contract was created with 4 slots and 2 tolerance.
This means from any 2 slots, we should be able to recover the original data.

As parameterized input to this test, we provide two distinct slotIndex variables that cover every combination of slotIndexes.

• Use chain events to find which hosts have filled the selected slotIndexes.
• Stop the selected hosts.

At this point half of the erasure-coded dataset exists in the network, but the full original data should be retrievable.

• Start a new node.
• Use the new node to retrieve the original dataset (using the verifiable-manifest CID, such that erasure-coding information and indexing information is available.)

This should always work. Testing reveals it doesn't always work.

From this we can see that if both slotIndex 0 and slotIndex 1 are removed, retrieval is likely to fail.
The test allows download to be retried once. Original dataset size: 32MB.

If you have any insights into what might be causing this problem: Please comment below!

[benbierens]

I've been running this test some more. It seems that the slotIndex combinations (0,2) and (0,3) never fail. It also seems that the probability of failure for the other combinations isn't as skewed as the screenshot suggests. These tests take a while so the sample size isn't that big yet. Still, there might be a clue here.

[benbierens]

I suspect there's an issue here:
erasure.nim:197 proc prepareDecodingData
At line 226:

  for fut in pendingBlocksIter:
    # Continue to receive blocks until we have just enough for decoding
    # or no more blocks can arrive
    if resolved >= encoded.ecK:
      break

    let (blkOrErr, idx) = await fut
    without blk =? blkOrErr, err:
      trace "Failed retrieving a block", idx, treeCid = encoded.treeCid, msg = err.msg
      continue

This looks like: We wait for each block in the iterator.
If we have enough blocks, we break out of the loop and start repair.
But we expect some futures to fail because the blocks cannot be downloaded. In those cases, we TRACE log "Failed retrieving a block" and move on to the next future.

In the test described above, we would expect to see plenty of "Failed retrieving a block" log messages, since half of the dataset is lost. But this message doesn't show up at all. (Maybe once in >50 test runs.)
Perhaps the block future is taking too long to fail and causing the download to time out.