Rank retains previous value across checkpoints

[upperwal]

Rank (or any other variable in user in user space [stack]) will retain its value across checkpointing.

Ex:

MPI_Comm_rank(MPI_COMM_WORLD, &rank);

...
...

printf("rank: %d\n", rank);

Could print rank in the previous job before checkpointing

So to counter this problem either the replication map should be retained [all update bit set to 1] or values from MPI_Comm_rank should be consumed ASAP or recalled whenever required.