Skip to content

Restart server sometimes doesn't pass for missed files #91

Description

@andreleblanc11

I've seen the same errors below on RHEL8 systems and on U22.04 systems (including from GitHub actions).

test 1 FAILURE: compare contents of downloaded_by_sub_amqp and downloaded_by_sub_cp had 14 differences
test 2 success: compare contents of downloaded_by_sub_cp and downloaded_by_sub_rabbitmqtt are the same
test 3 success: compare contents of downloaded_by_sub_rabbitmqtt and downloaded_by_sub_u are the same
test 4 success: compare contents of downloaded_by_sub_u and posted_by_shim are the same
test 5 success: compare contents of posted_by_shim and sent_by_tsource2send are the same
test 6 success: compare contents of cfile and cfr are the same
test 7 FAILURE: compare contents of downloaded_by_sub_amqp and recd_by_srpoll_test1 had 14 differences
broker state:
                 | dd.weather routing |
test  8 success: expected 2242 number of extended attributes in source tree 2270
test  9 success: post    count of posted files (1436) should be same those in the static data directory  (1435)
test 10 success: post    count of rejected files (10) should be same those in the static data directory  (10)
test 11 success: post    (1436) t_dd1 should have the same number of items as t_dd2      (1436)
test 12 success: sarra   (1436) should have the same number of items as one post         (1436)
test 13 success: sarra   (1436) should winnow the same number of items as one post       (1436)
test 14 success: subscribe amqp_f30      (1436) should have the same number of items as sarra            (1436)
                 | watch      routing |
test 15 success: watch                   (1418) should be the same as subscribe amqp_f30                  (1436)
test 16 success: sender                  (1418) should have the same number of items as watch  (1418)
test 17 success: rabbitmqtt              (1418) should have the same number of items as watch  (1418)
test 18 success: subscribe u_sftp_f60    (1418) should have the same number of items as sender (1418)
test 19 success: subscribe cp_f61        (1418) should have the same number of items as sender (1418)
                 | poll       routing |
 poll sftp_f62 posted 1393   sftp_f63 posted 24   total posted: 1417   # of duplicates posted: 0
test 20 success: poll sftp_f62+3         (1417) should have the same number of items of sender   (1418)
test 21 success: poll sftp_f62+3         (1417) should see the same number of items as poll sftp_f62 posted      (1417)
test 22 success: subscribe q_f71         (1417) should have the same number of items as poll sftp_f62+3 (1417)
                 | flow_post  routing |
test 23 success: post test2_f61          (1118) should have the same number of files of sender   (1118)
test 24 success: subscribe ftp_f70       (1118) should have the same number of items as post test2_f61 (1118)
test 25 success: post test2_f61          (1118) should post about the same number of files as shim_f63   (1118)
test 26 success: post test2_f61          (1118) should post about the same number of links as shim_f63   (1118)
test 27 success: static tree             (310) directories should be posted twice: for 1st copy and linked_dir by shim_f63       (592)
                 | py infos   routing |
test 28 success: 0 messages received that we don't know what happened.
                 | C          routing |
test 29 success: cpost both pelles should post the same number of messages (1437) (1437)
test 30 success: cpost both pelles should see same amount of post_rate_limit messages (285) (285)
test 31 success: cpost pelle04 should post 5 times the number of post_rate_limit messages (1437) (285)
test 32 success: cdnld_f21 subscribe downloaded (1125) the same number of files that was published by both van_14 and van_15 (1125)
test 33 success: veille_f34 should post as many files (1125) as subscribe cdnld_f21 downloaded (1125)
test 34 success: veille_f34 should post as many files (1125) as subscribe cfile_f44 downloaded (1125)
test 35 success: 0 there should be no unacknowledged messages left, but there are 0
test 36 success: 0 there should be no messages ready to be consumed but there are 0
test 37 success: there should be 1 process in wVip state
test 38 FAILURE: Overall restart_server 35 of 37 passed (sample size: 1435) !

There are some files that are not getting picked up from the watch that are being posted from the sarra component.

grep after_post.*SABZ24_SBCT_050300___51458 sarra_download_f20_01.log
2026-07-08 11:50:03,828 [INFO] sarracenia.flowcb.log after_post posted to amqp://tfeed@localhost/,xflow_public,v03.post.20200105.WXO-DD.bulletins.alphanumeric.20200105.SA.SBCT.03 a file with baseUrl: http://localhost:8001/ relPath: 20200105/WXO-DD/bulletins/alphanumeric/20200105/SA/SBCT/03/SABZ24_SBCT_050300___51458 size: 71 id: +2ZOr

grep SABZ24_SBCT_050300___51458 watch_f40_01.log.2026-07-08
2026-07-08 11:50:04,922 [DEBUG] sarracenia.flowcb.gather.file on_add create /home/sarra/sarra_devdocroot/downloaded_by_sub_amqp/20200105/WXO-DD/bulletins/alphanumeric/20200105/SA/SBCT/03/SABZ24_SBCT_050300___51458 None
2026-07-08 11:50:04,922 [DEBUG] sarracenia.flowcb.gather.file on_add modify /home/sarra/sarra_devdocroot/downloaded_by_sub_amqp/20200105/WXO-DD/bulletins/alphanumeric/20200105/SA/SBCT/03/SABZ24_SBCT_050300___51458 None
2026-07-08 11:50:04,925 [DEBUG] sarracenia.flowcb.gather.file on_add modify /home/sarra/sarra_devdocroot/downloaded_by_sub_amqp/20200105/WXO-DD/bulletins/alphanumeric/20200105/SA/SBCT/03/SABZ24_SBCT_050300___51458 None
2026-07-08 11:50:04,925 [DEBUG] sarracenia.flowcb.gather.file on_add modify /home/sarra/sarra_devdocroot/downloaded_by_sub_amqp/20200105/WXO-DD/bulletins/alphanumeric/20200105/SA/SBCT/03/SABZ24_SBCT_050300___51458 None

diff downloaded_by_sub_amqp.txt downloaded_by_sub_cp.txt
...
1006d999
< e27faf9fb3034c66f83d26faf149303e  ./20200105/WXO-DD/bulletins/alphanumeric/20200105/SA/SBCT/03/SABZ24_SBCT_050300___51458

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions