I found there is some room for improvements in the log handling.
- Some pipeline logs are gigantic, literally.
This makes our searching slow. I propose a solution for the above to be a log rotation or break up (e.g. per period). This could greatly improve the debugging by pre-filtering old stuff away. For example, a file structure like: logs/ACORN/yesterday, logs/ACORN/today, logs/ACORN/last_month or even something simple like logs.ACORN.2021 would already greatly improve searching.
- Some log outputs are not formatted. This is because the log is capturing both stderr/stdout from the harvesters, for example. In other words, some errors are not formatted with the uuid of a file, or the dates at the start of the line. This makes cutting and dicing the logs quite inefficient. For example, for me to actually cut off the part of the log that pertains to a particular file, including the unformatted output, I need to first:
a. search the first occurrence of the uuid
b. search the last occurrence of the uuid
c. cut the lines from a. to b.
Although a & b can be done in one operation, another read of the file is required by step (c). If all the logs are indeed formatted correctly, just searching for the uuid is enough. Note that the very big logs just turn this particular problem worse.
- The error msgs of emails and logs could improve.
It is clear that, for some people, the error msgs that goes into the email (or in the pipeline) is not enough, since they are not used to the pipeline. Maybe a good idea is to add an interpretation or more human friendly to the error msgs so people can follow up on the issues. For example, see aodn/content#483.
I found there is some room for improvements in the log handling.
This makes our searching slow. I propose a solution for the above to be a log rotation or break up (e.g. per period). This could greatly improve the debugging by pre-filtering old stuff away. For example, a file structure like:
logs/ACORN/yesterday, logs/ACORN/today, logs/ACORN/last_monthor even something simple likelogs.ACORN.2021would already greatly improve searching.a. search the first occurrence of the uuid
b. search the last occurrence of the uuid
c. cut the lines from a. to b.
Although a & b can be done in one operation, another read of the file is required by step (c). If all the logs are indeed formatted correctly, just searching for the uuid is enough. Note that the very big logs just turn this particular problem worse.
It is clear that, for some people, the error msgs that goes into the email (or in the pipeline) is not enough, since they are not used to the pipeline. Maybe a good idea is to add an interpretation or more human friendly to the error msgs so people can follow up on the issues. For example, see aodn/content#483.