Add retention time to keep data during the last X hours.

[CatherineF-dev]

Add retention time to keep data during the last X hours.

[leonzz]

The outdated files are deleted when diskqueue reaches size limit. in other words, they stay and will still be read as long as total size is within limit. In this case there seems not much improvement on setting total size limit alone.

I think it makes more sense (if a retention time is set) that expired files would not even be read and be deleted as soon as they expire. so that total size limit and the time limit both serves to reduce space usage and which ever comes first would take effect.

[kev1n80]

+1 I was wondering if you could check retention size:

• within the ioloop inside the select using a time.NewTicker() similar to syncTicker. The ticker will start if it is null and a log file exists, for the duration of retentionTime - (currentTime - oldestFileCreateTime). When the ticker goes off it will iterate starting from the oldest file and delete outdated files.
• before reading a log from a file in readOne()

This way. as Leon mentioned above, the retention size and retention time both serve to reduce space usage and whichever occurs first would take effect.

[leonzz]

If we want to use this now and go with the current approach I would suggest either:

• use a name other than retentionTime, because it's only enforcing retention time in the size limit exceeding case;
• or, keep the retentionTime name but add more comments/descriptions and a TODO as a reminder to implement the other half (enforcing retention time under the size limit).