Revive long-running nodes (mainnet and calibnet)

[LesnyRumcajs]

Issue summary

Forest had some hiccups recently, likely due to the new-ish chain follower implementation. See ChainSafe/forest#6042 as an example.

There is already code for deployment of such nodes (to be reviewed, some time has passed since it was last updated).

Still, an essential part is missing, being monitoring & alerting. There was supposed to be a follow up after refactoring this piece, but it never happened (for various reasons).

What needs to be done:

• Review the current deployment logic. Perhaps it can be simplified with new Forest features. Is there anything else missing? For example, it'd be great to create coredumps on panics.
• Decide on the monitoring & alerting solution. We already have some code in git history for integrating with New Relic, but perhaps it's better to use ChainSafe's Grafana, if possible.
• Implement monitoring & alerting based on the previous point. Discuss healthchecks with the team. Set up relevant alerts (so that the team is informed on real failures; we want to avoid false positives). Node's getting stuck, crash, OOM - likely a good candidate for such alerts.
• Decide on the schedule for handling the alerts. Polevik might be a good person to put this duty on.

Other information and links