memory limit to a parsl worker in HPCExecutor

[minxu74]

Description

Checklist

Please confirm that this pull request has done the following:

• Tests added
• Documentation added (where applicable)
• Changelog item added to changelog/

[codecov]

Codecov Report

❌ Patch coverage is 84.61538% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ckages/climate-ref/src/climate_ref/executor/hpc.py	84.61%	2 Missing ⚠️

Flag	Coverage Δ
core	93.20% <84.61%> (-0.03%)	⬇️
providers	89.65% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...ckages/climate-ref/src/climate_ref/executor/hpc.py	63.46% <84.61%> (+1.02%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[minxu74]

@lewisjared It is ready for your review. I changed the hard-coded memory limit to a dynamic one using a environment variable MEMORY_LIMIT_PARSL_JOB_GB. The memory constraint was implemented by setting the soft threshold of RLIMIT_AS resource. It is not very robust, but the test run on NERSC has showed its feasibility to avoid OOM errors.

[lewisjared]

Nice. This with_memory_limit function might be useful for the celery workers as well. Currently the resource limits are applied at the kubernetes layer

Does it OOM when it hits the soft limit?

[minxu74]

@lewisjared No. When a parsl worker hits the soft limit, a MemoryError will be thrown out for this worker, but the other parsl workers won't be affected and are still running. The memory constraint will keep the total memory used by all parsl workers in a node under its total physical memory size, thus it could prevent the OOM error in a node. The OOM in any nodes of a HPC job will kill all workers across multiple nodes and make the entire job quit.