TLS Socket leak when making a lot of requests using Proxy URL

[BenjD90]

Hello,

I'm having an issue using the library, most of the requests work fine, but over the time I have a leak of socket keeping open :

Those are two metrics provided from prom-client :

• https://github.com/siimon/prom-client/blob/c1d76c5d497ef803f6bd90c56c713c3fa811c3e0/lib/metrics/processHandles.js#L27
• https://github.com/siimon/prom-client/blob/c1d76c5d497ef803f6bd90c56c713c3fa811c3e0/lib/metrics/processResources.js#L24

The code used to get this result :

• Using p-queue, make 200 K requests (100 concurrently) using new proxy for each request

Does anyone got a leak like this one before ?
I tried to set all the timeouts on got, and to destroy manually the request stream and response stream, but that didn't change anything 😕

[barjin]

Hello @BenjD90, and thank you for your interest in this project!

Could you provide us with an example script that runs those requests? Feel free to omit the proxy / target URLs or replace them with a localhost server (if those are not important for the reproduction).

Also, note that we're currently sunsetting got-scraping in favor of impit, our new HTTP client for Node.JS and Python (see impit on npm). This client allows you to use HTTP proxies and impersonated headers / TLS settings directly from your Node.JS script. Feel free to give it a try and share your experience in the issues tab there.

[BenjD90]

Thank you @barjin for your answer 😄

For now the scraping project is in pause, but I'll try the https://github.com/apify/impit/ project next time 😃 (in a few weeks I hope).