result not saved on S3

[dancaspi]

Hey,
this is a great piece of code, but I seem to have a small issue:
The code ends successfully but the result appears to be removed from the designated S3 bucket. If I intentionally fail it (by passing filename with the "/tmp" prefix) the code creates the CSVs in S3 successfully but then fails to save the file locally on lambda.

1. I think the saving of data in the local lambda should be configurable. I can see a very common use case where the size of the CSVs are larger than what permitted on AWS Lambda local storage. if you are doing unloading you don't necessarily need a lambda-local copy of the CSVs

2. It seems i don't understand something, why is there a "delete" section:

[DEBUG] 2020-03-21T09:42:10.807Z dc09597d-e58f-1014-79e1-ecad5dce458f Remove all objects in S3

[DEBUG] 2020-03-21T09:42:10.808Z dc09597d-e58f-1014-79e1-ecad5dce458f Remove 6 object(s) from S3

[DEBUG] 2020-03-21T09:42:11.172Z dc09597d-e58f-1014-79e1-ecad5dce458f Remove temporary directory in local

how about adding those in a configurable params to unload?

Dan