Upload very big files

[gquintana]

At the moment, when uploading a file, the file content is generated in the heap memory prior to the Put Object request:
https://github.com/jenshadlich/S3-Performance-Test/blob/master/src/main/java/de/jeha/s3pt/operations/Upload.java#L39

This limits the file size to the maximum heap size (-Xmx) divided by the number of threads. With -Xmx1G and 4 threads, I can not send files bigger than 256M without generating an OOME.

2 solutions are possible, and both seem interesting to me:

1. Generate file content as it is uploaded. It should be possible to implement an InputStream which randomly generates bytes as they are requested.
2. Generate file content as a local temporary file. Even if the time taken to write the local file is a pure loss, it would allow S3 client to use a "send file" system call which is more efficient (no Java heap required).