include size of processed corpus in README

[erikfredner]

I think it would be good to include the current size of the corpus (as of date x) on the README.

When I started processing, I wasn't sure how much hard drive space I would need.

FWIW, I downloaded and processed the corpus today, and my gutenberg directory is 76G

[micimize]

@erikfredner do you mean the entire data directory? When I run it I get about 100GB total, but each subdir is maybe ~25GB

[erikfredner]

Yes, I would think include info about total size and maybe the sizes of subdirs too.

I just ran python get_data.py and here are the various directory sizes given by du -hs *:

 89G	data
 70M	metadata
 16M	src
 36K	LICENSE
8.0K	process_data.py
8.0K	get_data.py
4.0K	requirements.txt
4.0K	README.md

And for the data directory:

 25G	raw
 23G	text
 21G	tokens
4.3G	counts

[d-kleine]

The size will increase over the time new resources are added to the pg
I think it would also be good to have a progress bar displayed (e.g., with from tqdm import tqdm)