fix and improve: billion scale image search

[abhishekkrthakur]

billion image

I confirm that this contribution is made under the terms of the license found in the root directory of this repository's source tree and that I have the authority necessary to make this contribution on behalf of its copyright owner.

[thomasht86]

great job on fixing this! Running it as we speak.
The .parquet file is big. Not sure if this is suitable to run in CI, both due to time spent downloading, and size of github runners.
Could we maybe create our own (sampled down) .parquet file that we can host in our S3 with eg. 100k vectors?
Would make it more reasonable to run in decent time.
If you create it, I can show you/do the upload...?

[kkraune]

most excellent! we can test this once a week or so. an alternative to hosting ourselves can be to download and just extract the subset in the python code?

[thomasht86]

The file is 14.8GB, so won't be able to run it in default github action runner at all unfortunately.

[thomasht86]

But it seems we can stream the parquet file directly with pyarrow instead of downloading it first though

[abhishekkrthakur]

it might be slower. we are dealing with millions of rows. so storing on disk should be preferred. we are streaming rows in code though in batches of 50k. nevertheless, if you want me to make that change, please let me know

[abhishekkrthakur]

for ci we can also use a sample by the way

[thomasht86]

Yes. If we use a sample it would solve both ci and making it more user friendly imho.

[abhishekkrthakur]

the ci failure doesnt seem to be coming from this pr

[kkraune]

right, I fixed tue build in a separate pr