Directory listing caching

[ericvh]

We currently do not cache directory listings, add a notion of caching directory listings to the baseline.

[rickynils]

@ericvh May I ask if any progress has been made on the directory caching in v9fs? My use case is sharing read-only, immutable stores over 9P, where directory caching would really improve performance without any real downsides.

[ericvh]

I had started on an implementation but got derailed by some of the rework I did to prep for doing better caching across the board and ran out of the time I had allocated to this. Better directory caching remains one of the biggest gaps in our cache strategy. That being said -- if you are in a read-only, immutable store you should be able to use loose mode which should get you most of the benefit since it shouldn't be reissuing getattrs for the directory contents even though it will still request the file list which doesn't currently get cached. -eric

…

On Thu, Oct 3, 2024 at 7:55 AM Rickard Nilsson ***@***.***> wrote: @ericvh <https://github.com/ericvh> May I ask if any progress has been made on the directory caching in v9fs? My use case is sharing read-only, immutable stores over 9P, where directory caching would really improve performance without any real downsides. — Reply to this email directly, view it on GitHub <#37 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAQ7WELKWGBZXR23CVE5FTZZU5CTAVCNFSM6AAAAABHAJILUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJRGM2TANZQHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[rickynils]

That being said -- if you are in a read-only, immutable store you should be able to use loose mode which should get you most of the benefit since it shouldn't be reissuing getattrs for the directory contents even though it will still request the file list which doesn't currently get cached.

Yeah, stat calls etc are nicely cached, and with cache=fscache file contents are also cached. But if you have any kind of latency at all (even low latency like a 9P server running on localhost), the uncached directory listing kills many use cases, like grep --recursive. I will try to do some tracing to back up my claim that it is actually the directory listing that is the problem, but I'm pretty certain from simple timings.

[ericvh]

I can believe it still hurts for sure -- there may be a shortcut to try to loose cache directory contents (I was working towards proper cache). I'll see if I can get some time to look at it but I've been pretty busy elsewhere these days -- but it might be nice if we could come up with something for the next merge window.

…

On Thu, Oct 3, 2024 at 9:35 AM Rickard Nilsson ***@***.***> wrote: That being said -- if you are in a read-only, immutable store you should be able to use loose mode which should get you most of the benefit since it shouldn't be reissuing getattrs for the directory contents even though it will still request the file list which doesn't currently get cached. Yeah, stat calls etc are nicely cached, and with cache=fscache file contents are also cached. But if you have any kind of latency at all (even low latency like a 9P server running on localhost), the uncached directory listing kills many use cases, like grep --recursive. I will try to do some tracing to back up my claim that it is actually the directory listing that is the problem, but I'm pretty certain from simple timings. — Reply to this email directly, view it on GitHub <#37 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAQ7WFOTGQ4MODASC7TIRDZZVI3RAVCNFSM6AAAAABHAJILUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJRGU4DMNBRGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[rickynils]

@ericvh Loose caching of directory contents sounds promising. Should you find time to work on that, I'd be happy to try out any patches.