don't cache prefix-filtered listings as complete directory entries

[bruno-hays]

Problem

When glob("dir/train-*") is called, fsspec extracts train- as a stem prefix and calls _find(path, prefix="train-", maxdepth=1) for server-side filtering. Inside _find's maxdepth+prefix branch, _lsdir runs a delimiter-based S3 listing filtered to that stem, but then unconditionally writes the partial result into dircache[path] as if it were a complete directory listing.
A subsequent glob("dir/test-*") or ls("dir") hits this stale cache and misses every file that didn't match the first stem, returning empty results even though the files exist on S3.
The same poisoning happens without glob:

fs.find("bucket/data", prefix="train-", maxdepth=1)
fs.ls("bucket/data")  # returns only train-* files

Only s3fs has this defect. gcsfs already guards its equivalent write with if not prefix; adlfs never writes dircache from _find at all.

Fix

Add a partial=False keyword to _lsdir. When True, the dircache[path] = files write is skipped. _find's maxdepth+prefix branch is the only call site that passes partial=True. All other callers keep partial=False and continue to populate dircache normally. A pre-existing full cache entry is returned unchanged via the existing if path not in self.dircache guard.

Linked to fsspec:2054

[martindurant]

Why is prefix alone not enough to indicate whether the listing is partial or not?

[bruno-hays]

I think you are right. The only issue was that _lsdir overwrites the prefix argument with the key-derived directory prefix (prefix = key.lstrip("/") + "/" + prefix) before the cache-write, at which point it's non-empty for every non-root path (e.g. a plain ls("bucket/data") has prefix == "data/"). Gating on it there would disable caching for all subdirectories. Capturing the caller's prefix up front (partial = bool(prefix)) avoids that and lets us drop the extra kwarg.
I have updated the code accordingly.