`_isdir()` does not put entry in the dircache

[aabmass]

Thanks for the awesome project! I'm running into an issue where gcsfs is making many unexpected GET requests when I try to upload to a directory within a GCS bucket using put(). I believe the issue can be reproduced by just calling GcsFileSystem.put("somefile.txt", "gs://bucket/subdir/somefile.txt") with a subdirectory in a bucket. The network logs look like

2025-09-23 04:40:25,909 - gcsfs - DEBUG - _call -- GET: b/{}/o/{}, ('my-fake-bucket', 'v2/9bc1e04f-7b17-4da6-a32f-3d460e1e275c_outputs.json'), None
2025-09-23 04:40:26,067 - gcsfs - DEBUG - _call -- GET: b/{}/o, ('my-fake-bucket',), None
2025-09-23 04:40:26,213 - gcsfs - DEBUG - _call -- GET: b/{}/o/{}, ('my-fake-bucket', 'v2/9bc1e04f-7b17-4da6-a32f-3d460e1e275c_outputs.json'), None
2025-09-23 04:40:26,431 - gcsfs - DEBUG - _call -- POST: https://storage.googleapis.com/upload/storage/v1/b/my-fake-bucket/o, (), {'Content-Type': 'multipart/related; boundary="==0=="'}

_put() -> _isdir() -> _info() which calls GcsFileSystem._ls_from_cache() several times (1, 2) with cache misses. I did some debugging and dircache is remaining empty even though I'm doing this in a loop. My actual code is using simplecache and looks something like this:

for file in files:
  with fsspec.open(f"simplecache::gs://my-fake-bucket/v2/{file}", "w+") as file:
    json.dump(some_obj, file)

(which is calling GcsFileSystem.put() under the hood here)

[martindurant]

The reason for this, is that isdir() lists the target path with a maximum return length of 1 item; i.e., either nothing comes back (exists) or nothing (does not exist). That means, that we don't have a full listing to put in the cache.

So, in the case that isdir is the only operation you want, we get a faster response (assuming many files below that path). But if your next operation is to list the directory, having determined it exists, you end up making two calls. Of course, one could argue that the isdir operation is completely redundant in this case.

One possible remediation would be to keep track of directories known to exist separately, which would complicate the implementation.

[aabmass]

Understood thanks for the reply! Maybe my question was too specific, I don't really care about isdir() necessarily and subdirectories don't seem to matter either. The issue is really with put() making these GET requests even if lpath is a file. I included a minimal script that reproduces the problem (it's a little contrived I mentioned the real use case above).

Should I be making a different call beforehand to pre-fill the cache?

Script

# /// script
# requires-python = ">=3.11"
# dependencies = [
#     "fsspec",
#     "gcsfs",
# ]
# ///


import logging
import posixpath
import tempfile

import gcsfs

logging.basicConfig(level=logging.DEBUG)


def main() -> None:
    fs = gcsfs.GCSFileSystem()

    # Write two files into a bucket subdirectory
    for _ in range(2):
        with tempfile.NamedTemporaryFile("w") as file:
            file.write("hello world")
            file.flush()
            file.seek(0)

            fs.put(
                file.name,
                f"gs://some-example-bucket/{posixpath.basename(file.name)}",
            )


if __name__ == "__main__":
    main()

Log output

DEBUG:asyncio:Using selector: EpollSelector
DEBUG:google.auth._default:Checking None for explicit credentials as part of auth process...
DEBUG:google.auth._default:Checking Cloud SDK credentials as part of auth process...
DEBUG:gcsfs.credentials:Connected with method google_default
DEBUG:gcsfs:GET: b/{}/o/{}, ('some-example-bucket', 'tmpyz8xz1pf'), None
DEBUG:gcsfs.credentials:GCS refresh
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG:urllib3.connectionpool:https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
DEBUG:gcsfs:GET: b/{}/o, ('some-example-bucket',), None
DEBUG:gcsfs:GET: b/{}/o/{}, ('some-example-bucket', 'tmpyz8xz1pf'), None
DEBUG:gcsfs:POST: https://storage.googleapis.com/upload/storage/v1/b/some-example-bucket/o, (), {'Content-Type': 'multipart/related; boundary="==0=="'}
DEBUG:gcsfs:GET: b/{}/o/{}, ('some-example-bucket', 'tmpeqp7fkrd'), None
DEBUG:gcsfs:GET: b/{}/o, ('some-example-bucket',), None
DEBUG:gcsfs:GET: b/{}/o/{}, ('some-example-bucket', 'tmpeqp7fkrd'), None
DEBUG:gcsfs:POST: https://storage.googleapis.com/upload/storage/v1/b/some-example-bucket/o, (), {'Content-Type': 'multipart/related; boundary="==0=="'}

[aabmass]

For anyone hitting this, fsspec/filesystem_spec#1920 fixed this if you're using a cached FS chained over GCS like simplecache::gs://....

If you're calling put()/_put() directly and hitting this, you can workaround it by passing the files as a list put([a], [b]) to avoid path expansion.

Otherwise, the dircache solution described in this bug would be nice, but I would understand if it's infeasible based on Martin's comment.