Skip to content

fix(utils): download tokenizer models locally via HTTP to avoid TF GCS C++ segfault on macOS#661

Open
prince-shakyaa wants to merge 1 commit into
google-deepmind:mainfrom
prince-shakyaa:fix/macos-tokenizer-cache-download
Open

fix(utils): download tokenizer models locally via HTTP to avoid TF GCS C++ segfault on macOS#661
prince-shakyaa wants to merge 1 commit into
google-deepmind:mainfrom
prince-shakyaa:fix/macos-tokenizer-cache-download

Conversation

@prince-shakyaa
Copy link
Copy Markdown

@prince-shakyaa prince-shakyaa commented May 23, 2026

Description

Fixes #660
Fixes an issue where gemma fails to download/cache gs:// tokenizer paths efficiently, which triggers the TensorFlow C++ GCS client and results in a libc++abi: terminating due to uncaught exception of type std::__1::system_error: mutex lock failed crash on macOS Apple Silicon at Python interpreter exit.

By explicitly intercepting gs:// paths and downloading them via standard HTTP (urllib.request) directly into the local ~/.gemma/tokenizer/ cache, we completely bypass the C++ GCS client bug.

Additionally, this adds true local caching for tokenizer models (previously the code was checking the cache, but never actually saving to it if the file was missing, leading to the full model being fetched over the network on every run).

Changes

  • Modified gemma/gm/utils/_file_cache.py::maybe_get_from_cache to translate gs:// paths to https://storage.googleapis.com/ and download them natively to the cache directory before returning the local path.

@prince-shakyaa
Copy link
Copy Markdown
Author

prince-shakyaa commented May 23, 2026

While testing locally on macOS Apple Silicon, pytest workers were repeatedly segfaulting during teardown. I traced this to the TensorFlow C++ GCS client failing to shut down cleanly when loading the tokenizer directly from gs://.
This PR fixes the crash (and adds true local caching) by downloading the model via standard HTTP into ~/.gemma/tokenizer/ before loading it.

Let me know if you need me to make any changes.
Thank You.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

macOS Apple Silicon: C++ Segfault during test shutdown due to TF GCS client (tokenizer caching)

1 participant