PermissionError on Windows when a directory in the path contains non-ASCII characters (netCDF4 1.7.4 / libnetcdf 4.9.3)

[FBumann]

Hi, I found this bug using netcdf4. I hope its a helpful issue.

Note

AI assisted issue

Summary

On Windows, Dataset(path, "w") fails with PermissionError: [Errno 13] Permission denied when a directory component of the path contains non-ASCII characters (e.g. German umlauts äöü, Chinese characters). A non-ASCII filename in an ASCII directory works fine — only non-ASCII directories fail.

This still reproduces on the current release (netCDF4 1.7.4, libnetcdf 4.9.3), i.e. well after the Windows UTF-8 work in libnetcdf 4.7.4 (#1668) and 4.9.0 (#2222). It looks like the same problem reported in #1220 (closed, against the much older 1.4.0), #686, and #941 — reopening visibility against a current version.

Minimal reproduction

from pathlib import Path
from netCDF4 import Dataset

def make(path: Path):
    path.parent.mkdir(parents=True, exist_ok=True)
    with Dataset(str(path), "w") as ds:
        ds.createDimension("x", 1)
        ds.createVariable("value", "f4", ("x",))[:] = [42]

base = Path.home() / "nc_test"

make(base / "ascii"      / "test.nc")          # OK
make(base / "umlaut_äöü.nc")                    # OK  (non-ASCII *filename*)
make(base / "umlaut_äöü" / "test.nc")          # FAIL: PermissionError [Errno 13]  (non-ASCII *directory*)

Observed

PermissionError: [Errno 13] Permission denied: 'C:\\...\\umlaut_äöü\\test.nc'

(The directory exists and is writable — plain open(path, "wb") in the same directory succeeds, so it is not an actual OS permission problem; the path appears to be re-encoded incorrectly before being handed to the C library.)

Expected

The file is created in the non-ASCII directory, as it already is for a non-ASCII filename.

Isolation / extra data points

• Non-ASCII filename in ASCII dir → works.
• Non-ASCII directory → fails (PermissionError on create; FileNotFoundError on open).
• Verified on a GitHub Actions windows-latest runner as well as a local Windows 10/11 machine.
• The h5netcdf engine (h5py → HDF5 wide-char API) reads and writes the exact same non-ASCII directories without error, which points at the netCDF4/libnetcdf path-string encoding on Windows rather than HDF5 or the filesystem.
• macOS and Linux are unaffected (UTF-8 paths round-trip fine).

Environment

• OS: Windows 10/11 (and GitHub windows-latest)
• Python: 3.12
• netCDF4: 1.7.4
• libnetcdf: 4.9.3

Related

• PermissionError: [Errno 13] Permission denied: b'F:\\\xe5\x88\x9b\xe5\xbb\xba\xe6\x96\x87\xe4\xbb\xb6\\\\new.nc' #1220 (closed, netCDF4 1.4.0) — same error, old version
• filename is encoded to utf8, resulting in broken filenames on windows #686, Error opening netcdf in path with "special" characters #941 — earlier reports; filename is encoded to utf8, resulting in broken filenames on windows #686 suggests passing sys.getfilesystemencoding() to _strencode
• Fix open/create of UTF8 names netcdf-c#1668 (4.7.4), Improve UTF8 Support On Windows netcdf-c#2222 (4.9.0) — prior Windows UTF-8 fixes that do not cover this case

[jswhit]

sorry - I'm pretty ignorant of the details of windows and unicode so I don't have any suggestions for you.