gh-83765: indicate the maximum amount of memory on macOS

[nilleb]

Fixed following the suggestion #83765 (comment)

• Issue: [doc] creating large SharedMemory crashes on MacOs #83765

---

📚 Documentation preview 📚: https://cpython-previews--138552.org.readthedocs.build/

[AA-Turner]

At the very least please add some commas, but it might be more helpful to use a saner and more human readable unit:

Suggested change

	On macOS, the maximum size of a shared memory block is 140256418463744 bytes.
	On macOS, the maximum size of a shared memory block is 127 TiB (140,256,418,463,744 bytes).

[AA-Turner]

Also, it would be useful to add a reference to where this number comes from. I can't see anything on the linked issue mentioning it.

[nilleb]

Hi! Thank you for your comment! This number has been computed this way: https://gist.github.com/nilleb/e60e7adbfe2afaf9d727231c59b01100

I have tested it on macOS Sequoia 15.6.1.
That is the best I can afford for the moment, I don't have access to older operating systems.

[AA-Turner]

Ok, if this is an experimentally found number we should remove it, there are no guarantees.

Instead, say something like 'the maximum size is around 120 TiB'.

However, I'm now less sure about including this warning in the documentation, given the limit doesn't come from a documented or well-known property of macOS, and could change without warning. cc @python/macos-team for thoughts.

[nilleb]

Searching further, I have found this SO post https://stackoverflow.com/a/77078609

Where they mention 0x00007ffffe000000 that corresponds to 140737454800896 (so a bit more that the size I get).
This would be compatible with the fact that the memory for a python process is limited to 128TiB on a 64bits macOS OS (and hence the maximum allowed value depends on the memory already allocated by python itself).

I would propose to add a link to https://github.com/apple-oss-distributions/xnu/blob/xnu-8796.141.3/osfmk/mach/arm/vm_param.h#L137

And a paraphrase of the explanation above.

WDYT?

[picnixz]

Maybe I'm wrong here but AFAICT, 127 TiB is not 140,256,418,463,744 bytes. Indeed 1 TiB is $2^{40}$ bytes so $2^{40} * 127 = 139,637,976,727,552$. Note that $140,737,454,800,896$ is a bit below $2^{40} * 128$ but above $127$ TiB.

$2^{40}$ is still a reasonable amount of memory, at least in my research area where we try to limit memory to that number (up to some constant factor, but most of the time, we want to limit the overall complexity of an attack to $O(2^{40})$ bits).

Instead, I would rather say that the maximum amount of memory on a 64-bit architecture is upperbounded by $128$ TiB and indicate that it is $128 * 2^{40}$ bytes). If we have mathjax enabled, we can use the math role, otherwise, you can write it down in full numbers.

[nilleb]

OK, here is a further commit for your review! (the math role seems to be available, thank you!)

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[gpshead]

Why do we need to document this at all? It is platform specific and Apple will never ship hardware with anything approaching that much memory.

I don't think this is useful information for Python multiprocessing users to have.

FWIW other architectures have similar limits. The details of each aren't relevant here though.

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[gpshead]

left a comment on the issue. a more generic "btw your platform has limits" at most could make sense, otherwise this is too specific for cpython docs. (probably do a another PR for that if you want it at all)

[nilleb]

@gpshead may I just suggest to close the mentioned issue? That might be misleading for other wannabe fixers. Thank you!