Skip to content

Integrated the random allocation directory#414

Open
moshenfeld wants to merge 17 commits into
google:mainfrom
moshenfeld:random-alocation
Open

Integrated the random allocation directory#414
moshenfeld wants to merge 17 commits into
google:mainfrom
moshenfeld:random-alocation

Conversation

@moshenfeld

Copy link
Copy Markdown

Added a new pld/random_allocation package which implements the accounting algorithm proposed in "Efficient privacy loss accounting for subsampling and random allocation".
It contains public builders for Gaussian and explicit-realization random allocation PLDs in random_allocation_api.py.
It also wires the feature into dp_accounting as RandomAllocationDpEvent in dp_event.py, and makes PLDAccountant handle Gaussian inner events in pld_privacy_accountant.py.

@google-cla

google-cla Bot commented Apr 30, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@arung54 arung54 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Moshe,

No comments on the logic / implementation. Most of these comments are about style (and stuff that we will probably have to address anyway to pass the style checks) which are hopefully easy to fix. Some of these could have been prevented if I shared our style guide in advance, apologies for only pointing you to it during the review! If you haven't already, I'd also suggest fixing any issues raised by:

$ flake8 random_allocation/*.py
$ pyink random_allocation/*.py
$ pydocstyle --convention=google --add-ignore=D101,D102,D103,D105,D202,D402 random_allocation/*.py
$ pylint random_allocation/*.py
$ pytype random_allocation/*.py -k

Some top-level comments:

  • Since this library is already in a folder named 'random_allocation', consider renaming some of the libraries to be shorter (e.g. from 'random_allocation_convolution' to 'convolution') for brevity.
  • See https://google.github.io/styleguide/pyguide.html#22-imports - a lot of these libraries are importing methods, whereas the style guide suggests importing modules and then calling the method from the package name. Renaming to shorter module names might help with doing this while maintaining brevity.
  • For the test files, can you match the style of the other test files in dp_accounting here?:
    • Use absltest instead of pytest
    • Prefer absltest.TestCase or numpy.testing assertions over assert ..., since these will give more verbose failures.
    • Put "Test" at the end of test class names rather than beginning

Comment thread python/dp_accounting/dp_accounting/dp_event.py Outdated
Comment thread python/dp_accounting/dp_accounting/pld/pld_privacy_accountant.py
Comment thread python/dp_accounting/dp_accounting/pld/pld_privacy_accountant.py Outdated
Comment thread python/dp_accounting/dp_accounting/dp_event_test.py Outdated
Comment thread python/dp_accounting/dp_accounting/pld/random_allocation/utils.py
Comment thread python/dp_accounting/dp_accounting/pld/random_allocation/ra_utils.py Outdated
@ryan112358

Copy link
Copy Markdown

A few minor comments

@onlinedigitalservicexplacebank

Copy link
Copy Markdown

Comment thread python/dp_accounting/dp_accounting/pld/random_allocation/ra_convolution.py Outdated
Comment thread python/dp_accounting/dp_accounting/dp_event.py Outdated
@moshenfeld

moshenfeld commented Jun 7, 2026 via email

Copy link
Copy Markdown
Author

moshenfeld added 14 commits June 8, 2026 22:33
- Rename RandomAllocationDpEvent.k/.t to .num_selected/.num_steps (r3170917978).
- Add arxiv paper reference to the docstring (r3171223123).
- Remove ra_event alias in PLDAccountant._maybe_compose (r3170942975).
- Add test_random_allocation_basic to pld_privacy_accountant_test.py (r3170933999).
- Remove test_random_allocation_event_has_no_discretization_override (r3198173993). Instead, renamed AllocationSchemeConfig.loss_discretization to .value_discretization_interval for consistency with PLDAccountant's existing _value_discretization_interval.
Add soft-import guard (_HAS_NUMBA / _optional_njit() / has_numba()) in
random_allocation_types.py and replace all @njit(cache=True) decorators
with @_optional_njit(), so the package loads and runs without numba
installed (r3198378368, r3228974300).

For the two performance-critical kernels (_numba_geometric_kernel,
_numba_rediscretize_prob) add pure-NumPy fallbacks and dispatch via
has_numba() at call time. _add_single_zero_atom_cross_term is fully
vectorised with np.add.at, removing its hard numba dependency. Add parity
and fallback-dispatch tests for all three paths (r3171209246).
Convert all intra-package imports from relative to absolute module-level imports (dp_accounting.pld.random_allocation.module), and call internal helpers via module._fn() rather than importing them by name — matching the project's Google-style import convention (r4208613559 import bullet).

Add _ prefix to all implementation-only symbols across random_allocation_{types,distributions,utils,convolution,core}, so the public API is limited to the names in __init__.__all__ (r3229395649, part of r3229486152).
- Add paper link to algorithm docstrings (r3230294534, r3230302356, r3230234883, r3230282798, r3230097765, r3229713035).
- Add Attributes: sections to classes (r3229028609, r3229277278, r3229347201).
- Add inline comment explaining why math.fsum is preferred over np.sum for numerical stability (r3171163734).
- Rewrite mass-conservation ValueError (r3228994798, r3228999157).
- Inline trivial max_idx_plus_one temporary variable (r3171158087).
- Introduce assertion helpers in random_allocation_utils, replacing ad-hoc isinstance blocks (r3230280947, r3230069168).
- Remove post-convolution return-value check of _binary_self_convolve which is verified by pytype and is an overkill (r3230081238)
- Remove stale comment from truncate_edges docstring (r3229005737).
- Add del max_ind  # Unused. style comment (r3229414402).
- Fix typos: "distributsion" -> "distribution" (r3228981292),
  "TestDiscritizeRange" -> "TestDiscretizeRange" (r3229767210).
- Change branches to always use elif/else validation (r3229336820, r3229342023, r3229407683).
- Rename continuous-discretization helper and clarify its docstring (r3229679129).
- Rename lowercase variables and test alias (r3229709372, r3229794562).
- Use x_0 notation in discretization comments and remove the stale docstring blank line (r3229702549, r3229736292).
- Switch random-allocation tests to module imports instead of direct functions import (r4208613559 import bullet).
- Run pyink over the random-allocation package and close flake8, pydocstyle, and pytype gaps found during audit (r4208613559 style-check request).P
- Replace pytest assertions and free functions with absltest.TestCase classes, numpy.testing assertions, and Test-suffixed class names to match the rest of dp_accounting.
- Add absltest.main() for module execution. Resolves top-level review comment 4208613559 test-style bullets.
- Reorder functions in ra_distributions, ra_convolution, and ra_utils so that every helper is defined before the first call site that uses it, matching the project's internal style (all reviewer ordering comments).
- Rename _assert_dense_linear_dist and _assert_dense_geometric_dist to _validate_dense_linear_dist and _validate_dense_geometric_dist to match the _validate_* naming convention used by all other guard helpers in ra_utils.
…e pylint warning

- Place third-party numpy/scipy imports above first-party dp_accounting.
- Fix line-too-long violations.
- Add missing property docstrings.
- Add # pylint: disable=invalid-name on every T: int parameter (3230066401).
- Apply pyink reformatting.

Verified with:
- flake8 dp_accounting/pld/random_allocation --max-line-length=100 -> clean
- pyink --check dp_accounting/pld/random_allocation -> clean
- pydocstyle --convention=google ... dp_accounting/pld/random_allocation -> clean
- pylint dp_accounting/pld/random_allocation/ --ignore=*_test.py -> 9.06/10
- pytest ra_{api,distributions,convolution}_test.py -> 56 passed
… comments

- Replace get_x_array/x_array distribution API with private _x_array access
- Store sparse support in _x_arr and keep dense support derived from RegularGrid
- Use shared dense-linear validation in dp_accounting PMF conversion
- Use copy.deepcopy for rediscretization working distributions
…names

- Replace 9 inline DenseDiscreteDist spacing guards in ra_core with the
  _validate_dense_linear_dist/_validate_dense_geometric_dist helpers (3230280947).
- Import pld_pmf and common as modules rather than individual symbols.
- Strip leaked ra_*. prefixes from error messages, docstrings, and comments.
- Run pyink with --pyink-indentation 2 --line-length 80 to match the
  dp_accounting code style (files were 4-space).
- git mv ra_*.py -> *.py (api, convolution, core, distributions, utils, and their tests).
- Rename ra_types.py -> definitions.py (not types.py) to avoid shadowing the stdlib `types` module.
- Update all intra-package imports and qualified references (ra_* -> *) across the package.

@arung54 arung54 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Moshe,

Overall looks good. One minor comment below. In addition, I think the linter will complain if you are accessing protected methods from other classes (unless you are in a test class accessing a protected method from the tested class). I think the 'public API' methods in api.py / core.py are the main culprits for where the linter would complain.

Also, I think we will need to add a BUILD.bazel (similar to https://github.com/google/differential-privacy/blob/main/python/dp_accounting/dp_accounting/BUILD.bazel) to this folder, and update the BUILD.bazel in the parent folder. For the most part it should be pattern-matchable (also happy to help you with writing it), but since we are adding a numba dependency that may introduce some complications. Let me chat w/ other folks on our team at the right way to add numba (since right now it's not in the pip deps) and get back to you.

from dp_accounting import privacy_accountant
from dp_accounting.pld import common
from dp_accounting.pld import privacy_loss_distribution
from dp_accounting.pld import random_allocation as _random_allocation

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for consistency, I think it's fine to import without 'as _random_allocation'

- Drop the `_random_allocation` import alias in pld_privacy_accountant.py
- Fix pylint protected-access (W0212) in production code: de-underscore the helper functions that are called across modules within the package. Test-only internals keep their underscores.
- Make the DiscreteDist `x_array` property public (was `_x_array`).
- Rename the invalid-name `_st` locals in utils.py to `spacing`.
- Add random_allocation/BUILD.bazel (per-file py_library + py_test targets) and depend on it from pld/BUILD.bazel's pld_privacy_accountant target.
@moshenfeld

moshenfeld commented Jun 27, 2026 via email

Copy link
Copy Markdown
Author

@arung54

arung54 commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Sorry for the lack of clarity earlier! Yes, if a method in foo.py is only used in foo.py (and maybe foo_test.py) it should start with underscore. If it is used in bar.py in the same folder, the linter will complain if it starts with an underscore, which we need to avoid to pass style checks. And yes, if it is only used within the same folder, not including it in the init is good.

Thanks for your patience with the comments, at this point I think it's looking good generally. Let me check internally the right way to get numba added as a dependency, then I might suggest a change to one of the BUILD.bazels and then we are probably good to go to internal approval, which we can hopefully make fast. I will be OOO for the next few days due to the US holiday, but will try to check back in soon.

@moshenfeld

moshenfeld commented Jul 8, 2026 via email

Copy link
Copy Markdown
Author

…tations that are practically identical accuracy-wise and <= X2 slower.
@arung54

arung54 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Hi Moshe,

Thanks for the updates! I think to migrate this into an internal PR we need to resolve the merge conflicts? Otherwise I will try to keep things moving on our end once that's done.

@moshenfeld

moshenfeld commented Jul 10, 2026 via email

Copy link
Copy Markdown
Author

@arung54

arung54 commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Got it, I'll ask someone on our side who's more familiar with the OSS process for how to proceed then!

@ryan112358

Copy link
Copy Markdown

I'm not 100% sure, but one thing that might be happening is if the resolution of conflicts is non-trivial, you can't do it in the web interface, but you have to do it manually via the command line w/ git. I actually don't have a ton of experience with this, but I believe if you "git pull" it will sync to latest version at head and you will be forced to resolve merge conflicts before proceeding

@moshenfeld

moshenfeld commented Jul 11, 2026 via email

Copy link
Copy Markdown
Author

@ryan112358

Copy link
Copy Markdown

Can you try following these steps?

# 1. Add the upstream Google repository (if not done already)
git remote add upstream https://github.com/google/differential-privacy.git

# 2. Fetch the latest changes directly from Google's repository
git fetch upstream

# 3. Ensure you are on your feature branch
git checkout random-alocation

# 4. Merge the upstream main branch to trigger the conflicts locally
git merge upstream/main

# 5. Resolve the conflicts in your editor, then commit and push
git commit -m "Resolve merge conflicts with upstream main"
git push origin random-alocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants