[Feature] Add serveral differential privacy algorithms#369
[Feature] Add serveral differential privacy algorithms#369DavdGao wants to merge 6 commits intoalibaba:masterfrom
Conversation
xieyxclack
left a comment
xieyxclack
left a comment
There was a problem hiding this comment.
LGTM, but I have not checked the implementation details of Functional Mechanism: Regression Analysis under Differential Privacy. And please refer to the inline comments for minor suggestions, thx!
| """Returns the l2 norm of weight | ||
|
|
||
| Arguments: | ||
| p (int): The order of norm. |
There was a problem hiding this comment.
modified accordingly
There was a problem hiding this comment.
In the funcational mechanism original paper, the noise addition process is performed only once for the objective function, while your implementation performs noise addition in each iteration, is this supported by the theory?
|
|
||
| def forward(self, ctx): | ||
| l2_norm = 0. | ||
| for param in ctx.model.parameters(): |
There was a problem hiding this comment.
An interesting discussion about whether l2 regularization should be applied to beta and gamma of BN layers: https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994.
IMO, we can keep the implementation same as that in torch.norm
There was a problem hiding this comment.
The current torch.norm will calculate the l2 norm for all parameters including bn weight and bias.
For now, we use the parameter skip_bn to aovid calculating l2-norm for bn layer(with 'bn' in its name).
There was a problem hiding this comment.
Conflicts are solved
# Conflicts: # federatedscope/core/auxiliaries/optimizer_builder.py # federatedscope/register.py
3 participants
Main modifications:
fmtrainer