In file mlp.py, and similar MLP files, I believe the function select_activation_function should be called inside the forward function. Else, it would not change the state of MLP gradient by simply changing the config:
`model.cfg.use_lrp = True
model.cfg.LRP_rules = ['LN-rule', 'Identity-rule', 'Half-rule']
model.set_use_attn_result(True)
model.set_use_hook_mlp_in(True)
model.set_use_attn_in(True)
model.set_use_split_qkv_input(True)`, as in the notebook.
Thanks for reviewing.
In file mlp.py, and similar MLP files, I believe the function
select_activation_functionshould be called inside theforwardfunction. Else, it would not change the state of MLP gradient by simply changing the config:`model.cfg.use_lrp = True
model.cfg.LRP_rules = ['LN-rule', 'Identity-rule', 'Half-rule']
model.set_use_attn_result(True)
model.set_use_hook_mlp_in(True)
model.set_use_attn_in(True)
model.set_use_split_qkv_input(True)`, as in the notebook.
Thanks for reviewing.