Expose --attention_dropout in pretraining (consistency with classification/segmentation)#2
Open
alex-crlhmmr wants to merge 1 commit into
Conversation
…ream pretrain.py did not expose --attention_dropout, and models/pretraining.py built the graph encoder without passing it, so attention dropout stayed pinned at the DualGraphEncoder default (0.1) and --dropout did not affect it. classification.py and segmentation.py already expose the flag and thread it into DualGraphEncoder; this mirrors that pattern for pretraining. The default (0.1) preserves existing behavior; the flag now makes attention dropout controllable and visible in the pretraining config, which matters for reproducibility (the effective attention regularization was previously not derivable from the pretraining flags).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a
--attention_dropoutflag topretrain.pyand threads it into the graph encoder inmodels/pretraining.py, matching howclassification.pyandsegmentation.pyalready handle it.Why
The two downstream entrypoints expose
--attention_dropoutand pass it intoDualGraphEncoder:classification.py/segmentation.py:parser.add_argument("--attention_dropout", type=float, default=0.1)models/dual_classification.py/models/dual_segmentation.py:attention_dropout=getattr(args, "attention_dropout", args.dropout)pretrain.pyhas no such flag, andmodels/pretraining.pyconstructsDualGraphEncoderwithout passingattention_dropout. As a result, during pretraining the attention dropout is pinned at theDualGraphEncoderdefault (0.1), and--dropouthas no effect on it. The attention regularization actually used to pretrain a model is therefore not derivable from the pretraining flags/config, which is inconsistent with the fine-tuning entrypoints and awkward for reproducibility.Change
Two lines, mirroring the downstream pattern exactly:
pretrain.py: add--attention_dropout(default0.1).models/pretraining.py: passattention_dropout=getattr(args, "attention_dropout", args.dropout)toDualGraphEncoder.Behavior
The default (
0.1) reproduces current behavior exactly, so existing runs are unaffected. The flag simply makes attention dropout controllable and visible in the pretraining config, as it already is downstream.