Aligning Language Models to Explicitly Handle Ambiguity (EMNLP 2024)

Code for Alignment with Perceived Ambiguity (APA)

Run

Run sh scripts/main.sh.

• stage_0.sh : select ambgiuous queries and build train data.
• train.sh : train model.
• stage_1.sh : evaluate trained model.

Configurations

Change configs/main.yaml

• model.name : backbone
• model.offload_path : model offload path
• model.cache_path : huggingface cache path
• path.data : path to load dataset
• path.output : output path (logs, weights, ...)
• dataset.name : test dataset name
• pipeline.stage_index : set from 0 or 1
• explicit.template_id : explicit inference QA template
• explicit.evaluation_method : 'rouge' as default
• explicit.correct_threshold : generations with score above the threshold is evaluated as correct.
• implicit.method_id : how to measure INFOGAIN (default 0)
• implicit.disambiguation_template_id : template id for self-disambiguation
• implicit.generation_template_id
• implicit.threshold : threshold value to filter ambiguous queries
• implicit.aggregate_method
• explanation.template_id : template to generate explanations
• generation.num_generations_per_prompt : generation configs
• generation.num_single_generation : generation configs
• generation.max_new_tokens : generation configs
• generation.temperature : generation configs
• ablation_methods : data selection methods
• train.num_train_epochs : train configs (number of training epochs)
• train.per_device_train_batch_size : train configs (train batch size)
• train.gradient_accumulation_steps : train configs (gradient accumulation steps)
• train.learning_rate : train configs (learning rate)