Multiclass Classification via Binary Decomposition and Asymmetric Label Switching Ensemble (ALSE)

This repository implements a multiclass classification framework using Binary Decomposition and the Asymmmetric Label Switching Ensemble (ALSE or LSEnsemble). ALSE is employed for the binary classification tasks generated by ECOC, effectively improving accuracy, especially with imbalanced data.

Project Overview

Binarization decompose multiclass problems into multiple binary classification problems. Even if the original multiclass data is balanced, binarization—such as the One-versus-Rest (OvR) strategy—can introduce severe class imbalance. This imbalance complicates learning, particularly in Bayes-optimal classification, where accurate likelihood ratio estimation is crucial.

The Asymetric Label Switching Ensemble (ALSE) offers an alternative approach to handle imbalanced classification. It randomly swaps labels based on predefined switching rates, introducing diversity within the ensemble, which enhances performance. Additionally, label switching rebalances class distributions, improving the estimation of transformed a priori probabilities and, consequently, the likelihood ratio for base learners.

Asymmetric label switching can be combined with other neutral rebalancing strategies, such as cost-sensitive learning and population-based adjustments. This results in a transformed problem where the optimal Bayes threshold can be theoretically derived.

For further details on likelihood ratio estimation and Bayes threshold transformations, see: "Optimum Bayesian thresholds for rebalanced classification problems using class-switching ensembles," Pattern Recognition, 2022. https://doi.org/10.1016/j.patcog.2022.109158

Key Features

• Hyperparameter Optimization Script (optimize_hyperparameters.py):
  • This script performs the crucial task of optimizing the hyperparameters for the classification models.
  • It systematically explores different configurations of model parameters.
  • It evaluates the performance of each configuration using cross-validation or a similar technique.
  • The script saves the best-performing model configurations (parameters) for later use in the evaluation phase.
  • It is driven by the selected YAML file under config/, including the models to optimize and the ranges of their hyperparameters.
• Binarization Decomposition:
  • Implements binarization matrices for effective multiclass to binary transformation.
  • Offers flexible One-versus-One (OvO), OvR, and ECOC encoding options (complete, dense, sparse) configurable through the selected YAML file.
• ALS Ensemble Algorithm:
  • A specialized ensemble algorithm designed for label switching correction, enhancing model robustness.
  • Provides configurable optimization parameters for the ALS Ensemble through the selected YAML file.
• Comprehensive Evaluation:
  • Includes a thorough evaluation suite using key multiclass metrics: balanced accuracy, Cohen's kappa, geometric mean, and sensitivity.
  • Features detailed logging and output to facilitate performance analysis.
• Configuration-Driven Design:
  • Leverages YAML files under config/ for easy customization of datasets, models, and evaluation settings.
  • Employs streamlined model selection logic based on peak or average performance, configurable through the selected YAML file.
• Model Persistence:
  • Saves models configured with optimized hyperparameters, along with their configurations, using pickle for efficient testing and deployment.
  • Utilizes a naming convention for saved models that includes the parameters used for the ALS Ensemble.
• Performance Evaluation Script (evaluate_performance.py):
  • This script orchestrates the evaluation of models using pre-optimized hyperparameters.
  • It loads the best model configurations (parameters) saved during the optimization phase by optimize_hyperparameters.py.
  • Subsequently, it instantiates and trains the models using these optimal configurations.
  • The script concludes by performing testing on a separate dataset to obtain the final performance metrics.
  • It relies on the selected YAML file to load datasets, output paths, model definitions, and evaluation settings.
• Dataset Flexibility:
  • Engineered to accommodate a wide range of datasets stored within a designated data folder.

Getting Started

1. Clone the repository:

   git clone https://github.com/franjgs/MC_Label_Switching.git
   cd MC_Label_Switching

2. Choose a configuration file:

   • Use config/config_train.yaml for optimization runs.
   • Use config/config_test.yaml for final evaluation runs.
   • Use config/config_smoke.yaml for the minimal validated workflow.

3. Run the hyperparameter optimization script:

   python optimize_hyperparameters.py --config config/config_train.yaml

   • This script explores binary hyperparameter configurations per dichotomy and saves the best configurations as *_train.pkl files in paths.output_folder.

4. Run the performance evaluation script:

   python evaluate_performance.py --config config/config_test.yaml

   • This script loads the saved *_train.pkl files, retrains the models with the best binary configurations, computes final multiclass performance, and saves *_test.pkl files in paths.output_folder.

Configuration Files

• config/config_train.yaml: default configuration for optimize_hyperparameters.py.
• config/config_test.yaml: default configuration for evaluate_performance.py.
• config/config_smoke.yaml: minimal validated configuration for quick end-to-end checks.

Model grids can use either the legacy flat MC schema with prefixed dynamic parameters (LS_, LR_, RF_, MLP_, LGBM_, SVM_, kNN_, C45_, RB_) or the structured shared LSEnsemble configuration format. The structured format is recommended for new LSEnsemble configurations and uses params.base_learner, params.base_learner_configs, and canonical dynamic parameters such as alpha, beta, Q_C, Q_RB_S, Q_RB_C, and num_experts.

Smoke Test

The minimal validated workflow is:

python optimize_hyperparameters.py --config config/config_smoke.yaml
python evaluate_performance.py --config config/config_smoke.yaml

The first command writes *_train.pkl files to the configured output folder. The second command reads those files, performs the final multiclass evaluation, and writes *_test.pkl files.

Dependencies

• Python 3.x
• imbalanced-learn == 0.13.0
• Keras == 2.15.0
• NumPy == 1.26.4
• Pandas == 2.2.3
• PyYAML == 6.0.2
• scikit-learn == 1.5.2
• SciPy == 1.15.1
• Torch == 2.6.0
• TorchEnsemble == 0.2.0

Datasets

• Place versioned datasets in the datasets folder, or point paths.data_folder to a local dataset folder in the selected YAML file.

Output

• Trained configurations and evaluation metrics are saved in the configured paths.output_folder.
• Do not version generated outputs or local datasets such as results/, results*/, results_smoke/, datasets_lowC_complete/, *.pkl, *.joblib, *.npy, *.npz, *.csv, or log files.

Contributing

• Contributions are welcome! Please feel free to submit pull requests or open issues.

License

• Copyright (c) 2025 UC3M