GitHub - Mengda-Li/attention: Code for experimentation in Attention Degeneration

Code for the paper Spectral Degeneration of Softmax Attention under Isotropic Score Geometry

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
.gitignore		.gitignore
README.md		README.md
compute_erank_quantiles.py		compute_erank_quantiles.py
eigenvalue_uniformity_analysis.py		eigenvalue_uniformity_analysis.py
gpt2_eigen_angle_statistics.py		gpt2_eigen_angle_statistics.py
plot_eigen_angle_statistics.py		plot_eigen_angle_statistics.py
plot_gpt2_longest_spectral_histogram.py		plot_gpt2_longest_spectral_histogram.py
plot_vit_layer_boxplot.py		plot_vit_layer_boxplot.py
plot_vit_sample_spectral_histogram_from_a.py		plot_vit_sample_spectral_histogram_from_a.py
pyproject.toml		pyproject.toml

Provide feedback