Can't reproduce the results of Figure.6

[farisxiong]

Amazing work!

However, when I tried to reproduce Figure 6 based on commit b1dd507, I found that it differs quite a lot from the one in the paper on Llama-3.1 8B. What could be the reason for this? Additionally, I found that the code for reproducing Figure 6 in commit 0412fc2 contains quite a few bugs, so I tried using the latest commit to reproduce it.

Method	token_sparse_method	Perplexity	Arc-Easy Accuracy	Hellaswag Accuracy	PIQA Accuracy	Winogrande Accuracy
Oracle	fixed_60pc	6.7961	0.732	0.496	0.745	0.595
Quest	fixed_60pc	79.74	0.342	0.322	0.558	0.498
ExpPred	fixed_60pc	102.1971	0.359	0.328	0.568	0.511

[akhauriyash]

Hey,

Can you tell me what bugs are there in the earlier commit?
This isnt published work yet, so we are still modifying the predictor architecture. The weights from the old model are not at all compatible with the new arch (which has layer norms etc.)

If you are using the latest commit ID, may i recommend using the latest models from Google Drive?
(https://drive.google.com/file/d/1w2wq23dy7vNA5Dn7an-5kHeNTYtc3HKB/view)

[akhauriyash]

Also I should recommend, if you are quickly validating basic performance, just do wikitext perplexity, its way faster and you can start downstream eval only once the perplexity number makes sense.

[farisxiong]

I apologize for the delay, but I would like to ask that for the baselines other than exppred, it seems that the download links you provided are not necessary. However, I am still struggling to reproduce the results of these baselines.

model_path	eval_wk2_seqlen	eval_llm_mode	token_sparse_method	task_list	true_token_sparsity	perplexity	arc_easy_acc	hellaswag_acc	piqa_acc	winogrande_acc
meta-llama/Llama-3.2-3B	1024	oracle	fixed_10pc	['winogrande', 'hellaswag', 'piqa', 'arc_easy']	12.190641	8.119585	0.731	0.488	0.763	0.681
meta-llama/Llama-3.2-3B	1024	oracle	fixed_20pc	['winogrande', 'hellaswag', 'piqa', 'arc_easy']	21.591681	8.124074	0.732	0.488	0.764	0.685
meta-llama/Llama-3.2-3B	1024	oracle	fixed_30pc	['winogrande', 'hellaswag', 'piqa', 'arc_easy']	30.947378	8.129761	0.727	0.48	0.755	0.678
meta-llama/Llama-3.2-3B	1024	oracle	fixed_40pc	['winogrande', 'hellaswag', 'piqa', 'arc_easy']	40.292094	8.147371	0.714	0.481	0.747	0.624
meta-llama/Llama-3.2-3B	1024	oracle	fixed_50pc	['winogrande', 'hellaswag', 'piqa', 'arc_easy']	49.693164	8.173347	0.696	0.474	0.746	0.577
meta-llama/Llama-3.2-3B	1024	oracle	fixed_60pc	['winogrande', 'hellaswag', 'piqa', 'arc_easy']	58.916751	8.231646	0.654	0.459	0.729	0.555
meta-llama/Llama-3.2-3B	1024	quest	fixed_10pc	['winogrande', 'hellaswag', 'piqa', 'arc_easy']	12.292053	10.000510	0.513	0.404	0.664	0.506
meta-llama/Llama-3.2-3B	1024	quest	fixed_20pc	['winogrande', 'hellaswag', 'piqa', 'arc_easy']	21.676278	11.873290	0.473	0.391	0.627	0.508
meta-llama/Llama-3.2-3B	1024	quest	fixed_30pc	['winogrande', 'hellaswag', 'piqa', 'arc_easy']	30.989876	15.459391	0.428	0.375	0.599	0.492
meta-llama/Llama-3.2-3B	1024	quest	fixed_30pc	['winogrande', 'hellaswag', 'piqa', 'arc_easy']	30.989876	15.459391	0.428	0.375	0.599	0.492
meta-llama/Llama-3.2-3B	1024	quest	fixed_40pc	['winogrande', 'hellaswag', 'piqa', 'arc_easy']	40.289951	22.858629	0.391	0.347	0.573	0.493
meta-llama/Llama-3.2-3B	1024	quest	fixed_50pc	['winogrande', 'hellaswag', 'piqa', 'arc_easy']	49.652446	38.651443	0.366	0.337	0.57	0.5
meta-llama/Llama-3.2-3B	1024	quest	fixed_60pc	['winogrande', 'hellaswag', 'piqa', 'arc_easy']	58.850169	73.413918	0.337	0.321	0.555	0.501
meta-llama/Llama-3.2-3B	1024	snapkv	fixed_10pc	['winogrande', 'hellaswag', 'piqa', 'arc_easy']	14.353249	931.751526	0.315	0.316	0.574	0.502

[akhauriyash]

https://github.com/abdelfattah-lab/TokenButler/blob/main/ablation_results/evalresults/L3_3B_2k_1PC_quest.csv

Your oracle and quest results look pretty much the same as in our ablation results? With a slight +/- on the 'effective sparsity'.

However, the snapkv baseline definitely looks wrong. This might be a bug that I pushed since the earlier version. I will try to run the SnapKV perplexity baseline at 10% sparsity, I expect to see perplexity ~ 9.132674217224121 (https://github.com/abdelfattah-lab/TokenButler/blob/main/ablation_results/evalresults/L3_3B_2k_1PC_snapkv.csv)

Will update here. Thanks!

[akhauriyash]

apologies on the delay, i havent found a free slot to run the baseline snapKV test. Will try to update you ASAP. Thanks