KeyError(f"{labels[mask].tolist()} not found in axis") KeyError: "['index'] not found in axis" when generating heatmaps

[sosmarr]

Trying to generate heatmaps with following code:

# convert rMATS output to TSV format
python3 splice/Jutils/jutils.py convert-results \
	--rmats-dir splice/output \
	--out-dir splice/jutils_output

# heatmap: visualize DSR events
python3 splice/Jutils/jutils.py heatmap \
	--tsv-file splice/jutils_output/rmats_JC_results.tsv \
    --meta-file splice/meta_file.tsv \
	--p-value 0.05 \
	--q-value 0.05 \
	--out-dir splice/jutils_output 

but keep getting following errors:

Traceback (most recent call last):
  File "/mnt/scratch/Precision/CD/Lynn/SCalphaM15KD/splice/Jutils/jutils.py", line 182, in <module>
    main()
  File "/mnt/scratch/Precision/CD/Lynn/SCalphaM15KD/splice/Jutils/jutils.py", line 172, in main
    run_heatmap_module(args, parser_dict)
  File "/mnt/scratch/Precision/CD/Lynn/SCalphaM15KD/splice/Jutils/jutils.py", line 130, in run_heatmap_module
    plot_heatmap_pca_mds(Path(args.tsv_file), Path(args.meta_file), Path(args.out_dir), args.p_value,
  File "/mnt/scratch/Precision/CD/Lynn/SCalphaM15KD/splice/Jutils/heatmap_pca_mds_utils.py", line 89, in plot_heatmap_pca_mds
    data_df = process_data_supervised(data_df, samples, sample_cond_dict, conditions, original_columns,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/scratch/Precision/CD/Lynn/SCalphaM15KD/splice/Jutils/heatmap_pca_mds_utils.py", line 255, in process_data_supervised
    data_df = data_df.drop(columns=['dPSI', 'index'])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/BCRICWH.LAN/samantha.mar/.local/lib/python3.12/site-packages/pandas/core/frame.py", line 6288, in drop
    return super().drop(
           ^^^^^^^^^^^^^
  File "/home/BCRICWH.LAN/samantha.mar/.local/lib/python3.12/site-packages/pandas/core/generic.py", line 4644, in drop
    obj = obj._drop_axis(labels, axis, level=level, errors=errors)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/BCRICWH.LAN/samantha.mar/.local/lib/python3.12/site-packages/pandas/core/generic.py", line 4686, in _drop_axis
    new_axis = axis.drop(labels, errors=errors)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/BCRICWH.LAN/samantha.mar/.local/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 7268, in drop
    raise KeyError(f"{labels[mask].tolist()} not found in axis")
KeyError: "['index'] not found in axis"

Was able to generate PCA with no problems. Thank you!

[edwwlui]

Hi sosmarr,

Thanks for reporting this! It looks like a pandas versioning issue. I've pushed a patch to ignore the missing 'index' column during that step. Please pull the latest commit and let me know if it resolves the issue. If the error persists, sharing a small sample input would be helpful for reproducing the error on my end.

[sosmarr]

Hi edwwlui,

Thank you for helping me with the previous issue. After git pull, I am now able to generate supervised heatmaps. But now I get the same error when I try to generate unsupervised heatmaps with following code:

python3 splice/Jutils/jutils.py heatmap \
	--tsv-file splice/jutils_output/JC/rmats_JC_results.tsv \
    --meta-file splice/meta_file.tsv \
	--unsupervised \
	--out-dir splice/jutils_output/JC

Here is the error:

Traceback (most recent call last):
  File "/mnt/scratch/Precision/CD/Lynn/SCalphaM15KD/splice/Jutils/jutils.py", line 182, in <module>
    main()
  File "/mnt/scratch/Precision/CD/Lynn/SCalphaM15KD/splice/Jutils/jutils.py", line 172, in main
    run_heatmap_module(args, parser_dict)
  File "/mnt/scratch/Precision/CD/Lynn/SCalphaM15KD/splice/Jutils/jutils.py", line 130, in run_heatmap_module
    plot_heatmap_pca_mds(Path(args.tsv_file), Path(args.meta_file), Path(args.out_dir), args.p_value,
  File "/mnt/scratch/Precision/CD/Lynn/SCalphaM15KD/splice/Jutils/heatmap_pca_mds_utils.py", line 87, in plot_heatmap_pca_mds
    data_df = process_data_unsupervised(data_df, samples, original_columns, avg_threshold, aggregate, top)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/scratch/Precision/CD/Lynn/SCalphaM15KD/splice/Jutils/heatmap_pca_mds_utils.py", line 169, in process_data_unsupervised
    data_df = data_df.drop(columns=['variance', 'index'])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/BCRICWH.LAN/samantha.mar/.local/lib/python3.12/site-packages/pandas/core/frame.py", line 6288, in drop
    return super().drop(
           ^^^^^^^^^^^^^
  File "/home/BCRICWH.LAN/samantha.mar/.local/lib/python3.12/site-packages/pandas/core/generic.py", line 4644, in drop
    obj = obj._drop_axis(labels, axis, level=level, errors=errors)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/BCRICWH.LAN/samantha.mar/.local/lib/python3.12/site-packages/pandas/core/generic.py", line 4686, in _drop_axis
    new_axis = axis.drop(labels, errors=errors)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/BCRICWH.LAN/samantha.mar/.local/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 7268, in drop
    raise KeyError(f"{labels[mask].tolist()} not found in axis")
KeyError: "['index'] not found in axis"

[edwwlui]

Thanks for letting me know! Now the latest commit should have those KeyErrors fixed. Feel free to reach out if you run into any further issues.