Error in topic binarization after topic modeling with polars_1xx branch

[Sebdumas]

Hi,

I used the polars_1xx branch to run models, which worked well, but then I get the following error when I attempt to binarize the topics with "3k" and "otsu" methods:

Example of error I get for "otsu" (I have the same error for "3k" unless I set plot to False):

region_bin_topics_otsu = binarize_topics(
    cistopic_obj, method='otsu',
    plot=False, num_columns=5)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[14], line 1
----> 1 region_bin_topics_otsu = binarize_topics(
      2     cistopic_obj, method='otsu',
      3     plot=False, num_columns=5
      4 )

File ~/miniconda3/envs/pycistopic_alone/lib/python3.11/site-packages/pycisTopic/topic_binarization.py:108, in binarize_topics(cistopic_obj, target, method, smooth_topics, ntop, predefined_thr, nbins, plot, figsize, num_columns, save)
    106     thr = predefined_thr["Topic" + str(i + 1)]
    107 elif method == "otsu":
--> 108     thr = threshold_otsu(l_norm, nbins=nbins)
    109 elif method == "yen":
    110     thr = threshold_yen(l_norm, nbins=nbins)

File ~/miniconda3/envs/pycistopic_alone/lib/python3.11/site-packages/pycisTopic/topic_binarization.py:268, in threshold_otsu(array, nbins)
    247 def threshold_otsu(array, nbins=100):
    248     """
    249     Apply Otsu threshold on topic-region distributions [Otsu, 1979].
    250 
   (...)
    266     cybernetics, 9(1), pp.62-66.
    267     """
--> 268     hist, bin_centers = histogram(array, nbins)
    269     hist = hist.astype(float)
    270     # Class probabilities for all possible thresholds

File ~/miniconda3/envs/pycistopic_alone/lib/python3.11/site-packages/pycisTopic/topic_binarization.py:336, in histogram(array, nbins)
    320 """
    321 Draw histogram from distribution and identify centers.
    322 
   (...)
    333         Histogram values and bin centers.
    334 """
    335 array = array.ravel().flatten()
--> 336 hist, bin_edges = np.histogram(array, bins=nbins, range=None)
    337 bin_centers = (bin_edges[:-1] + bin_edges[1:]) / 2.0
    338 return hist, bin_centers

File ~/miniconda3/envs/pycistopic_alone/lib/python3.11/site-packages/numpy/lib/histograms.py:780, in histogram(a, bins, range, density, weights)
    680 r"""
    681 Compute the histogram of a dataset.
    682 
   (...)
    776 
    777 """
    778 a, weights = _ravel_and_check_weights(a, weights)
--> 780 bin_edges, uniform_bins = _get_bin_edges(a, bins, range, weights)
    782 # Histogram is an integer or a float array depending on the weights.
    783 if weights is None:

File ~/miniconda3/envs/pycistopic_alone/lib/python3.11/site-packages/numpy/lib/histograms.py:426, in _get_bin_edges(a, bins, range, weights)
    423     if n_equal_bins < 1:
    424         raise ValueError('`bins` must be positive, when an integer')
--> 426     first_edge, last_edge = _get_outer_edges(a, range)
    428 elif np.ndim(bins) == 1:
    429     bin_edges = np.asarray(bins)

File ~/miniconda3/envs/pycistopic_alone/lib/python3.11/site-packages/numpy/lib/histograms.py:323, in _get_outer_edges(a, range)
    321     first_edge, last_edge = a.min(), a.max()
    322     if not (np.isfinite(first_edge) and np.isfinite(last_edge)):
--> 323         raise ValueError(
    324             "autodetected range of [{}, {}] is not finite".format(first_edge, last_edge))
    326 # expand empty range to avoid divide by zero
    327 if first_edge == last_edge:

ValueError: autodetected range of [nan, nan] is not finite 

Originally posted by @Sebdumas in #195

[09oulin]

Oh, I have recently used the polars_1xx branch, and faced a similar problem when trying to binarize the topics with "otsu" method. Could you please tell me how you solve the problem?

[bmacnabb]

Hi all,

I recently encountered the same problem and I have a potential fix (feel free to skip to the end for the code).
Based on the error message and the fact that the code runs if smooth_topics is set to False, the problem seemed to be in smooth_topics_distributions() within topic_binarization.py. The issue is that the smoothing equation x * (np.log(x + 1e-100) - np.sum(np.log(x + 1e-100)) / x.shape[0] returns infinite and zero values when applied to the topic_region_distributions, which are stored in the cisTopic object as cistopic_obj.selected_model.topic_region. This appears to be due to the fact that these data are stored as float32 in the outputs of the polars forked version of the code, but are stored as float64 in the main distribution. If the values are converted to float64, they are no longer rounded to zero/infinity.

The fix that worked for me was changing topic_region_distributions.values to topic_region_distributions.values.astype(np.float64) in line 249 of topic_binarization.py.

Good luck and I hope this helps.