Error when using different values for latent_dim parameter in tadgan model

[LuSchnitt]

• Orion version: 0.6.1
• Python version: 3.11.10
• Operating System: Ubuntu 24.04.1 LTS

Description

When using 40 for the latent_dim parameter in the tadgan model, the following error occurs:

File ~/anaconda3/envs/orion_env/lib/python3.11/site-packages/orion/core.py:286, in Orion.evaluate(self, data, ground_truth, fit, train_data, metrics)
    282     mlpipeline = self._mlpipeline
    284 if train_data is not None:
    285     # Fit first and then predict
--> 286     mlpipeline.fit(train_data)
    287     method = mlpipeline.predict
    288 else:
    289     # Fit and predict at once

File ~/anaconda3/envs/orion_env/lib/python3.11/site-packages/mlblocks/mlpipeline.py:802, in MLPipeline.fit(self, X, y, output_, start_, debug, **kwargs)
    799         LOGGER.debug('Skipping block %s fit', block_name)
    800         continue
--> 802 self._fit_block(block, block_name, context, debug_info)
    804 if fit_pending or output_blocks:
    805     self._produce_block(
    806         block, block_name, context, output_variables, outputs, debug_info)

File ~/anaconda3/envs/orion_env/lib/python3.11/site-packages/mlblocks/mlpipeline.py:644, in MLPipeline._fit_block(self, block, block_name, context, debug_info)
    642 memory_before = process.memory_info().rss
    643 start = datetime.utcnow()
--> 644 block.fit(**fit_args)
    645 elapsed = datetime.utcnow() - start
    646 memory_after = process.memory_info().rss

File ~/anaconda3/envs/orion_env/lib/python3.11/site-packages/mlblocks/mlblock.py:311, in MLBlock.fit(self, **kwargs)
    309 fit_kwargs.update(kwargs)
    310 fit_kwargs = self._get_method_kwargs(fit_kwargs, self.fit_args)
--> 311 getattr(self.instance, self.fit_method)(**fit_kwargs)

File ~/anaconda3/envs/orion_env/lib/python3.11/site-packages/orion/primitives/tadgan.py:358, in TadGAN.fit(self, X, y, **kwargs)
    356     kwargs = self._augment_hyperparameters(X, y, **kwargs)
    357     self._set_shapes()
--> 358     self._build_tadgan(**kwargs)
    360 self._fit((X, y))
    361 self.fitted = True

File ~/anaconda3/envs/orion_env/lib/python3.11/site-packages/orion/primitives/tadgan.py:233, in TadGAN._build_tadgan(self, **kwargs)
    230 hyperparameters.update(kwargs)
    232 # Models
--> 233 self.encoder = self._build_model(
    234     hyperparameters, self.layers_encoder, self.encoder_input_shape, name='encoder')
    235 self.generator = self._build_model(
    236     hyperparameters, self.layers_generator, self.generator_input_shape, name='generator')
    237 self.critic_x = self._build_model(
    238     hyperparameters, self.layers_critic_x, self.critic_x_input_shape, name='critic_x')

File ~/anaconda3/envs/orion_env/lib/python3.11/site-packages/orion/primitives/tadgan.py:98, in TadGAN._build_model(hyperparameters, layers, input_shape, name)
     95     built_layer = build_layer(layer, hyperparameters)
     96     model.add(built_layer)
---> 98 return Model(x, model(x))

File ~/anaconda3/envs/orion_env/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py:70, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     67     filtered_tb = _process_traceback_frames(e.__traceback__)
     68     # To get the full stack trace, call:
     69     # `tf.debugging.disable_traceback_filtering()`
---> 70     raise e.with_traceback(filtered_tb) from None
     71 finally:
     72     del filtered_tb

File ~/anaconda3/envs/orion_env/lib/python3.11/site-packages/keras/src/layers/reshaping/reshape.py:118, in Reshape._fix_unknown_dimension(self, input_shape, output_shape)
    116     output_shape[unknown] = original // known
    117 elif original != known:
--> 118     raise ValueError(msg)
    119 return output_shape

ValueError: Exception encountered when calling layer 'reshape_6' (type Reshape).

total size of new array must be unchanged, input_shape = [20], output_shape = [40, 1]

Call arguments received by layer 'reshape_6' (type Reshape):
  • inputs=tf.Tensor(shape=(None, 20), dtype=float64)

Hyperparameter used:

 {'mlstars.custom.timeseries_preprocessing.time_segments_aggregate#1': {'interval': 3600},
   'mlstars.custom.timeseries_preprocessing.rolling_window_sequences#1': {'window_size': 168,
    'target_column': 0,
    'step_size': 1},
   'orion.primitives.tadgan.TadGAN#1': {'epochs': 5,
    'batch_size': 16,
    'latent_dim': 40,
    'verbose': False,
    'input_shape': [168, 16],
    'target_shape': [168, 1]},
   'orion.primitives.tadgan.score_anomalies#1': {'score_window': 24,
    'rec_error_type': 'dtw'},
   'orion.primitives.timeseries_anomalies.find_anomalies#1': {'window_step_size': 24,
    'fixed_threshold': False,
    'window_size': 168}}

[sarahmish]

Hi @LuSchnitt! There are additional parameters that must be changed if you are altering the latent_dim parameter. For one, edit the dense_units to be similar to the latent_dim. In this case, 40.

Full parameters are available here: https://github.com/sintel-dev/Orion/blob/master/orion/primitives/jsons/orion.primitives.tadgan.TadGAN.json

Please let me know if you have any additional questions!