Skip to content

An error when training #6

@Scxw010516

Description

@Scxw010516

Thank you for your great work!
When I run the training code, I get an error like the following

Traceback (most recent call last):
  File "C:\Apps\Python39\lib\site-packages\torch\multiprocessing\spawn.py", line 69, in _wrap
    fn(i, *args)
  File "D:\Projects\GRAM\train.py", line 33, in train
    training_process(rank, world_size, opt, device)
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "C:\Apps\Python39\lib\site-packages\torch\nn\parallel\distributed.py", line 1110, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])  # type: ignore[index]
  File "C:\Apps\Python39\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Projects\GRAM\generators\generators.py", line 112, in forward
    img, _ = self.renderer.render(self._intersections, self._volume(z, truncation_psi), img_size, camera_origin, camera_pos, fov, ray_start, ray_end, z.device)
  File "D:\Projects\GRAM\generators\renderers\manifold_renderer.py", line 146, in render
    transformed_points_sample, transformed_ray_directions, transformed_ray_origins, _ = transform_sampled_points(points_cam, rays_d_cam, camera_origin, camera_pos, device=device)
  File "D:\Projects\GRAM\generators\renderers\manifold_renderer.py", line 92, in transform_sampled_points
    cam2world_matrix = create_cam2world_matrix(forward_vector, camera_origin, device=device)
  File "D:\Projects\GRAM\generators\renderers\manifold_renderer.py", line 123, in create_cam2world_matrix
    cam2world = translation_matrix @ rotation_matrix
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`

Please help solve,thank you very much!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions