Evaluation error and interpretation in 'pc_metric.py'

[JasonShiii]

Hi, thanks for your work! The application of difussion model on brep file generation is quite interesting!

I'm trying to directly do evaluation on ABC dataset, here's what I did:

1. I use OCC to convert ABC subdataset from .step to .stl
2. Run 'sample_points.py' to sample point cloud from .stl files and save the point cloud data
3. Run 'pc_metric.py' to evaluate the two point cloud ABC dataset.

The first bug appeared was:

/home/group/cad_codebased/eval_ckpt/Shape$ python pc_metric_original.py  --fake /home/group/cad_code
based/eval_ckpt/Shape/sampled_pointcloud --real /home/group/cad_codebased/eval_ckpt/Shape/sampled_pointcloud2
Traceback (most recent call last):
  File "/home/group/cad_codebased/eval_ckpt/Shape/pc_metric_original.py", line 13, in <module>
    from chamfer_distance import ChamferDistance
ModuleNotFoundError: No module named 'chamfer_distance'

I followed all the steps in README guidance to set up the environment and install the packages. After checking the pip list, I found the installed package was 'chamfer_distance', so I assumed it was a typo in the original code and changed 'chamfer_distance' to 'chamferdist' and re-ran the script. But this time, the error was reported as:

/home/group/cad_codebased/eval_ckpt/Shape$ python pc_metric_original.py  --fake /home/group/cad_codebased/eval_ckpt/Shape/sampled_pointcloud --real /home/group/cad_codebased/eval_ckpt/Shape/sampled_pointcloud2
n_test: 1000, multiplier: 3, repeat times: 10
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5446/5446 [00:00<00:00, 6247.06it/s]
real point clouds: (5446, 2000, 3)
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4929/4929 [00:00<00:00, 6311.65it/s]
fake point clouds: (4929, 2000, 3)
iteration 0...
  0%|                                                                                                                                                             | 0/3000 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/group/cad_codebased/eval_ckpt/Shape/pc_metric_original.py", line 355, in <module>
    main()
  File "/home/group/cad_codebased/eval_ckpt/Shape/pc_metric_original.py", line 339, in main
    result = compute_cov_mmd(rand_sample_pcs, rand_ref_pcs, batch_size=args.batch_size)
  File "/home/group/cad_codebased/eval_ckpt/Shape/pc_metric_original.py", line 84, in compute_cov_mmd
    all_dist = _pairwise_CD(sample_pcs, ref_pcs, batch_size)
  File "/home/group/cad_codebased/eval_ckpt/Shape/pc_metric_original.py", line 68, in _pairwise_CD
    dl, dr, idx1, idx2 = chamfer_dist(sample_batch_exp,ref_batch)
  File "/home/xiangyu/miniconda3/envs/brepgen_env/lib/python3.9/site-packages/torch/_tensor.py", line 1154, in __iter__
    raise TypeError("iteration over a 0-d tensor")
TypeError: iteration over a 0-d tensor

It turned out that the line in _pairwise_CD(sample_pcs, ref_pcs, batch_size):
dl, dr, idx1, idx2 = chamfer_dist(sample_batch_exp,ref_batch)
returned only one dimensional value but it used 4 variables.

I don't know if it was because I didn't pre-process the data in a right way, or my assumption that chamferdist and chamferdist are the same was incorrect, or it's a bug in the original code. Do you have any idea on it?

But anyway, I then modified the _pairwise_CD() function as follows:

def _pairwise_CD(sample_pcs, ref_pcs, batch_size):
    print("pairwise chamfer distance...")
    print("sample_pcs.shape", sample_pcs.shape)
    print("ref_pcs.shape", ref_pcs.shape)
    N_sample = sample_pcs.shape[0]
    N_ref = ref_pcs.shape[0]
    all_cd = []
    all_emd = []
    iterator = range(N_sample)
    print("iterator", iterator)
    matched_gt = []
    pbar = tqdm(iterator)
    chamfer_dist = ChamferDistance()

    for sample_b_start in pbar:
        sample_batch = sample_pcs[sample_b_start]
        # print("sample_batch.shape", sample_batch.shape)
        cd_lst = []
        emd_lst = []
        for ref_b_start in range(0, N_ref, batch_size):
            ref_b_end = min(N_ref, ref_b_start + batch_size)
            ref_batch = ref_pcs[ref_b_start:ref_b_end]
            # print("ref_batch.shape", ref_batch.shape)

            batch_size_ref = ref_batch.size(0)
            sample_batch_exp = sample_batch.view(1, -1, 3).expand(batch_size_ref, -1, -1)
            sample_batch_exp = sample_batch_exp.contiguous()
            # print("sampe_batch_exp.shape",sample_batch_exp.shape)
            # print("chamfer_dist",chamfer_dist(sample_batch_exp, ref_batch).shape, chamfer_dist(sample_batch_exp, ref_batch))   #original code
            # print("chamfer_dist",chamfer_dist(sample_batch_exp, ref_batch, batch_reduction=None, point_reduction=None).shape, chamfer_dist(sample_batch_exp, ref_batch, batch_reduction=None, point_reduction=None))
            # print(type(chamfer_dist(sample_batch_exp, ref_batch, batch_reduction=None, point_reduction=None)))
            # dl, dr, idx1, idx2 = chamfer_dist(sample_batch_exp,ref_batch)  #original code
            dl = chamfer_dist(sample_batch_exp,ref_batch, batch_reduction=None, point_reduction=None)
            # print("dl.shape",dl.shape)
            # cd_lst.append((dl.mean(dim=1) + dr.mean(dim=1)).view(1, -1))  #original code
            cd_lst.append((dl.mean(dim=1)).view(1, -1))

        cd_lst = torch.cat(cd_lst, dim=1)
        all_cd.append(cd_lst)

        hit = np.argmin(cd_lst.detach().cpu().numpy()[0])
        matched_gt.append(hit)
        pbar.set_postfix({"cov": len(np.unique(matched_gt)) * 1.0 / N_ref})

    all_cd = torch.cat(all_cd, dim=0)  # N_sample, N_ref
    print("all_cd.shape", all_cd.shape, all_cd)
    return all_cd

This time the code rans successfully and got the following result:

/home/group/cad_codebased/eval_ckpt/Shape$ python pc_metric_original.py  --fake /home/group/cad_codebased/eval_ckpt/Shape/sampled_pointcloud --real /home/group/cad_codebased/eval_ckpt/Shape/sampled_pointcloud2
n_test: 1000, multiplier: 3, repeat times: 10
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5446/5446 [00:00<00:00, 6359.25it/s]
real point clouds: (5446, 2000, 3)
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4929/4929 [00:00<00:00, 6257.02it/s]
fake point clouds: (4929, 2000, 3)
iteration 0...
pairwise chamfer distance...
sample_pcs.shape torch.Size([3000, 2000, 3])
ref_pcs.shape torch.Size([1000, 2000, 3])
iterator range(0, 3000)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3000/3000 [01:17<00:00, 38.48it/s, cov=0.673]
all_cd.shape torch.Size([3000, 1000]) tensor([[0.2168, 0.2938, 0.0885,  ..., 0.0519, 0.1731, 0.0713],
        [0.1101, 0.3016, 0.1435,  ..., 0.0908, 0.0073, 0.3110],
        [0.3095, 0.4385, 0.0653,  ..., 0.0629, 0.1822, 0.0713],
        ...,
        [0.2182, 0.8228, 0.0326,  ..., 0.0546, 0.2333, 0.0794],
        [0.2023, 0.7430, 0.0146,  ..., 0.0200, 0.2030, 0.0156],
        [0.3126, 0.4789, 0.0630,  ..., 0.0571, 0.1900, 0.0595]],
       device='cuda:0')
{'MMD-CD': 0.0029969392344355583, 'COV-CD': 0.6729999780654907, 'JSD': 0.010725033826304298}
...

Since here the fake dataset and real dataset I passed in for python pc_metric.py --fake sampled_pointcloud --real deepcad_test_pcd were two irrelavant dataset, there should not be much similarity or relation between them. But I wonder if the generated value of those metrics make sense or valid because they're obtained differently from the way of original code.

[tb2-sy]

please install this chamfer_distance

[Peter-Panhongyu]

This is an automatic reply, confirming that your e-mail was received.Thank you！Hongyu PanEmail: ***@***.***:+86-17353995408