Questions about smooth upsample and weighted downsample

[lisc199]

I am trying to reproduce the paper. And I have some questions:

1. How to perform smooth upsample when s=16?
   For a dowmsampling factor s=64, I can use 4x4 triangle kernel three times. But for s=16, should I use 4x4 twice or use [4x4, 2x2, 2x2] or somethind else?

2. How can I perform modified guided filter after model inference?
   In my opinion, the model output is already a low-resolution map. How to apply weighted downsample in this case? How to use confidence map which is gotten by formula (1) and (2) if I don't use weight downsample? I guess that the model's output will be down-sampled to 64x, is that right? But this process increases the computational workload.

[lisc199]

About question 1, I used [2x2, 2x2, 4x4] for upsampling, and get a good result. But I found another question:

The most value of calculated result A and b is binary, i.e. 0 (or lower than 0) or 1 (or larger than 1), I think it might have been caused by weighted_downsample because the result of ldl_decompositoin and matrix inversion and matrix multiply are same.

Have you ever encountered this problem and how did you solove it?

My weighted_downsample code as this:

XC = X * C
xc_ds = bilinear_downsample(xc, s)
c_ds = bilinear_downsample(c, s)
res = xc_ds / c_ds

where c is the confidence map and x is the input image. I perfromed it by pytorch.

I compared downsampled weighted_downsample image and bilinear_downsample image. It is hard to get the difference with the eyes.

[orlyliba]

1. For s=16, you can try both options that you mentioned and see which results you like best.
   Regarding the followup, please clarify, are you seeing that A and b are mostly binary?
   What values do you see after weighted_downsample?
   Your pseudo code looks generally ok, but are the multiplication and division element-wise?
   You can try not using the weighted guided filter for a first step and then add the weights, the difference depends on the weights and in some cases there won't be a lot of difference..

2. You can upsample your model output with nearest neighbors or bilinear and then apply the guided filter. The confidence map is only relevant if you do the weighted guided filter, which includes weighted downsampling.

[lisc199]

Regarding the followup, please clarify, are you seeing that A and b are mostly binary?

I tested on an image which has tree, building and sky. The resulotion is 3648*2736. In my reproducing, there only 3.15% of pixels which values are in (-1, 1) in calculated A. In b, the number is 5.67%. However, in the traditional guided filter that I reproduced, the number is 41.92% and 22.35%.

I save calculated A and b as images which clipped to (0, 255). They look like binary images.

What values do you see after weighted_downsample?

Just look like bilinear downsample

Your pseudo code looks generally ok, but are the multiplication and division element-wise?

Yes, I am sure they are element-wise operator.

You can upsample your model output with nearest neighbors or bilinear and then apply the guided filter.

Do you mean using upsampling first and then applying the weighted guided filter?

[orlyliba]

Do you mean using upsampling first and then applying the weighted guided filter?

Yes

[yggs1401]

@lisc199 have you managed to implement the wighted_guided_filter?

[lisc199]

@yggs1401 No. I didn't reproduce the results of the paper. But if you use Boxfilter instead of weighted_downsample, you can get a result which is almost the same.

[yggs1401]

@lisc199 Can you provide your code and results?

[orlyliba]

Hi! Please see this code submission for the guided filter: 8a35938