Loss of sharpness with larger resolutions

[sapoluri]

The compensated images obtained after inferencing lose their sharpness. As I understand, autoencoders due to their nature of encoding and then decoding, will result in the loss of detail. Do you observe this behavior as well in your experiments? Is it possible to somehow reduce this loss? A way that we preserve the edge detail or perhaps combine the original and the generated compensation image to get the best of both worlds.

I am curious as to what the results in your lab looked like with larger resolutions. Although compensation works to hide the screen imperfections, the loss in sharpness results in an objective observer preferring the sharp uncompensated image over a compensated image.

[BingyaoHuang]

@sapoluri
Many factors can affect sharpness, e.g., underfitting, training data and loss function, etc. I don't think it's due to the autoencoder structure. In fact, some super resolution networks use autoencoder-like structures and generate sharp high resolution images, e.g., X. Mao, C. Shen, and Y.-B. Yang. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In NIPS 2016.

We have not tried larger resolutions. Can you post some exemplar input and output images?

[sapoluri]

share.zip
I have left a readme file in there explaining the different folders and the images contained within.

I have trained the model with 250 images of 800x600 size. All the metrics like SSIM, PSNR etc. are close to the results posted in the paper. The only issue is the sharpness of the images overall degrades.

[BingyaoHuang]

@sapoluri
Here is what I find:
In train_captured, the camera resolution is 800x600 and the images in Original are 800x600 (I assume your projector is set to this resolution).

If above information is correct, a potential problem is in train_original, where CompenNet's projector input images are resized from 256x256 to 800x600, and these images are not sharp as those in Original. Try using the same sharp images as Original for training.

[sapoluri]

I do set the projector to 800x600 and regarding the resized images, I did initially try with original 800x600 images for training except with just 75 images rather than the 250 with the resized images. I will retry with 250 original 800x600 images and see how the results look.

I am assuming the problem with the resized images is that the autoencoder might be learning the blurred look of the resized images. Is that your thought as well?

[BingyaoHuang]

@sapoluri
Yes. And after you try with 250 original 800x600 images, can you post the training curve and validation results in the visdom web page? Just want to make sure the model is well trained.

[sapoluri]

Will do