U Network(pix2pix)

Introduction

pix2pix network is very successful in transforming images of one style into images of another style. The full paper and github page can be found here. The network concatenate intermediate encoder layer outputs with decoder layer outputs and then feed into another decoder. Therefore decoder layer gain information from both the encoded vectors (features of the image) and decoded vectors (features of the output image classes). It seems like such network can work with any style transfers as long as there is sufficient training data and that's the hypothesis I want to test and hopefully prove with empirical results.

Tests on edge->cat, edge->anime character color filling, drawing->building, day->night are successful. Since acquiring test data with labels are difficult, one another approach is using CycleGAN with pix2pix on semi-supervised learning.