Data parallelism across multiple GPUs

[dennybritz]

Allow the user to replicate the model on multiple GPUs. Still WIP and untested.

[dennybritz]

This code works, but it is currently very slow. Need to very the op placement on different GPUs to figure out why it is slow.

[dennybritz]

Ref #44

[dennybritz]

A few things:

• Variables need to be placed on CPU
• Optimizer ops and preprocessing need to be on CPU

It's probably cleaner to put this into the Estimator class. For example, subclass estimator and add support for model replicas.

[bhack]

See also tensorflow/tensorflow#2126

[SvensBigData]

I get the following error on this branch:

InvalidArgumentError (see above for traceback): Cannot assign a device to node 'save/ShardedFilename_1': Could not satisfy explicit device specification '/device:GPU:1' because no supported kernel for GPU devices is available.
Colocation Debug Info:
Colocation group had the following types and devices:
Identity: CPU
ShardedFilename: CPU
[[Node: save/ShardedFilename_1 = ShardedFilename[_device="/device:GPU:1"](save/StringJoin, save/ShardedFilename_1/shard, save/num_shards)]]

[hrishikeshvganu]

@dennybritz : wanted to know if the branch is usable now. If there are some specific ToDos I can help with the implementation