Summary: Caffe2: allow nets that don't use all input in net.ClonePartial
Differential Revision: D5535564
fbshipit-source-id: 0ec8fb3ade4d7d6cd4a702c9c265d9c77f27a627
Summary: Add tensor inference function for squeeze, refactor a bit
Reviewed By: asaadaldien
Differential Revision: D5518880
fbshipit-source-id: 5b8cb9154f5f777d4be3612a96d7ed76a9068c0c
Summary:
Feed team uses distributed training and wants to also use transfer learning.
Currently, transfer learning implements by overwriting the layer parameter
initializer. Therefore, PS builder can't infer correctly the parameter shape.
To fix this, add a field 'shape' in `layer_parameter` and set the shape if we
overwrite its initializer.
We also enforce the check of parameter shape between the original initializer
and the loaded blob. (this adds extra cost)
Differential Revision: D5520541
fbshipit-source-id: 80547dbd328b3f6cbfcea0b2daaf4004703dfe81
Summary: Several refinements to seq2seq example code, including support for multilayer LSTM.
Reviewed By: jamesr66a
Differential Revision: D5460372
fbshipit-source-id: d2eabf6aa9a5b5df7bbc341fd99c4e7d8322e717
Summary: Memonger did not properly track the number of times a blob output has to be produced before an operator can be visited. Actually I remember fixing this before, but well. This bug was manifested in Priya's model, so thanks prigoyal, and benz's model verifier nicely caught the wrong output.
Reviewed By: asaadaldien
Differential Revision: D5524912
fbshipit-source-id: 10f4d7056b84aba0274a918af508ea043e6026f9
Summary: This method runs a train net multiple times therefore enables testing layers with iteration-dependent behavior.
Differential Revision: D5493750
fbshipit-source-id: a7fb967a66f799aaf82acfadc4ecf66e0744da20
Summary: One of my workflows was stuck before everstore/hive data input was experiencing networking issues (No route to host etc.). But it is hard to know this is happening because the errors were logged to stdout. Anyway, added a simple logging to warn if the data workers enqueue thread is not getting new data for over 10 secs.
Reviewed By: panshen1
Differential Revision: D5522816
fbshipit-source-id: a036c4afdfbbafea130a4251c1ca02c138d19a83
Summary: The diff adds support for rank_loss operator to support computing loss for multiple sessions (batch).
Reviewed By: kittipatv
Differential Revision: D5515465
fbshipit-source-id: 55a01cd5ad21eaeae82875ad136c392fed0dbb26
Summary:
Optimised SparseLengthsSum (fp32) for now
1) Specialized reducer
2) created fast routine with prefetches, loop unrolling, block specailization and register tiling
3) added more variety of block sizes to segment_ops_test.py
Reviewed By: Yangqing
Differential Revision: D5392472
fbshipit-source-id: 8ed9baf1b12ec05bd391cabb390024e6bc60a6f6
Summary: to support an operation needed by D5507205
Reviewed By: xianjiec
Differential Revision: D5512522
fbshipit-source-id: a9b3a668c28eff71d1e106dbbb572184df4a7638
Summary:
The renames were only being applied to the main net, if step_net has an
external input that is part of renames, running the model would fail with 'blob
not found in workspace' error.
Differential Revision: D5511953
fbshipit-source-id: ba262a094c3263978dfe173f2cab00301131b57f
Summary:
Updated the semi-random layer model for multi-layer models using semi-random layers.
Notable changes:
- Input and outputs for the semi-random layer is now a Struct with "full" and "random" components
- Flag was added to choose to initialize output schema in Arc Cosine or not (if output schema initialization will happen in Semi Random layer)
Reviewed By: chocjy
Differential Revision: D5496034
fbshipit-source-id: 5245e287a5b1cbffd5e8d2e3da31477c65b41e04
Summary: It is common mistake to create test/validation model with init_params=True. When its param_init_net is run, it will overwrite training models' params, and with DPM, those won't be synchronized to all GPUs. I don't want to make this an assertion yet, since it might break people's trainers (it is ok to have init_params=True if you never run the param_init_net...).
Reviewed By: asaadaldien
Differential Revision: D5509963
fbshipit-source-id: 63b1a16ec0af96e3790e226850f6e0e64689143f
Summary:
As per rushabhmshah99 request: he wants to append a pre-trained model (without training that) to the model.
So added data_parallel_model.ConvertNetForDevice() to enable that. The unit test shows example how to use this with
AppendNet, and I also added a blurb to the function.
Differential Revision: D5503335
fbshipit-source-id: b2a5db5c1739dc97f46dd0d7606ed555d99255b8
Summary: Use romain-intel's ContextFactory to create common worlds from existing common worlds, thus bypassing KV store completely. Changed data_parallel_model to automatically find if there is already a CW we can work. CreateCommonWorldOp takes optional second parameter, which is existing CW.
Reviewed By: andrewwdye
Differential Revision: D5494956
fbshipit-source-id: 5f7a840bcd5fe4ea756fafeacc746bc2cf5078b0
Summary:
Nothing gets changed - this would allow us to more easily deal with build
systems. Also now everything that is MKL related lives under mkl/.
Reviewed By: dzhulgakov
Differential Revision: D5505157
fbshipit-source-id: ddb2e6ac290a146a7cb495da23bb0e5b5594bd2a
Summary:
Use smaller step size for GradientChecks and pass seed to help reproducing the
test from logged inputs.
Reviewed By: Yangqing
Differential Revision: D5505698
fbshipit-source-id: fc308efe72d535695ba628944aee1913ba16b2f1
Summary:
The original issue was that the initialized parameters for randomized layers (Arc Cosine and Semi-Random) were not fixed across distributed runs of the layers. Moreover, as the weights are initialized as (constant) parameters, when the layer is added to the preprocessing part, these weights won't be saved after training since they don't exist on the trainer.
I fixed the issue here by building an option to add the randomized parameters to the model global constants so that the same parameter values can be accessed. Also, the parameters can be saved when the training is finished.
In this diff, I've:
- Updated randomized parameters to be added as a global constant across distributed runs of Arc Cosine Feature Map and Semi Random Feature layers
- Updated unit tests
- Ran an end-to-end test, enabling multiple readers to test the fixed issue
Reviewed By: chocjy
Differential Revision: D5483372
fbshipit-source-id: b4617f9ffc1c414d5a381dbded723a31a8be3ccd
Summary:
Moved distance_op_test from hypothesis_test to distance_op_test and
refactored
Reviewed By: akyrola, asaadaldien
Differential Revision: D5495104
fbshipit-source-id: 4a90c75eabeb380ae9d150d6258e9b5b0fbfc5ca
Summary:
Data type conversion between Numpy Array and Caffe2 Tensor currently only support 3 types: FLOAT, DOUBLE and INT32. Support 8bit and 16bit date types will help reduce the model size in some circumstance. I benefit from this to reduce size of a data set from 8GB to 1GB by using INT8.
Closes https://github.com/caffe2/caffe2/pull/930
Reviewed By: Yangqing
Differential Revision: D5440929
Pulled By: akyrola
fbshipit-source-id: 3762da1d845e62a13ba384d1c144328b19dd663b
Summary: When creating parameters for modelhelper, we should use create_param instead of using param_init_net and model.params directly. The diff rewrite some of these cases in rnn_cell.py in order to make model._parameter_info and model.params consistent.
Reviewed By: kittipatv
Differential Revision: D5477724
fbshipit-source-id: 28c4aaf8f98d9d89125af6a42ad328008f0079e1
Summary:
Need it for some reference comparison for c2isl.
Also there's an argument that it might be faster on GPU with int32. Doesn't seem to be the case now, but haven't tested with Jeff's changes yet.
Reviewed By: kennyhorror
Differential Revision: D5405482
fbshipit-source-id: dc1a983dce5f06f1111c5634ec475647c94848cc
Summary:
In order to get dimensions right, correctly identify gradients, etc., DropoutCell should call the _prepare_output and _prepare_output_sequence methods of its internal cell for its own such methods.
This bug was identified by NVIDIA intern Syed Tousif Ahmed.
Reviewed By: akyrola
Differential Revision: D5483082
fbshipit-source-id: f6df5b4a0502ed0771056638aab219fb5cc7d964
Summary: TSIA - this makes it a bit easy to benchmark sparse lengths sum.
Reviewed By: dzhulgakov
Differential Revision: D5477844
fbshipit-source-id: 89e25c5e0dbf3538877ba1a9abc75a10abfa2757
Summary:
This is needed for us to do more fine grained dispatch based on CPU arch, so
I figured we should just add it. Can help Dima and Misha doing optimization
I think?
Reviewed By: dzhulgakov
Differential Revision: D5477444
fbshipit-source-id: 48aaf8bd799e9755493cd51c793ceec080a8846c
Summary:
For RNN attention, we should not include the invalid parts of the encoder output (based on encoder_lengths) in the computation. This diff accomplishes that by forcing logits for those positions to be negative infinity.
Note that the this step can be bypassed by passing encoder_lengths=None, which is what we do for beam search, thus incurring no extra overhead for inference.
Reviewed By: jamesr66a
Differential Revision: D5402547
fbshipit-source-id: 1863d6050b5129e4df829c6357f0aa9ded0715dc
Summary: fixing the case where the init net will initialize same blob twice. I made an exception by allowing inplace blob among ops if the blob keeps on the same device. This should fix this problem in a generalized way as most of our training is only on CPU now.
Reviewed By: dzhulgakov
Differential Revision: D5450564
fbshipit-source-id: 525c4c9a2e5216a70dbd1229da2d9f8a58b89e47
Summary: Saving 2 nets at offline training and loading the correct net the user want. The keep_device=false will help us load gpu blobs to CPU memory.
Reviewed By: dzhulgakov
Differential Revision: D5396689
fbshipit-source-id: ff26bf3759856b07f3a1bbefac4a1e613a8a02e1
Summary:
===Update log 7/10===
We are now restrained from problem of connection. Will post if this problem does not fix in 2hrs.
===Update 7/6===
Luke is experimenting on the convergence of this diff. Hopefully he could present results next week
Right now this is not affecting our original CPU training pipeline because the loading op is still correct in CPU situation now.
I will need final test to make sure. But that is now blocked by log device issue t19952135
I will do CPU/GPU nets saved in a separate diff.
====Update before 7.4====
It's actually working! Include local run screenshot
{F67959016}
dogscience
Reviewed By: dzhulgakov
Differential Revision: D5307058
fbshipit-source-id: cad5d9324c239419530f4b120392ec2ccbb72280
Summary: CopyGPUToGPU does not exist. Copy seems to do the trick. Didn't go into details of how copy works, not sure if it ends up triggering UVA.
Reviewed By: akyrola
Differential Revision: D5471014
fbshipit-source-id: d8bc1aed9b19070c92f3ffc76f5617bdd0054563
