Summary:
DPer example have been creating multiple copies of the transform config in net
defition till this moment, that resulted in the fact that I've hit the limit of
ProtoBuf (64MB) for a certain Task requests (especially visible because of the
ValidationPipeline that I was adding).
After this diff we're going to store SigridTransforms in one instance per
machine for training (or 1 instance per reading).
Difference in sizes of the plans for some simple SparseNN model ~30 MB (even including the fact that second model have validation plan as well).
TODO: Do similar logic for NNPreProc as well (it's also pretty large).
Reviewed By: dzhulgakov
Differential Revision: D4441441
fbshipit-source-id: 4452dd86a4dc49b2c7f5b7642f443aed5720b047
Summary: As title. We want to have request_only net which runs on user_only sparse features. Submitting to get early feedback.
Reviewed By: dzhulgakov
Differential Revision: D4282783
fbshipit-source-id: 71241bf5444550075884c788c2da4783659bc1e0
Summary:
We want to implement request only net and to do this we decided to split the work into two parts. The first part will propagate required metadata and the second part will cut the nets properly.
This diff is to propagate request_only metadata across the layers.
A few notes about implementation:
- Each layer contains a field request_only which can be set based on the input_record. If all the scalars from the input_record are marked request_only we mark a layer as request_only;
- Sparse-To-Dense layer sets request_only metadata;
- SigridTransformation and SparseLookup layers propagate request_only status;
- As for now we join request_only and other sparse features together in input_record, but ideally we may want to separate this, because request_only should be served separately;
Reviewed By: xianjiec
Differential Revision: D4259505
fbshipit-source-id: db8a30ef92cba84f1a843981b9dde3a8b9633608
