Summary:
As described in T19378176 by kittipatv, in this diff, we fix the issue of __getitem__() of schema.List.
For example, given Map(int32, float) (Map is a special List), field_names() will return "lengths", "values:keys", & "values:values". "values:keys" and "values:values" are not accessible via __getitem__(). __getitem__() bypasses the values prefix and directly access the fields in the map. Other APIs (e.g., _SchemaNode & dataset_ops) expect "values:keys" and "values:values" as it simplifies traversal logic. Therefore, we should keep field_names() as is and fix __getitem__().
Reviewed By: kittipatv
Differential Revision: D5251657
fbshipit-source-id: 1acfb8d6e53e286eb866cf5ddab01d2dce97e1d2
Summary:
The current version of schema.py has a Metadata class with three fields. The default for it is set to
four Nones. This is just changing that to three Nones so that the number of default values matches the number
of actual fields.
Reviewed By: kennyhorror
Differential Revision: D5250463
fbshipit-source-id: 42e5650d270f5f63662614d8445b4819ed370dec
Summary: Previous implementation relied on the order of fields for some reason.
Reviewed By: azzolini
Differential Revision: D5164478
fbshipit-source-id: 12717310860584e18ce4ca67d0bd5048354cdc0a
Summary:
fixing missing future package issue.
Recently we found some of our users does not have future module support. So we might need a try/catch wrapper around all past import
Reviewed By: Yangqing
Differential Revision: D5183547
fbshipit-source-id: 262fdf2940ee1be4454bf0b0abb9e6a0f1a0ee82
Summary:
Split the Caffe2 memory based model into to parts
- Dimension reduction MLP
- DNN with concatenation of memory and obj feature
Currently only implement simple mean
Differential Revision: D4866825
fbshipit-source-id: d2f6813402513ec9af30dbe29a50593e2d3cdb3b
Summary: This diff is one step towards enabling python 3 build by making it be more diligent in its handling of strings.
Reviewed By: salexspb
Differential Revision: D4893083
fbshipit-source-id: 28b8adf3280e8d1f0a7dc9b0fee5ad53f2fada57
Summary: The code snippet below is invalid in the add unit test is invalid but it may or may not cause exception. Disable the syntax so people don't accidentally use it.
Reviewed By: dzhulgakov
Differential Revision: D4985030
fbshipit-source-id: ffa2b26f7b29128b196aba1b1001a97c87e381cf
Summary: I ran into this earlier and the debug messages were not helpful enuogh
Reviewed By: kennyhorror
Differential Revision: D4985754
fbshipit-source-id: b3d12b5e2cfa1b54fca9126768c84c902664ef28
Summary: Calling `set()` or `set_value()` on Scalar is dangerous as something might be holding a reference to it. This is especially true with `LayerModel`, where instantiation is delayed. The code may still run but it will produce unexpected results, i.e., values maybe written to the wrong blob.
Reviewed By: kennyhorror
Differential Revision: D4955366
fbshipit-source-id: f5e8694a9a411ee319ca9f39a0fed632d180b8a5
Summary:
This diff contains the following changes:
- implementing __repr__ on Field types; this makes it a little easier to see what broken in the unit tests
- preserve the shape of ndarray input to schema; previously, empty and scalar arrays lose their shape, while other keeps the shape.
- type-checking ndarray input; this ensures basic integrety of schema
Reviewed By: xianjiec
Differential Revision: D4913030
fbshipit-source-id: bd0f6b8722d95bfe800edf98ba05029c5b99d2af
Summary: `not field` calls `__len__()`, causing the field to appear to be missing even when it's not
Differential Revision: D4910587
fbshipit-source-id: bc2b2fadab96571ae43c4af97b30e50c084437af
Summary:
as desc.
small fix in the feature_proc layer for the case when we only have one preproc type
Reviewed By: chocjy
Differential Revision: D4908933
fbshipit-source-id: 1338048fc395f85c3724721a9996ad1ee51f0f20
Summary:
Add distributed training to dper2 and keep the dper1 working.
* Created a ModelDelegator to wrap ModelHelper and LayerModelHelper to mitigate the difference.
* To get the average length for sparse feature, I extracted some information in feature_processor. There should be some better way to do it after we have new compute_meta.
* metric right now only runs on the first trainer.
* The model is saved correctly for evaluation. But I'm still not sure how to handle the weights for adagrad.
Reviewed By: kennyhorror
Differential Revision: D4767745
fbshipit-source-id: 0559d264827a7fd9327071e8367d1e84a936bea9
Summary:
D4690225 added support for nested field name lookup in nested
`schema.Struct`s. It would throw a KeyError if trying to access a nested
`List`s field. Writing the lookup recursively avoids the need to enumerate
all complex field types in the lookup.
Differential Revision: D4719755
fbshipit-source-id: 37c87a32d730f0f45f72fb20894da3e32f820999
Summary:
1. migrate the basic mtml model to dper 2
2. test dper 2 mtml model
3. test all optimizers
Reviewed By: kittipatv
Differential Revision: D4680215
fbshipit-source-id: 7aac5c59bdac22fcad8ed869b98e9e62dca1d337
Summary:
We are having more and more nested Struct schema. There is increasing need to get/adda field by nested name, e.g., for the following nest Struct schema:
st = Struct(
('a': Scalar()),
('b': Struct(
('c': Scalar()),
)),
)
We may want to get the field "b:c" and/or insert a new field "b:x". The immediate need is for dper2 metrics.
This diff is to achieve this.
Reviewed By: kittipatv
Differential Revision: D4690225
fbshipit-source-id: 71d4a74b36bd1228a2fefd901db2f200602152b7
Summary: Whe debug using LayerModelHelper, adding Print to model will trigger this assert.
Reviewed By: xianjiec
Differential Revision: D4687859
fbshipit-source-id: 6932e38f8dd17ba0b80da18a20943ecdb2e8af0a
Summary:
This diff is trying to address one of the concerns that Xianjie have had - requirements create a layer for all operators and attach pass shapes and other info around.
The basic idea of the diff:
1. Try to create a layer with a given name, but if it's not available try to fallback on operator with that name (that is expected to have no parameters).
2. For all operators that we're adding through this functional style of creation - try to use C2 Shape/Type inference logic to get output type. If we fail to get - it just return untyped record and expect user to annotate it when it's really needed.
Reviewed By: xianjiec
Differential Revision: D4408771
fbshipit-source-id: aced7487571940d726424269970df0eb62670c39
Summary: Do I understand correctly? It must be of size 1 for sigrid
Reviewed By: kennyhorror
Differential Revision: D4576541
fbshipit-source-id: 92fa8dc62e36ff095e14cceeb80b03c0028f5695
Summary:
Remove the use of `NextName` in layer model helper, so that the same function return `model_helper` that should construct identical `Net`, when under the same NameScope.
The `NextScopedBlob` should only take effect when there is real name conflicting, otherwise it returns ScopedBlobReference.
This is critical for parameter blobs. In long run, we need to be able to specify parameter blobs more explicitly. (kennyhorror is working on this). This solution works in short term for e.g., two tower sparse nn models.
Reviewed By: kennyhorror
Differential Revision: D4555423
fbshipit-source-id: 2c4b99a61392e5d51aa878f7346466a8f14be187
Summary:
We want to train models with user sequence data for mobile side ranking.
The operators are for preprocessing the sequence based data. They read in a sequence with a batch and convert the examples with different method.
I also add a new loader for connecting the operator to existing trainers
Differential Revision: D4485411
fbshipit-source-id: 0cf17206704995f2ce079e1594607bea70b1ed0c
Summary:
Ievgen ran into this bug with his dper work - we didn't preserve metadata on lengths field.
Also, we didn't take keep_blobs into account for List's main field. Now fixed.
Also, reformat the file to be nice.
Differential Revision: D4357859
fbshipit-source-id: 1c26c533a10d38afab13b46ccbcb541f5fa9074a
Summary: att. part of the effort to unify loader configueration.
Differential Revision: D4342147
fbshipit-source-id: bb021112f61d4838b0ccc7a5a8bcaf272cb35cd8
Summary:
We want to implement request only net and to do this we decided to split the work into two parts. The first part will propagate required metadata and the second part will cut the nets properly.
This diff is to propagate request_only metadata across the layers.
A few notes about implementation:
- Each layer contains a field request_only which can be set based on the input_record. If all the scalars from the input_record are marked request_only we mark a layer as request_only;
- Sparse-To-Dense layer sets request_only metadata;
- SigridTransformation and SparseLookup layers propagate request_only status;
- As for now we join request_only and other sparse features together in input_record, but ideally we may want to separate this, because request_only should be served separately;
Reviewed By: xianjiec
Differential Revision: D4259505
fbshipit-source-id: db8a30ef92cba84f1a843981b9dde3a8b9633608
