`Sparsity` as a term doesn't reflect the tools that are developed by the AO. The `torch/ao/sparsity` also has utilities for structured pruning, which internally we always referred to as just "pruning". To avoid any confusion, we renamed `Sparsity` to `Prune`. We will not be introducing the backwards compatibility, as so far this toolset was kept under silent development.
This change will reflect the changes in the documentation as well.
**TODO:**
- [ ] Change the tutorials
- [ ] Confirm no bc-breakages
- [ ] Reflect the changes in the trackers and RFC docs
Fixes #ISSUE_NUMBER
Pull Request resolved: https://github.com/pytorch/pytorch/pull/84867
Approved by: https://github.com/supriyar
The stored mask is dumped as `torch.sparse_coo` while serializing. While restoring the state,
the mask is converted to a dense tensor again.
Test Plan:
```python test/test_ao_sparsity.py TestActivationSparsifier```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/82181
Approved by: https://github.com/z-a-f
Implemented dumping and loading of state_dicts and __get_state__ and __set_state__ functions.
hook and layer are removed from the data_groups dictionary before serializing.
In the future, might have to treat functions differently before serializing. Currently, it is being
treated similar to other types while serializing.
Test Plan:
```python test/test_ao_sparsity.py TestActivationSparsifier```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80890
Approved by: https://github.com/z-a-f
Unregisters aggreagate hook that was applied earlier and registers sparsification hooks.
The sparsification hook will apply the mask to the activations before it is fed into the
attached layer.
Test Plan:
```python test/test_ao_sparsity.py TestActivationSparsifier```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80889
Approved by: https://github.com/z-a-f
The step() internally calls the update_mask() function for each layer
The update_mask() applies reduce_fn and mask_fn to compute the sparsification mask.
Note:
the reduce_fn and mask_fn is called for each feature, dim over the data
Test Plan:
```python test/test_ao_sparsity.py TestActivationSparsifier```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80888
Approved by: https://github.com/z-a-f
The register_layer() attaches a pre-forward hook to the layer to aggregate
activations over time. The mask shape is also inferred here.
The get_mask() returns the computed mask associated to the attached layer.
The mask is
- a torch tensor is features for that layer is None.
- a list of torch tensors for each feature, otherwise
Test Plan:
```python test/test_ao_sparsity.py TestActivationSparsifier```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80887
Approved by: https://github.com/z-a-f
The Activation sparsifier class aims to sparsify/prune activations in a neural
network. The idea is to attach the sparsifier to a layer (or layers) and it
zeroes out the activations based on the mask_fn (or sparsification function)
input by the user.
The mask_fn is applied once all the inputs are aggregated and reduced i.e.
mask = mask_fn(reduce_fn(aggregate_fn(activations)))
Note::
The sparsification mask is computed on the input **before it goes through the attached layer**.
Test Plan:
```python test/test_ao_sparsity.py TestActivationSparsifier```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80886
Approved by: https://github.com/HDCharles
