OSSForks/pytorch
1
0
Fork 0
mirror of https://github.com/zebrajr/pytorch.git synced 2025-12-07 12:21:27 +01:00
Code Issues Actions 38 Packages Projects Releases Wiki Activity
64,482 Commits 8,304 Branches 150 Tags 1.5 GiB
1b51d29b66
Commit Graph

345 Commits

Author SHA1 Message Date
Denis Vieriu
1a0738f599 [MPS] Add support for torch.linalg.cross (#91642) 
* Add support for torch.linalg.cross
* Make use of `metal::cross` for float and half. For the other dtypes implement cross manually

Pull Request resolved: https://github.com/pytorch/pytorch/pull/91642
Approved by: https://github.com/razarmehr, https://github.com/malfet
 2023-01-05 14:48:34 +00:00
Ramin Azarmehr
229f12bf6a [MPS] Implement nan_to_num() for MPS backend (#91110) 
Added a test case, and also enabled it in TestConsistency

Pull Request resolved: https://github.com/pytorch/pytorch/pull/91110
Approved by: https://github.com/malfet, https://github.com/kulinseth
 2023-01-05 02:17:48 +00:00
Ramin Azarmehr
b44d46702a [MPS] Fix correctness issues with Upsample 1D and 2D (#91669) 
- Implemented following new ops: upsample_nearest1d_backward
upsample_nearest_exact1d
upsample_nearest_exact1d_backward
- Moved Upsample code from Shape.mm to Upsample.mm
- Fallback to CPU for nearest mode on Monterey

Pull Request resolved: https://github.com/pytorch/pytorch/pull/91669
Approved by: https://github.com/malfet
 2023-01-05 00:48:54 +00:00
Ramin Azarmehr
7dd28e9e83 [MPS] Fix data type and shape issues in Scatter and Gather ops (#91514) 
- Clean up redundant code and headers
- Move scatter/gather ops from block list to allow list in TestConsistency
Pull Request resolved: https://github.com/pytorch/pytorch/pull/91514
Approved by: https://github.com/kulinseth
 2023-01-04 23:20:01 +00:00
Kulin Seth
fc59664ef4 [MPS] Add Unique and unique_consecutive ops. (#88532) 
Add check for macos 13.0

Fixes #88487

Pull Request resolved: https://github.com/pytorch/pytorch/pull/88532
Approved by: https://github.com/malfet
 2023-01-04 22:15:13 +00:00
Ramin Azarmehr
13de5a0150 [MPS] Fix the right padding bug in Monterey (#91522) 
- Workaround for the bool type bug in padding (needed for both Monterey and Ventura)
- Move the recently fixed padding tests of TestConsistency to AllowList

Pull Request resolved: https://github.com/pytorch/pytorch/pull/91522
Approved by: https://github.com/DenisVieriu97, https://github.com/kulinseth, https://github.com/malfet
 2023-01-04 22:00:37 +00:00
Denis Vieriu
80394bb734 [MPS] Register norm_dtype_out_mps and cdist (#91643) 
Add support for `norm_dtype_out` and `cdist` ops
Pull Request resolved: https://github.com/pytorch/pytorch/pull/91643
Approved by: https://github.com/razarmehr
 2023-01-04 02:20:53 +00:00
Denis Vieriu
38de981e16 [MPS] Add nonzero mps support (#91616) 
Adds nonzero support for mps:

  **Pseudocode**:
  ```
  //
  // inputTensor   = [1,  0,  0,  3]
  // inputNonZero  = [1,  0,  0,  1] (input != 0)
  // scan          = [1,  1,  1,  2] (prefix sum)
  // maskedIndices = [0, -1, -1,  1] (select)
  // coordinates   = [0,  1,  2,  3] (coordinateAlongAxis)
  // scatterResult = [0,  3]         (scatter)
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/91616
Approved by: https://github.com/razarmehr
 2023-01-04 00:02:24 +00:00
Ramin Azarmehr
688e351970 [MPS] Implement MPSGenerator to enable manual random seeding (#91348) 
This patch adds support for creating torch.Generator for MPS device, and enables its functions such as manual_seed, get_state, and set_state.
Fixes #84288 and #84516
Pull Request resolved: https://github.com/pytorch/pytorch/pull/91348
Approved by: https://github.com/malfet, https://github.com/albanD
 2023-01-03 16:01:19 +00:00
Denis Vieriu
f7939b21e1 [MPS] Add bincount support for mps (#91267) 
Add support for bincount on MPS
Pull Request resolved: https://github.com/pytorch/pytorch/pull/91267
Approved by: https://github.com/razarmehr
 2023-01-03 06:01:07 +00:00
Denis Vieriu
dbf96164be [MPS] Add suport for casting updatesTensor directly in scatter (#91197) 
Fixes copies into slices where the input data type is different than the output dtype.

This change removes the cast done before scatter, so we don't have to allocate additional memory to perform the casting. Scatter handles the casting directly now.

device = "mps"
shape = (4, 4)
tensor = torch.randint(10, shape, device=device)
tensor_before = tensor.clone()
res = torch.empty(shape[0], shape[1] * 2, device=device)[:, ::2].copy_(tensor)
torch.testing.assert_close(tensor, tensor_before)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/91197
Approved by: https://github.com/razarmehr
 2023-01-02 16:31:27 +00:00
Denis Vieriu
bdbf188c80 [MPS] Exclude int64 dtype from reduction ops (#91272) 
Reduction ops don't support int64 data type. This PR takes care to assert when int64 is used for min / max reductions ops.
All other integer dtypes are casted to int32.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/91272
Approved by: https://github.com/razarmehr, https://github.com/malfet
 2022-12-23 17:30:42 +00:00
Ramin Azarmehr
6485d2609a [MPS] Fix data type issues in Binary Ops (#91151) 
- Cast to unsigned type when comparing signed vs. unsigned integers
- Refactor and cleanup logaddexp() ops

Pull Request resolved: https://github.com/pytorch/pytorch/pull/91151
Approved by: https://github.com/malfet
 2022-12-23 17:11:55 +00:00
Denis Vieriu
4477a5b691 [MPS] Register unfold key for MPS (#91266) 
Register unfold key for MPS (uses generic implementation that's already existent).
Pull Request resolved: https://github.com/pytorch/pytorch/pull/91266
Approved by: https://github.com/razarmehr
 2022-12-22 21:21:04 +00:00
Nikita Shulga
fd3a7264ae [MPS] Add group_norm[fwd+backward] and mean_var (take 2) (#91190) 
Use Prims to implement group_norm, group_norm_backward and mean_var

Use `torch._ops.ops` instead of `torch.ops` in numerous subpackages in
order to be able to make them importable from `torch/backend/mps/__init__.py` as this alias is defined in
15af4b1cee/torch/__init__.py (L1095)
is executed last during init process.

Add `__all__` to `torch/backends/mps/__init__.py` as well as alias all imports as private

Add `TestNNMPS.test_group_norm_backward` that validates no NaNs are generated during the backward pass

Fixes https://github.com/pytorch/pytorch/issues/88331
Pull Request resolved: https://github.com/pytorch/pytorch/pull/91190
Approved by: https://github.com/albanD
 2022-12-22 08:54:37 +00:00
Denis Vieriu
81a9a0ac07 [MPS] Fix gather for uint8 dtype in index_select (#91047) 
Use int8 instead of uint8 for MPS Gather/Scatter (uint8 is broken in macOS Monterey)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/91047
Approved by: https://github.com/razarmehr
 2022-12-21 19:48:46 +00:00
Li-Huai (Allan) Lin
b7f35e4104 [MPS] Fix index_add with non-f32 inputs (#88542) 
The `multiplicationWithPrimaryTensor` and/or `scatterWithDataTensor` api has issues with handling two f16 tensor inputs, resulting in zeros outputs. With int16 or int64 inputs, there are issues as well.

This PR conditionally casts inputs to f32 if they're not and then casts the output back to the source's datatype.

Fixes #82645.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/88542
Approved by: https://github.com/kulinseth
 2022-12-21 05:31:03 +00:00
Ramin Azarmehr
a274b5b99e [MPS] Fix data type issues in Unary ops (#91120) 
Refactored sigmoid() and log1p()

Pull Request resolved: https://github.com/pytorch/pytorch/pull/91120
Approved by: https://github.com/DenisVieriu97, https://github.com/kulinseth
 2022-12-21 02:42:59 +00:00
Nikita Shulga
dd735b96df [MPS] Fix torch.std/torch.var default/correction handling (#91203) 
If `torch.std`, `torch.var` are invoked without any arguments, it should be assumed that `unbiased` is `True`.

Also, if `correction` parameter is specified it should be use in correction computation.

Test by adding `std` and `var` to consistency tests

Fixes https://github.com/pytorch/pytorch/issues/91198

Pull Request resolved: https://github.com/pytorch/pytorch/pull/91203
Approved by: https://github.com/kit1980
 2022-12-21 02:23:50 +00:00
Ramin Azarmehr
b63f0311a5 [MPS] Add floor_divide() op and its test case (#91126) 
Pull Request resolved: https://github.com/pytorch/pytorch/pull/91126
Approved by: https://github.com/malfet
 2022-12-20 17:02:29 +00:00
Kulin Seth
8ecb49b8fb [MPS] Add Inverse op. (#90428) 
Pull Request resolved: https://github.com/pytorch/pytorch/pull/90428
Approved by: https://github.com/DenisVieriu97, https://github.com/malfet
 2022-12-19 22:00:12 +00:00
Nikita Shulga
3859aace20 [MPS] Skip tests broken on Ventura (#90843) 
Also add `torch.backends.mps.is_macos13_or_newer`
See https://github.com/pytorch/pytorch/issues/85758

Pull Request resolved: https://github.com/pytorch/pytorch/pull/90843
Approved by: https://github.com/kulinseth, https://github.com/albanD
 2022-12-14 19:51:00 +00:00
Li-Huai (Allan) Lin
544756ae5e Fix mps constant pad (#89864) 
Support arbitrary dimensions for constant padding on MPS

Fixes #89624
Fixes #87277

Pull Request resolved: https://github.com/pytorch/pytorch/pull/89864
Approved by: https://github.com/kulinseth, https://github.com/malfet
 2022-12-13 17:28:54 +00:00
Denis Vieriu
b71c710db1 Add additional tests for view slice tensors (#86282) 
Fixes https://github.com/pytorch/pytorch/issues/83995 and https://github.com/pytorch/pytorch/issues/84489

Pull Request resolved: https://github.com/pytorch/pytorch/pull/86282
Approved by: https://github.com/kulinseth
 2022-12-08 17:59:55 +00:00
PyTorch MergeBot
cba96366a2 Revert "remove torch.equal usages (#89527)" 
This reverts commit 4095ef8b80.

Reverted https://github.com/pytorch/pytorch/pull/89527 on behalf of https://github.com/clee2000 due to broke periodic multigpu tests 4095ef8b80 https://github.com/pytorch/pytorch/actions/runs/3592806602/jobs/6049368502
 2022-12-02 21:36:13 +00:00
Philip Meier
4095ef8b80 remove torch.equal usages (#89527) 
Preparation for the next PR in this stack: #89559.

I replaced

- `self.assertTrue(torch.equal(...))` with `self.assertEqual(..., rtol=0, atol=0, exact_device=True)`,
- the same for `self.assertFalse(...)` with `self.assertNotEqual(...)`, and
- `assert torch.equal(...)` with `torch.testing.assert_close(..., rtol=0, atol=0)` (note that we don't need to set `check_device=True` here since that is the default).

There were a few instances where the result of `torch.equal` is used directly. In that cases I've replaced with `(... == ...).all().item()` while sometimes also dropping the `.item()` depending on the context.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/89527
Approved by: https://github.com/mruberry
 2022-12-01 11:22:52 +00:00
Sergii Dymchenko
09f2373ec0 Fix TODOs related to #38095 in test_mps.py (#89815) 
Pull Request resolved: https://github.com/pytorch/pytorch/pull/89815
Approved by: https://github.com/weiwangmeta, https://github.com/kulinseth
 2022-11-30 17:00:36 +00:00
Thomas
4935b597ac Added implementation and tests for MPS Hardswish (#87952) 
## What?
Fixes issue #86807 by adding MPS backend support for aten::hardswish.

## How?
Registered mps hardswish functions in native_functions.yaml, and added the code implementation to Activations.mm.

Added functions:
- hardswish_mps
- hardswish_mps_
- hardswish_backward_mps
- hardswish_out_mps

## Testing
Added test in test/test_mps.py and tested code using the command `python3 test/test_mps.py -k test_hardswish`

Pull Request resolved: https://github.com/pytorch/pytorch/pull/87952
Approved by: https://github.com/kulinseth, https://github.com/kit1980
 2022-11-23 02:18:03 +00:00
Edward Z. Yang
dbeacf1182 Fix cat striding in PrimTorch (#89332) 
Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/89332
Approved by: https://github.com/ngimel
 2022-11-20 04:05:33 +00:00
PumeTu
fc1c0cd3ef Add support trace on MPS backend (#87910) 
Fixes [#87221](https://github.com/pytorch/pytorch/issues/87221)
`trace` now supported on MPS

Pull Request resolved: https://github.com/pytorch/pytorch/pull/87910
Approved by: https://github.com/kulinseth, https://github.com/malfet
 2022-11-18 07:24:33 +00:00
Raman kumar
fd0efb01a7 [MPS] Support for median with dim (#88807) 
## Summary ⚡

**Aim**: Add support for aten::median for MPS backend (Fixes #87220)

This is fresh clean PR from the previous [PR](https://github.com/pytorch/pytorch/pull/88554)

- Implementing the new median function in aten/src/ATen/native/mps/operations/ReduceOps.mm
- Adding it to aten/src/ATen/native/native_functions.yaml
- Adding it to existing test_median

### **this will works like this** 🪶
median of entire input tensor on MPS
`torch.median(mps_inputTensor)`
median of along a dim
`torch.median(mps_inputTensor, dim=[int], keepdim=[Bool])`
Pull Request resolved: https://github.com/pytorch/pytorch/pull/88807
Approved by: https://github.com/kulinseth
 2022-11-18 02:53:42 +00:00
Lukas Hoenig
81a8fdc40d [MPS] Add binary operations dtype precedence test case (#87545) 
See https://github.com/pytorch/pytorch/pull/84742 and https://github.com/pytorch/pytorch/pull/78319.

The test case tests that
- for the binary operations (add, sub, mul, div),
- for all data types (dtypes),
- for a range of representative values and their combinations,
- for various shapes and ways of creating the test tensors,

the contents and dtype of the result tensor is identical for the MPS and CPU backends.

It adds about 15-18s runtime to `test_mps.py`.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/87545
Approved by: https://github.com/kit1980
 2022-11-17 04:54:27 +00:00
Nikita Shulga
62ef15e320 [MPS] Fix test_embedding_dense_backward (#88847) 
By copying randomly initialized weights distribution from MPS `nn.Embedding` to `cpu`

Test plan: `python test_mps.py -k test_embedding_dense_backward --repeat 150`

Fixes https://github.com/pytorch/pytorch/issues/88679

Pull Request resolved: https://github.com/pytorch/pytorch/pull/88847
Approved by: https://github.com/seemethere
 2022-11-10 23:52:27 +00:00
Li-Huai (Allan) Lin
7c353eb395 [MPS] Fix softplus (#88555) 
1. Fixes #87780
2. Fixes mps graph cache issue
3. Adds proper tests

Pull Request resolved: https://github.com/pytorch/pytorch/pull/88555
Approved by: https://github.com/kulinseth
 2022-11-10 09:40:08 +00:00
Nikita Shulga
078c25df13 [MPS][BE] Code cleanup (#88529) 
Various code cleanup in MPS operations:
 - Per @kulinseth suggestion move `mpsSupportsCumsum` to `MPSDevice.h` and rename it to
   `is_macos_13_or_newer()`
 - Move Ventura MPSGraph new operators to `MPSGraphVenturaOps.h` header
 - Use `LookupAs` and `CreateCachedGraphAs` to make code more compact
 - Formatting

Pull Request resolved: https://github.com/pytorch/pytorch/pull/88529
Approved by: https://github.com/kulinseth
 2022-11-08 21:10:07 +00:00
Li-Huai (Allan) Lin
15e54293ef [MPS] Fix embedding backward with scalar index (#82809) 
### Description
Previously the embedding backward always expands `-1` dim to indices, resulting in the following error when the indices is a scalar:

```
 error: Rank of data array must equal number of outer dimensions in indices array + rank of slice to update, 2 != 1 + 0
-:8:10: note: see current operation: %5 = "mps.scatter_nd"(%0, %arg1, %4) {batch_dims = 0 : ui32, mode = 0 : i32} : (tensor<10x5xf16>,
```

Now makes it conditional.

Reproducer:

```python
def repro():
    w = torch.tensor([[-2.6465,  2.5859,  0.4688,  1.7949,  3.2676],
        [-3.1641,  8.9375,  5.7578, -2.9453, -6.5469],
        [ 2.0469,  1.3516, -8.7344,  6.0000,  1.3906],
        [ 6.5781,  7.8438,  6.9766,  3.2891, -5.1172],
        [-7.9414,  7.7344,  4.1875,  2.8574,  2.9531],
        [-0.4844, -5.6328, -6.8359, -4.5156,  3.7891],
        [ 4.9375,  6.6094,  6.7031,  0.6719, -6.4219],
        [ 7.0469,  8.2031,  4.4453,  1.7129, -2.4688],
        [ 1.2207, -3.3750, -2.4531,  7.4062, -6.0469],
        [-8.9688,  2.2656,  2.4160, -1.0176,  8.4531]], dtype=torch.float32, requires_grad=True)
    x = torch.tensor(5)
    out = torch.nn.functional.embedding(x, w)
    out.sum().backward()

    w_mps = w.detach().clone().to("mps").requires_grad_()
    x_mps = x.to("mps")
    out = torch.nn.functional.embedding(x_mps, w_mps)
    out.sum().backward() # error
```

### Issue
<!-- Link to Issue ticket or RFP -->

### Testing
<!-- How did you test your change? -->

Pull Request resolved: https://github.com/pytorch/pytorch/pull/82809
Approved by: https://github.com/malfet
 2022-11-04 19:43:56 +00:00
Nikita Shulga
657f2e12f0 [MPS] Add native cumsum implementation (#88319) 
Using https://developer.apple.com/documentation/metalperformanceshadersgraph/mpsgraph/4057333-cumulativesumwithtensor?language=objc

Fall back to CPU if running on older MacOS versions
In `unary_op` add output tensor dims/dtype to the graph key (as even in default op we check output graph type)
Also, upcast int16 to int32 as MPS cumsum op on Ventura returns incorrect results for Int16 type (and it makes total sense for int8, as chances for overflow are very high)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/88319
Approved by: https://github.com/kulinseth
 2022-11-04 01:22:41 +00:00
Philip Meier
bc73affdad prepare removal of deprecated functionality in torch.testing (#87969) 
_Redo of #86586 with all BC breaking changes granularly placed into separate commits._

---

Per title. Deprecation happened on Feb 25, 2022 in c6f1bbc0ac, which made it into the 1.12 release. Since it is now 245 days later and the next release will be 1.14, the removals later in the stack comply with the [BC policy](https://github.com/pytorch/pytorch/wiki/PyTorch's-Python-Frontend-Backward-and-Forward-Compatibility-Policy#minimizing-the-disruption-of-bc-breaking-changes).
Pull Request resolved: https://github.com/pytorch/pytorch/pull/87969
Approved by: https://github.com/mruberry
 2022-11-02 14:04:48 +00:00
arnaudstiegler
16e35bd179 Adding expm1 to MPS (#87147) 
Fixes #86744

- Implementing the new `expm1_out_mps` function in `aten/src/ATen/native/mps/operations/UnaryOps.mm`
- Adding it to `aten/src/ATen/native/native_functions.yaml`
- Adding it to existing `test.test_mps.TestNLLLoss.test_unary_ops`

Pull Request resolved: https://github.com/pytorch/pytorch/pull/87147
Approved by: https://github.com/kulinseth
 2022-10-26 17:45:46 +00:00
Daniel Falbel
e818574e78 Support signbit in MPS. (#87214) 
Implements the signbit operator for MPS. Links to #77764

Pull Request resolved: https://github.com/pytorch/pytorch/pull/87214
Approved by: https://github.com/kulinseth, https://github.com/kit1980
 2022-10-25 07:12:31 +00:00
Alex
620dbc43d8 Slowly introduce ops to be tested by test_numpy_ref on MPS backend (#87342) 
Enable a test that would have caught https://github.com/pytorch/pytorch/issues/86239

Prior to the fix for that bug, this test fails with

```
_____________________________ TestCommonMPS.test_numpy_ref_mps_where_mps_float32 _____________________________
Traceback (most recent call last):
  File "/Users/alex/git/pytorch/test/test_ops.py", line 197, in test_numpy_ref_mps
    self.compare_with_reference(
  File "/Users/alex/git/pytorch/torch/testing/_internal/common_utils.py", line 2366, in compare_with_reference
    actual = torch_fn(t_inp, *t_args, **t_kwargs)
  File "/Users/alex/git/pytorch/torch/testing/_internal/opinfo/core.py", line 1068, in __call__
    return self.op(*args, **kwargs)
  File "/Users/alex/git/pytorch/torch/testing/_internal/common_methods_invocations.py", line 15167, in <lambda>
    op=lambda self, condition, other: torch.where(condition, self, other),
RuntimeError: 0'th index 3 of x tensor does not match the other tensors
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/87342
Approved by: https://github.com/albanD
 2022-10-21 19:03:00 +00:00
Nikita Shulga
ae62cf7c02 [MPS] Revamp copy_to_mps_ implementation (#86956) 
Tensor's view in linear storage is represented by the following parameters: `.shape`, `.stride()` and `.storage_offset()`.

Only tensors that are representable as 1d-views can be copied from host to device (and vice versa) using single  [`copy(from:sourceOffset:to:destinationOffset:size:)`](https://developer.apple.com/documentation/metal/mtlblitcommandencoder/1400767-copyfrombuffer?language=objc) call.

Modify `copy_to_mps_` function to do the following steps:
- Cast `src` tensor to dst data type if needed
- Expand `src` tensor to `dst` tensor shape
- Clone `src` tensor if it is not stride contiguous (i.e. can not be represented by `src.view(src.numel())`)
- Create an empty tensor if `dst` is not stride-contiguous or if its strides are different then potentially cloned `src` strides
- Do 1d copy for `src` to (potentiall temp) `dst`
- Finally do re-striding/copy on MPS if needed

Add test to cover cases where stide-contiguous permuted tensor is copied to MPS, non-stride-contiguous tensor is copied to MPS and if permuted CPU tensor is copied to differently permuted MPS tensor

Fixes https://github.com/pytorch/pytorch/issues/86954

Pull Request resolved: https://github.com/pytorch/pytorch/pull/86956
Approved by: https://github.com/kulinseth
 2022-10-21 14:10:05 +00:00
Nikita Karetnikov
1b8af28fe8 [primTorch] Add refs for softmax, softmin, log_softmax (#84956) 
cc @ezyang @mruberry @ngimel @Lezcano @fdrocha
Pull Request resolved: https://github.com/pytorch/pytorch/pull/84956
Approved by: https://github.com/lezcano, https://github.com/mruberry
 2022-10-20 12:29:04 +00:00
Nikita Shulga
13cff2ee8e [MPS] Copy from CPU always add storageOffset (#86958) 
Because why wouldn't it?
Fixes https://github.com/pytorch/pytorch/issues/86052

Pull Request resolved: https://github.com/pytorch/pytorch/pull/86958
Approved by: https://github.com/kulinseth
 2022-10-14 17:35:18 +00:00
Nikita Shulga
692b525b71 [MPS] Extend unary ops to int64 (#86615) 
Most of them are already supported for `int64` except for:
 - rounding operations (`floor`, `ceil` and `round`), which are no-ops for integral types anyway
 - sign operation, when it can be emulated by clamping it tensor to [-1, 1] range

Test new types by test MPS

Fixes https://github.com/pytorch/pytorch/issues/86319

Pull Request resolved: https://github.com/pytorch/pytorch/pull/86615
Approved by: https://github.com/DenisVieriu97, https://github.com/huydhn
 2022-10-12 00:32:53 +00:00
Nikita Shulga
b7b5bd47ae [MPS] Implement frac operator (#86625) 
As combination if self-trunc

Fixes #ISSUE_NUMBER

Pull Request resolved: https://github.com/pytorch/pytorch/pull/86625
Approved by: https://github.com/kulinseth, https://github.com/albanD
 2022-10-10 20:36:22 +00:00
Alex
ca69ddb4f7 Fix broadcasting to implicit leading dimensions in torch.where on MPS (#86240) 
Fixes #86239

Pull Request resolved: https://github.com/pytorch/pytorch/pull/86240
Approved by: https://github.com/kulinseth
 2022-10-07 01:38:57 +00:00
Nikita Shulga
fa799132d8 [MPS] Better error message for slow_conv2d_forward (#86303) 
Error `Could not run 'aten::_slow_conv2d_forward' with arguments from the 'MPS' backend.` is very misleading as usually this method is only invoked if input is on CPU but weights are on MPS device.
Raise a more user friendly error in this case

Add test to `test_invalid_conv2d` to check for those conditions.

Fixes https://github.com/pytorch/pytorch/issues/77931

Pull Request resolved: https://github.com/pytorch/pytorch/pull/86303
Approved by: https://github.com/kulinseth
 2022-10-06 15:38:57 +00:00
Nikita Shulga
97d2e1df55
[MPS] Fix GELU for torch.half (#86218) 
Also, make sure it raises catcheable errors if invoked with integral types

Otherwise, it used to fail with following fatal error  invoked for `torch.half` and with similar signatures if invoked for integral types
```
loc("mps_multiply"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/4883e71d-37bd-11ed-b0ef-b25c5e9b9057/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":228:0)): error: input types 'tensor<2xf16>' and 'tensor<1xf32>' are not broadcast compatible
LLVM ERROR: Failed to infer result type(s).
```

Modified `test_gelu_simple` to check both fwd and backward gradients for gelu
 2022-10-05 09:09:17 -07:00
Kulin Seth
6a842e33c6 MPS: Add multinomial op (#80760) 
Add multinomial with replacement

Pull Request resolved: https://github.com/pytorch/pytorch/pull/80760
Approved by: https://github.com/razarmehr, https://github.com/malfet
 2022-10-03 21:05:30 +00:00
Abhishek Pathak
8860e48994 [MPS] Handle compatible inputs to where (#85946) 
Inputs with different number of dimensions but compatible shapes were being rejected

e.g. x.shape = [10,1,10]
y.shape = [10,10]
cond.shape = [10,10,1]
Pull Request resolved: https://github.com/pytorch/pytorch/pull/85946
Approved by: https://github.com/malfet
 2022-10-03 18:12:48 +00:00
Nikita Shulga
b9b24c31fd [MPS] Fix non-contig to contig tensor copy (#86056) 
This handles a rare case when MPS tensor is constructed from non-contiguous CPU tensor.
Fixes https://github.com/pytorch/pytorch/issues/85967

Pull Request resolved: https://github.com/pytorch/pytorch/pull/86056
Approved by: https://github.com/janeyx99
 2022-10-02 20:13:05 +00:00
Ramin Azarmehr
bf667c63e7 Fix the error with constant_pad_nd for 4D+ padding (#85991) 
- We warn the user and fall back to default implementation for 4D+ constant padding

Fixes #84535

Pull Request resolved: https://github.com/pytorch/pytorch/pull/85991
Approved by: https://github.com/kulinseth
 2022-10-01 00:33:23 +00:00
Ramin Azarmehr
334686bde7 Fix the dimension of padding to match the input's dimension (#85990) 
Fixes #85143

Pull Request resolved: https://github.com/pytorch/pytorch/pull/85990
Approved by: https://github.com/malfet, https://github.com/kulinseth
 2022-09-30 22:57:57 +00:00
Ramin Azarmehr
a4cc63991a [MPS] Enable caching for random ops with Philox engine (#85833) 
Also Fix type cast issue in Bernoulli (Fixes #85611)

Pull Request resolved: https://github.com/pytorch/pytorch/pull/85833
Approved by: https://github.com/kulinseth, https://github.com/malfet
 2022-09-30 22:40:50 +00:00
Denis Vieriu
be327ec08f [MPS] Fix base shape size for view ops in case of multiple slices (#85934) 
Fixes https://github.com/pytorch/pytorch/issues/84364, https://github.com/pytorch/pytorch/issues/85592

Fixes bug for view ops where the base shape would be incorectly determined.
E.g for the following tensor `torch.tensor([0.5, 0.5], device="mps")[1][None]`, we could consider the base shape of the parent tensor as 1, while the actual base shape is 2.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/85934
Approved by: https://github.com/kulinseth
 2022-09-30 18:51:43 +00:00
Abhishek Pathak
81b366a9dd [MPS] Handle scalar input for scatter and gather (#85842) 
Issue noticed in test consistency - "Indexing dim 0 is out of bounds of tensor"
Pull Request resolved: https://github.com/pytorch/pytorch/pull/85842
Approved by: https://github.com/kulinseth
 2022-09-30 00:24:16 +00:00
Abhishek Pathak
62a4fd7907 [MPS] Handle output shape for empty input in binary ops (#85836) 
Output of input shape [0,1,2] should be [0,1,2], not [0] i.e. delay returning from empty input condition to resize/reshape the output accordingly
Pull Request resolved: https://github.com/pytorch/pytorch/pull/85836
Approved by: https://github.com/DenisVieriu97, https://github.com/kulinseth
 2022-09-30 00:19:14 +00:00
Denis Vieriu
6a14fcb922 [MPS] Add support for aten::masked_select on mps (#119) (#85818) 
Reuse the `index.Tensor_out` implementation since it's already expanding the bool/byte indices to long tensors.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/85818
Approved by: https://github.com/kulinseth
 2022-09-29 23:23:00 +00:00
Denis Vieriu
ce4f187f15 [MPS] Add tensor::index_put op (#85672) 
Pull Request resolved: https://github.com/pytorch/pytorch/pull/85672
Approved by: https://github.com/malfet
 2022-09-28 00:47:52 +00:00
Kulin Seth
7ff6a00a9a [MPS] Handle 1D weight in linear layer (#85752) 
Fixes https://github.com/pytorch/pytorch/issues/84591

Pull Request resolved: https://github.com/pytorch/pytorch/pull/85752
Approved by: https://github.com/malfet
 2022-09-28 00:43:11 +00:00
Abhishek Pathak
e746fff8ba [MPS] Enable adaptive avg pool 2d with larger output size (#85726) 
* Handle adpative pool 2d forward and backward when ouptut size is larger than input size
* Disallow larger output size if not a multiple of input size
Fixes: https://github.com/pytorch/pytorch/issues/80732
Pull Request resolved: https://github.com/pytorch/pytorch/pull/85726
Approved by: https://github.com/malfet
 2022-09-27 19:08:22 +00:00
Nikita Shulga
1367f2409f [MPS] Fix placeholder behavior for transposed view (#85689) 
Looks like the expectation in that code were that `.clone` will return contiguous tensor, so explicitly specify memory format

Fixes https://github.com/pytorch/pytorch/issues/85675 and https://github.com/pytorch/pytorch/issues/85224

Pull Request resolved: https://github.com/pytorch/pytorch/pull/85689
Approved by: https://github.com/kulinseth
 2022-09-27 15:44:53 +00:00
Abhishek Pathak
7a5449f148 [MPS] Clamp op - fix shape issues (#114) (#85673) 
* Handle shape mismatch
* Handle case where 1 occurs in input shape; fix fill_new_shapes
* Move clamp ops to allowlist

Pull Request resolved: https://github.com/pytorch/pytorch/pull/85673
Approved by: https://github.com/malfet
 2022-09-27 01:54:42 +00:00
George Qi
686555b663 [maskedtensor] port torch/_masked into torch/masked (#85515) 
Pull Request resolved: https://github.com/pytorch/pytorch/pull/85515
Approved by: https://github.com/cpuhrsch
 2022-09-26 23:41:13 +00:00
Abhishek Pathak
f0570354dd [MPS] Fix memory error in var (#85571) 
* Fix memory corruption + wrong handling of negative dims
* Use vector for shape

Pull Request resolved: https://github.com/pytorch/pytorch/pull/85571
Approved by: https://github.com/malfet
 2022-09-25 19:03:58 +00:00
Kulin Seth
e1ed485c65 [MPS] Handle reduction of scalars in edge-cases (#83743) 
The issue was found as part of fixing Test consistency issues.
Test case coming up.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/83743
Approved by: https://github.com/razarmehr, https://github.com/malfet
 2022-09-20 17:53:45 +00:00
Kulin Seth
077db3de92 [MPS] Fix conv1d backwards crash for channels last case (#85283) 
Fixes pytorch#84511

Use the same logic as in the forward pass for the backward pass (in case of channels last, manually set the shape to NHWC)

Pull Request resolved: https://github.com/pytorch/pytorch/pull/85283
Approved by: https://github.com/malfet, https://github.com/razarmehr
 2022-09-20 06:19:40 +00:00
Nikita Shulga
1a6cf6ea88 [MPS] Fix int rounding div crash on M1 (#85016) 
Fixes  https://github.com/pytorch/pytorch/issues/84995

Pull Request resolved: https://github.com/pytorch/pytorch/pull/85016
Approved by: https://github.com/kulinseth
 2022-09-14 23:40:20 +00:00
Denis Vieriu
4247cc98a2 [MPS] Fix mps to cpu casting from a smaller dtype to a bigger dtype (#84928) 
Fixes #82566 , #80800

- mps->cpu casts from a smaller dtype to a bigger dtype mps->mps cast from smaller/bigger dtype to another dtype in case of scatter
- For mps->cpu copies where we don't have a source/destination offset, we can save the cast result directly in the destTensor, so we can skip the additional overhead of the blit.
- In case we can return the data without doing the blit, we need to check if it's blocking call, case in which we'd need a synchronize(SyncType::COMMIT_AND_WAIT); call (previously this was done by the blit).

Pull Request resolved: https://github.com/pytorch/pytorch/pull/84928
Approved by: https://github.com/razarmehr
 2022-09-14 17:24:25 +00:00
Nikita Shulga
9b16bf04af Fix MPS test sanity (#84889) 
Follow up after https://github.com/pytorch/pytorch/pull/84834

Pull Request resolved: https://github.com/pytorch/pytorch/pull/84889
Approved by: https://github.com/tugsbayasgalan, https://github.com/janeyx99, https://github.com/ZainRizvi
 2022-09-12 22:25:26 +00:00
Abhishek Pathak
bccc26f365 [MPS] Handle casting for div operation (#84742) 
* Handle casting for div operation
* Update divmode test to test for rounding mode in div

cc. @lhoenig
Pull Request resolved: https://github.com/pytorch/pytorch/pull/84742
Approved by: https://github.com/razarmehr
 2022-09-10 03:10:04 +00:00
Peter Bell
b9793a66b5 Fix linalg.norm sample inputs function and related failures (#84452) 
Due to an indentation error, the return statement happens after just 1
loop of `for test_size in test_sizes` so only one shape was ever
tested.

This also revealed several cases where the provided shapes don't work
so I've disabled the generation of those sample inputs.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/84452
Approved by: https://github.com/Lezcano, https://github.com/zou3519
 2022-09-08 17:07:58 +00:00
soulitzer
bfdfeecd15 Add per-op MPS gradient tests and update skips (#84242) 
Follow up:
- ~Remove non-float dtypes from allow-list for gradients~
- ~Map dtypes to short-hand so there aren't so many lines, i.e. float16 should be f16.~
- ~There were a lot of linting issues that flake8 wouldn't format for me, so I reformatted with black. This makes the diff a little trickier to parse.~

Observations:
- there are entries in the allow-list that weren't there before
- some forward that we previously passing now fail with requires_grad=True
- Because the allow list does not know about variants, a special skip was added for that in the block list

Pull Request resolved: https://github.com/pytorch/pytorch/pull/84242
Approved by: https://github.com/kulinseth, https://github.com/malfet
 2022-09-01 16:41:52 +00:00
Ramin Azarmehr
d1be36ceab [MPS] Fix the index error in constant_pad_nd() with single-dimension input (#83745) 
* Fix the index error in constant_pad_nd() with single-dimension input (#83343)
- Also added a test case in test_mps for it

* Move padding code into new file Pad.mm

Fixes https://github.com/pytorch/pytorch/issues/83343

Pull Request resolved: https://github.com/pytorch/pytorch/pull/83745
Approved by: https://github.com/razarmehr
 2022-08-22 17:07:09 +00:00
Denis Vieriu
a6b75bb099 [MPS] Fix placeholder case for missing gather graph (#83744) 
Fixes https://github.com/pytorch/pytorch/issues/82543, https://github.com/pytorch/pytorch/issues/83230

The current Placeholder code relies to find a gather graph in order to make the data contiguous, otherwise we'll try calling into tensor.contiguous() directly, which for slice elements, won't do anything.

E.g consider the following basic case where we index a 2 element tensor:
```
tensor_list = torch.tensor([1.2, 1.0], device="mps")

for scalar in tensor_list:
  r_mps = torch.ceil(scalar)
  r_cpu = torch.ceil(scalar.to("cpu"))
  self.assertEqual(r_mps.cpu(), r_cpu)
```

The second element 1.0 is a contiguous view tensor (similar to slicing), but it has no gather graph created behind. In the placeholder, we won't be able to find the graph, thus relying on the fallback case where we call _tensor = src.contiguous();. For an already contiguous tensor, this won't do anything, thus we end up creating the NDArray with all the values of the tensor (1.2 and 1.0 instead of just 1.0). Doing clone instead of contiguous will actually perform a blit behind and take into consideration the storage_offset of the view when performing the copy.

Similarly, the following basic case is also failing because of this issue:

```
x = torch.tensor([1.0, 0.49], device="mps")
print(x) # prints 1.0 and 0.0
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/83744
Approved by: https://github.com/razarmehr
 2022-08-22 17:05:53 +00:00
Nikita Shulga
ff533b1efa [MPS] Fix torch.full for uint8 (#83697) 
By creating uint32  tensor and then downcasting it to uint8

Workaround https://github.com/pytorch/pytorch/issues/83692

Pull Request resolved: https://github.com/pytorch/pytorch/pull/83697
Approved by: https://github.com/albanD
 2022-08-18 21:59:15 +00:00
Kulin Seth
ce7177f88a [MPS] Register index.Tensor_out (#82507) 
* Add more tests from test_indexing into test_mps
* Cache the indexing library on the MPSDevice
Pull Request resolved: https://github.com/pytorch/pytorch/pull/82507
Approved by: https://github.com/malfet
 2022-08-18 06:03:16 +00:00
Kulin Seth
31d4b6f52a [MPS] Fix conv1D and conv2D with non-matching strides/paddings (#83522) 
* Add reference to the github issue in test_mps.py

Fixes https://github.com/pytorch/pytorch/issues/83180, https://github.com/pytorch/pytorch/issues/82921, https://github.com/pytorch/pytorch/issues/82711, https://github.com/pytorch/pytorch/issues/82563

Pull Request resolved: https://github.com/pytorch/pytorch/pull/83522
Approved by: https://github.com/albanD, https://github.com/malfet
 2022-08-17 00:26:41 +00:00
Kulin Seth
02cfefb48c [MPS] Fix for matmul errors in test consistency (#83124) 
* Handle empty input with non-empty output
* Remove transpose options from mm op

Pull Request resolved: https://github.com/pytorch/pytorch/pull/83124
Approved by: https://github.com/malfet
 2022-08-12 23:28:38 +00:00
Kulin Seth
b3dea3e413 Add the Conv1D support for NHWC format. (#83121) 
Fixes https://github.com/pytorch/pytorch/issues/81557

cc @DenisVieriu97

Pull Request resolved: https://github.com/pytorch/pytorch/pull/83121
Approved by: https://github.com/malfet
 2022-08-10 14:30:20 +00:00
Nikita Shulga
dcf5188561 [MPS] Fix scatter for bool type (#82685) 
By typecasting bool tensors to int8 before scattering

See https://github.com/pytorch/pytorch/issues/82663

Pull Request resolved: https://github.com/pytorch/pytorch/pull/82685
Approved by: https://github.com/albanD, https://github.com/kulinseth
 2022-08-03 14:54:47 +00:00
Nikita Shulga
420c576809 [MPS] Unary ops over empty tensors should be no-op (#82650) 
Fixes https://github.com/pytorch/pytorch/issues/82531

Pull Request resolved: https://github.com/pytorch/pytorch/pull/82650
Approved by: https://github.com/kulinseth
 2022-08-02 21:15:37 +00:00
Nikita Shulga
bdd0a4a84c [MPS] Fix torch.full for boolean types (#82575) 
By creating int8 tensor and casting it to bool later

Workaround for MPSGraph deficiency reported in https://github.com/pytorch/pytorch/issues/82427

Pull Request resolved: https://github.com/pytorch/pytorch/pull/82575
Approved by: https://github.com/kulinseth
 2022-08-01 19:42:24 +00:00
qqaatw
7496937e3f [MPS] Add prelu (#82401) 
### Description
Adds `prelu` MPS support.

I'm wondering whether the generic test framework implemented in `test_mps.py` checks the correctness of gradients.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/82401
Approved by: https://github.com/kulinseth
 2022-07-30 05:16:25 +00:00
Ramin Azarmehr
38b4114278 [MPS] Add MPS implementation for constant_pad_nd() (#75) (#82366) 
MPS has a native implementation of the constant pad nd. Adding that instead of going through the view ops helps improve performance in several benchmarks in torchbench.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/82366
Approved by: https://github.com/malfet, https://github.com/razarmehr
 2022-07-29 16:34:07 +00:00
Abhishek Pathak
d7210e6129 [MPS] Fixes for MPS testConsistency (#81735) 
Fixes #ISSUE_NUMBER

Pull Request resolved: https://github.com/pytorch/pytorch/pull/81735
Approved by: https://github.com/razarmehr, https://github.com/albanD
 2022-07-20 16:31:44 +00:00
Kulin Seth
596bb41163 [MPS] Get the correct size of the view tensor when copying from cpu to mps (#81730) 
Fixes: https://github.com/pytorch/pytorch/issues/81567, https://github.com/pytorch/pytorch/issues/80844
* Get the correct size of the view tensor when copying from cpu to mps

* Use 'computeStorageNbytesContiguous' to get the size just when src is a view

* Add asserts and tests to check for storage_offset

* Add testcase for https://github.com/pytorch/pytorch/issues/80844

* Replace assert_allclose with assertEqual

* Replace TORCH_CHECK with TORCH_INTERNAL_ASSERT
Pull Request resolved: https://github.com/pytorch/pytorch/pull/81730
Approved by: https://github.com/razarmehr, https://github.com/albanD
 2022-07-20 14:27:54 +00:00
qqaatw
1caa25ebcb [MPS] Add aten::index_add.out (#79935) 
Fixes #ISSUE_NUMBER

Pull Request resolved: https://github.com/pytorch/pytorch/pull/79935
Approved by: https://github.com/kulinseth
 2022-07-14 23:40:00 +00:00
Denis Vieriu
e3b98ba7a4 [MPS] Handle Boolean inputs in the cat op (#81480) 
Fixes https://github.com/pytorch/pytorch/issues/80850

Pull Request resolved: https://github.com/pytorch/pytorch/pull/81480
Approved by: https://github.com/razarmehr
 2022-07-14 22:00:57 +00:00
Denis Vieriu
0adc2e35f2 Add testcase for MPS issue #80856 (#81455) 
Testcase for #80856.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/81455
Approved by: https://github.com/kulinseth
 2022-07-14 19:54:18 +00:00
Kulin Seth
067c8067a3 [MPS]: Added op upsample_nearest1d (#81303) 
Fixes #ISSUE_NUMBER

Pull Request resolved: https://github.com/pytorch/pytorch/pull/81303
Approved by: https://github.com/malfet
 2022-07-13 21:39:50 +00:00
Abhishek Pathak
ae83e44c5f [MPS] Handle 1D inputs for NLL (#81290) 
* Add test for NLL 1d
* Fix forward NLL for 1D case
* Handle NLL backward for 1d

Pull Request resolved: https://github.com/pytorch/pytorch/pull/81290
Approved by: https://github.com/razarmehr
 2022-07-12 19:46:59 +00:00
qqaatw
b0b24b4285 [MPS] Fix LSTM batch_first output transposed (#80597) 
The output of LSTM with `batch_first` should be transposed back to batch first format.

Fixes #80306

Pull Request resolved: https://github.com/pytorch/pytorch/pull/80597
Approved by: https://github.com/kulinseth
 2022-07-07 07:18:00 +00:00
qqaatw
2458b3cd5f [MPS] Add argmin (#80828) 
This PR

1. adds argmin
2. refactors `reduction_type` in `ReduceOps.mm` with enum.

Co-authored by Kulin Seth <kulinseth@gmail.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80828
Approved by: https://github.com/malfet
 2022-07-07 00:04:49 +00:00
PyTorch MergeBot
255649984c Revert "[MPS] Add argmin (#80828)" 
This reverts commit 37ebcc8a80.

Reverted https://github.com/pytorch/pytorch/pull/80828 on behalf of https://github.com/suo due to Broke Mac tests on master: 37ebcc8a80
 2022-07-06 16:20:18 +00:00
qqaatw
c48e964a04 [MPS] Add huber loss (#80163) 
Fixes #ISSUE_NUMBER

Pull Request resolved: https://github.com/pytorch/pytorch/pull/80163
Approved by: https://github.com/kulinseth, https://github.com/malfet
 2022-07-06 15:36:47 +00:00
qqaatw
37ebcc8a80 [MPS] Add argmin (#80828) 
This PR

1. adds argmin
2. refactors `reduction_type` in `ReduceOps.mm` with enum.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80828
Approved by: https://github.com/malfet
 2022-07-06 15:14:04 +00:00
qqaatw
8745118ccb [MPS] Add softplus backward (#79873) 
Fixes #ISSUE_NUMBER

Pull Request resolved: https://github.com/pytorch/pytorch/pull/79873
Approved by: https://github.com/malfet
 2022-07-06 06:13:21 +00:00
Kulin Seth
5436134f32 [MPS] Move the View ops to a separate file and reduce the number of graphs created (#80491) 
This is dependent on the PR to go in first: https://github.com/pytorch/pytorch/pull/79939

Remove the data_ptr from the View Graph key which reduces the number of
graphs created significantly.

Don't wait when copying from MPS to MPS tensors

Pull Request resolved: https://github.com/pytorch/pytorch/pull/80491
Approved by: https://github.com/malfet
 2022-07-06 03:39:20 +00:00
Kulin Seth
76cff18242 [MPS] Add test consistency from OpInfo based tests from PR 78504 (#79532) 
Pull Request resolved: https://github.com/pytorch/pytorch/pull/79532
Approved by: https://github.com/albanD, https://github.com/malfet
 2022-07-04 06:41:39 +00:00
Ramin Azarmehr
0e3953fc52 MPS: Fix handling of 1D tensors in linear backward (#80759) 
Fixes #https://github.com/pytorch/pytorch/issues/79784

Pull Request resolved: https://github.com/pytorch/pytorch/pull/80759
Approved by: https://github.com/ezyang
 2022-07-04 02:06:14 +00:00
Kulin Seth
b744e1c8ef Add scatter support for view operations (#79939) 
* Add scatter support for view operations; #78074, #78886, #79672
* Update test_slicing_replace_column to properly test different sizes
* Handle in-place changes for binary ops; add new testcase
* Add new view ops testing scatter; add MPSDebugConfig.h config file for debugging purposes
* Merge gatherViewTensor and scatterViewTensor into a generic function
* Add scatter on demand in scatterViewOperation instead of caching it into a generic graph
* Create separate graphs for scatter and gather;
* Create scatter graph at scatter time

Fixes #ISSUE_NUMBER

Pull Request resolved: https://github.com/pytorch/pytorch/pull/79939
Approved by: https://github.com/razarmehr
 2022-07-01 15:10:56 +00:00
PyTorch MergeBot
b1943e01e2 Revert "[MPS] Add test consistency from OpInfo based tests from PR 78504 (#79532)" 
This reverts commit c71886e048.

Reverted https://github.com/pytorch/pytorch/pull/79532 on behalf of https://github.com/malfet due to Unintended submodules updates
 2022-06-30 16:37:11 +00:00
qqaatw
ae6f07e7d5 [MPS] Fix std/var cache issue (#80502) 
Use `getTensorsStringKey` which has tensor shape info added as part of the key to prevent cache lookup issue when the shape of input tensor is changed.

Fixes #80499

Pull Request resolved: https://github.com/pytorch/pytorch/pull/80502
Approved by: https://github.com/malfet, https://github.com/kulinseth
 2022-06-30 12:56:56 +00:00
qqaatw
c980fc3d3c [MPS] Add glu (#79866) 
Adds mps op for `aten::glu.out`.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/79866
Approved by: https://github.com/kulinseth, https://github.com/albanD
 2022-06-30 08:58:42 +00:00
Kulin Seth
c71886e048 [MPS] Add test consistency from OpInfo based tests from PR 78504 (#79532) 
Pull Request resolved: https://github.com/pytorch/pytorch/pull/79532
Approved by: https://github.com/albanD
 2022-06-30 01:50:17 +00:00
qqaatw
5943aaa0c4 [MPS] Add logical ops (#80216) 
This PR adds `logical_not`, `logical_and`, `logical_or`, `logical_xor`.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80216
Approved by: https://github.com/albanD, https://github.com/kulinseth
 2022-06-29 02:44:35 +00:00
qqaatw
c4da23ed1b [MPS] Add flip (#80214) 
Fixes #ISSUE_NUMBER

Pull Request resolved: https://github.com/pytorch/pytorch/pull/80214
Approved by: https://github.com/DenisVieriu97, https://github.com/albanD
 2022-06-28 19:51:45 +00:00
qqaatw
e1b15b7a04 [MPS] add aten::normal.Tensor_float aten::normal.float_Tensor aten::normal.Tensor_Tensor (#80297) 
Fixes #ISSUE_NUMBER

Pull Request resolved: https://github.com/pytorch/pytorch/pull/80297
Approved by: https://github.com/albanD, https://github.com/kulinseth
 2022-06-28 15:19:39 +00:00
Nikita Shulga
f11cce309b [MPS] Add equal operator (#80195) 
Which is, in essence is composite of `eq`->`all`->`item`
`native/mps/operators/Equal.cpp` is an almost verbatim copy of `native/cuda/Equal.cpp`

Fix codegen by generating MPSFunctions headers

Pull Request resolved: https://github.com/pytorch/pytorch/pull/80195
Approved by: https://github.com/albanD
 2022-06-25 12:40:52 +00:00
Nikita Shulga
06f874e276 [MPS] Fix binary ops between int32 tensor with int64 scalar (#80220) 
For some reason, tensor *op* scalar does not follow the normal binary promotion rules
So cast output tensor to expected type if needed
It seems that one should have casted input tensors to expected output tensor type, but it does not really work for boolean binary ops, so...
Add output tensor type/shape to cached graph key
Extend `TestMPS. test_add_scalars` to test for this regression

Fixes #79835

Pull Request resolved: https://github.com/pytorch/pytorch/pull/80220
Approved by: https://github.com/albanD
 2022-06-25 02:21:34 +00:00
qqaatw
ff44bfa1ea [MPS] Add L1 loss test (#80010) 
Fixes #ISSUE_NUMBER

Pull Request resolved: https://github.com/pytorch/pytorch/pull/80010
Approved by: https://github.com/albanD
 2022-06-24 17:18:31 +00:00
Nikita Shulga
4390546f86 [MPS] Fix torch.uint8 support (#80049) 
`ScalarType.Byte` should be cast to `MPSDataTypeUInt8`
And support for `torch.int8` as well as test those conversions in `TestMPS.test_to`

Fixes #80006

Pull Request resolved: https://github.com/pytorch/pytorch/pull/80049
Approved by: https://github.com/albanD
 2022-06-22 18:41:21 +00:00
Abhishek Pathak
074dc7465e MPS: Add amax and amin Ops with tests (#79682) 
* Add amax and amin with tests

Fixes #ISSUE_NUMBER

Pull Request resolved: https://github.com/pytorch/pytorch/pull/79682
Approved by: https://github.com/albanD
 2022-06-18 00:14:05 +00:00
Kulin Seth
4615f6aa97 [MPS]: Add fix for squeezed input axes handling in BCE loss (#79676) 
Fixes #79527

Pull Request resolved: https://github.com/pytorch/pytorch/pull/79676
Approved by: https://github.com/razarmehr, https://github.com/albanD
 2022-06-16 20:21:31 +00:00
Kulin Seth
355a1c8c3f MPS: TopK raise an error if K>16 (#79677) 
* Error out in TopK when k>16.
* Add a test case too.

Fixes #78915

Pull Request resolved: https://github.com/pytorch/pytorch/pull/79677
Approved by: https://github.com/albanD
 2022-06-16 16:06:45 +00:00
Nikita Shulga
81cd276d61 [MPS] Support stride of stride 
Fixes https://github.com/pytorch/pytorch/issues/79181

Pull Request resolved: https://github.com/pytorch/pytorch/pull/79521

Approved by: https://github.com/kulinseth
 2022-06-14 18:49:44 +00:00
Alban Desmaison
0a651a231d Add full support for serialization of MPS Tensors (#79465) 
Fix https://github.com/pytorch/pytorch/issues/79384
Pull Request resolved: https://github.com/pytorch/pytorch/pull/79465
Approved by: https://github.com/kulinseth, https://github.com/malfet
 2022-06-14 17:54:30 +00:00
PyTorch MergeBot
ce6ce74703 Revert "Add full support for serialization of MPS Tensors (#79465)" 
This reverts commit 64c2a275c4.

Reverted https://github.com/pytorch/pytorch/pull/79465 on behalf of https://github.com/zengk95 due to this broke X linux-xenial-py3.7-clang7-onnx / test (default, 1, 2, linux.2xlarge). Not sure why since it passed on pull.
 2022-06-14 16:42:36 +00:00
Alban Desmaison
64c2a275c4 Add full support for serialization of MPS Tensors (#79465) 
Fix https://github.com/pytorch/pytorch/issues/79384
Pull Request resolved: https://github.com/pytorch/pytorch/pull/79465
Approved by: https://github.com/kulinseth, https://github.com/malfet
 2022-06-14 14:20:09 +00:00
Kulin Seth
77b6885a22 MPS: add layer_norm_backward (#79189) 
Layernorm backward

Fixes #ISSUE_NUMBER

Pull Request resolved: https://github.com/pytorch/pytorch/pull/79189
Approved by: https://github.com/razarmehr, https://github.com/albanD
 2022-06-10 13:25:41 +00:00
Kulin Seth
83239351c5 MPS: add exponential op (#79188) 
Add exponential distribution

Fixes #ISSUE_NUMBER

Pull Request resolved: https://github.com/pytorch/pytorch/pull/79188
Approved by: https://github.com/razarmehr, https://github.com/albanD
 2022-06-10 13:16:21 +00:00
Kulin Seth
50f7b40ad9 MPS: Binary cast fix by proper type promotion and remove spurious copy warning (#79185) 
Fixes #78019, #78020
Fixes https://github.com/pytorch/pytorch/pull/79185
Pull Request resolved: https://github.com/pytorch/pytorch/pull/79185
Approved by: https://github.com/albanD, https://github.com/razarmehr
 2022-06-09 17:33:06 +00:00
Nikita Shulga
97594a24b4 Print output during MPS test import tests (#79163) 
Simplify `test_no_warnings_on_input` to simply capture any output.
Copy its implementation to `test_testing.py` as this is not specific to MPS
Pull Request resolved: https://github.com/pytorch/pytorch/pull/79163
Approved by: https://github.com/janeyx99, https://github.com/kulinseth
 2022-06-09 13:07:05 +00:00
Philip Meier
32593ef2dd move MPS compat into common comparison machinery (#77836) 
Addresses https://github.com/pytorch/pytorch/issues/77144#issuecomment-1128168082.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/77836
Approved by: https://github.com/albanD
 2022-06-08 08:09:18 +00:00
Kulin Seth
a6347f5467 MPS: Fixes (#78930) 
Cast integer to float in UnaryOps
Add tensor dtype in key generation
Enable FP16 scalars and use placeholder for alpha tensor in add/sum ops

Fixes #ISSUE_NUMBER

Pull Request resolved: https://github.com/pytorch/pytorch/pull/78930
Approved by: https://github.com/albanD
 2022-06-07 18:22:10 +00:00
Nikita Shulga
55cac22cdf [MPS] Add arange_mps_out implementation (#78789) 
Mostly by factoring out shader logic from `linspace_out_mps` implementation
Pull Request resolved: https://github.com/pytorch/pytorch/pull/78789
Approved by: https://github.com/albanD, https://github.com/kulinseth
 2022-06-03 21:54:41 +00:00
Kulin Seth
4858c56334 MPS: Fix issues with view tensors and linspace. (#78690) 
Fixes: #https://github.com/pytorch/pytorch/issues/78642, https://github.com/pytorch/pytorch/issues/78511
Pull Request resolved: https://github.com/pytorch/pytorch/pull/78690
Approved by: https://github.com/razarmehr, https://github.com/DenisVieriu97
 2022-06-02 06:17:19 +00:00
Kulin Seth
a3bdafece3 MPS: add linespace op (#78570) 
Fixes #ISSUE_NUMBER

Pull Request resolved: https://github.com/pytorch/pytorch/pull/78570
Approved by: https://github.com/malfet
 2022-06-01 13:47:14 +00:00
Ramin Azarmehr
aa62b3e003 Add test case for issue: https://github.com/pytorch/pytorch/issues/77851 (#78547) 
The test works fine now.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/78547
Approved by: https://github.com/kulinseth
 2022-05-31 19:15:45 +00:00
Rohan Mitchell
f42b42d3eb MPS: Implement aten::count_nonzero.dim_IntList (#78169) 
- See: #77764

Implements the `aten::count_nonzero.dim_IntList` operator (as used by [torch.count_nonzero](https://pytorch.org/docs/stable/generated/torch.count_nonzero.html)) for [MPS](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/).

Pull Request resolved: https://github.com/pytorch/pytorch/pull/78169
Approved by: https://github.com/malfet, https://github.com/kulinseth, https://github.com/albanD
 2022-05-31 18:23:25 +00:00
Kulin Seth
017b0ae943 MPS: Fix crashes in view tensors due to buffer size mismatch (#78496) 
Fixes #78247, #77886

Pull Request resolved: https://github.com/pytorch/pytorch/pull/78496
Approved by: https://github.com/albanD, https://github.com/malfet
 2022-05-31 02:09:03 +00:00
Alban Desmaison
bde246fcc6 Speed up test_mps from 9min to 25s 
Pull Request resolved: https://github.com/pytorch/pytorch/pull/78488

Approved by: https://github.com/kulinseth
 2022-05-30 18:16:53 +00:00
Alban Desmaison
02551a0025 Remove prints and add proper asserts 
Pull Request resolved: https://github.com/pytorch/pytorch/pull/78454

Approved by: https://github.com/kulinseth
 2022-05-30 18:16:53 +00:00
Kulin Seth
d63db52349 MPS: Fixes the as_strided_mps implementation for contiguous view operations (#78440) 
Fixes https://github.com/pytorch/pytorch/issues/78107; https://github.com/pytorch/pytorch/issues/77750

Pull Request resolved: https://github.com/pytorch/pytorch/pull/78440
Approved by: https://github.com/malfet
 2022-05-28 14:41:56 +00:00
Nikita Shulga
437ecfc461 [MPS] Fix copy_kernel_mps (#78428) 
By passing `storage_offset` of source and destination Tensors
This fixes following simple usecase:
```
python3` -c "import torch;x=torch.zeros(3, 3, device='mps'); x[1, 1]=1;print(x)"
```

Add test to validate it would not regress in the future

Pull Request resolved: https://github.com/pytorch/pytorch/pull/78428
Approved by: https://github.com/kulinseth
 2022-05-27 20:46:53 +00:00
Kulin Seth
8552acbd74 MPS: Eye op (#78408) 
This can be used as a reference PR was to add Op in MPS backend.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/78408
Approved by: https://github.com/albanD
 2022-05-27 17:07:02 +00:00
Kulin Seth
2e32d5fcd8 MPS: Add adaptive max pool2d op (#78410) 
Adaptive max pool 2d forward and backward with test

Pull Request resolved: https://github.com/pytorch/pytorch/pull/78410
Approved by: https://github.com/albanD
 2022-05-27 11:59:07 +00:00
Nikita Shulga
705082656a Fix typo in testname (#78258) 
`test_linear2D_no_bias_backwarwd` -> `test_linear2D_no_bias_backward`

Pull Request resolved: https://github.com/pytorch/pytorch/pull/78258
Approved by: https://github.com/kulinseth, https://github.com/janeyx99
 2022-05-25 16:23:10 +00:00
Lukas Hoenig
a52bfe2c5d Convert MPS Tensor data using MPSGraph API (#78092) 
Fixes #78091
If you are already working on this, simply disregard this or take what may be helpful. This is my attempt at MPS-native Tensor datatype conversion. It works for everything tested ~~but is currently only implemented for MPS-to-MPS copy, not MPS-to-X or X-to-MPS, but the same approach could easily be used~~.

Before:
```python
In [5]: pt.full((40,), -10.3, device="mps")
Out[5]:
tensor([-10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000,
        -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000,
        -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000,
        -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000,
        -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000,
        -10.3000, -10.3000, -10.3000, -10.3000, -10.3000], device='mps:0')

In [6]: pt.full((40,), -10.3, device="mps").int()
Out[6]:
tensor([-1054552883, -1054552883, -1054552883, -1054552883, -1054552883,
        -1054552883, -1054552883, -1054552883, -1054552883, -1054552883,
        -1054552883, -1054552883, -1054552883, -1054552883, -1054552883,
        -1054552883, -1054552883, -1054552883, -1054552883, -1054552883,
        -1054552883, -1054552883, -1054552883, -1054552883, -1054552883,
        -1054552883, -1054552883, -1054552883, -1054552883, -1054552883,
        -1054552883, -1054552883, -1054552883, -1054552883, -1054552883,
        -1054552883, -1054552883, -1054552883, -1054552883, -1054552883],
       device='mps:0', dtype=torch.int32)

In [7]: pt.full((40,), -10.3, device="mps").int().float()
Out[7]:
tensor([-10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000,
        -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000,
        -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000,
        -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000,
        -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000,
        -10.3000, -10.3000, -10.3000, -10.3000, -10.3000], device='mps:0')

In [8]: pt.full((40,), -10.3, device="mps").int().float().bool()
Out[8]:
tensor([ True, False, False,  True,  True, False, False,  True,  True, False,
        False,  True,  True, False, False,  True,  True, False, False,  True,
         True, False, False,  True,  True, False, False,  True,  True, False,
        False,  True,  True, False, False,  True,  True, False, False,  True],
       device='mps:0')
```

After:
```python
In [3]: pt.full((40,), -10.3, device="mps")
Out[3]:
tensor([-10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000,
        -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000,
        -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000,
        -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000,
        -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000, -10.3000,
        -10.3000, -10.3000, -10.3000, -10.3000, -10.3000], device='mps:0')

In [4]: pt.full((40,), -10.3, device="mps").int()
Out[4]:
tensor([-10, -10, -10, -10, -10, -10, -10, -10, -10, -10, -10, -10, -10, -10,
        -10, -10, -10, -10, -10, -10, -10, -10, -10, -10, -10, -10, -10, -10,
        -10, -10, -10, -10, -10, -10, -10, -10, -10, -10, -10, -10],
       device='mps:0', dtype=torch.int32)

In [5]: pt.full((40,), -10.3, device="mps").int().float()
Out[5]:
tensor([-10., -10., -10., -10., -10., -10., -10., -10., -10., -10., -10., -10.,
        -10., -10., -10., -10., -10., -10., -10., -10., -10., -10., -10., -10.,
        -10., -10., -10., -10., -10., -10., -10., -10., -10., -10., -10., -10.,
        -10., -10., -10., -10.], device='mps:0')

In [6]: pt.full((40,), -10.3, device="mps").int().float().bool()
Out[6]:
tensor([True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True], device='mps:0')
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/78092
Approved by: https://github.com/kulinseth, https://github.com/malfet
 2022-05-24 20:09:45 +00:00
Alban Desmaison
04ac80c73a Fix a few issues on assert/double error/legacy constructor (#77966) 
Fixes https://github.com/pytorch/pytorch/issues/77960, https://github.com/pytorch/pytorch/issues/77957, https://github.com/pytorch/pytorch/issues/77781
Pull Request resolved: https://github.com/pytorch/pytorch/pull/77966
Approved by: https://github.com/soulitzer, https://github.com/kulinseth
 2022-05-20 20:25:12 +00:00
Kulin Seth
3d83321b44 MPS Fixes: copy operations, addmm and baddmm (#77791) 
Fixes for the copy operations and GEMM operations on MPS backend.

Fixes https://github.com/pytorch/pytorch/issues/77819
Pull Request resolved: https://github.com/pytorch/pytorch/pull/77791
Approved by: https://github.com/albanD
 2022-05-20 03:18:11 +00:00
Kulin Seth
978304fc9c MPS: fixes (#77462) 
- Fix the is_available flag for x86 machines
- Fix the tensor creation for older MacOS platforms
- Addmm fixes for transposition

Pull Request resolved: https://github.com/pytorch/pytorch/pull/77462
Approved by: https://github.com/albanD
 2022-05-14 13:33:16 +00:00
Kulin Seth
e011a8e18b Enable PyTorch operations on MPS Backend. (#77343) 
Add PyTorch operations to MPS backend.

- https://github.com/pytorch/pytorch/issues/77394
Pull Request resolved: https://github.com/pytorch/pytorch/pull/77343
Approved by: https://github.com/albanD
 2022-05-13 18:28:53 +00:00