Summary:
OrderedImporters is supposed to be an importer which tries out every single importer in self._importers. However the get_name API does not follow this behavior and only uses the get_name from the basic Importer class.
This change is to update the OrderedImporters get_name API so that it tries the get_name API of every single importers.
Differential Revision: D76463252
Pull Request resolved: https://github.com/pytorch/pytorch/pull/155743
Approved by: https://github.com/jcwchen, https://github.com/jingsh
Summary:
Applies new import merging and sorting from µsort v1.0.
When merging imports, µsort will make a best-effort to move associated
comments to match merged elements, but there are known limitations due to
the diynamic nature of Python and developer tooling. These changes should
not produce any dangerous runtime changes, but may require touch-ups to
satisfy linters and other tooling.
Note that µsort uses case-insensitive, lexicographical sorting, which
results in a different ordering compared to isort. This provides a more
consistent sorting order, matching the case-insensitive order used when
sorting import statements by module name, and ensures that "frog", "FROG",
and "Frog" always sort next to each other.
For details on µsort's sorting and merging semantics, see the user guide:
https://usort.readthedocs.io/en/stable/guide.html#sorting
Test Plan: S271899
Reviewed By: lisroach
Differential Revision: D36402110
Pull Request resolved: https://github.com/pytorch/pytorch/pull/78973
Approved by: https://github.com/osalpekar
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/72533
The current check that we have for dummy packages is too expansive; it
will skip anything without a `__file__`, including extension modules in
the standard library.
So first check if a module was created by torch.package before skipping
it, which should rule out anything accidentally getting skipped (as the
only time torch.package creates something without a `__file__` is the
dummy case).
Test Plan: Imported from OSS
Reviewed By: PaliC
Differential Revision: D34082792
Pulled By: suo
fbshipit-source-id: 18b17eb0f693927697657b20843ec5cd8bcccb47
(cherry picked from commit f571370078)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/71520
Specifically this pr is to deal with adverse cases such as D33662192
It is possible for someone to export a package, change something in it, and then attempt to repackage it. In this case it's possible that dependencies of the package are no longer interned. In this case it is not obvious where we would look for these packages, therefore, we throw an error.
Test Plan: Imported from OSS
Reviewed By: bradleyhd
Differential Revision: D33675557
Pulled By: PaliC
fbshipit-source-id: 807962bfb340d30d418617d6e78661a033828314
(cherry picked from commit 1b10c23807)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/57651
We've gone back and forth on whether to emulate the `sys.modules` lookup
behavior in our own `whichmodule`, the provided test is a concrete case
for doing so.
An additional minor cleanup is to make the type of `self.modules` in
importers `Dict[str, ModuleType]`. Modules could only be None in the
dictionary in older versions of the import system
Test Plan: Imported from OSS
Reviewed By: SplitInfinity
Differential Revision: D28226536
Pulled By: suo
fbshipit-source-id: c2e6da91651ddaa4fbf7171555df9e5cbe1060fd
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/53783
Use isort + black on torch/package/
Test Plan: Imported from OSS
Reviewed By: SplitInfinity
Differential Revision: D26969020
Pulled By: suo
fbshipit-source-id: e2c0738e79bf41b6342355eb7025998178c35dc9
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/51676
We offer the ability to access the importer from within packaged modules by doing
`import resources`. This behavior is nice (and more powerful than the
importlib resources API), but I think `resources` is too common a name
(pip has a package for it)
Change to `import torch_package_importer` but open to bikeshedding
Test Plan: Imported from OSS
Reviewed By: jamesr66a
Differential Revision: D26620314
Pulled By: suo
fbshipit-source-id: 0942c99f02c0f55f5f3a1b2566961018b796bdd4
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/53061
We only care about evaluating the string return version. If `reduce()`
throws an error, we should just continue on with pickling.
Test Plan: Imported from OSS
Reviewed By: Lilyjjo
Differential Revision: D26737652
Pulled By: suo
fbshipit-source-id: 0b6fbbe345ad0b6a33330b2efa39d7bab703193d
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/51975
See comments in code.
Test Plan: Imported from OSS
Reviewed By: zdevito
Differential Revision: D26340592
Pulled By: suo
fbshipit-source-id: 61b16bafad15e19060710ad2d8487c776d672847
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/52320
as title
Test Plan: Imported from OSS
Reviewed By: zdevito
Differential Revision: D26468416
Pulled By: suo
fbshipit-source-id: 890eecea76426918daff900402fbcbc149e48535
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/51754
This API allows you to manage multiple python interpreters in a single
process to deploy PyTorch models packaged with torch.package.
torch/csrc/deploy/deploy.h contains the API definition
torch/csrc/deploy/test_deploy.cpp has some examples.
Notes:
* mutex is added to PyTorchStreamReader to make it safe to use from multiple threads at once.
* USE_DEPLOY is only true for the special libtorch_deployinterpreter.so library, when enabled
we use a hash table to maintain PyObject <> at::Tensor mappping rather than the internal pointer
in Tensor since >1 interpreter may have a reference to the tensor.
* serialization.py has some additional functions for creating pickle objects
but keeping storages in memory for use transfering tensors between interpreters
Test Plan: Imported from OSS
Reviewed By: wconstab
Differential Revision: D26329468
Pulled By: zdevito
fbshipit-source-id: d75f4ebb9a27f1d911179d9996041bcb3ca04a07
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/51826
Looks like this:
```
resnet.pt
├── .data # Data folder named so it can't clash with torch.package codemodules.
│ │ # Names/extensions automatically added to avoid namingconflicts.
│ ├── 94286146172688.storage # tensor data
│ ├── 94286146172784.storage
│ ├── extern_modules # torch.package metadata
│ ├── version # version metadata
│ └── ...
├── model # package pickled model created w/
│ │ # exporter.save_pickel('model','model.pkl', resnet_model)
│ └── model.pkl
└── torchvision # all code dependencies for packaged picked
└── models # models are captured as source files
├── resnet.py
└── utils.py
```
Since `version` is hardcoded in our zip reader/writer implementation,
add it as an option that defaults to "version" but accepts other
locations for putting the version metadata.
Test Plan: Imported from OSS
Reviewed By: zdevito
Differential Revision: D26295649
Pulled By: suo
fbshipit-source-id: 2d75feeb7de0f78196b4d0b6e2b814a7d58bd1dd
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/51595
Right now `PackageExporter` defines its own `persistent_id` but
`PackageImporter` uses the one defined in `torch.serialization`. I have
some downstream plans to customize this so this PR just splits it out.
Not to fear! I know this introduces some duplication and potential for
different behavior between `torch.save` and `torch.package`, but I have
plans to re-unify them soon.
Test Plan: Imported from OSS
Reviewed By: zdevito
Differential Revision: D26211578
Pulled By: suo
fbshipit-source-id: 48a2ccaefb2525e1498ad68b75c46d9de3d479b7
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/51666
This ensures the modules will get properly unloaded when all references
to them die
Test Plan: Imported from OSS
Reviewed By: Lilyjjo
Differential Revision: D26232574
Pulled By: suo
fbshipit-source-id: a9889965aa35ba2f6cbbfbdd13e02357cc706cab
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/51568
The default behavior of inspect.getfile doesn't work on classes imported
from PackageImporter, because it returns the following.
sys.modules[kls.__module__].__file__
Looking in `sys.modules` is hard-coded behavior. So, patch it to first
check a similar registry of PackageImported modules we maintain.
Test Plan: Imported from OSS
Reviewed By: yf225
Differential Revision: D26201236
Pulled By: suo
fbshipit-source-id: aaf5d7ee8ca0155619c8185e64f70a30152ac567
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/50838
Similar to `torch.save` and `torch.jit.save`, accept a IO-like object
instead of just a file.
Test Plan: Imported from OSS
Reviewed By: nikithamalgifb
Differential Revision: D25982719
Pulled By: suo
fbshipit-source-id: 42f3665932bbaa6897215002d116df6338edae50
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/50711
As title, missed a few of these.
Test Plan: Imported from OSS
Reviewed By: yf225
Differential Revision: D25949363
Pulled By: suo
fbshipit-source-id: 197743fe7097d2ac894421a99c072696c3b8cd70
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/50049
Rationale and implementation immortalized in a big comment in
`torch/package/mangling.md`.
This change also allows imported modules to be TorchScripted
Test Plan: Imported from OSS
Reviewed By: pbelevich
Differential Revision: D25758625
Pulled By: suo
fbshipit-source-id: 77a99dd2024c76716cfa6e59c3855ed590efda8b
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/48799
Python's IMPORT_FROM bytecode will bypass the import infrastructure
when a packaging being loaded as part of a cirular dependency is being
accessed from the module _before_ that package has finished loading
and is installed on the module. Since we cannot override the lookup
on sys.modules, this PR pre-emptively does the module assignment before
running the submodules initialization code.
Note: this appears to work, but it is not clear to me why python doesn't
do this by default. It is possible that the logic for creating modules
is flexible enough in generic python that this interception between creating
the module and running its code is not always possible.
Test Plan: Imported from OSS
Reviewed By: suo
Differential Revision: D25312467
Pulled By: zdevito
fbshipit-source-id: 6fe3132af29364ccb2b3cabdd2b847d0a09eb515
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/45015
torch.package allows you to write packages of code, pickled python data, and
arbitrary binary and text resources into a self-contained package.
torch.package.PackageExporter writes the packages and
torch.package.PackageImporter reads them.
The importers can load this code in a hermetic way, such that code is loaded
from the package rather than the normal python import system. This allows
for the packaging of PyTorch model code and data so that it can be run
on a server or used in the future for transfer learning.
The code contained in packages is copied file-by-file from the original
source when it is created, and the file format is a specially organized
zip file. Future users of the package can unzip the package, and edit the code
in order to perform custom modifications to it.
The importer for packages ensures that code in the module can only be loaded from
within the package, except for modules explicitly listed as external using :method:`extern_module`.
The file `extern_modules` in the zip archive lists all the modules that a package externally depends on.
This prevents "implicit" dependencies where the package runs locally because it is importing
a locally-installed package, but then fails when the package is copied to another machine.
Test Plan: Imported from OSS
Reviewed By: SplitInfinity
Differential Revision: D23824337
Pulled By: zdevito
fbshipit-source-id: 1247c34ba9b656f9db68a83e31f2a0fbe3bea6bd
