pytorch/docs/source
Stas Bekman 7f474da6bc [docs] start a new FSDP notes doc (#117323)
As discussed on [slack](https://pytorch.slack.com/archives/C3PDTEV8E/p1703699711772289) adding Andrew Gu's advanced FSDP design notes with a few additions from myself based on our discussion.

I hope I did the RST right, I haven't done RST in a while.

- The first section is Andrew's words verbatim + formatting
- The second section is Andrew's words verbatim + formatting + a few of my additions that were confirmed by Andrew, and which hopefully should help understand the process better.

tagging @albanD as requested.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/117323
Approved by: https://github.com/albanD, https://github.com/awgu
2024-01-21 15:11:24 +00:00
..
_static Removing HTA documentation (#116513) 2023-12-28 23:04:23 +00:00
_templates
community Add thiagocrepaldi as person of interest for onnx exporter (#113402) 2023-11-10 15:19:58 +00:00
elastic [TorchElastic] Support for overprovisioning in C10 based rendezvous (#117066) 2024-01-18 01:16:55 +00:00
notes [docs] start a new FSDP notes doc (#117323) 2024-01-21 15:11:24 +00:00
rpc
scripts Fixed typo in build_activation_images.py (#117458) 2024-01-15 03:27:40 +00:00
amp.rst Add torch.utils.deterministic.fill_uninitialized_memory flag (#111377) 2023-11-01 16:10:09 +00:00
autograd.rst Allow specifiying inputs as GradientEdge in autograd APIs (#110867) 2023-10-12 04:08:44 +00:00
backends.rst Revert "[CUDNN][SDPA] Experimental cuDNN Flash Attention v2 Inference (#115663)" 2024-01-18 23:40:30 +00:00
benchmark_utils.rst
bottleneck.rst
checkpoint.rst Add set_checkpoint_debug_enabled that overrides local setting (#110728) 2023-10-11 02:12:31 +00:00
complex_numbers.rst Update mentions of deprecated functions if complex_numbers.rst (#113391) 2023-11-09 22:32:26 +00:00
cond.rst [HigherOrderOp] expose torch.cond (#110293) 2023-10-07 20:39:52 +00:00
conf.py Initial torchbind support in PT2 (#117697) 2024-01-19 06:28:20 +00:00
config_mod.rst
cpp_extension.rst
cpp_index.rst
cpu.rst Add current_device() to torch.cpu (#110987) 2023-10-11 05:13:10 +00:00
cuda._sanitizer.rst
cuda.rst Doc test non packages (#110568) 2023-10-06 14:16:01 +00:00
cudnn_persistent_rnn.rst
cudnn_rnn_determinism.rst
data.rst Revert "reseed all Generators in Dataloader's _worker_loop() -- via GC (#107131)" 2023-08-23 17:08:07 +00:00
ddp_comm_hooks.rst [DOCS][DDP]Fix the simple of saving and reloading PowerSGD state and hook. (#102721) 2023-06-10 00:15:00 +00:00
deploy.rst
deterministic.rst Add torch.utils.deterministic.fill_uninitialized_memory flag (#111377) 2023-11-01 16:10:09 +00:00
distributed.algorithms.join.rst
distributed.checkpoint.rst adds async save, makes checkpointer private (#116293) 2023-12-22 05:22:39 +00:00
distributed.elastic.rst
distributed.optim.rst
distributed.rst adds async save, makes checkpointer private (#116293) 2023-12-22 05:22:39 +00:00
distributed.tensor.parallel.rst [tp] improve documentation (#115880) 2023-12-15 18:44:22 +00:00
distributions.rst Add inverse gamma distribution and fix sign bug in PowerTransform. (#104501) 2023-11-01 02:26:25 +00:00
dlpack.rst
docutils.conf
export.ir_spec.rst [export][refactor][4/n] Make equality_constraints optional (#116233) 2024-01-05 00:50:52 +00:00
export.rst [export] Add lifted constant obj to input (#116985) 2024-01-18 22:10:53 +00:00
fft.rst
fsdp.rst [FSDP][state_dict] Expose optimizer state_dict config (#105949) 2023-08-21 07:29:49 +00:00
func.api.rst
func.batch_norm.rst
func.migrating.rst
func.rst
func.ux_limitations.rst
func.whirlwind_tour.rst
futures.rst
fx.rst Split SymNode into its own file (#112037) 2023-10-26 23:32:27 +00:00
hub.rst
index.rst Removing HTA documentation (#116513) 2023-12-28 23:04:23 +00:00
jit_builtin_functions.rst
jit_language_reference_v2.rst
jit_language_reference.rst
jit_python_reference.rst
jit_unsupported.rst Add support for torch.Generator type in TorchScript (#110413) 2023-11-21 23:07:21 +00:00
jit_utils.rst
jit.rst Doc test non packages (#110568) 2023-10-06 14:16:01 +00:00
library.rst Rewrite torch.library's documentation (#111310) 2023-10-23 23:02:41 +00:00
linalg.rst
logging.rst [Easy] log graphs in compiled_autograd if TORCH_LOGS=compiled_autograd (#108991) 2023-09-12 00:15:02 +00:00
masked.rst Doc test non packages (#110568) 2023-10-06 14:16:01 +00:00
math-quantizer-equation.png
mobile_optimizer.rst
model_zoo.rst
monitor.rst
mps.rst Doc test non packages (#110568) 2023-10-06 14:16:01 +00:00
multiprocessing.rst Doc test non packages (#110568) 2023-10-06 14:16:01 +00:00
name_inference.rst [docs] Properly link register_post_accumulate_grad_hook docs (#108157) 2023-08-29 22:13:33 +00:00
named_tensor.rst fixing named tensor unflatten example (#106921) 2023-08-22 18:00:10 +00:00
nested.rst
nn.attention.bias.rst Add an attention bias subclass for a lower right causal masking (#114823) 2023-12-06 08:29:26 +00:00
nn.functional.rst Add python and C++ support for LPPool3d (#114199) 2023-12-08 18:18:44 +00:00
nn.init.rst
nn.rst WIP Add compatibility with channels_last_3d for conv3d (#114790) 2023-12-20 19:28:25 +00:00
onnx_dynamo_onnxruntime_backend.rst Follow-up #108379 (#108905) 2023-09-09 01:38:36 +00:00
onnx_dynamo.rst [ez][doc] Fix sample code in onnx_dynamo.rst (#114770) 2023-11-29 19:27:52 +00:00
onnx_torchscript_supported_aten_ops.rst Refactor torch.onnx documentation (#108379) 2023-09-08 18:23:48 +00:00
onnx_torchscript.rst Follow-up #108379 (#108905) 2023-09-09 01:38:36 +00:00
onnx.rst Doc test non packages (#110568) 2023-10-06 14:16:01 +00:00
optim.rst Doc test non packages (#110568) 2023-10-06 14:16:01 +00:00
package.rst Doc test non packages (#110568) 2023-10-06 14:16:01 +00:00
pipeline.rst
profiler.rst Doc test non packages (#110568) 2023-10-06 14:16:01 +00:00
quantization-accuracy-debugging.rst
quantization-backend-configuration.rst
quantization-support.rst [quant][pt2e] Add generate_numeric_debug_handle pass (#114315) 2023-12-01 03:38:17 +00:00
quantization.rst [ao] updating embedding_bag support for fx and eager (#107623) 2023-11-21 03:54:00 +00:00
random.rst
rpc.rst [BE] RPC is missing RRef docs (#106902) 2023-08-10 16:26:27 +00:00
signal.rst
sparse.rst Fix typo in https://pytorch.org/docs/stable/sparse.html (#115282) 2023-12-08 18:31:33 +00:00
special.rst
storage.rst
tensor_attributes.rst
tensor_view.rst
tensorboard.rst
tensors.rst add float8 types to dtypes table (#117375) 2024-01-15 00:23:07 +00:00
testing.rst
torch_cuda_memory.rst Fix typo under docs directory (#110359) 2023-10-03 16:36:05 +00:00
torch.ao.ns._numeric_suite_fx.rst
torch.ao.ns._numeric_suite.rst
torch.compiler_aot_inductor.rst [AOTI][refactor] Organize model runner files (#116022) 2023-12-20 15:35:34 +00:00
torch.compiler_api.rst Add cudagraph_mark_step_begin in torch.compiler, reference in error message (#111722) 2023-10-25 21:53:21 +00:00
torch.compiler_best_practices_for_backends.rst Restructure torch.compile docs (#105376) 2023-07-28 20:58:57 +00:00
torch.compiler_cudagraph_trees.rst [docs] add mode="reduce-overhead" into torch.compile to enable cuda g… (#116529) 2024-01-05 22:54:20 +00:00
torch.compiler_custom_backends.rst [docs, dynamo] fix typos in dynamo custom backend docs (#115444) 2023-12-08 23:58:26 +00:00
torch.compiler_deepdive.rst [Dynamo]Expose bytecode hooks and add example usage for decompilation in docs (#110714) 2023-10-13 12:36:00 +00:00
torch.compiler_dynamic_shapes.rst Update dynamic shapes documentation (#109764) 2023-09-21 13:53:43 +00:00
torch.compiler_fake_tensor.rst Restructure torch.compile docs (#105376) 2023-07-28 20:58:57 +00:00
torch.compiler_faq.rst Add a wrapper to transform a NumPy function into a PyTorch function (#114610) 2024-01-02 18:35:29 +00:00
torch.compiler_fine_grain_apis.rst Restructure torch.compile docs (#105376) 2023-07-28 20:58:57 +00:00
torch.compiler_get_started.rst [Reland2] [inductor][BE] split triton_meta and inductor_meta (#112351) 2023-11-02 00:40:12 +00:00
torch.compiler_guards_overview.rst Do not use a specific LOC in link (#108957) 2023-09-13 19:21:45 +00:00
torch.compiler_inductor_profiling.rst Restructure torch.compile docs (#105376) 2023-07-28 20:58:57 +00:00
torch.compiler_ir.rst [export] torch.export landing page (#108783) 2023-09-10 01:40:42 +00:00
torch.compiler_nn_module.rst Revert "Reland 3rd try [finishing colesbury's PR 100642] Guard on nn.Module dicts and type (#109323)" + Forward fixes + test (#110964) 2023-10-11 05:16:47 +00:00
torch.compiler_performance_dashboard.rst Restructure torch.compile docs (#105376) 2023-07-28 20:58:57 +00:00
torch.compiler_profiling_torch_compile.rst Restructure torch.compile docs (#105376) 2023-07-28 20:58:57 +00:00
torch.compiler_transformations.rst Fix typo under docs directory (#110359) 2023-10-03 16:36:05 +00:00
torch.compiler_troubleshooting.rst Update torch.compiler_troubleshooting.rst (#114530) 2023-11-25 23:15:47 +00:00
torch.compiler.rst [docs] Fix torch.compile "tensorrt" backend docs (#113711) 2023-11-15 08:42:53 +00:00
torch.overrides.rst Doc test non packages (#110568) 2023-10-06 14:16:01 +00:00
torch.rst Some tiny modification about torch.set/get_default_device (#116014) 2023-12-19 05:08:06 +00:00
type_info.rst
utils.rst New swap function (#111747) 2023-12-08 18:49:35 +00:00