ESM resolution and loading is now always synchronous from a
non-loader-hook thread. If no asynchrnous loader hooks are
registered, the resolution/loading is entirely synchronous.
If asynchronous loader hooks are registered, these would be
synchronous on the non-loader-hook thread, and asynchronous
on the loader hook thread.
This avoids several races caused by async/sync loading sharing
the same cache. In particular, asynchronous loader hooks
now works with `require(esm)` - previously it tends to break
due to races.
In addition, when an asynchronous loader hook
returns a promise that never settles, the main thread no longer
silently exits with exit code 13, leaving the code below
any module loading calls silently ignored without being executed.
Instead, it now throws ERR_ASYNC_LOADER_REQUEST_NEVER_SETTLED
which can be caught and handled by the main thread. If the module
request comes from `import()`, the never-settling promise is
now relayed to the result returned by `import()`.
Drive-by: when annotating the error about importing undetectable
named exports from CommonJS, it now no longer reload the source
code of the CommonJS module, and instead reuses format information
cached when the module was loaded for linking.
PR-URL: https://github.com/nodejs/node/pull/60380
Fixes: https://github.com/nodejs/node/issues/59666
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Jacob Smith <jacob@frende.me>
Otherwise if the ESM happens to be cached separately by the ESM loader
before it gets loaded with `require(esm)` from within an imported
CJS file (which uses a re-invented require() with a couple of quirks,
including a separate cache), it won't be able to load the esm properly
from the cache.
PR-URL: https://github.com/nodejs/node/pull/59679
Refs: https://github.com/nodejs/node/issues/59666
Refs: https://github.com/nodejs/node/issues/52697
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
This completes the TODO to compile WASM synchronously and thus
making translation (i.e. compilation + instantiation) synchronous.
PR-URL: https://github.com/nodejs/node/pull/59453
Refs: https://github.com/nodejs/node/issues/55782
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/58521
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: LiviaMedeiros <livia@cirno.name>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Obtain sourceURL magic comments via V8 API to avoid a second round of
extraction of magic comments.
Updates source map snapshot normalization to allow testing full path
names inside the test outputs.
PR-URL: https://github.com/nodejs/node/pull/58389
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Boolean value to check if an ES Module is the entrypoint of the
current process.
Implements: #57226
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/57804
Fixes: https://github.com/nodejs/node/issues/57226
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Guy Bedford <guybedford@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
The two proposals reached stage 4 at the October 2024 meeting.
PR-URL: https://github.com/nodejs/node/pull/55333
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/53848
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
This lays the foundation for supporting synchronous hooks proposed
in https://github.com/nodejs/loaders/pull/198 for ESM.
- Corrects and adds several JSDoc comments for internal functions
of the ESM loader, as well as explaining how require() for
import CJS work in the special resolve/load paths. This doesn't
consolidate it with import in require(esm) yet due to caching
differences, which is left as a TODO.
- The moduleProvider passed into ModuleJob is replaced as
moduleOrModulePromise, we call the translators directly in the
ESM loader and verify it right after loading for clarity.
- Reuse a few refactored out helpers for require(esm) in
getModuleJobForRequire().
PR-URL: https://github.com/nodejs/node/pull/54769
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
The synchronous CJS translator can handle entrypoints now, this
can be hit when --import is used, so lift the bogus assertions and
added tests.
PR-URL: https://github.com/nodejs/node/pull/54592
Fixes: https://github.com/nodejs/node/issues/54577
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
This patch:
1. Refactor the routines used to compile and run an embedder
entrypoint. In JS land special handling for SEA is done
directly in main/embedding.js instead of clobbering the CJS
loader. Add warnings to remind users that currently the
require() in SEA bundled scripts only supports loading builtins.
2. Don't use the bundled SEA code cache when compiling CJS
loaded from disk, since in that case we are certainly not
compiling the code bundled into the SEA. Use a is_sea_main
flag in CompileFunctionForCJSLoader() (which replaces an unused
argument) to pass this into the C++ land - the code cache is
still read directly from C++ to avoid the overhead of
ArrayBuffer creation.
3. Move SEA loading code into
MaybeLoadSingleExecutableApplication() which calls
LoadEnvironment() with its own StartExecutionCallback().
This avoids more hidden switches in StartExecution() and
make them explicit. Also add some TODOs about how to support
ESM in embedded applications.
4. Add more comments
PR-URL: https://github.com/nodejs/node/pull/53573
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Unifies the CJS and ESM source map cache map with SourceMapCacheMap
and allows the CJS cache entries to be queried more efficiently with
a source url without iteration on an IterableWeakMap.
Add a test to verify that the CJS source map cache entry can be
reclaimed.
PR-URL: https://github.com/nodejs/node/pull/51711
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
This patch:
1. Adds ESM syntax detection to compileFunctionForCJSLoader()
for --experimental-detect-module and allow it to emit the
warning for how to load ESM when it's used to parse ESM as
CJS but detection is not enabled.
2. Moves the ESM detection of --experimental-detect-module for
the entrypoint from executeUserEntryPoint() into
Module.prototype._compile() and handle it directly in the
CJS loader so that the errors thrown during compilation *and
execution* during the loading of the entrypoint does not
need to be bubbled all the way up. If the entrypoint doesn't
parse as CJS, and detection is enabled, the CJS loader will
re-load the entrypoint as ESM on the spot asynchronously using
runEntryPointWithESMLoader() and cascadedLoader.import(). This
is fine for the entrypoint because unlike require(ESM) we don't
the namespace of the entrypoint synchronously, and can just
ignore the returned value. In this case process.mainModule is
reset to undefined as they are not available for ESM entrypoints.
3. Supports --experimental-detect-module for require(esm).
PR-URL: https://github.com/nodejs/node/pull/52047
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Symbol properties are typically more GC-efficient than using WeakMaps,
since WeakMap requires ephemeron GC. `module[kModuleExportNames]`
would be easier to read than `importedCJSCache.get(module).exportNames`
as well.
PR-URL: https://github.com/nodejs/node/pull/52095
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
This patch disallows CJS <-> ESM edges when they come from
require(esm) requested in ESM evalaution.
Drive-by: don't reuse the cache for imported CJS modules to stash
source code of required ESM because the former is also used for
cycle detection.
PR-URL: https://github.com/nodejs/node/pull/52264
Fixes: https://github.com/nodejs/node/issues/52145
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Guy Bedford <guybedford@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
This refactors the code that compiles SourceTextModule for the
built-in ESM loader to use a common routine so that it's easier
to customize cache handling for the ESM loader. In addition
this introduces a common symbol for import.meta and import()
so that we don't need to create additional closures as handlers,
since we can get all the information we need from the V8 callback
already. This should reduce the memory footprint of ESM as well.
PR-URL: https://github.com/nodejs/node/pull/52291
Refs: https://github.com/nodejs/node/issues/47472
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
This patch adds `require()` support for synchronous ESM graphs under
the flag `--experimental-require-module`
This is based on the the following design aspect of ESM:
- The resolution can be synchronous (up to the host)
- The evaluation of a synchronous graph (without top-level await) is
also synchronous, and, by the time the module graph is instantiated
(before evaluation starts), this is is already known.
If `--experimental-require-module` is enabled, and the ECMAScript
module being loaded by `require()` meets the following requirements:
- Explicitly marked as an ES module with a `"type": "module"` field in
the closest package.json or a `.mjs` extension.
- Fully synchronous (contains no top-level `await`).
`require()` will load the requested module as an ES Module, and return
the module name space object. In this case it is similar to dynamic
`import()` but is run synchronously and returns the name space object
directly.
```mjs
// point.mjs
export function distance(a, b) {
return (b.x - a.x) ** 2 + (b.y - a.y) ** 2;
}
class Point {
constructor(x, y) { this.x = x; this.y = y; }
}
export default Point;
```
```cjs
const required = require('./point.mjs');
// [Module: null prototype] {
// default: [class Point],
// distance: [Function: distance]
// }
console.log(required);
(async () => {
const imported = await import('./point.mjs');
console.log(imported === required); // true
})();
```
If the module being `require()`'d contains top-level `await`, or the
module graph it `import`s contains top-level `await`,
[`ERR_REQUIRE_ASYNC_MODULE`][] will be thrown. In this case, users
should load the asynchronous module using `import()`.
If `--experimental-print-required-tla` is enabled, instead of throwing
`ERR_REQUIRE_ASYNC_MODULE` before evaluation, Node.js will evaluate the
module, try to locate the top-level awaits, and print their location to
help users fix them.
PR-URL: https://github.com/nodejs/node/pull/51977
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Guy Bedford <guybedford@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
So that we can use it to handle code caching in a central place.
Drive-by: use per-isolate persistent strings for the parameters
and mark GetHostDefinedOptions() since it's only used in one
compilation unit
PR-URL: https://github.com/nodejs/node/pull/52016
Refs: https://github.com/nodejs/node/issues/47472
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Split the `internal/process/esm_loader` file which contains the
singleton cascaded loader:
- The the singleton cascaded loader now directly resides in
`internal/modules/esm/loader`, where the constructor also lives.
This file is the root of most circular dependency of ESM code,
(because components of the loader need the singleton itself),
so this makes the dependency more obvious. Added comments about
loading it lazily to avoid circular dependency.
- The getter to the cascaded loader is also turned into a method
to make the side effect explicit.
- The sequence of `loadESM()` and `handleMainPromise` is now merged
together into `runEntryPointWithESMLoader()` in
`internal/modules/run_main` because this is intended to run entry
points with the ESM loader and not just any module.
- Documents how top-level await is handled.
PR-URL: https://github.com/nodejs/node/pull/51999
Fixes: https://github.com/nodejs/node/issues/42868
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
This patch adds support for using
`vm.constants.USE_MAIN_CONTEXT_DEFAULT_LOADER` as
`importModuleDynamically` in all APIs that take the option
except `vm.SourceTextModule`. This allows users to have a shortcut
to support dynamic import() in the compiled code without missing
the compilation cache if they don't need customization of the
loading process. We emit an experimental warning when the
`import()` is actually handled by the default loader through
this option instead of requiring `--experimental-vm-modules`.
In addition this refactors the documentation for
`importModuleDynamically` and adds a dedicated section for it
with examples.
`vm.SourceTextModule` is not supported in this patch because
it needs additional refactoring to handle `initializeImportMeta`,
which can be done in a follow-up.
PR-URL: https://github.com/nodejs/node/pull/51244
Fixes: https://github.com/nodejs/node/issues/51154
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
This puts it together with the cjsParseCache and reduces the
circular dependency on the singleton loader, which is the
only place where this cache is stored.
Drive-by: remove always-false module status check because there's
no longer a local module variable after
https://github.com/nodejs/node/pull/34605 which is now invalid
leftover code at this point and only doesn't throw because
we happen to have a top-level variable called module.
PR-URL: https://github.com/nodejs/node/pull/51157
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/51033
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/50466
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
When using the Modules Customization Hooks API to load CommonJS modules,
we want to support the returned value of `defaultLoad` which must be
nullish to preserve backward compatibility. This can be achieved by
fetching the source from the translator.
PR-URL: https://github.com/nodejs/node/pull/50825
Fixes: https://github.com/nodejs/node/issues/50435
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Jacob Smith <jacob@frende.me>
PR-URL: https://github.com/nodejs/node/pull/50127
Reviewed-By: Guy Bedford <guybedford@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Instead of using the public versions of the vm APIs internally,
use the internal versions so that we can skip unnecessary
argument validation.
The public versions would need special care to the generation
of host-defined options to hit the isolate compilation cache
when imporModuleDynamically isn't used, while internally it's
almost always used, so this allows us to handle the host-defined
options separately.
PR-URL: https://github.com/nodejs/node/pull/50137
Refs: https://github.com/nodejs/node/issues/35375
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
The old import assertions proposal has been
renamed to "import attributes" with the follwing major changes:
1. The keyword is now `with` instead of `assert`.
2. Unknown assertions cause an error rather than being ignored,
This commit updates the documentation to encourage folks to use the new
syntax, and add aliases for module customization hooks.
PR-URL: https://github.com/nodejs/node/pull/50140
Fixes: https://github.com/nodejs/node/issues/50134
Refs: 159c82c5e6
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
