ES Modules: suggestions for improvement

There has been a recent bout of comments about ECMAScript (ES) harmony modules on twitter and elsewhere. Here is my attempt to explain parts of it, some of the design tradeoffs, and perhaps a middle ground that would open up some options that may bridge some gaps.

Modules are one of those things that seem very simple, but involve quite a lot of decisions and tradeoffs. This post is mostly just about module linking and module ID resolution, and even with that, it is quite long.

If ES Modules do not come up with different ways to work (or maybe explain where I have it wrong), they are not competing well with what can be done with a combination of CommonJS/Node and AMD.

My background: I work on RequireJS and AMD.

What is it

First, some links to the specs. The "harmony" moniker means it is in process for the next version of the ECMAScript (JavaScript) language:

• Harmony Modules: The basic module spec.
• Harmony Module Loaders: Specifies how you can create isolated loaders for modules. Very useful.

The module examples page is suggested if you want to get a feel for it, but it is good to read the other docs too. It can be a bit daunting though, unless you speak the spec language.

Points of reference

One way to evaluate how ES Modules works is to compare it to something you may already know:

• CommonJS / Node. Node implements a version of the CommonJS module API.
• AMD / RequireJS. RequireJS implements the AMD module API.

Run time vs compile time

ES is a "compile time" approach where the formats mentioned above are "run time" approaches. Maybe not precise terms, but here is a definition of what is meant by those terms for the purposes of this post:

"Compile time" means:

• JS text is parsed, and the "module" "import", and "export" syntax is found.
• Any dependencies are fetched and parsed.
• Once the dependency tree has been all fetched, the ES module loader will wire up the exports from a dependency to a module's "module" or "import" use, and do type checking on that export type and how it is referenced in the module
• The module code is then evaluated/executed.

"Run time" means: there is usually no pre-parse stage. The JS text is evaluated, and any module API that is encountered is run as it is encountered.

AMD will actually do a small parse step if the module looks like:

define(function (require) {
    var a = require('a');
});

In that case, it will parse the function to look for require() dependencies, and then load and execute the dependencies first before running the function above.

"Compile time" was chosen for ES because:

• it is familiar from other scripting languages
• sets the way for other possible static features, like macros
• ensures that "import *" are static, *not* dynamic bindings
• allows some type checking on the values that are explicitly "export"ed.
• generally seen as safer and easier to reason about that run time.

The CommonJS/Node style of pure runtime, no parse, was hard to get to work with some edge cases as I understand it, but I heard that second-hand, I did not see that discussion.

Impact of compile time

For compile time to work well, it should use new keywords in the language, to have clear markers on what is participating in the module system.

Although, it could work with a module API instead of new syntax, by only recognizing literal use of that API, and do not support variable assignment of the API or dependencies to other names. This is what AMD does for the "sugared CommonJS form" mentioned above.

For "import *", static binding is critical because anything that is a runtime scope lookup gets into "with" territory, and "with" has been seen as a mistake by the committee. ES5's 'use strict' bars its use.

Since new syntax is involved, ES Modules cannot be "shimmed" into existing JS libraries. There is a Module Loader "runtime" registration call that can used for a module to register its, but it means those libraries cannot participate in the compile time linking stage, so they need to be pre-loaded by a script loader before an ES Module can effectively reference it with module syntax.

Module ID resolution

One other consideration, one that is usually overlooked when talking about modules, is how a module ID like "jquery" is resolved to a file path and loaded.

Both CommonJS/Node and AMD/RequireJS support "short, logical names" for dependencies. So, you can say require('jquery') and that jquery gets converted to a path using some algorithm. Node uses multiple paths to find jquery.js, and AMD in the browser relies on a declarative configuration to do so.

ES modules do not really have support for this, unless you also implement an imperative resolver. They support full URLs, like:

module foo at 'http://example.com/scripts/foo.js'

but we have found in AMD that it is useful to be able to say require('jquery'), but then declaratively map that to zepto.js. 

So, an individual module specifies a dependency on an API provider, but how that provider is satisfied is resolved using the declarative configuration.

If there is only an imperative resolve API, no simple declarative API to resolve short names, it will mean shipping a userland "loader library" to effectively use modules. This opens the door to balkanization in module ID resolution since there is not built in support.

Special factors in JavaScript

There are a few special factors with JavaScript that are not usually in other programming languages, and they have an impact on the design:

• The largest deployed use case of JavaScript, the browser, is async, network IO. File size and number of requests are very important to performance. So combining modules together into one file, and minifying/transforming the source for smaller delivery is common.
• There is a large legacy codebase of browser-based JavaScript that just use browser globals, and no real module format. Some small uses of JavaScript do not need modules, and browsers will support those use cases indefinitely.

My goals

I want AMD and RequireJS to go away.

They solve a real problem, but ideally the language and runtime should have similar capabilities built in.

Native support should be able to cover the 80% case of RequireJS usage, to the point that no userland "module loader" library should be needed for those use cases, at least in the browser.

If the ES module format requires a web developer to use a script loader to use existing, non-AMD/CommonJS, non-ES JS in a project for those 80% use cases, it is a failure.

Example: If I cannot use jquery and backbone in an ES 6 module without needing another library to preload or prep those libraries for ES 6 module use, then existing JS users will not see much advantage over using AMD.

If the web developer needs to code any imperative logic to wire up the ES Module Loader, that will result in a loader library. That is a failure condition.

As compared to AMD: if the ES approach cannot do the above without a helper loader library and the ES approach does not allow something like loader plugins, then there is no contest -- AMD will still be more useful to a developer than the built in system. Small savings in the amount to type and a thin layer of type checking is not enough.

This may very well not be the goal of ES modules, and it would be great if the specs or some background material acknowledge that, and list out the mitigation strategies developers are expected to use.

Shortcomings of ES modules

Right now, ES harmony modules do not improve an AMD user's workflow because of the following:

New syntax makes it very hard to optionally upgrade

If I am the author of something like jQuery or Backbone, I cannot optionally add in a way to register as an ES module because ES modules use new syntax. However, there are many uses of those libraries which will not be in ES module-capable browsers.

The Node and AMD communities have found optional opt-in via a runtime API very useful for adoption of code that works with their module systems, but still work in older "use plain script tags with browser globals" approach.

There is a runtime API in the ES module loader proposal that would allow a legacy script to register something as a module, but that requires the end developer to use another script loader library to load that library so it can do that runtime call, then start loading ES module code.

The developer may as well just stick with AMD. Complexity has not been reduced.

Register module and a global

Backbone originally had trouble adopting AMD because if it called define() to register a module, they found other libraries, like Backbone plugins, would break. The plugins were expecting to find a Backbone global variable but when Backbone called define() it was not also exporting a global.

This same problem will exist in ES-mixed code. Any dynamic registration also needs to allow an export of a global so that downstream libraries will work until they are also converted to optional module registration.

There should be a migration path, one that allows gradual rollout of modules without requiring a project to go whole hog on module syntax.

Declarative module ID resolution

While I have made tools to allow a developer to "convert" an existing library to AMD, there are many developers that did not want to touch existing libraries. It makes it difficult to compare against new versions and there is a concern that the conversion introduces breaking scope changes (rightly so).

So the "shim" configuration for requirejs was introduced to allow specifying dependencies and an export value for JS code that does not call a module API. This has been well received in the community. More background on shim here.

"shim" with "paths" and "map" make it possible to declaratively set up a configuration that allows for one file IO lookup per module ID, an easy way to "shim" old libraries, and to load more than one version of a module for use by different modules. That covers the 80% case for using old and new code with a module loader.

By using a declarative configuration that is supported by the "default" module ID resolution mechanism in the language, then it avoids having to ship a userland loader library for browser use. This is a big win because it will help kill AMD.

It is fine if the Module Loaders API still has an imperative API to set up different  module ID resolution logic. That would allow Node to maintain its current multiple IO, nested directory lookup logic. However, the default should favor the harsher browser environment in such a way that an extra loader library is not needed.

Loader plugins

I can appreciate that supporting Loader plugins may seem out of scope for the default module loader, but they have been incredibly useful for AMD. Node has seen a use for them, even though they are done in a different way via require.extensions. They effectively allow use of transpiled languages.

I find the AMD loader plugins better than Node's approach because:

• load behavior vs. file format: since a prefix is used on the resource ID instead of just using a file extension suffix, it allows multiple plugins to deal with the same type of file extension. For example, "text!index.html" and "template!index.html" can be used in the same app, the first one just giving the raw text, the second one "compiling" some text for use as a template. The developer, not the plugin provider, chooses the right use. It still allows "single extension" plugins too, and for those, they can omit the file extension in the ID, so no increase in ID length.
• one IO lookup: For a resource ID "foo", node may do a lookup for "foo.js", "foo.coffee" and "foo.node". By specifying the loading mechanism via the prefix, it avoids multiple IO lookups, which are important for browser use. It also makes it clear what handles the loading.

AMD loader plugins can participate in build steps, so the "text!" plugin can inline the text as module in a built file:

define('text!index.txt', function() {
   return 'hello world';
});

That is incredibly useful for getting good network performance in the browser.

Even for local file environments like Node, being able to combine all the assets for a program into one file is really great for distribution. It is conceptually simpler to reason about tracking one file vs. "nested directory of directory" installs. Think of it as a way to easily share shell scripts.

The middle way

For developer workflow, right now AMD is a better alternative than ES harmony modules, given the choices around compile time linking, new syntax, and the use of imperative ID resolution.

Here are some suggestions on how to allow some of the benefits of the compile time approach with the run time ones used by AMD. The goals are reuse non-module code in modular systems, allow for a way to get some static version of import *, and perhaps even macros.

Fetch dependencies, execute, modify, execute

The core of the middle way for compile time vs run time:

• do not force compile time operations to be all up front, before any evaluation.
• evaluate dependencies before executing the current module.
• provide an API for modules, not just new syntax

These are effectively what AMD does today, except it does not have a way to alter the AST before final execution. Well, an AMD loader could do that, but AMD loaders have traditionally avoided it. However, the harmony loader plugin I made effectively does this to support "import *". More below:

The ES module loader would operate like so:

• Load JS text. Parse out dependency references.
• Load dependencies, parse out their dependencies, load them, etc...
• Before executing a given module, execute its dependencies, and wait for the dependencies to finish exporting their module values.
• Take that exported value and if there is an "import *" in the current module, modify the AST of the current module such that it gets a locally bound variable to any of the hasOwnProperties of the dependency that are known at that time. So, any properties added to the module after this point are not visible. This should avoid concerns about dynamic scope.
• Once the module AST has been fixed up for any import *, then evaluate it.

When parsing out dependency references, look for any new keywords, but also any API that corresponds to that keyword. So, look for at('moduleID') for dependency references in addition to at 'moduleID'.

The runtime API for the module should be something like at('moduleID') for dependencies and exports.propertyName for specifying export properties. I am not arguing for that specific API, just mentioning that there would be an API alternative to the new syntax. The API alternative does not need an import alternative though.

Since a module is executed before giving the exports to a module that depends on it, and since there is a runtime API for modules, then that allows existing JS code to opt-in to ES modules without getting bitten by new syntax.

Since all dependencies are executed before executing the current module, an "import *" can be supported, and I believe that would allow for macros later.

There are some limitations around circular dependencies, but they are still possible, and the restrictions are minor in comparison to allowing existing code to opt in to ES modules and still work in non-ES module environments.

Declarative configuration

Support something like the "paths", "map" and "shim" config as used in RequireJS. This allows easier use of old code, and scales up to very large code without requiring a developer to ship a library that sets up an imperative resolution API.

Support loader plugins

Now that all dependencies are executed before executing the current module, then it is easier to support loader plugins, as the loader will have the exported value for that plugin resource before running the current module.

These environment-based loading, like an "env!" plugin that can load a module for Node and a different API-compatible one for the browser. See also a "has!" plugin for feature detection-based loading, and plugins to enable transpilers.

Yes, it is more to sort out, but they provide a lot of benefit. AMD has already primed this pump. It even works with a build/optimization step for inlining resources.

Use string IDs for module identifiers

This allows the module references in dependencies to be the same as the ID that is inlined when modules are combined and named in built files. Right now it is weird to use a JS identifier, like module Foo {} to name a module, but then see module Foo at "Foo". It is hard to match up at "Foo" with module Foo.

The extreme positions

The following is based on my limited experience. I am not a language designer. I am but a simple plumber that uses the pipes that available to build things. I may not have the right long term thinking involved, but I think the following make the ES module proposal simpler.

To be clear though, I think the middle way above is enough to bridge the gap. Please, do not read the following and then discount the middle way. The middle way is separate from these more extreme measures.

No new syntax

If there is a runtime API available to allow existing code to opt in, just shed the new syntax. Just have one way to do it via API that can also then be shimmed.

no import *

import * makes it difficult to determine where code comes from. If
this type of construct is allowed:


module foo {
   var sin = function () {};


   module bar {
       import * from "Math";
       sin();
   }
}

for a minifier, it now needs access to Math to do its work correctly. This has not been the case in the past. It would suck to need all of the code for all the modules used in a system just to complete a minifier pass.

For developers, if you have two modules that do import * it can be difficult to know where something comes from.

Destructuring provides enough benefit for these use cases, just do the
comma separated list for things you really use:

   import {sin, cos} from "Math";

import * is a bad pattern and it does not save much.

If you get rid of import * then with the "middle way" of evaluating modules, then regular var/let-based destructuring is enough, there is no need for an import keyword.

No macros

Similarly, rethink the need for macros long term. They suffer from the same "where did this come from" problem as import * does. The function capabilities in JavaScript are good enough to get the job done for the "don't repeat yourself" task.

A way forward for today's code

The nice thing is that we can prototype this new world by combining what CommonJS/Node does today with AMD. So we can just use the require() and define() as used today to get there. The ES committee does not have to ratify it, and we get the benefit of real world implementation and use before committing to default language support.

Cajon is my attempt from the AMD side to bridge the gap with plain Node code and a runtime browser loader. LinkedIn's Inject is another AMD loader that uses a similar approach. So, just use CommonJS/Node modules in the browser in dev, use the r.js optimizer to compile down to AMD for final deployment.

The cjsTranslate capability in the r.js optimizer allows a developer that always likes to do builds, even in dev, can code in Node syntax but output to AMD and load it in the browser either by the small Almond AMD shim, or the full dynamic loader via RequireJS. Or choose Dojo or curl.js.

browserify can be updated to use AMD as its transport format instead of its home-grown require.define() API, and then not have to ship a loader, but use one of the AMD loaders/API shims. browserify is nice in that, unlike the r.js optimizer+cjsTranslate, it provides browser module shims for the native node modules. It would be great to break those out as a separate project that could be consumed by a project just using the r.js optimizer.

If Node adds define() support, callback-require for use within a module for dynamically calculated dependencies, and supports at least a limited form of loader plugins, then we're done. The amdefine project is an implementation proof of that support. There are details to sort out, but it is doable. Any node committers are interested, give me holler. We can work out the details.

Summary

For developer workflow, the current ES module spec is not competing well with a combination of CommonJS/Node and AMD with loader plugins. Or even just AMD with loader plugins.

Using the middle way for module execution and getting a good declarative module ID to path configuration in the ES spec will level the playing field. Add loader plugins to get language transpiler support and environment/feature detection loading that is efficient for the browser.

I have given some of this feedback to the es-discuss list, but I think some of it, in particular the "middle way" module evaluation flow, got lost in my poor communication where it seemed like I was proposing a dynamically scoped import *. Hopefully this post clarifies what I was trying to achieve with that earlier feedback.

Finally, I appreciate working on the ES committee is very difficult work. I do not envy them. I do not mean for this feedback to come across harshly, but the committee is running out of time, and I do not feel like it has made the case very well for how what is being proposed is better than what we have cobbled together with existing technology. To be clear, I want an ES Modules proposal to succeed because I do not want to do AMD or RequireJS forever. Hopefully this feedback can be viewed as loyal opposition, and as a challenge to do better, or at least to do it in a way that is explained more clearly.

On inventing JS module formats and script loaders

There was some recent twitter tweets over the last couple months that indicated some folks want to experiment with some JavaScript module formats that work in the browser. In addition, making a script loader is almost as popular as making your own template engine. This is my response to both of those endeavors.

AMD is Winning

If you are one of those people that think they want to invent a module format or a script loader that works well in today's browsers, let me save you some trouble: a module format has already been worked out. Check out the Asynchronous Module Definition (AMD). There is still some room for innovation though, see near the end of this post.

If your script loader does not support AMD, it will be of very limited usefulness and in a sea of competitors that will make it hard for you to gain adoption.

So, for the module syntax, that problem is solved, and the solution is AMD.

A Solved Problem

Why is it solved?

• It has been discussed for more than a year and a half by multiple people, in the Dojo world, and on the CommonJS list.
• It has been implemented more than 5 times by different people.
• It has been deployed in real projects, the BBC iPlayer being a very visible one.
• It works in the browser, web workers, Node, Rhino, Mozilla Add-On SDK-based add-ons, and can be used in traditional Mozilla add-ons (Firebug will be using it).
  
• Many AMD implementations support loader plugins. You can add modules that know how to load other things as dependencies. While not strictly part of AMD, plugin systems greatly enhance a base technology.
  

You will be facing an uphill battle trying to get your format accepted. AMD has at least a half-year head start and has been seriously battle-tested. The ECMAScript folks are considering separate, different language changes to support modules, so you have that on the event horizon. Time is really running out pushing for something new.

I do not want to harsh your mellow, and I definitely want to follow your bliss, but just know the actual cost involved, and do your research first. Make sure you understand why AMD is constructed how it is, and what it really is.

It is not the traditional CommonJS Module format, but it can support similar concepts. However, it is different, take the time to figure out why.

Here is a handy gist with a summary of the AMD API, loader plugins, and some thoughts as to how it relates to modules implemented in Node, which are more like traditional CommonJS modules. In addition, you can read the RequireJS API docs, and the AMD wrapper format for traditional CommonJS modules.

If you think something was missed, feel free to ask on the RequireJS list, or on the CommonJS list, and enlighten us all. Or just send me an email or GitHub message if you want to talk privately. However, I believe most questions will be in the minor bikeshed/personal preference arena that do not ultimately matter for adoption. In particular, do not propose anything that requires the developer to do more typing that what is in AMD already. The format needs to be usable and easy to type. Discoverability is not a goal, daily usability is, and the AMD format is easy to explain.

Requirements

This leads to a discussion of other requirements. Here is what I believe are the minimum requirements for a module format in the browser, and they well-met by AMD:

• No globals: The module format needs a way for the developer to avoid creating global variables for each module they create. The format needs to be web-scalable, which means that some sites need the ability to embed two different versions of a module on a page. This normally means embedding in different "contexts" (the different versions do not need to talk to each other, just co-exist on the page), but it is a web-scale need. Seriously. We have seen it in Dojo. jQuery has seen it, it is the reason they have noConflict(). While noConflict() is nice, there are edge cases where it falls down. Know why that happens (in particular, dynamic script loading in IE: more than one script can execute before the first script's onload event fires).
  
• Allow for globals: at the same time, you want developers to be able to do want they want, in particular, some like to modify global object prototypes. Prototype and MooTools are valid ways to build web sites. So allow for it. This is one of my concerns with the ECMAScript harmony modules experimentation, but I think they know they need to allow for this, even if within a module loader instance.
• Do not require server transforms or think that using XHR+text transform+eval() is usable across all of web development. It is not. Here are some reasons why. They are fine for your boutique rails projects, but do not assume that translates well to the web at large, for instance, to serving JS files from disk for mobile widgets.
• You will need to use a function wrapper and a way to specify dependencies before that function wrapper is executed. You will be using script elements to load content (seriously, don't use eval, that is an evolutionary dead path), and dynamic script elements will load scripts async and out of order. A function wrapper will need to be part of the format.
  
• You need a way for modules to have a name. Otherwise you cannot combine them all together for optimization purposes.
• At the same time, it is best to allow source modules (the non-optimized ones) to not be named. This makes it much easier to move code around. This is not a hard requirement though, and I think one of the easiest to try to not support, see next section.

Places for Innovation

So you are not going to come up with a better module format. I know that sounds harsh, and for some of you that will be just the thing you need to hear to try something different. But I really am trying to save you some work. Again though, go for it if it really is your bliss, just know the amount of work you are taking on.

A better use of your time might be doing the best AMD implementation/script loader. In particular, you can choose some limiting design choices to help make it easier to meet/more compact:

• Do not support the simplified wrapper for traditional CommonJS Modules.
• Only allow named modules.
• Do not support loader plugins.
• Do not worry about supporting more than one version of a module in a page. The module format allows for it, which is the most important aspect, so module authors can code for it if they wish. However, most sites can get away without needing a loader that can actually support loading multiple versions.
  

If you do those things, you still can consume the built/optimized AMD modules that someone made using another implementation, and any modules created while working with your implementation will work with other AMD implementations. And with that, you are part of a larger ecosystem.

If you make a script loader that does not understand AMD, it will be a boutique loader, and not something that fits all of front-end development, and limits the ability to share code with other JS environments. Which is fine, boutique businesses can be really satisfying, just realize you are in a boutique, and do not over-sell the loader.

FAQ

These are odds and ends that do not fit above.

Why use modules or a script loader at all? Just server-side concat all your scripts!

The goal is to establish a larger ecosystem of shared code that works without puking globals all over the place. JavaScript is also not just jQuery (although I greatly appreciate the project), so it cannot rely on a jQuery implementation.

Not all environments can use server-side script concatenation (file-served mobile widgets), and it does not allow for more nuanced optimization like loading built layers on demand after initial page load. On-demand loading is really needed for large web apps like a webmail UI.

With tools like the RequireJS optimizer, you can still get the same effect of "concat all files". If you do not want to take the hit of downloading a script loader in that scenario, there is work underway to allow a simplified stub that removes the need for a full script loader. The Ace project uses a "mini-require" after they build the code to avoid loading the RequireJS script loader.

What about YUI 3 modules?

YUI 3 got a lot of things right with their modules, in particular designing for async up front. However, I believe the way they implemented the module name-to-JS-variable-name mapping is not right. It requires knowledge of two names that allows for easy typos and creates the opportunity for a naming conflict on the Y object. For instance:

YUI().use("io-base", "my-module", function(Y) {

//Where does .on() come from? Did 'my-module' add it?
//Did 'io-base'? What if both tried to add it?
Y.on(...);

//'io-base' creates the 'io' property on Y.
//How is that known? Should it be 'ioBase'?
//What else does 'io-base' add to Y?
var request = Y.io(...);
});

In short, too much magic that requires extra documentation. YUI 3 has very nice documentation so that helps, but I do not believe that approach scales as well as AMD.

Is AMD a CommonJS specification?

No, it is a proposal, but with lots of implementations and real use. Not all participants on the CommonJS list believe it should be a elevated to "endorsed spec" status. Those participants wish browsers would grow better base technology and/or they want to maintain stricter compatibility with traditional CommonJS modules, which were not designed primarily for async loading environments like the browser.

Any browser technology change will likely be for harmony module support which operate differently than traditional CommonJS modules, and AMD works well today in all JS environments. AMD is proven, and it can work on the server side. The RequireJS Node adapter even allows using NPM-installed modules in Node.

How does AMD relate to ECMAScript harmony modules?

They are unrelated, although hopefully the AMD work and the traditional CommonJS module work will help inform the harmony effort. I believe AMD modules will be easy to convert to harmony modules, but as the harmony work is still in progress it is hard to know for sure. The return value options in AMD may be more flexible than harmony modules, and it is unclear if loader plugins will work in a harmony module world.

AMD has the benefit of working in all the JS environments now, and it will work in the future, and I do not believe the syntax gains in harmony modules are a vast improvement over the typing cost in AMD, particularly if sharp functions are supported. That said, I hope to inform the harmony effort where I can to help make it the best it can be.

Who are you?

I'm James Burke (GitHub, Twitter). I maintained the original Dojo XHR-based script loader, maintained Dojo Core, created the Dojo xdomain script element-based loader, created RequireJS (formerly RunJS), and rewrote RequireJS about three times. I have strongly advocated for browser-friendly modules on the Dojo and CommonJS lists. I like Pina Coladas and getting caught in the rain.