Tuesday, December 28, 2010

Standards and proposals for JavaScript Modules and jQuery

This commit to add registeration of jQuery as a module via the Asynchronous Module Definition (AMD) API has brought up some questions that I should have addressed previously before the commit happened. The jQuery team may decide to back out the commit, as is their prerogative, but I hope not. This post will be the justification for the AMD API in general and for its suitability for jQuery in particular.

What is the Standard?

There was a concern brought up by Kevin Smith that module loading is still up in the air, and that there is no "Standard" yet. To me a standard is something that is written down, understood by people, and has a good level of market adoption. Multiple implementations help too.

CommonJS Modules is a standard in that way. It is written down, understood by a few people, and has a a good level of market adoption via Node. There are a few implementations for other JS engines.

The CommonJS group also has a process where they have voted before to call it a Standard vs. a Proposal. Anyone can join the CommonJS discussion list, and I think it is still an ongoing discussion on how best to vote on proposals, who should get to vote, and if voting makes any sense. It is helpful to know if most of the people on the list like a particular proposal though, it gives some confidence that the proposal has been thought out.

I have a concern about the diversity of the list participants though. Most joined when it was still ServerJS, and the CommonJS modules "standard" does not work well for use in web browsers that use script src="" to load scripts. So, you could hear that "CommonJS Modules are a standard", but it does not mean it works well in all CommonJS environments, most notably, the largest distribution of a JS environment, the browser.

That is the reason the AMD proposal exists. It works well in the browser environment where module loading is by default asynchronous, and with script src="" loading (the easiest to debug and fastest way to load code), there is no control over modifying the script source before it executes. AMD is called a "proposal" in the CommonJS list because people on the CommonJS list have not voted to call it a "standard".

There is recognition on the CommonJS list that the CommonJS Modules 1.1 is not sufficient for script src="" browser loading, but there is some disagreement over how to solve it. AMD is one proposal. Kevin Smith has put together a Module Wrappings proposal, based on some discussions on the CommonJS list. The Wrappings proposal is incomplete -- there is more to it based on the list discussions. I will highlight the main difference between AMD and Wrappings below, but first, another module API contender:

ECMAScript Simple Modules. This is a "strawman", which is like a proposal in the CommonJS group: it is still something being worked on. This one is different because it is being worked on by the ECMAScript committee, and it seems like the strongest module proposal in that committee so far. If adopted, some future version of the ECMAScript (JavaScript) standard will have support for modules natively. So, any module API that used today should be aware of what is going on in the ECMAScript arena.

So, for talking about a module standard, particularly for use in the browser, these are the main contenders:
  • AMD
  • Wrappings
  • Simple Modules
There are other modules systems like the one used in Dojo, and the one used by YUI, but those are too tied to those existing toolkits to be broadly adopted. Dojo has started to move to AMD.

AMD and Wrappings Differences

AMD and Module Wrappings are mostly the same. There needs to be a function wrapping around the actual module, to make sure the dependencies are fetched first before executing the module. There is some bikeshedding around names that I do not think is important (define vs. module.declare).

The main difference: AMD executes the module functions for dependencies before executing the current module function, where Wrappings just makes sure the module function is available, but does not execute it until the first require("") call inside the module function asks for it.

For shorthand, I will call the AMD approach the "execution" approach, and the Wrappings approach "availability". Both relate to how dependencies are handled.

Because AMD executes the module functions of dependencies before calling the function wrapping of the current module, it can pass the dependency as an argument to the factory function. This is a side benefit of execution.

The "availability" approach was used for Wrappings to keep it most similar to how CommonJS modules deal with dependencies, but I believe it is the wrong choice for the future.

Problems with "availability" model

"Availability" is not supported in the ECMAScript Simple Modules strawman. Simple Modules uses a model that is closest to the "execution" model: you use "module" and "import" keywords to reference dependencies, but they cannot be used like require("") is used in the "availability" model. "module" and "import" are more like directives, and not something that can by dynamically assigned, as part of conditionals. Examples:

In the process of working on RequireJS and trying to translate modules written for Node to a wrapped format, I have seen the following constructs in Node modules (which assume the "availability" model):
try {
var constants = require("constants");
...
} catch (e) {
//Older version of node here
}
and:
var foo;
if (someCondition) {
foo = require("foo1");
} else {
foo = require("foo2");
}

Those work in the "availability" model, but not in Simple Modules. AMD's execution model does not allow the above constructs, so modules written for AMD will be more portable to Simple Modules.

2) The "execution" model fits better for projects that use libraries like jQuery, Prototype or MooTools, where many of the modules augment other objects, and they are assumed to have already run before executing the current module function. jQuery plugins augment the jQuery object, Prototype and MooTools augment JavaScript object prototypes.

Choose AMD

AMD is the API for today that translates to the future best, and it should be the one that jQuery should support because:
  • There is a document that defines the API.
  • It is understood by a few people, and discussed in public on the CommonJS list.
  • It is not ratified as a "Standard" in CommonJS because there are list participants that still want to hold on to CommonJS "availability" semantics. However, that availability model is not as future proof as AMD, at least for the Simple Modules future.
  • AMD translates better to Simple Modules due to the execution model. It maps better to the the related Module Loaders strawman, so if Simple Modules with the Module Loaders API becomes a standard and gets implemented in browsers, code written for AMD is more likely to continue to work.
  • For browsers versions that probably never support Simple Modules (IE6-8, even 9?), code continues to work.
  • The "execution" approach for dependencies fits better with the mental model of existing browser toolkits that want to augment other objects, like jQuery plugins.
  • AMD is supported by a module loader (RequireJS) that has had some level of successful adoption, and it is fairly robust now, with over a year in development with frequent releases. RequireJS has an adapter that allows the same modules to be run in Node and Rhino. The Node adapter allows using existing Node modules.
  • There are other AMD implementations.
  • RequireJS in particular works hard to make sure it works well with jQuery, including integration with jQuery's readyWait to hold off DOM ready callbacks until all scripts are loaded.
  • AMD has more real world adoption than Wrappings. The Dojo Toolkit now supports using an AMD-compliant loader in their trunk code for Dojo Core and Dijit. There is a thriving RequireJS list with real people using it. BBC iPlayer uses it. The RequireJS implementation of the AMD API has been promoted as a useful tool in multiple jQuery-related conferences.

11 comments:

Kris Zyp said...

http://wiki.commonjs.org/wiki/Modules/AsynchronousDefinition lists like 5 or 6 implementations, it's adoption goes far beyond just RequireJS and Dojo.

James Burke said...

Kris Zyp: you are correct, and I tried to put links to three implementations I knew of specifically in the final summary area, but they are a bit hidden as links behind individual words. Good to call that out better.

Alex S said...

Specifically RequireJS (and therefore AMD) is used in the Bazaarvoice suite of products (that I work on) so it's being used on several thousand (fairly large) sites (BestBuy, Walmart, Dell, etc) - Just in case current adoption was a question.

Dave Methvin said...

@James, thanks very much for this explanation!

khs4473 said...

Hi James,

I'm not trying to kill your commit. And your product is no doubt a great one. But you can't go around equating RequireJS (or AMD) with CommonJS when half of the list participants are in disagreement with AMD. If the jQuery team decides that they want to have built-in support for RequireJS/AMD, that's awesome, but it's not fair to give them the impression that they are buying "CommonJS" support with the commit.

On the other hand, maybe "CommonJS" support isn't all that important. You have lots of people using RequireJS out there who will stand behind you - why not just leave "CommonJS" out of the commit message?

Kevin

James Burke said...

Hi Kevin: I'm open to changing how to present AMD. It is something specified on the CommonJS wiki and discussed on the list. It has implementations by CommonJS participants, and it is listed as a proposal, not a spec. I see it similar to have people refer to CommonJS Promises/A and Promises/B on the CommonJS wiki.

One option would be to remove it from the CommonJS list, but I'm just not sure how that buys much. I hope that this blog post actually explains how this proposal relates to CommonJS, and its standing with that group.

Maybe I could put a link to this blog post on the proposal page, so if people look at the AMD proposal, they would be able to read more up on the particulars of the list agreement? Or what else do you think would help?

Anonymous said...

James,

I appreciate your efforts in writing RequireJS and helping to create the AMD proposal but for me this blogpost feels like a kick in the back...

Why? because I think that the last 3 months there are a lot more browser liking people involved trying to make the AMD proposal less 'intrusive' to the current module specs without throwing out the good parts of AMD...

Myself together with Kevin and to some less extent Christoph and some others where willing to write down a proposal based on the (mostly fruitful) discussions that took place on the mailing list. In my opinion all 'differences in opinion' can or are already resolved and I think that also your new point (or implementation differentiation) with the simple_module proposal can be resolved.

Hope you are willing to join the writing group in shaping the AMD proposal to a server and browser side success...

Regards,

Sander

James Burke said...

Sander: I did not intend for it to feel like a kick in the back.

From my perspective, I have wanted to solve this issue for a little over a year now, but when I first brought it up, no one on the CommonJS list wanted to talk about it.

I feel like it only became important to others on the list once I showed an implementation getting adoption, and then Kris Zyp worked up the AMD proposal. I had to change some things to get in line with it, but we discussed on the CommonJS list, and got it settled as a proposal with some implementations.

I think others only started to pay attention when it was clear it was not going away, and RequireJS started to pick up speed.

The previous discussions related to AMD and talk of a module.declare Wrappings proposal did not uncover anything new for me, and stated so on the list.

I made a point to post this blog entry to the CommonJS list, I am not trying to do anything in a sneaky way, I am just tired of waiting for an alternate proposal that has a tangible benefit above AMD.

So far all I have seen are different style issues, with the choice of when to execute the dependency's factory function and whether to allow injection as the only major differences. To me, they make sense, and I have not seen any compelling arguments to move away from them.

I am trying to respond when it makes sense on the CommonJS list and state my concerns with the Wrappings proposal, but I do not expect an alternate proposal to be noticeably better than AMD.

Now that there are more and more users of AMD it is also harder to move away from it, but I think that is a good sign -- it has been tried by real people in real projects and it seems to hold up.

TeMs@ said...

I like AMD and I'm pushing a big project to use it using RequireJS.

Actually what I don't like about current AMD isn't the principle, but it's "almost" cryptic syntax for any newcomer (verified less than an hour ago in front of some people which has to define a script loader to use on his project, and the favorite one right now is far from CommonJS as it's labjs).

My main concern is: have a look at any language you know, and point me to imports detailed by an Array without even telling some that this array are the imports? When you look at this without knowing the spec, you won't know what this array means.

It is probably not perfect, but I'd like much much more a syntax like this gist : https://gist.github.com/764653

module .name (for id) and .import (for defining one dependency at a time) are pretty self explanatory, easy to chain and widely improves readability IMHO.

I don't care about calling module.declare or module.define or anything else for the module definition function itself, it "just" has to be :
- simple
- readable
- self-explanatory enough
- pure JavaScript
- it should be easy to make wrappers for any browser library (e.g. RequireJS + jQuery + jQuery-ui + some components/widgets + jquery-ui theme css file)

James Burke said...

TeMs@ you can use this form which is supported:

define(function (require) {
var foo = require("bar/foo"),
bazbar = require("baz").bar;

return moduleValue;
});

The .import calls in the gist look like extra typing that does not need to be there. The require calls express the same information.

The loader uses Function.prototype.toString to find the require calls in the above snippet. Some folks may not feel comfortable with that, but for the case where .imports are done up front, the above form works just as well and is less typing for the developer.

TeMs@ said...

I'm not very comfortable with the toString option, despite its efficiency regarding "semantics".

Actually this isn't part of AMD( and I don't believe the toString opt will ever be), and .imports are extra typing but meaningful, and they could be part of AMD if it has any support in the community.

If we would like not to do any extra typing, we probably would just use such a notation :

d(function(r){
var foo = r("foo");
})

This would be horrible, we just need more meaning than that when programming.

So ok, ".imports" are not mandatory but would be much more pleasant to use IMHO than a dependency array or an unreliable toString ( what if I would like to do var r=require, then it wouldn't work despite it would be correct JS).