Tuesday, December 28, 2010

Standards and proposals for JavaScript Modules and jQuery

This commit to add registeration of jQuery as a module via the Asynchronous Module Definition (AMD) API has brought up some questions that I should have addressed previously before the commit happened. The jQuery team may decide to back out the commit, as is their prerogative, but I hope not. This post will be the justification for the AMD API in general and for its suitability for jQuery in particular.

What is the Standard?

There was a concern brought up by Kevin Smith that module loading is still up in the air, and that there is no "Standard" yet. To me a standard is something that is written down, understood by people, and has a good level of market adoption. Multiple implementations help too.

CommonJS Modules is a standard in that way. It is written down, understood by a few people, and has a a good level of market adoption via Node. There are a few implementations for other JS engines.

The CommonJS group also has a process where they have voted before to call it a Standard vs. a Proposal. Anyone can join the CommonJS discussion list, and I think it is still an ongoing discussion on how best to vote on proposals, who should get to vote, and if voting makes any sense. It is helpful to know if most of the people on the list like a particular proposal though, it gives some confidence that the proposal has been thought out.

I have a concern about the diversity of the list participants though. Most joined when it was still ServerJS, and the CommonJS modules "standard" does not work well for use in web browsers that use script src="" to load scripts. So, you could hear that "CommonJS Modules are a standard", but it does not mean it works well in all CommonJS environments, most notably, the largest distribution of a JS environment, the browser.

That is the reason the AMD proposal exists. It works well in the browser environment where module loading is by default asynchronous, and with script src="" loading (the easiest to debug and fastest way to load code), there is no control over modifying the script source before it executes. AMD is called a "proposal" in the CommonJS list because people on the CommonJS list have not voted to call it a "standard".

There is recognition on the CommonJS list that the CommonJS Modules 1.1 is not sufficient for script src="" browser loading, but there is some disagreement over how to solve it. AMD is one proposal. Kevin Smith has put together a Module Wrappings proposal, based on some discussions on the CommonJS list. The Wrappings proposal is incomplete -- there is more to it based on the list discussions. I will highlight the main difference between AMD and Wrappings below, but first, another module API contender:

ECMAScript Simple Modules. This is a "strawman", which is like a proposal in the CommonJS group: it is still something being worked on. This one is different because it is being worked on by the ECMAScript committee, and it seems like the strongest module proposal in that committee so far. If adopted, some future version of the ECMAScript (JavaScript) standard will have support for modules natively. So, any module API that used today should be aware of what is going on in the ECMAScript arena.

So, for talking about a module standard, particularly for use in the browser, these are the main contenders:
  • AMD
  • Wrappings
  • Simple Modules
There are other modules systems like the one used in Dojo, and the one used by YUI, but those are too tied to those existing toolkits to be broadly adopted. Dojo has started to move to AMD.

AMD and Wrappings Differences

AMD and Module Wrappings are mostly the same. There needs to be a function wrapping around the actual module, to make sure the dependencies are fetched first before executing the module. There is some bikeshedding around names that I do not think is important (define vs. module.declare).

The main difference: AMD executes the module functions for dependencies before executing the current module function, where Wrappings just makes sure the module function is available, but does not execute it until the first require("") call inside the module function asks for it.

For shorthand, I will call the AMD approach the "execution" approach, and the Wrappings approach "availability". Both relate to how dependencies are handled.

Because AMD executes the module functions of dependencies before calling the function wrapping of the current module, it can pass the dependency as an argument to the factory function. This is a side benefit of execution.

The "availability" approach was used for Wrappings to keep it most similar to how CommonJS modules deal with dependencies, but I believe it is the wrong choice for the future.

Problems with "availability" model

"Availability" is not supported in the ECMAScript Simple Modules strawman. Simple Modules uses a model that is closest to the "execution" model: you use "module" and "import" keywords to reference dependencies, but they cannot be used like require("") is used in the "availability" model. "module" and "import" are more like directives, and not something that can by dynamically assigned, as part of conditionals. Examples:

In the process of working on RequireJS and trying to translate modules written for Node to a wrapped format, I have seen the following constructs in Node modules (which assume the "availability" model):
try {
var constants = require("constants");
...
} catch (e) {
//Older version of node here
}
and:
var foo;
if (someCondition) {
foo = require("foo1");
} else {
foo = require("foo2");
}

Those work in the "availability" model, but not in Simple Modules. AMD's execution model does not allow the above constructs, so modules written for AMD will be more portable to Simple Modules.

2) The "execution" model fits better for projects that use libraries like jQuery, Prototype or MooTools, where many of the modules augment other objects, and they are assumed to have already run before executing the current module function. jQuery plugins augment the jQuery object, Prototype and MooTools augment JavaScript object prototypes.

Choose AMD

AMD is the API for today that translates to the future best, and it should be the one that jQuery should support because:
  • There is a document that defines the API.
  • It is understood by a few people, and discussed in public on the CommonJS list.
  • It is not ratified as a "Standard" in CommonJS because there are list participants that still want to hold on to CommonJS "availability" semantics. However, that availability model is not as future proof as AMD, at least for the Simple Modules future.
  • AMD translates better to Simple Modules due to the execution model. It maps better to the the related Module Loaders strawman, so if Simple Modules with the Module Loaders API becomes a standard and gets implemented in browsers, code written for AMD is more likely to continue to work.
  • For browsers versions that probably never support Simple Modules (IE6-8, even 9?), code continues to work.
  • The "execution" approach for dependencies fits better with the mental model of existing browser toolkits that want to augment other objects, like jQuery plugins.
  • AMD is supported by a module loader (RequireJS) that has had some level of successful adoption, and it is fairly robust now, with over a year in development with frequent releases. RequireJS has an adapter that allows the same modules to be run in Node and Rhino. The Node adapter allows using existing Node modules.
  • There are other AMD implementations.
  • RequireJS in particular works hard to make sure it works well with jQuery, including integration with jQuery's readyWait to hold off DOM ready callbacks until all scripts are loaded.
  • AMD has more real world adoption than Wrappings. The Dojo Toolkit now supports using an AMD-compliant loader in their trunk code for Dojo Core and Dijit. There is a thriving RequireJS list with real people using it. BBC iPlayer uses it. The RequireJS implementation of the AMD API has been promoted as a useful tool in multiple jQuery-related conferences.

Monday, December 20, 2010

RequireJS 0.2.1 Released

RequireJS 0.2.1 is available for download.

This is a bug fix release related to issues with the refactoring done in 0.2.0. Probably the most noticeable bug that has been fixed: the "all plugins" builds did not include the plugins.

Thursday, December 16, 2010

RequireJS 0.2.0 Released

RequireJS 0.2.0 is available for download. The highlights: stronger, faster, and prettier.
  • Refactored core.
  • New loader plugin API support.
  • Improved Rhino and Node adapters.
  • Bug fixes around loader plugins and the optimizer.
  • require.modify removed.
  • Removed Transport D files.
Stronger: The refactored core is more robust vs. the 0.1x releases, and the new plugin API makes it much easier to construct loader plugins. The plugin API is an implementation of Kris Zyp's plugin API proposal. The optimizer/build support is different than what he proposed, and I plan on discussing the changes on the CommonJS list where he sent the proposal. But the basic API for defining a plugin in that proposal is supported.

The Rhino and Node adapters are even better in this release. I want to improve the Node adapter in a future release by allowing the use of npm installed modules without setting up a package path config.

Faster: The refactored core responds faster to loaded modules, and it will execute a module as soon as its dependencies load. In practice you will probably not notice an appreciable speed change, but it allows the new plugin API to work well.

Prettier: The web site got a much needed facelift and a logo, thanks to the talented Andy Chung. So much better than my feeble attempt.

As part of the refactoring, I removed the Transport D files. I believe they are no longer in use by other projects. I also removed require.modify. It could not support relative path names, and was always an odd thing that did not quite fit in with the rest of the code.

The refactored code is also more strict. If you do this in main.js:
require(['foo'], function () {});
And then in foo.js, it does:
require(['bar'], function () {});
The require callback in main.js will be called before the require callback in foo.js, because foo.js did not define a module. The loader thinks it just needs execute foo.js in order to call the require callback in main.js. To make sure the require callback in main.js waits for foo's factory function to get called, change the require call in foo.js to define:
//In foo.js:
define(['bar'], function () {});
Thanks to the community for filing bugs and contributing to the discussions on the mailing list. Thanks to Ben Hockey for drilling in deep and working out how to load the Dojo trunk code with RequireJS. It prompted some great bug fixes, and a few patches from Ben.

Rawld Gill, along with Kris Zyp, has done a lot of work to convert the Dojo trunk code to use the AMD API proposal. Rawld has even done an alternative loader that implements the AMD API. Check out his work if you want a different look at an AMD implementation.

What's next? I want to move closer to a 1.0 for RequireJS, but I would like to get these things settled first:
  • Get feedback from the community on the loader plugin API. I need to do some better documentation for it too.
  • I want to do a has.js plugin generator, and I want to integrate support in the optimizer to trim if(has['test']){} if the configuration passed to the optimizer indicates that has['test'] will always be true.
  • Allow the optimizer to run on top of Node. Right now the optimizer runs on top of Java via Rhino, but that is increasingly becoming a liability.
  • Sort out full packages support. Right now RequireJS can load modules in packages, but it cannot handle two versions of the same package in a project. While this is a minority use case, I would like to have a story for it. I think it is workable in source form, but I am not sure how it will work out yet for optimized scripts that want to include modules from two different versions of the same package.