AMD, the question, and a JavaScript sugar analogy

I want to expand more on the feedback from Tom Dale and Patrick Mueller (see part 2 too), who do not think AMD is the answer, and Dave Geddes asking at the end of his post what was the question?

More on the question, and an analogy:

How can we get to a future where we can easily share JS code, one that works on the server and in the browser? 

I have been talking about the advantages of AMD because it answers that question very well, and I have been reading people's responses to suggest CommonJS (CJS) modules as the format as impractical because it cannot work everywhere. Maybe what we can do is agree on some basics that still allow for sugar.

In other words, consider AMD the "base format" for distributing JS code, because it can work anywhere. That was its design goal. However, it is fine to use sugar on top of that, and even distribute source in that sugared form. But it would be great if the runtimes/tools you use have the capability to ingest AMD (if you take in outside code) and the tools that output your code should be be able to register as AMD modules.

An analogy:

JavaScript is winning because it is widely deployed. However, not everyone wants to program in plain JS. Some folks like to use sugar, like CoffeeScript and Stratified JS. In the end though, those sugared forms reduce to JS so that it can run in the most widely deployed environment.

AMD is to JavaScript as CJS modules/browser globals are to CoffeeScript/Stratified JS

It is fine to use sugar, it just helps if that sugar can take input/generate output that works everywhere. A bit more on why AMD can work everywhere:

Wrap It Up

Any complete JS module solution needs to have a wrapped format to allow:

1) serving optimized code, where multiple modules are combined into one file.
2) dynamically loading code from CDNs.

AMD solves this need, by allowing string IDs and a function wrapper, the shortest form is here:

define('module/id/here', function (require) {

    //Module code in here.

});

The current ES harmony module proposal may be able to get around a wrapped format for case #2 (cross domain script fetching), but they need a "wrapped" format to allow #1, and it looks like:

module moduleIdHere {

    //Module code in here.

}

I think it is better to use string IDs for the harmony case, and harmony is by no means done, but still, the need is the same: when combining multiple modules together. they need to be demarcated and named.

Given old browsers and old code that cannot participate in harmony, I believe any harmony module format should be designed to be translatable to AMD since AMD works everywhere. At the same time, we should make converters from AMD to harmony. I can appreciate these conversions may not allow 100% conversion, but it should be a very large percentage. I see any harmony proposal as sugar given it will not be ubiquitous. Maybe in 2-5 years after ES.next comes out the story will be different.

Give Me Some Sugar

Tom and Patrick object to having to write too much boilerplate, or ceremony, to code a module, and that is great goal. It is the same reason why some people like CoffeeScript or Stratified JS. But, we need to have a common wrapped format if we want to share code in one file bundles and from CDNs.

The Middle Way

Viewed in this light, I'm hoping we can all code in our preferred sugar, but still be able to share modular code at the base level. Here are some suggestions for us to get to that goal, some for browser library authors, node library authors, JS environment authors, and AMD Toolmakers:

Browser Library Authors

Not all of these may work for you. If you cannot do the first or second one, at least consider the third option.

Consider writing your modules using AMD. The define(function(require){}) wrapper is really light on boilerplate/ceremony, particularly if you are already using a CJS-style of code. There are AMD tools you can use to help with the building of your library, and it allows others with AMD tools to easily use your code in source form.

You get a free "custom builder" capability with this approach. Consumers of your source, particularly if it consists of multiple modules, can just drop your library code in their project, and when they do a build with AMD tools, it will only pull out the modular pieces of your library that they actually use. That is awesome.

You will be using a common idiom that people understand. It will be easier to translate skills.

CJS modules have this quality too, so feel free to use them if you prefer that style. In that case, I hope you will still consider the rest of these suggestions.

Consider using AMD if you need to bundle more than one module together. Almond can be used to provide the AMD API shim. Ace's mini-require might be another option.

If you limit yourself to just define() declarations, no loader plugins or callback-style require, then you can easily make your own wrapper. If you just need a way to concatenate the modules together, but do not need the require calls after a build because you are using globals (I think Tom fits in this group) then you might be able to get by with plain anonymous function wrappings, so this item may not apply to you.

For the top-level, public API for your module, feature detect for define.amd, and register your code as an AMD module. This is particularly helpful when you have dependencies. You can also get away from using globals and allowing better named dependencies.

You can still export a global for now, even in the AMD case, because AMD is still new to some people, and as underscore discovered, it can lead to problems. I expect this type of problem will go away after AMD support has been around a while and the global export can be removed. Since modules are still a new thing to many JS programmers, there will be a transition period while developers get used to not relying completely on globals. This will be true even if/when harmony modules ship.

By calling define() an AMD consumer now has the option now to get direct references for your code under a name of their choosing, and properly state and wait for its dependencies before running with them. The umdjs/umd project has some boilerplate examples to help you get started with this item.

Node Library Authors

If you author user-land node modules for others to use, in other words, do not develop Node core, and you think your library may be usable in the browser, then:

Stick with declarative dependencies. Avoid using calculated values for dependency names:

var a = require(someCondition ? 'a' : 'a1');

Ideally node would support a callback-style require(), as AMD does, to help when you do have a calculated dependency (see JS environment authors section below), but in the meantime, the module will translate very easily to AMD with just declarative dependencies:

var a = require('a'),

    b = require('b');

These type of calculated dependencies will not work in harmony modules either, so it is good to get used to this now.

Avoid __dirname and __filename. Or at least, be aware that in AMD loaders, you have access to module.uri which is like __filename, and a __dirname can be calculated from that value. So this type of detection is a good way to go: var uri = module.uri || __filename.

Use amdefine to create a define() method in your module, and call define() to define your module. This will make it easier for browser authors to directly consume your code, and the RequireJS optimizer removes the if(){} block that uses amdefine, so browser developers will not take the hit of that code or that module.

Consider using AMD as the bundling format, if you make a bundling tool, like browserify. If you need help with this or have questions on how best to do it, feel free to contact me.

The wrapper can be as simple as define('id', function(require, exports, module){});. You may need to provide an option to wrap your output in a function and use an internal define inside that function, but call the global define to expose the public API for the bundled set of modules.

By using the amdefine module as a dependency in your package.json, this will help test a real implementation of define(), and help core Node contributors get a feel for how many modules would like to have define() as an API option directly in Node.

JS Environment Authors

Do you make an environment that can run JavaScript? Like Node, Rhino, Mozilla's Jetpack?

Support AMD APIs for easily ingesting code. You can support the sugared CJS modules as a default, but allow JS library authors to reduce their feature detection boilerplate and allow easier code transfer by allowing define()'d modules.

This does not mean that you must load modules asynchronously. The nice thing about the basic define() API is that it works well in synchronous environments too, particularly with the declarative dependencies.

For Node in particular, I know this has been a contentious thing to consider, and it can be done in phases. Not all the phases need to be done, particularly loader plugins, but each one allows easier code sharing:

• Support module.uri and encourage that instead of __filename. In AMD, module.id is a module ID that is not a full path, so there is explicit module.uri for getting the module's path.
• Support a simple define() call, like the one Isaac had in previously. It can still operate synchronously.
• Support callback-style require([], function(){}) inside define() calls, one that fires the callback on nextTick. This is important for calculated dependencies.
• Consider loader plugin support. It could be limited to loader plugins that can only be run synchronously. This one might also help with the sort of transpiler support you have in Node now.

The previously mentioned amdefine package for Node, which consists of one file, does all of the above, and I am happy to continue work on it, build up unit tests for it, do whatever code changes you think are needed that would allow something like its define() implementation to land in core.

I think Node's synchronous module behavior can be maintained, but still open up easier code sharing pathways with code written by browser-based developers.

I hope my previous posts show why Node's module system as-is is not the right general solution for module-based development in the browser. The ES Harmony module proposal set has similar high level features as AMD, like a module in a block, the equivalent of a callback-style require. The need for those things are not going away.

Plus, the more browser developers (the biggest JS audience there is) can easily transfer their knowledge/idioms/code to run under Node, the more it helps ensure Node's longevity.

Support in other JS environments is also encouraged. It can take a similar path to the one outlined for Node. Mozilla's Jetpack right now only supports a simple define() call, and that is great. It still allows for some basic code sharing. Even getting a little bit of support, doing small steps is fine.

AMD Toolmakers

For those of us that make AMD tools, there are some things we can do that can help people share their code easier. Here is what I am doing:

Create translation tools. I'm working on volo, which can do this, among other things. With volo, you can download your favorite libraries from GitHub and optionally specify AMD wrapping for the library. Right now it is set up to wrap a library that uses globals. I want to add support for converting CJS/Node modules to AMD. CJS conversion is cleaner than the globals wrapping -- the globals wrapping needs more information from the developer to work.

Consider configuration of global import and export of module values. In a runtime loader, allow the developer to configure something like "when underscore loads, grab the global _ and use it as the value for the module". And conversely, "when this AMD module loads, also assign the module export value to a global named 'foo'". This may be better served by the translation tool -- it can be more direct, precise -- but this might help for cases in which the developer does not want a tool to modify the library source. It would only help with libraries that have no dependencies though, so it may not be useful enough on its own.

It is still best for library authors to call define() themselves since they can be more precise on the dependencies and allow getting direct references instead of using globals. However, the AMD toolmakers can help bridge the gap.

Globalize All The Things

All of the above assumes you think getting local handles on other modules/scripts is better than using globals. Not all JS developers hold that position though. In particular, I think Thomas Fuchs, Jeremy Ashkenas and to some extent Scott Gonzalez have all expressed a desire for a system that does not require changing the source files from using globals.

I'm open to hearing how that might work, but I think it is unrealistic as a general approach because:

We already have a problem with globals today, where multiple third party libraries want to use the same global but at different versions. For anything larger than a simple app, it is not scalable to try to do a "noConflict()" approach on nested dependencies.

You could try to get around that by executing module sets in a "sandbox", but the tech to do that with today's browsers, iframes, is riddled with land mines and it has trouble sharing things across frames.

It is bad practice for modules to name themselves. Case in point: Zepto. Zepto is targeted as a jQuery replacement. However, to use it as-is, it means libraries like Backbone have to do manual detection for it: var $ = root.jQuery || root.Zepto. This is not scalable, and it would be to Zepto's advantage to register as an anonymous module, and then allow the end developer, not every library author, to map require('jquery') to a path that resolves to Zepto instead.

It does not map well to other environments like Node and Jetpack, which have gone away from globals.

To be clear here though -- I want a module format that allows for the use of globals, and AMD does allow that. It is just not good to enforce use of globals as the path to code sharing/use. There needs to be a way to get local references via string names that can be mapped to other providers of that functionality.

In the end, I see globals as another kind of sugar. The tricky part is that it is hard to auto-convert given the variability between what a file is called, what global(s) it can export and what dependencies it uses, and mapping those dependencies effectively to files. Systems like AMD/CJS have a much more consistent approach to this, which leads to cleaner code and more effective sharing and tooling.

Summary

Hopefully this gives some concrete ways we can work together even if you prefer some sort of sugar over plain AMD.

Supporting AMD, even just on the output/input edges, is not because you think dynamic loading of scripts is the way to go, or that you have to throw away your favorite sugar. It is about getting a base level of code understanding that allows effective sharing, tooling and reuse across all JS environments.