Web dev with two turntables and a microphone

There is a development style for web-based user interfaces a little up the road, from the habitations and the towns we know. It is not new or unusual, but I think it needs to be spelled out a bit since it is the "why" behind tools like RequireJS and volo, and it is what I expect to be the defacto development style for the future, particularly for mobile web-based apps:

Do not use servers to build web-based user interfaces.

What do I mean by "do not use servers"?

• Servers should be restricted to providing data API services only.
• A simple web server to serve static files for UI can be used, but it is only to support serving static resources that can be heavily cached. The UI serving is strongly decoupled from the API services.

There is a JavaScript corollary for this approach, and a "red pill" variant, but first more on the benefits.

Design advantages

This separation of concerns gives the following benefits:

• Better scaling characteristics. Static web UI can be offloaded to Content Delivery Networks (CDNs) and heavily cached. Even if the UI is served from the same domain as the data API services, the UI can be served from separate, simpler servers that can be scaled independently from the API servers.
• Forces good API design. Getting your data from a network API enforces strong constraints on what are data needs and what are UI needs.
• Better offline. This is really critical for mobile UIs. You can set up the UI to use AppCache and give the user feedback if the network is not available.
• Better division of labor between teams. Front end UI developers and back end data developers can have different personalities and scaling concerns. Having a strict API contract between them allows them to each focus on what they do best.

Ideal use cases

Basically, the future:

• JavaScript-driven web apps. Message-based/social applications, maps, games.
• Mobile web. Sites served to users on mobile devices, devices where connectivity is not always guaranteed.
• Installed "web apps". Phonegap/Cordova-backed apps, or something like Mozilla Web Apps that will allow you to install your web app on an OS so that it has its own desktop icon and may launch the UI from files on the local computer/device.

Use cases that may not fit

Some content-heavy UIs, like a CMS or blog, might work well as a PHP/Rails/Django/Node app where the data (content) is injected into the page. This is primarily to satisfy search engines or deal with older browsers that, for instance, do not support the history API.

However, for more static content, like blogs, much of the content can be generated once and saved to disk, and could still fit into the "serve static resources" approach. memcached is often used to achieve this effect.

So even for these cases, see how much a data API can be used to generate static output for the UI instead of doing a runtime merge for every request. It will make it easier to deal with the request-hungry googlebots, and enable the other benefits above.

Of course this will not work for everything, but for the next project, see if it does fit.

The JavaScript corollary

Any tools you need to build the UI should be written in JavaScript.

You are already using JavaScript in the page, might as well use it for your tools. It makes it easier for front end developers to share and improve the tools.

While we may not be at this end state yet, we can now get there now with Node.

Wait, isn't Node a server? It is an event-based JavaScript runtime that happens to have good networking and file I/O support. You could use it to build an API server, but you can just treat it as a command line tool environment. That is how the RequireJS optimizer and volo use Node -- a bootstrap for single file command line tools.

By using fs.watch(), you can set up tool actions to occur on file changes, no need to run the UI through Node.

If you prefer sugar like CoffeeScript and SASS-like systems, go for it. Just use command line/watch tools to use them. And ideally the tools are written in JavaScript. If they are not, that is OK, consider JS the next time a rewrite is in order. Or better yet, create JS-backed versions of the project now and use it for future development.

You do not need to subscribe to this corollary, the big step is to just embrace serverless UI design.

The red pill

If you want to go with the "use JS for tooling", you can go a step further:

Avoid tools/dependencies/sugar to help you make UI, and embrace plain JS/CSS/HTML. It is all you really need to know. Sure, you need some tools around that to help with source code management/deployment, but those are needed in any case. You can simplify by avoiding abstractions.

Of course I still support you if you want the extra sugar or tools to make web development more palatable for yourself. I can see using tools like bicycles to help you get places. Even if using plain JS, it is helpful to use some libraries to smooth over browser differences. However it is great to keep the extra gear you need to understand and maintain to a minimum.

The tools I make will focus on getting as close to the JS/CSS/HTML bare metal as possible, but they should also work well if you want to go up the abstraction ladder.

RequireJS and AMD modules are about sticking to the JS metal but still provide a basic capability missing in JavaScript. AMD is a great "transpile" target if you want sugar on top of AMD. volo is about getting your web project set up easily with the dependencies you choose to have.


Summary

Serverless UIs supported by JS-backed tools are where it's at.

RequireJS 1.0.5 released

RequireJS 1.0.5 is available.
 
The notable changes:

• Small fix for running the optimizer on Windows with Node 0.6.8.
• Fix bug with loading Dojo code from source.
• New config option that allows using the RequireJS config from your main JS file as part of the optimizer options.  It is mentioned in a new "common pitfalls" section in the optimizer doc.

Complete set of changes:

• require.js
• r.js optimizer

AMD support for Underscore and Backbone

As mentioned previously, at the moment there is no support for AMD in Underscore, a decision that also affects Backbone.

However, there have been enough developers interested in using Underscore and Backbone with AMD that forks were set up under the amdjs organization that have optional define() calls in them:

https://github.com/amdjs/underscore

https://github.com/amdjs/backbone

Ryan Florence suggested the AMD community maintain these forks, and if you would like to see AMD support as part of the main repos, click on the links above and click the Watch button. But only do it if you actually use the libraries above in an AMD context.

Hopefully this will give the DocumentCloud folks some indication of how many people might benefit from having support directly in the main repos, and maybe over time AMD support can be added to the main repos.

Both forks optionally register as AMD modules but also export global objects. This is to address the difficulty Underscore saw when only doing the AMD call, which led to the AMD removal from Underscore.

If you want an easy way to fetch these AMD versions from the command line, or to have a tool to make it easy to AMD-wrap older releases of those libraries, use volo. The README's Usage section has specific instructions on how to fetch these libraries.

If you would like to try an AMD+Backbone option that does to not modify the source files, check out Tim Branyen's explorations.

AMD, the question, and a JavaScript sugar analogy

I want to expand more on the feedback from Tom Dale and Patrick Mueller (see part 2 too), who do not think AMD is the answer, and Dave Geddes asking at the end of his post what was the question?

More on the question, and an analogy:

How can we get to a future where we can easily share JS code, one that works on the server and in the browser? 

I have been talking about the advantages of AMD because it answers that question very well, and I have been reading people's responses to suggest CommonJS (CJS) modules as the format as impractical because it cannot work everywhere. Maybe what we can do is agree on some basics that still allow for sugar.

In other words, consider AMD the "base format" for distributing JS code, because it can work anywhere. That was its design goal. However, it is fine to use sugar on top of that, and even distribute source in that sugared form. But it would be great if the runtimes/tools you use have the capability to ingest AMD (if you take in outside code) and the tools that output your code should be be able to register as AMD modules.

An analogy:

JavaScript is winning because it is widely deployed. However, not everyone wants to program in plain JS. Some folks like to use sugar, like CoffeeScript and Stratified JS. In the end though, those sugared forms reduce to JS so that it can run in the most widely deployed environment.

AMD is to JavaScript as CJS modules/browser globals are to CoffeeScript/Stratified JS

It is fine to use sugar, it just helps if that sugar can take input/generate output that works everywhere. A bit more on why AMD can work everywhere:

Wrap It Up

Any complete JS module solution needs to have a wrapped format to allow:

1) serving optimized code, where multiple modules are combined into one file.
2) dynamically loading code from CDNs.

AMD solves this need, by allowing string IDs and a function wrapper, the shortest form is here:

define('module/id/here', function (require) {

    //Module code in here.

});

The current ES harmony module proposal may be able to get around a wrapped format for case #2 (cross domain script fetching), but they need a "wrapped" format to allow #1, and it looks like:

module moduleIdHere {

    //Module code in here.

}

I think it is better to use string IDs for the harmony case, and harmony is by no means done, but still, the need is the same: when combining multiple modules together. they need to be demarcated and named.

Given old browsers and old code that cannot participate in harmony, I believe any harmony module format should be designed to be translatable to AMD since AMD works everywhere. At the same time, we should make converters from AMD to harmony. I can appreciate these conversions may not allow 100% conversion, but it should be a very large percentage. I see any harmony proposal as sugar given it will not be ubiquitous. Maybe in 2-5 years after ES.next comes out the story will be different.

Give Me Some Sugar

Tom and Patrick object to having to write too much boilerplate, or ceremony, to code a module, and that is great goal. It is the same reason why some people like CoffeeScript or Stratified JS. But, we need to have a common wrapped format if we want to share code in one file bundles and from CDNs.

The Middle Way

Viewed in this light, I'm hoping we can all code in our preferred sugar, but still be able to share modular code at the base level. Here are some suggestions for us to get to that goal, some for browser library authors, node library authors, JS environment authors, and AMD Toolmakers:

Browser Library Authors

Not all of these may work for you. If you cannot do the first or second one, at least consider the third option.

Consider writing your modules using AMD. The define(function(require){}) wrapper is really light on boilerplate/ceremony, particularly if you are already using a CJS-style of code. There are AMD tools you can use to help with the building of your library, and it allows others with AMD tools to easily use your code in source form.

You get a free "custom builder" capability with this approach. Consumers of your source, particularly if it consists of multiple modules, can just drop your library code in their project, and when they do a build with AMD tools, it will only pull out the modular pieces of your library that they actually use. That is awesome.

You will be using a common idiom that people understand. It will be easier to translate skills.

CJS modules have this quality too, so feel free to use them if you prefer that style. In that case, I hope you will still consider the rest of these suggestions.

Consider using AMD if you need to bundle more than one module together. Almond can be used to provide the AMD API shim. Ace's mini-require might be another option.

If you limit yourself to just define() declarations, no loader plugins or callback-style require, then you can easily make your own wrapper. If you just need a way to concatenate the modules together, but do not need the require calls after a build because you are using globals (I think Tom fits in this group) then you might be able to get by with plain anonymous function wrappings, so this item may not apply to you.

For the top-level, public API for your module, feature detect for define.amd, and register your code as an AMD module. This is particularly helpful when you have dependencies. You can also get away from using globals and allowing better named dependencies.

You can still export a global for now, even in the AMD case, because AMD is still new to some people, and as underscore discovered, it can lead to problems. I expect this type of problem will go away after AMD support has been around a while and the global export can be removed. Since modules are still a new thing to many JS programmers, there will be a transition period while developers get used to not relying completely on globals. This will be true even if/when harmony modules ship.

By calling define() an AMD consumer now has the option now to get direct references for your code under a name of their choosing, and properly state and wait for its dependencies before running with them. The umdjs/umd project has some boilerplate examples to help you get started with this item.

Node Library Authors

If you author user-land node modules for others to use, in other words, do not develop Node core, and you think your library may be usable in the browser, then:

Stick with declarative dependencies. Avoid using calculated values for dependency names:

var a = require(someCondition ? 'a' : 'a1');

Ideally node would support a callback-style require(), as AMD does, to help when you do have a calculated dependency (see JS environment authors section below), but in the meantime, the module will translate very easily to AMD with just declarative dependencies:

var a = require('a'),

    b = require('b');

These type of calculated dependencies will not work in harmony modules either, so it is good to get used to this now.

Avoid __dirname and __filename. Or at least, be aware that in AMD loaders, you have access to module.uri which is like __filename, and a __dirname can be calculated from that value. So this type of detection is a good way to go: var uri = module.uri || __filename.

Use amdefine to create a define() method in your module, and call define() to define your module. This will make it easier for browser authors to directly consume your code, and the RequireJS optimizer removes the if(){} block that uses amdefine, so browser developers will not take the hit of that code or that module.

Consider using AMD as the bundling format, if you make a bundling tool, like browserify. If you need help with this or have questions on how best to do it, feel free to contact me.

The wrapper can be as simple as define('id', function(require, exports, module){});. You may need to provide an option to wrap your output in a function and use an internal define inside that function, but call the global define to expose the public API for the bundled set of modules.

By using the amdefine module as a dependency in your package.json, this will help test a real implementation of define(), and help core Node contributors get a feel for how many modules would like to have define() as an API option directly in Node.

JS Environment Authors

Do you make an environment that can run JavaScript? Like Node, Rhino, Mozilla's Jetpack?

Support AMD APIs for easily ingesting code. You can support the sugared CJS modules as a default, but allow JS library authors to reduce their feature detection boilerplate and allow easier code transfer by allowing define()'d modules.

This does not mean that you must load modules asynchronously. The nice thing about the basic define() API is that it works well in synchronous environments too, particularly with the declarative dependencies.

For Node in particular, I know this has been a contentious thing to consider, and it can be done in phases. Not all the phases need to be done, particularly loader plugins, but each one allows easier code sharing:

• Support module.uri and encourage that instead of __filename. In AMD, module.id is a module ID that is not a full path, so there is explicit module.uri for getting the module's path.
• Support a simple define() call, like the one Isaac had in previously. It can still operate synchronously.
• Support callback-style require([], function(){}) inside define() calls, one that fires the callback on nextTick. This is important for calculated dependencies.
• Consider loader plugin support. It could be limited to loader plugins that can only be run synchronously. This one might also help with the sort of transpiler support you have in Node now.

The previously mentioned amdefine package for Node, which consists of one file, does all of the above, and I am happy to continue work on it, build up unit tests for it, do whatever code changes you think are needed that would allow something like its define() implementation to land in core.

I think Node's synchronous module behavior can be maintained, but still open up easier code sharing pathways with code written by browser-based developers.

I hope my previous posts show why Node's module system as-is is not the right general solution for module-based development in the browser. The ES Harmony module proposal set has similar high level features as AMD, like a module in a block, the equivalent of a callback-style require. The need for those things are not going away.

Plus, the more browser developers (the biggest JS audience there is) can easily transfer their knowledge/idioms/code to run under Node, the more it helps ensure Node's longevity.

Support in other JS environments is also encouraged. It can take a similar path to the one outlined for Node. Mozilla's Jetpack right now only supports a simple define() call, and that is great. It still allows for some basic code sharing. Even getting a little bit of support, doing small steps is fine.

AMD Toolmakers

For those of us that make AMD tools, there are some things we can do that can help people share their code easier. Here is what I am doing:

Create translation tools. I'm working on volo, which can do this, among other things. With volo, you can download your favorite libraries from GitHub and optionally specify AMD wrapping for the library. Right now it is set up to wrap a library that uses globals. I want to add support for converting CJS/Node modules to AMD. CJS conversion is cleaner than the globals wrapping -- the globals wrapping needs more information from the developer to work.

Consider configuration of global import and export of module values. In a runtime loader, allow the developer to configure something like "when underscore loads, grab the global _ and use it as the value for the module". And conversely, "when this AMD module loads, also assign the module export value to a global named 'foo'". This may be better served by the translation tool -- it can be more direct, precise -- but this might help for cases in which the developer does not want a tool to modify the library source. It would only help with libraries that have no dependencies though, so it may not be useful enough on its own.

It is still best for library authors to call define() themselves since they can be more precise on the dependencies and allow getting direct references instead of using globals. However, the AMD toolmakers can help bridge the gap.

Globalize All The Things

All of the above assumes you think getting local handles on other modules/scripts is better than using globals. Not all JS developers hold that position though. In particular, I think Thomas Fuchs, Jeremy Ashkenas and to some extent Scott Gonzalez have all expressed a desire for a system that does not require changing the source files from using globals.

I'm open to hearing how that might work, but I think it is unrealistic as a general approach because:

We already have a problem with globals today, where multiple third party libraries want to use the same global but at different versions. For anything larger than a simple app, it is not scalable to try to do a "noConflict()" approach on nested dependencies.

You could try to get around that by executing module sets in a "sandbox", but the tech to do that with today's browsers, iframes, is riddled with land mines and it has trouble sharing things across frames.

It is bad practice for modules to name themselves. Case in point: Zepto. Zepto is targeted as a jQuery replacement. However, to use it as-is, it means libraries like Backbone have to do manual detection for it: var $ = root.jQuery || root.Zepto. This is not scalable, and it would be to Zepto's advantage to register as an anonymous module, and then allow the end developer, not every library author, to map require('jquery') to a path that resolves to Zepto instead.

It does not map well to other environments like Node and Jetpack, which have gone away from globals.

To be clear here though -- I want a module format that allows for the use of globals, and AMD does allow that. It is just not good to enforce use of globals as the path to code sharing/use. There needs to be a way to get local references via string names that can be mapped to other providers of that functionality.

In the end, I see globals as another kind of sugar. The tricky part is that it is hard to auto-convert given the variability between what a file is called, what global(s) it can export and what dependencies it uses, and mapping those dependencies effectively to files. Systems like AMD/CJS have a much more consistent approach to this, which leads to cleaner code and more effective sharing and tooling.

Summary

Hopefully this gives some concrete ways we can work together even if you prefer some sort of sugar over plain AMD.

Supporting AMD, even just on the output/input edges, is not because you think dynamic loading of scripts is the way to go, or that you have to throw away your favorite sugar. It is about getting a base level of code understanding that allows effective sharing, tooling and reuse across all JS environments.

Reply to Tom on AMD

Tom Dale had a good post about why he dislikes AMD. Please go check it out. We talked about it a bit while he was writing it. It is great he took the time to write it out. We need more of that. We are all trying to find the best path, and it is great we are talking this out.

While I appreciate the critique, what is missing is a solid alternative to AMD. In particular, "use build tools to wrap up CommonJS modules" is not a generic solution. It does not allow for generic dynamic loading of resources, particularly from CDNs. Dynamic loading is needed for big sites that still do builds but want to stage loading of the site as the user uses it.

On his concern about too much ceremony -- this is the minimum amount of ceremony:

define(function(require) {

    //Module code in here

});

By doing that, it is easier for other people to consume your module since module authors do not have to provide "built" versions of their module to use on CDNs. And if they did have to do builds, what format would they publish it in? It would need a function wrapper. Hey, AMD has that!

That simple function wrapper allows moving away from globals, although you still can use globals if you want. It allows for dynamic loading. Recall I'm talking about built layers for staged loading/performance, not "load each module with a separate HTTP request". It allows very simple module sharing now. It allows easier sharing of built files too, since AMD specifies the built file format.

With AMD, you can still use build tools if that is how you like to roll in dev. Go for it. Almond is a great AMD one for that purpose. Here is an example of using the RequireJS optimizer for runtime HTTP-served builds. And since there are multiple AMD implementations, you can choose how big of an AMD lib you want.

I'm sure Tom could adjust his 50 line loader to process the define calls, since his built code probably has function wrappers. He could even adapt his "hydrate from strings" loader to use the AMD syntax.

By using that function wrapper, others could consume his individual modules and use their own choice of AMD loader to deal with their particular loading needs.

On the "hydrate from strings" build approach: I would prefer using appcache and using built layers of AMD code instead. This gives much better overall mobile app performance since more than just JS can be cached. But that is more of a side point. Feel free to use the "hydrate from strings" approach, AMD could be used in that fashion too.


My summary: it sounds like he just wants to avoid a small function wrapping and a level of indent. And to be fair, many simple AMD introductions use the array of dependencies syntax, not the sugared form above, so I can appreciate the confusion.

But trying to avoid that function wrap+indent comes with some code sharing costs, and the alternative he mentions does not handle the breadth of JS module consumption.

Simplicity and JavaScript modules

All of us are looking for simplicity, but there are different levels to simplification. This is a story of what could be considered simple for modules in JavaScript. This post was prompted by the removal the optional AMD define() call in underscore.

For a post on simplicity, it is a bit long, but I'm not a great writer, and I find I normally edit myself so much as to lose interest in posting, so then I end up not communicating. Better to start communicating even if imperfect.

I want to lay out why AMD modules are the simplest overall module solution for JavaScript at the moment, and where other approaches are not as simple as they may appear.

Script Tags

JavaScript does not have syntax for modules, but most programming languages do. "import", "require", "include" seem like popular choices.

JavaScript on the web has meant using script tags and manually ordering those script tags so that dependencies on global objects are worked out correctly. It is also important that those dependencies execute in order.

That sucks for the following reasons:

• You use a separate language, HTML, to specify your JS dependencies and their order.
• A script's dependencies can be unclear.
• If you want to later load some code on demand, it is very difficult to work it out so that the scripts execute in order. This has gotten better over time -- newer browsers that support the async="false" attribute help with this. But older browsers still suck.

So what happens?

• You constrain yourself to only creating simpler applications that can get by with concatenating all the scripts in a certain order, and depend on server tools to help you write out your HTML with the script tags and the rewrite the page after a build to not have those script tags. Think Ruby on Rails or Django.
• Toolkits start building APIs to do this work for you. Dojo and YUI were some of the very first to do so. The Closure library does too, although it was more of a cousin to Dojo.

Option 1 is clearly not appropriate for the full spectrum of web development. There are HTML game engines, webmail/office suites, and then browser-based "no server" development like Phonegap-backed mobile apps.

So it is best to have some API or syntax in JavaScript to get units of code. For it to work well, we should all be using the same API. Otherwise things get messy real fast.

CommonJS

The CommonJS group tried to work out a system for doing this. It is pretty simple too:

• require('some/id') to reference a bit of code. Since a string literal is used, it allows for easy static analysis of dependencies -- no more manually ordering script tags! The ID also has a convention of mapping to a partial path. So you can get by without having to specify full URLs and it opens up for ways to map an ID to a path does not fit the ID-to-path convention. This is really helpful most immediately for mock testing, but it has many other uses.
• Modules do not give themselves a name, they are anonymous.  This is great -- if you load require('jquery') then you do not have to have special knowledge to access some global with different spelling, like jQuery. Same for 'backbone.js' loading an object called Backbone. The end user is in control of the name.
• exports is handy for circular dependencies. Yes you should avoid circular dependencies. If you have one, there is a good chance you are doing it wrong. But there are valid circular dependencies, and a script referencing system needs to support them.
• module is important because modules are not named. Sometimes though you need to know the name they ended up with, and sometimes from what path, because you may have some non-JS assets you need to reference. module.id and module.uri give you that ability.

There were some wrinkles though:

• They did not formalize a way to export a module value that was not just a plain object with properties. It is extremely common and useful to want to export a function as the module value. jQuery is probably the most known example. Constructor functions are another set of useful export values.
• It was not so simple to get it to work in the browser.

AMD

The original participants on the CommonJS list thought it was better to blue-sky the development of a module syntax, not be held back by what might work in the browser, just what might work with existing JS syntax. The group also started off as ServerJS, so they were also in the mindset of what would work best on the server, where file I/O is cheaper/easier.

The hope was that if they worked out a module system that worked well with the existing JS syntax but had problems in the browser, hopefully they could convince browser makers to plug the holes to make it possible. In the meantime, since they were developing servers that could run JS -- they could just bundle up/transform the JS using server tools, or offer a compile step, while browsers caught up.

However, for those of us who came from Dojo, requiring a server tool or compile step to just develop in JS was a complication. I'm going to mangle Alex Russell's quote on this, but "the web already has a compile step. It's called Reload".

Why force the use of a tool to just start developing? It should be simple: just a browser and a text editor.

This was accomplished by taking the CommonJS format and allowing a function wrapper around the code:

define(function (require){

    var dep = require('dep');

    return value;

});

Or, the shorter form::

define(['dep'], function (dep) {

    return value;

});

Since a function wrapper was used, it meant:

• the scripts could load in any order they want, easy to parallelize even on old browsers.
• "return" could be used to return the module value, even functions.
• No special tooling was needed to convert source to an acceptable browser format.
• It worked in browsers, today, and would in the future.

So the story around modules gets very simple, no special tooling to start, no worrying about "I need to run a converter to share this module", no special sets of instructions for your server of choice to do conversions.

Yes, a loader library is needed, but one is needed in any case, there is no native JS syntax. That is the basic ante for any module system that scales up beyond a simple JS concatenation for a Rails application.

For some people, a function wrapper with a level of indent was not seen as simple enough. However, designing a system without it meant a bunch of complexity once you stopped looking at an individual file. A miscalculation of the overall complexity cost.

Loader Plugins

Another simplicity in AMD systems: loader plugins. Loader plugins would not be considered if you had copious, synchronous IO capabilities. However, by fully embracing the network/async loading on the web, you start to see how creating a loader plugin that treats a dependency as a simple string as useful.

Some dependencies are not static scripts, but could have more complex loading (Google Maps code) or simple HTML templates that need to be loaded for the module to be useful.

In AMD, this can look like so:

define(function (require) {

    var maps = require('googleapi!maps?sensor=false'),

        template = require('text!form.html');

    return function (data) {

        //You can  synchronously return a value

        //based on the template.

        return template.replace(...);

    });

});

Compare that with a non-plugin approach. Do you want to load those two resources in parallel? That would be ideal, but then to do that, you probably need something like a promise library to help out. And now you have two problems. I mean that partly in jest -- I use a promise library for some types of code -- but it is definitely a complexity hurdle to jump.

The async networking also means your module is not completely ready until those resources are available, so your public module API now must be a callback API. For "simplicity" assume you just load the dependencies serially, to avoid some of the promise-isms.

define(function (require) {

    var googleapi = require('googleapi'),

        text = require('text');

        googleapi.fetch('maps?sensor=false', function (map) {

            text.fetch('form.html'), function (text) {

                //dependencies are now loaded.

            });

        });

    return function (data, callback) {

        waitForDependenciesToLoad(function (data, callback) {

            callback(template.replace(...));

        });

    }

});

Loader plugins simplify async development, and some of their resources, like the text templates, can be inlined in a build file with other JS modules. Loader plugins give you simplicity and speed.

If you like transpiling other languages into JS, they are great for that purpose too.

ECMAScript

The ECMAScript group wants to get modules in for the harmony effort, the next ECMAScript release. They chose to not go with the CommonJS syntax, but what did they did choose looks fairly similar on the surface. It favors the introduction of new syntax to get some advantages with compile time checks. You can check out the following for more information:

• harmony:modules
• harmony:module_examples
• harmony:module_loaders

Those links have changed over time, so the rest of this feedback may not be valid in the future. As I read it today, I do not believe harmony modules give much benefit over what AMD can do now, but harmony modules do introduce more complexity, and has some unanswered questions:

• It uses module Foo {} syntax for declaring an inline module, where the module ID, Foo, is a JS identifier. However dependencies can string names/URLs. How do you optimize this code for delivery in a web browser? In AMD, string names are used both for references and for the module names. This is better because it allows for loader plugin IDs that can have their output inlined in build output, and it ties module IDs more directly to the string names used in dependency names.
• It does not support a loader plugin API out of the box. You can construct one by using the module_loaders API, but at that point you need to ship a .js file for the "loader", so now the developer has the same complexity cost as AMD today.
• New syntax means the module support cannot be shimmed in older browser via a runtime library. It requires a compile step/server transform to work in older browsers. This is one the same problems the CommonJS format had.
• It is unclear how a JS library that works in browsers without ES module support "opts in" to register as an an ES module since ES modules use new syntax. Maybe the suggestion is to use the module_loader's loader.createModule syntax? I cannot see how that fits with the compile-time export checking though. If a runtime capability check cannot be used to opt in to ES module registration, it makes it very hard to upgrade web libraries to ES module syntax. We're back to the problems that made it hard to use CommonJS syntax on the web.

The new syntax in harmony modules is used to enable some compile time checking of export names and if they are referenced correctly.

However, compile time checking to see if an export name is use correctly is a very small benefit for the end developer given the other costs above. Furthermore, I want more than just an export name/type check. I want intellisense on the arguments that can be passed to functions, general data type information and comments on usage.

Working out a comment-based system that can reflect this info into text editors will provide much more value. Since it is comment based it fits in with old browsers. I know it is harder to agree on that comment syntax (mostly because it is easy to bikeshed), but it will simplify developers' lives more, and lead to faster turn-around time on development.

If something like loader plugins are not natively supported, and the optimization story is not sorted out, it does not have an advantage for AMD today, and AMD is simpler.

However, the harmony module_loaders API is really useful. I'm not sure it needs to be as big as it currently is. I would be fine with something like Node's vm module API as a start. Basically, some container or vat I can load scripts into without interfering with other scripts. This is one area that is very hard to do on the web today.

So for me, the "simple" solution for any ES6 module-related work:

• Make it a module API, not new syntax. Then we can shim it easier for older browsers, and existing libraries and capability check it and opt in. If you need a suggested API, I hear AMD is quite nice. It has a few implementations and has been used in the real world.
• Support loader plugins natively. It really helps with async programming, and optimizations. If I need to ship a library to deliver the benefits that loader plugins give for async programming, then there is very little motivation for developers to switch away from AMD.
• Provide something like Node's VM API, maybe with a little bit of the intrinsics stuff in the current module_loaders API.
• Do not bother with compile time checking of export names. Instead put effort into a comment based system that can reflect more information for use in editors. That will save developer more time. Since it is comment based, it will work in older browsers and optimizes out cleanly.

I will post this feedback to the es-discuss group. I wanted to hone my feedback before giving it to the es-discuss group, but this will have to do, otherwise I may never give it.

Underscore and AMD

This brings us to the recent code change to remove the AMD block from underscore. The arguments for removing it are listed here, but I think the main argument is really about what is simple for Jeremy: He does not need it for the kinds of sites he builds, and by adding it, he has gotten reports of problems.

Those problem reports go away if he also exports underscore as a global, but he does not think he should have to do that if AMD is available. I do not think that is a fair bar to hold for AMD, since he would have to do the same for a harmony syntax, so his lib could be used in cases where ES loading and old style global loading is still in play.

Some feedback on his specific reasons:

Folks requesting other module formats for other loaders.

No other module format comes close to the level of support AMD has: AMD has multiple implementations, better support in other libraries (Dojo, jQuery, libraries friendly to Ender-bundling, MooTools), it is used in real sites, and has a thriving amd-implement list and thriving implementations. What else can claim all of those things? What else is being asked for inclusion?

I can appreciate it is easy for someone to come up with a loader syntax and want people to use it. I think AMD has gone the extra steps though to be considered more of a standard. But it depends on what you want to use as for standards of legitimacy.

If any library that depends on Underscore (and there are many of them: http://search.npmjs.org/) does not yet 'support' AMD, but Underscore does, things get royally screwed up.

npm is for node modules, not browser modules, so they would not have the error that was being reported for Underscore.

That said, I do believe Underscore is a common dependency for browser-based code. But for the browser, as mentioned, registering a global is perfectly acceptable for this transition period, something I believe will need to happen anyway no matter what modular format is chosen. There will always be a transition period.

In an ideal situation, libraries do not have to be modified to support a particular script loader (or group of loaders).

The point is to make developer's lives simpler. A script loader like that does nothing to help order the dependency tree correctly, or use a naming convention that can be parsed easily by tools like optimizers.

The browser globals with implicit dependencies just do not scale well past maybe 15 dependencies? Not everything fits as "lets concat all the scripts" Rails app. But I'm open to seeing a design that might scale up.

JavaScript's upcoming native module support is entirely incompatible with AMD.

They are very compatible as far as base semantics. As mentioned above, ES harmony modules are still baking, not ready for prime time. But even with that, it would be very easy to convert AMD code to harmony module code -- since the dependencies are all string names it fits in. Loader plugins would be a problem for sure, but basic modules are easy.

It is much more compatible than the current "browser globals and implicit dependencies" approach.

Loading individual modules piecemeal is a terrifically inefficient way to built a website. Because of this, there's the great RequireJS optimizer, which will turn your modules into ordinary packages.

The browser globals approach already depends on build tools, so I'm not sure why this is a knock against using AMD. A bunch of manually typed HTML script tags perform just as poorly.

Fortunately, since dependencies can be easily statically determined with AMD calls, the developer no longer has to worry about manually figuring out the load order, and there can be (and are!) many different build tools built on top of the standardized AMD API.

Summary

In the end, though I just think it boils down to Jeremy not needing this personally based on the scope of his work, and he has other things he would rather work on. It is hard to get a standard of legitimacy in this area, so it is easier to just wait it out. For the particular issue in Underscore, the simple export of a global even in the AMD case would have solved the issue, but such is life.

I'm a bit sad because Backbone and RequireJS have been a very popular combination. They fit very well together. The thought of maintaining a fork/branch is distasteful, particularly since the AMD patch for Backbone was smaller than the code it replaced.

Auto-wrapping tools are difficult to do generically given how scripts want to grab globals. The dependency name can have weirder names that do not match the file name, so it loses the nice, simple dependency parsing of AMD module IDs. It means creating a centralized list of global names that map to IDs/paths. Not very webby/distributed. Not very simple.

Oh well. It probably means AMD just needs a bit more time out in the wild, even more adoption, and we'll see how people feel in 6 months.

Another Approach?

I have heard complaints from folks on the internet about AMD from time to time, but they have not offered anything better, particularly given the simplicity tradeoffs mentioned above. I think it is just an NIH thing most of the time, or getting it mixed up with generic browser script loader.

Here is a survey of things I know about:

Ender is fine if you want to just build a file that you will not change often and use it in place of jQuery, but it really does not help with the larger site structure and loading issues, being able to dynamically load. It is not a general module system for the web. It basically is just like the CommonJS "do a build before starting development" approach. Same as SproutCore/Ember. Same complexity problems as mentioned above, and complicates individual module debugging.

Dan Webb started something with loadrunner. It supports part of AMD, and the "native format" it supported do not seem better than AMD, maybe just another function nesting and different API names.

Ext has something similar to AMD, but mixes in a particular class declaration syntax into the format, so that will not fly as a general solution.

YUI is close, but obscuring what a dependency name loads on a Y instance makes it hard to associate what came from what module, and the API to add a module relies on naming the module and putting named fields on the Y instance. This will lead to name collisions.

What else am I missing?

One thing these approaches all have in common: they get away from the browser globals and implicit dependency approach used by traditional browser scripts. If you think something using that traditional pattern is the future, please describe how it might work. Remember, requiring build step to just start developing is a complication. That does not scale well across all the types of JS development mentioned above.

While Jeremy and I were talking in the documentcloud room, he mentioned that Ryan Dahl, if he could do it over again, would prefer not to use CommonJS modules system for Node, but do something closer to how browsers load scripts in script tags.

I think I have heard that comment too, but in the context I heard it, it seemed like an off-hand comment. I'm not sure how much that was just about having to wade through CommonJS discussions or actual problems with the module API. I would love to hear more about it though. Ryan or someone how has more info on this, if you happen to see this post, please clue me in. I'm on github, the amd-implement list, or on gmail as jrburke.

I want to get to a workable, simple solution that works well in web browser too. I only do AMD because I need it and I think it is the simplest overall path for end developers. But I want to solve other higher level problems. I want something to good to win. It does not have to be AMD, but I do think it hits the right simplicity goals particularly given browser use. It will not be brain-dead simple because upgrading the web is not simple. But it does a pretty good job considering.

I feel like the folks doing AMD have done their due diligence on the matter. ES harmony may be able to do something a bit different, but the basic mechanisms will be the same: get a handle on a piece of code via a string ID and export a value. The rest starts to look like arguing paint colors.

Anyway, enough of that. Back to making things to make things better.

RequireJS 1.0.4 released

RequireJS 1.0.4 is available.

The big fix for this release is making sure the config options are passed correctly to uglifyjs. There was a problem with 1.0.3 that would result in some files not being parsed correctly.

The small set of bug fixes for:

• require.js
• r.js optimizer

volo: A JS tool for JS projects

Over the last month I have been playing with a tool to help me make JS-based projects. It is still very early in its development, but it can do a trick or two now, so it might be fun for you to try out too.

volo is a command line tool. It is written as a set of AMD modules that are built into one JS file that runs in node.

Check out the README for more info, but what I like about this project:

• Just one JS file to install.
• It uses GitHub as the code registry.
• You can create new commands for it (using volo) and if you put your commands on GitHub it is easy for others install them.

Still lots to do and there are lots of sharp edges to it, but it already feels pretty sweet. If you would like to help out, check out the design goals and working the code. Also feel free to give some feedback on some of the more important issues I would like to sort out:

• Windows support
• "Short name" dependency lookups, possibly using some sort of GitHub search API.

On a side note: I have been on vacation and not checking email. If you are waiting on a reply from me, I should get to it this week.