RequireJS 2.0.6 released

RequireJS 2.0.6 is available.

The main focus of this release was cleaning up some rough edges in the r.js optimizer after switching to esprima for all module parsing/tracing. Most notably, the findNestedDependencies build option should work correctly again. The bundled UglifyJS was updated to 1.3.3 too.

Complete list of fixes:

• Fixed require.js issues
• Fixed r.js optimizer issues

On client components for web apps

This is a response to a blog post by TJ Holowaychuk about browser-based components for web applications, and Isaac's notes on TJ's post.

I am going to try to make this brief because I get tired of people in the Node community wanting to apply the same patterns from Node in the browser. I feel like I say these things on a periodic basis, but human communication is hard, and I certainly could do better. But I also want to get back to just making things. So this will be terser than I normally would like.

Web components

I suggest TJ look at volo, my attempt in this space. It does lots of what he describes already, and it can even be used as a module in another command line tool. We use volo for some things in Mozilla already.

volo uses GitHub as the component registry. It does so without the downsides that TJ mentions.

Specifically, volo uses the GitHub HTTP API to get version tags, do registry searches. I grabs .zip snapshots for a given version/github commit/branch, so the command line tool (the consumer) does not need use git. Git is not necessary on the client side.

This means the downloaded code is smaller -- no need to get a full repo and all of its commits.

volo also understands dependencies via the shorter "owner/repo/tag" IDs instead of the full github URLs.

It has a "shim" repo that means it can support installing components without needing the author of the component to conform to some new publishing system. Since it allows {version} replacement in URLs, the registry setup just needs to be done once. From then on, normal best-practice versioning via git tags is enough.

Some other notes in this post.

Base module format

Node bros, the AMD trolling is getting tiresome. Node's module system is woefully under-specified for web-based loading. While you can limp along with browserify, there are still these issues:

* For builds you need a wrapped format. For CDN deployment you need a wrapped format. browserify uses a wrapped format. AMD anyone? For that reason alone, AMD will never go away. Get used to it already.
* Web code needs a callback-style require for on-demand loading.
* Browserify's uses of file paths for IDs is awful for mixed local and CDN-based loading. Module IDs need to stay in ID format, not translated to a specific file path.
* Loader plugins reduce the need for callback-style APIs, and callback pyramid of doom, or inside-out callback hell, or the need for promise based programs. This more than makes up for the extra level of indent in AMD.

Loader plugins solve the translation issues TJ talks about, and they can participate in optimization builds, meaning templating engines can inline the JS function form of the template. Ditto for language transpilers like CoffeeScript.

By doing this:

define(function (require) {
    //node module code in here.
    //Return module value instead
    //of needing `module.exports`
});

you have an AMD module.

Quit dismissing AMD for surface issues. AMD avoids mandating translation layers that lead to more things for the developer to understand and fix, and more process for the user to go through to deploy code. It is a net win when the source file works when deployed anywhere, without requiring specialized builds/converters.

Even if you want to personally use Node style and always do builds before loading in the browser, AMD is a great target for the built, wrapped format. You can even use the requirejs optimizer to do this, with the cjsTranslate option.

The universal module boilerplate gets simpler when Node supports AMD's define along with Node's existing module format. If you want to help improve the ugliness, start there.

AMD comes from real world exprience in Dojo with trying to deploy an unwrapped module format that depended on XHR+eval in dev and a wrapped format for builds. Yes, you can something to to work but the second order translation and support costs are not worth it. Some environments disallow eval. CORS configuration is awkward, and potentially hazardous if your API is on the same domain and CORS is done incorrectly.

The simplicity of the complete module lifecycle is worth the function wrapping. Quit looking just at what you type once, and consider the complete code lifecycle, and how much time could be wasted there.

npm's registry as the component registry

The implied rules with npm and node's module behavior are not good fits for front end web development:

• Forcing a directory structure is complicating project layout and loading for web-based projects. It should be possible to publish and install single JS libraries as single files. volo can do this.
• Related: the "index.js" convention is awful for web development and debugging. Debugging 'jquery.js' instead of trying to find 'jquery/index.js' in the web tools? No thank you.
• npm's registry namespace is already polluted. Check searches for 'jquery'. Maybe that just means having a separate npm registry-based registry for client code. But if there needs to be a separate repo, might as well use one that can adapt better to front end development. Like single JS/CSS file installs without extra Java-esque directory structures and metadata debris on the file system.

Ender is not a success story for using npm. Ask the Ender folks how well that worked out. It only works because it is small scale.

By using GitHub, it comes with user auth handled, private repos, and robust social tools that will not be matched by a something like npm because the financial incentives are not there. Plus developers already use it. For simple open source sharing, make it easy without introducing more things in the middle.

Github as the registry is not perfect, and we still need some standalone servers that can be run inside corporations/for mirrors, but I would model those standalone servers on the github API. At least the default case of a public repo can be bootstrapped very quickly.

RequireJS 2.0.4 released

 RequireJS 2.0.4 is available.

 

And hot on the heels of 2.0.3, there is a 2.0.4 release! Unfortunately a bug with a fix that was in 2.0.3 caused a different bug, and it was not found until after the 2.0.3 release. It made the optimizer unusable in certain situations, so the change was rolled back and 2.0.4 is out, with the only change over 2.0.3 being the rollback of that previous change.


I apologize for the extra noise.

RequireJS 2.0.3 released

 RequireJS 2.0.3 is available.

Just a maintenance bug fix release. Most notable changes are probably:

• optimizer now does not fully resolve "paths" until all config sources (mainConfigFile, build profile and comand line args) have been merged.
• a fix for data-main resolution for require.js.

Full list of fixed bugs:

• Fixed require.js issues
• Fixed r.js optimizer issues

ES Modules: suggestions for improvement

There has been a recent bout of comments about ECMAScript (ES) harmony modules on twitter and elsewhere. Here is my attempt to explain parts of it, some of the design tradeoffs, and perhaps a middle ground that would open up some options that may bridge some gaps.

Modules are one of those things that seem very simple, but involve quite a lot of decisions and tradeoffs. This post is mostly just about module linking and module ID resolution, and even with that, it is quite long.

If ES Modules do not come up with different ways to work (or maybe explain where I have it wrong), they are not competing well with what can be done with a combination of CommonJS/Node and AMD.

My background: I work on RequireJS and AMD.

What is it

First, some links to the specs. The "harmony" moniker means it is in process for the next version of the ECMAScript (JavaScript) language:

• Harmony Modules: The basic module spec.
• Harmony Module Loaders: Specifies how you can create isolated loaders for modules. Very useful.

The module examples page is suggested if you want to get a feel for it, but it is good to read the other docs too. It can be a bit daunting though, unless you speak the spec language.

Points of reference

One way to evaluate how ES Modules works is to compare it to something you may already know:

• CommonJS / Node. Node implements a version of the CommonJS module API.
• AMD / RequireJS. RequireJS implements the AMD module API.

Run time vs compile time

ES is a "compile time" approach where the formats mentioned above are "run time" approaches. Maybe not precise terms, but here is a definition of what is meant by those terms for the purposes of this post:

"Compile time" means:

• JS text is parsed, and the "module" "import", and "export" syntax is found.
• Any dependencies are fetched and parsed.
• Once the dependency tree has been all fetched, the ES module loader will wire up the exports from a dependency to a module's "module" or "import" use, and do type checking on that export type and how it is referenced in the module
• The module code is then evaluated/executed.

"Run time" means: there is usually no pre-parse stage. The JS text is evaluated, and any module API that is encountered is run as it is encountered.

AMD will actually do a small parse step if the module looks like:

define(function (require) {
    var a = require('a');
});

In that case, it will parse the function to look for require() dependencies, and then load and execute the dependencies first before running the function above.

"Compile time" was chosen for ES because:

• it is familiar from other scripting languages
• sets the way for other possible static features, like macros
• ensures that "import *" are static, *not* dynamic bindings
• allows some type checking on the values that are explicitly "export"ed.
• generally seen as safer and easier to reason about that run time.

The CommonJS/Node style of pure runtime, no parse, was hard to get to work with some edge cases as I understand it, but I heard that second-hand, I did not see that discussion.

Impact of compile time

For compile time to work well, it should use new keywords in the language, to have clear markers on what is participating in the module system.

Although, it could work with a module API instead of new syntax, by only recognizing literal use of that API, and do not support variable assignment of the API or dependencies to other names. This is what AMD does for the "sugared CommonJS form" mentioned above.

For "import *", static binding is critical because anything that is a runtime scope lookup gets into "with" territory, and "with" has been seen as a mistake by the committee. ES5's 'use strict' bars its use.

Since new syntax is involved, ES Modules cannot be "shimmed" into existing JS libraries. There is a Module Loader "runtime" registration call that can used for a module to register its, but it means those libraries cannot participate in the compile time linking stage, so they need to be pre-loaded by a script loader before an ES Module can effectively reference it with module syntax.

Module ID resolution

One other consideration, one that is usually overlooked when talking about modules, is how a module ID like "jquery" is resolved to a file path and loaded.

Both CommonJS/Node and AMD/RequireJS support "short, logical names" for dependencies. So, you can say require('jquery') and that jquery gets converted to a path using some algorithm. Node uses multiple paths to find jquery.js, and AMD in the browser relies on a declarative configuration to do so.

ES modules do not really have support for this, unless you also implement an imperative resolver. They support full URLs, like:

module foo at 'http://example.com/scripts/foo.js'

but we have found in AMD that it is useful to be able to say require('jquery'), but then declaratively map that to zepto.js. 

So, an individual module specifies a dependency on an API provider, but how that provider is satisfied is resolved using the declarative configuration.

If there is only an imperative resolve API, no simple declarative API to resolve short names, it will mean shipping a userland "loader library" to effectively use modules. This opens the door to balkanization in module ID resolution since there is not built in support.

Special factors in JavaScript

There are a few special factors with JavaScript that are not usually in other programming languages, and they have an impact on the design:

• The largest deployed use case of JavaScript, the browser, is async, network IO. File size and number of requests are very important to performance. So combining modules together into one file, and minifying/transforming the source for smaller delivery is common.
• There is a large legacy codebase of browser-based JavaScript that just use browser globals, and no real module format. Some small uses of JavaScript do not need modules, and browsers will support those use cases indefinitely.

My goals

I want AMD and RequireJS to go away.

They solve a real problem, but ideally the language and runtime should have similar capabilities built in.

Native support should be able to cover the 80% case of RequireJS usage, to the point that no userland "module loader" library should be needed for those use cases, at least in the browser.

If the ES module format requires a web developer to use a script loader to use existing, non-AMD/CommonJS, non-ES JS in a project for those 80% use cases, it is a failure.

Example: If I cannot use jquery and backbone in an ES 6 module without needing another library to preload or prep those libraries for ES 6 module use, then existing JS users will not see much advantage over using AMD.

If the web developer needs to code any imperative logic to wire up the ES Module Loader, that will result in a loader library. That is a failure condition.

As compared to AMD: if the ES approach cannot do the above without a helper loader library and the ES approach does not allow something like loader plugins, then there is no contest -- AMD will still be more useful to a developer than the built in system. Small savings in the amount to type and a thin layer of type checking is not enough.

This may very well not be the goal of ES modules, and it would be great if the specs or some background material acknowledge that, and list out the mitigation strategies developers are expected to use.

Shortcomings of ES modules

Right now, ES harmony modules do not improve an AMD user's workflow because of the following:

New syntax makes it very hard to optionally upgrade

If I am the author of something like jQuery or Backbone, I cannot optionally add in a way to register as an ES module because ES modules use new syntax. However, there are many uses of those libraries which will not be in ES module-capable browsers.

The Node and AMD communities have found optional opt-in via a runtime API very useful for adoption of code that works with their module systems, but still work in older "use plain script tags with browser globals" approach.

There is a runtime API in the ES module loader proposal that would allow a legacy script to register something as a module, but that requires the end developer to use another script loader library to load that library so it can do that runtime call, then start loading ES module code.

The developer may as well just stick with AMD. Complexity has not been reduced.

Register module and a global

Backbone originally had trouble adopting AMD because if it called define() to register a module, they found other libraries, like Backbone plugins, would break. The plugins were expecting to find a Backbone global variable but when Backbone called define() it was not also exporting a global.

This same problem will exist in ES-mixed code. Any dynamic registration also needs to allow an export of a global so that downstream libraries will work until they are also converted to optional module registration.

There should be a migration path, one that allows gradual rollout of modules without requiring a project to go whole hog on module syntax.

Declarative module ID resolution

While I have made tools to allow a developer to "convert" an existing library to AMD, there are many developers that did not want to touch existing libraries. It makes it difficult to compare against new versions and there is a concern that the conversion introduces breaking scope changes (rightly so).

So the "shim" configuration for requirejs was introduced to allow specifying dependencies and an export value for JS code that does not call a module API. This has been well received in the community. More background on shim here.

"shim" with "paths" and "map" make it possible to declaratively set up a configuration that allows for one file IO lookup per module ID, an easy way to "shim" old libraries, and to load more than one version of a module for use by different modules. That covers the 80% case for using old and new code with a module loader.

By using a declarative configuration that is supported by the "default" module ID resolution mechanism in the language, then it avoids having to ship a userland loader library for browser use. This is a big win because it will help kill AMD.

It is fine if the Module Loaders API still has an imperative API to set up different  module ID resolution logic. That would allow Node to maintain its current multiple IO, nested directory lookup logic. However, the default should favor the harsher browser environment in such a way that an extra loader library is not needed.

Loader plugins

I can appreciate that supporting Loader plugins may seem out of scope for the default module loader, but they have been incredibly useful for AMD. Node has seen a use for them, even though they are done in a different way via require.extensions. They effectively allow use of transpiled languages.

I find the AMD loader plugins better than Node's approach because:

• load behavior vs. file format: since a prefix is used on the resource ID instead of just using a file extension suffix, it allows multiple plugins to deal with the same type of file extension. For example, "text!index.html" and "template!index.html" can be used in the same app, the first one just giving the raw text, the second one "compiling" some text for use as a template. The developer, not the plugin provider, chooses the right use. It still allows "single extension" plugins too, and for those, they can omit the file extension in the ID, so no increase in ID length.
• one IO lookup: For a resource ID "foo", node may do a lookup for "foo.js", "foo.coffee" and "foo.node". By specifying the loading mechanism via the prefix, it avoids multiple IO lookups, which are important for browser use. It also makes it clear what handles the loading.

AMD loader plugins can participate in build steps, so the "text!" plugin can inline the text as module in a built file:

define('text!index.txt', function() {
   return 'hello world';
});

That is incredibly useful for getting good network performance in the browser.

Even for local file environments like Node, being able to combine all the assets for a program into one file is really great for distribution. It is conceptually simpler to reason about tracking one file vs. "nested directory of directory" installs. Think of it as a way to easily share shell scripts.

The middle way

For developer workflow, right now AMD is a better alternative than ES harmony modules, given the choices around compile time linking, new syntax, and the use of imperative ID resolution.

Here are some suggestions on how to allow some of the benefits of the compile time approach with the run time ones used by AMD. The goals are reuse non-module code in modular systems, allow for a way to get some static version of import *, and perhaps even macros.

Fetch dependencies, execute, modify, execute

The core of the middle way for compile time vs run time:

• do not force compile time operations to be all up front, before any evaluation.
• evaluate dependencies before executing the current module.
• provide an API for modules, not just new syntax

These are effectively what AMD does today, except it does not have a way to alter the AST before final execution. Well, an AMD loader could do that, but AMD loaders have traditionally avoided it. However, the harmony loader plugin I made effectively does this to support "import *". More below:

The ES module loader would operate like so:

• Load JS text. Parse out dependency references.
• Load dependencies, parse out their dependencies, load them, etc...
• Before executing a given module, execute its dependencies, and wait for the dependencies to finish exporting their module values.
• Take that exported value and if there is an "import *" in the current module, modify the AST of the current module such that it gets a locally bound variable to any of the hasOwnProperties of the dependency that are known at that time. So, any properties added to the module after this point are not visible. This should avoid concerns about dynamic scope.
• Once the module AST has been fixed up for any import *, then evaluate it.

When parsing out dependency references, look for any new keywords, but also any API that corresponds to that keyword. So, look for at('moduleID') for dependency references in addition to at 'moduleID'.

The runtime API for the module should be something like at('moduleID') for dependencies and exports.propertyName for specifying export properties. I am not arguing for that specific API, just mentioning that there would be an API alternative to the new syntax. The API alternative does not need an import alternative though.

Since a module is executed before giving the exports to a module that depends on it, and since there is a runtime API for modules, then that allows existing JS code to opt-in to ES modules without getting bitten by new syntax.

Since all dependencies are executed before executing the current module, an "import *" can be supported, and I believe that would allow for macros later.

There are some limitations around circular dependencies, but they are still possible, and the restrictions are minor in comparison to allowing existing code to opt in to ES modules and still work in non-ES module environments.

Declarative configuration

Support something like the "paths", "map" and "shim" config as used in RequireJS. This allows easier use of old code, and scales up to very large code without requiring a developer to ship a library that sets up an imperative resolution API.

Support loader plugins

Now that all dependencies are executed before executing the current module, then it is easier to support loader plugins, as the loader will have the exported value for that plugin resource before running the current module.

These environment-based loading, like an "env!" plugin that can load a module for Node and a different API-compatible one for the browser. See also a "has!" plugin for feature detection-based loading, and plugins to enable transpilers.

Yes, it is more to sort out, but they provide a lot of benefit. AMD has already primed this pump. It even works with a build/optimization step for inlining resources.

Use string IDs for module identifiers

This allows the module references in dependencies to be the same as the ID that is inlined when modules are combined and named in built files. Right now it is weird to use a JS identifier, like module Foo {} to name a module, but then see module Foo at "Foo". It is hard to match up at "Foo" with module Foo.

The extreme positions

The following is based on my limited experience. I am not a language designer. I am but a simple plumber that uses the pipes that available to build things. I may not have the right long term thinking involved, but I think the following make the ES module proposal simpler.

To be clear though, I think the middle way above is enough to bridge the gap. Please, do not read the following and then discount the middle way. The middle way is separate from these more extreme measures.

No new syntax

If there is a runtime API available to allow existing code to opt in, just shed the new syntax. Just have one way to do it via API that can also then be shimmed.

no import *

import * makes it difficult to determine where code comes from. If
this type of construct is allowed:


module foo {
   var sin = function () {};


   module bar {
       import * from "Math";
       sin();
   }
}

for a minifier, it now needs access to Math to do its work correctly. This has not been the case in the past. It would suck to need all of the code for all the modules used in a system just to complete a minifier pass.

For developers, if you have two modules that do import * it can be difficult to know where something comes from.

Destructuring provides enough benefit for these use cases, just do the
comma separated list for things you really use:

   import {sin, cos} from "Math";

import * is a bad pattern and it does not save much.

If you get rid of import * then with the "middle way" of evaluating modules, then regular var/let-based destructuring is enough, there is no need for an import keyword.

No macros

Similarly, rethink the need for macros long term. They suffer from the same "where did this come from" problem as import * does. The function capabilities in JavaScript are good enough to get the job done for the "don't repeat yourself" task.

A way forward for today's code

The nice thing is that we can prototype this new world by combining what CommonJS/Node does today with AMD. So we can just use the require() and define() as used today to get there. The ES committee does not have to ratify it, and we get the benefit of real world implementation and use before committing to default language support.

Cajon is my attempt from the AMD side to bridge the gap with plain Node code and a runtime browser loader. LinkedIn's Inject is another AMD loader that uses a similar approach. So, just use CommonJS/Node modules in the browser in dev, use the r.js optimizer to compile down to AMD for final deployment.

The cjsTranslate capability in the r.js optimizer allows a developer that always likes to do builds, even in dev, can code in Node syntax but output to AMD and load it in the browser either by the small Almond AMD shim, or the full dynamic loader via RequireJS. Or choose Dojo or curl.js.

browserify can be updated to use AMD as its transport format instead of its home-grown require.define() API, and then not have to ship a loader, but use one of the AMD loaders/API shims. browserify is nice in that, unlike the r.js optimizer+cjsTranslate, it provides browser module shims for the native node modules. It would be great to break those out as a separate project that could be consumed by a project just using the r.js optimizer.

If Node adds define() support, callback-require for use within a module for dynamically calculated dependencies, and supports at least a limited form of loader plugins, then we're done. The amdefine project is an implementation proof of that support. There are details to sort out, but it is doable. Any node committers are interested, give me holler. We can work out the details.

Summary

For developer workflow, the current ES module spec is not competing well with a combination of CommonJS/Node and AMD with loader plugins. Or even just AMD with loader plugins.

Using the middle way for module execution and getting a good declarative module ID to path configuration in the ES spec will level the playing field. Add loader plugins to get language transpiler support and environment/feature detection loading that is efficient for the browser.

I have given some of this feedback to the es-discuss list, but I think some of it, in particular the "middle way" module evaluation flow, got lost in my poor communication where it seemed like I was proposing a dynamically scoped import *. Hopefully this post clarifies what I was trying to achieve with that earlier feedback.

Finally, I appreciate working on the ES committee is very difficult work. I do not envy them. I do not mean for this feedback to come across harshly, but the committee is running out of time, and I do not feel like it has made the case very well for how what is being proposed is better than what we have cobbled together with existing technology. To be clear, I want an ES Modules proposal to succeed because I do not want to do AMD or RequireJS forever. Hopefully this feedback can be viewed as loyal opposition, and as a challenge to do better, or at least to do it in a way that is explained more clearly.

Cajon: a browser module loader for CommonJS/node/AMD modules

I just released the first version of Cajon. From the README:

Cajon is a JavaScript module loader for the browser that can load CommonJS/node and AMD modules. It is built on top of RequireJS.

You can use it to code modules for your project in CommonJS/node style, then use the RequireJS Optimizer to build all the modules into an AMD-compliant bundle. This allows you to then use a small AMD API shim, like almond, to get nicely optimized code without needing a full runtime loader.

See the README for more information and restrictions.

Why do this?

It is an experiment, to see what people like to build with. The attributes of AMD are needed for any comprehensive JavaScript module solution, but some people really like sugar, and the sugared form of AMD may not be enough for them. They also may want to use existing CommonJS/node modules as-is but still want to get a good, optimized/built story, and something that works well cross-domain in the browser.

This experiment goes along with the latest requirejs 2.0.2 optimizer setting, cjsTranslate, which will automatically convert CommonJS/node modules to have a define() wrapper, so they can be consumed by the optimizer. This would allow you, for example, to build a node command that watched your js lib folder as you did changes to modules in the CommonJS/node format, and build them into an optimized AMD bundle.

End result, if you cannot bring yourself to use AMD:

If you do not want to do builds during your CommonJS/node-based module development, use Cajon. If you like doing builds, you can now use the RequireJS optimizer (with the almond AMD shim) to do that.

To be clear: CommonJS/node modules as-is are not enough for a comprehensive JS module solution. These tools allow you to use them though and fill in the gaps by "compiling down" the code to AMD.

RequireJS 2.0.2 released

 RequireJS 2.0.2 is available.

More bug fixes, thanks to the community for really putting it through its paces.

Notable change for the optimizer: The "dir" output directory is now deleted before each build run, if it exists. This is done to avoid problems with transforms via onBuildRead/onBuildWrite that made it difficult to do incremental builds. If you want to keep the "dir" directory between optimizer runs, then set "keepBuildDir" to true.

Complete list of changes:

• Fixed require.js issues
• Fixed r.js optimizer issues

RequireJS 2.0.1 released

 RequireJS 2.0.1 is available.

A few fixes to clean up some edges from the 2.0 release. If you are using 2.0.0, you should upgrade to 2.0.1 right away. Changes:

• Fixed require.js issues
• Fixed r.js optimizer issues

RequireJS 1.0.7 released

RequireJS 1.0.7 is available.

The fixes are almost all optimizer fixes. There was a fix for a bad source overwriting bug when "appDir" and "mainConfigFile" were used together when mainConfigFile had a paths config.

Release Notes

Template for responsive mobile, offline web apps

It can be difficult to create a web app that is responsive to different screen resolutions and works well in mobile environments or environments with spotty network availability.

In order to make it easier, Jason Strimpel and I created a project template, create-responsive-template, that takes care of a few things for you:

• Uses Twitter Bootstrap for a baseline style and responsive UI
• A couple of small JS helpers for AppCache and network events
• A build step that optimizes JS and CSS and generates the AppCache manifest for you

A bonus: the tooling is in JavaScript, run in Node. Some background on that choice.

Here is a video that shows how you can use the template:



The next steps I would like to see for this project template:

• An IndexedDB data layer: A JS helper library that looks like the IndexedDB API, but will use an API shim over Web SQL if IndexedDB is not available. The shim does not have to implement 100% of the IndexedDB API, but something that allows a good start to local data storage. I am looking at Lawnchair at the moment, but open to other ideas.
• An easy way to generate app store manifests for Mozilla Web Apps and Google Chrome Store.

If you have any thoughts on how best to accomplish those steps, open an issue with details, or leave a comment here on the blog post.

RequireJS 1.0.1 released

RequireJS 1.0.1 is available for download.

This just has three small bug fixes in it (one in require.js, two in r.js). It was prompted by the release of jQuery 1.7 and wanting to make sure people could use the "namespace" optimizer option with it, now that jQuery 1.7 registers as an AMD module.

The bug fixes were related to:

• allowing full URLs for simplified CommonJS wrapped modules
• AST parsing of dependencies for modules that use a variable for the factory function
• catching more cases that should have the "namespace" optimizer option applied

Detailed list of changes for require.js and the r.js optimizer:

• require.js 1.0.1 changes
• r.js 1.0.1 changes

RequireJS 1.0 released

Hell yeah. Get it now.

This release is basically the same as the 0.27.1 release. There was just one change, to a regexp used by the optimizer when it converts CommonJS modules to AMD modules via its -convert flag.

This release has been two years in the making. The code and APIs have changed over time based on feedback from the community, and now there are some very strong AMD loaders besides RequireJS. The AMD API and RequireJS have been proven useful and used in real projects.

I created a new Why AMD document if you want to learn more about why the AMD API exists and how you can use it.

I still plan on doing RequireJS releases and pushing modular JavaScript forward. This is not the end, but it is an important milestone. Thank you to the RequireJS and AMD communities for making this release.

Go use it already!

• Use it to build apps!
• Upgrade your library code to optionally call define()!
• Implement AMD in your favorite JS environment!

Let's make modular JS a reality on the web today.

RequireJS 0.27.1 released

RequireJS 0.27.1 is available for download.

Just a small update with some bug fixes for 0.27.0. See the 0.27.0 release for the major functionality changes over previous releases.

I will wait about a week to see if this release seems stable for a 1.0 release.

Fixes:

• define(id, function () {}) where the function has require('') dependencies will now be scanned for dependencies. Allows for smaller universal module adapters.
• Loader plugin that depends on a different plugin's loaded resource works as it did in 0.26.0.
• Optimizer: update UglifyJS to 1.1.
• Optimizer: semicolons are inserted between files if concatenating would cause errors.
• Optimizer: always strip BOM files on all platforms when file transforms/concatenations are done.
• Optimizer: allow override of modules used in optimizer. Example.
• Optimizer: allow copying of .directories via build config option.
• Optimizer: Resolving paths for .js dependencies might fail if an appDir was not part of the config.

RequireJS 0.27.0 released, 1.0 release candidate

RequireJS 0.27.0 is available for download.

This is the 1.0 release candidate. Unless things go horribly awry, I expect to do the 1.0 release in about a week. So please be sure to try it out soon if you have concerns. If you need more time to evaluate it, let me know.

From the release notes:

• require.ready() has been removed. In its place, use the domReady plugin. This allows better interoperability with other AMD loaders and better separation of concerns.
• A new wrap config option for the optimizer is available, for wrapping built code in a function. Allows for better API hiding and tiny builds with the almond API shim.
• The order plugin is improved for IE.
• Loader plugins can now have dependencies and they will work in the optimizer, as long as the dependencies work in the optimizer environment (Node, Rhino).
• The namespace config option for the optimizer is more robust.
• Removed require.def(), use define() instead.
• Removed module.setExports, use module.exports instead.

almond: a small AMD shim loader

I just put up release 0.0.1 of almond, a tiny shim loader for the AMD JavaScript module API. It can be used as a replacement for RequireJS after an optimized build. It is great for standalone libs or tiny apps that want a very minimal footprint with all the benefits of AMD.

From the README:

Some developers like to use the AMD API to code modular JavaScript, but after doing an optimized build, they do not want to include a full AMD loader like RequireJS, since they do not need all that functionality. Some use cases, like mobile, are very sensitive to file sizes.

By including almond in the built file, there is no need for RequireJS. almond is 948 bytes when minified with Closure Compiler and gzipped.

Since it can support certain types of loader plugin-optimized resources, it is a great fit for a library that wants to use text templates or CoffeeScript as part of their project, but get a tiny download in one file after using the RequireJS Optimizer.

If you are building a library, the wrap=true support in the RequireJS optimizer will wrap the optimized file in a closure, so the define/require AMD API does not escape the file. Users of your optimized file will only see the global API you decide to export, not the AMD API. See the usage section below for more details.

So, you get great code cleanliness with AMD and the use of powerful loader plugins in a tiny wrapper that makes it easy for others to use your code even if they do not use AMD.

RequireJS 0.26.0 released, npm install requirejs

RequireJS 0.26.0 is available for download.

The big feature is being able to npm install requirejs to allow require("requirejs"). This allows you to:

1) Load AMD modules inside node without running a bootstrap script. It also fixes some path issues using traditional Node modules alongside AMD modules. So, now you can use requirejs inside Node to load AMD modules and use loader plugins:

var requirejs = require("requirejs");



requirejs.config({

    nodeRequire: require

});



requirejs(["module/a", "module/b", "text!templates/one.html"],

 function (a,          b,          template) {

     //use a and b with the text template

 });

2) Exposes the optimizer as require("requirejs").optimize() to allow dynamic server builds for people who like to do "only one script tag before end of body tag" development. With the "excludeShallow" optimizer config, you can still debug a single module/script while having the rest combined into one script.

Pretty sweet -- using JavaScript to run a server to optimize JavaScript on the fly but still get fine-grained JavaScript debugging. JavaScript turtles all the way down (until you hit the C turtle).

You can still use the r.js script to do command-line build optimizations. If you npm install -g requirejs, then you can use r.js as an executable (the requirejs package replaces the old requirejs-r package).

More information on the Use with Node page.

The requirejs npm package includes require.js for the browser. So, for web pages, load node_modules/requirejs/require.js to use RequireJS in the browser. As always, it is easy to get require.js for the browser without using npm.

Also notable in the release: UglifyJS in the minifier is updated to 1.0.6. The upside: has() branch trimming now works with the default minifier.

Other items from the release notes:

• Fixes for running under Node on Windows using the native node.exe builds that are available in the Node 0.5.x series. Now there is less of a need to use Java to drive the RequireJS Optimizer!
• Configuration is now done via a require.config({}) call, to get in line with the amdjs require API. The old require({}) method works on the global require() for backwards compatibility, but the suggested API going forward is require.config({}). The API doc has been updated to show proper usage.
• There is a namespace option now for builds, to allow moving require() and define() calls under a different namespace. This allows you to build an optimized file that uses RequireJS but does not interfere with any other AMD loader on the page, and you can make sure only your modules are loaded in that namespaced object.
• The default error behavior when a define() factory function throws an error is to not catch it. The catching done in 0.25.0 made it more difficult to debug. However, there are some situations where catching the errors is preferred. Setting the config value catchError.define = true will switch to catching the errors and allow processing via require.onError()
• Closure Compiler in the optimizer was updated. As a result, the code to invoke Closure Compiler changed, and will likely only work with the latest Closure Compiler release. You can grab a version known to work with the optimizer in the optimizer's lib/closure directory.
• There is now a pragmasOnSave build option, which is used in the require-cs CoffeeScript loader plugin build profile to strip out the CoffeeScript compiler after a build. The end result: tiny build layers of converted CoffeeScript code.

RequireJS node adapter now in npm

[Update August 17, 2011: This post is out of date, see the 0.26.0 release for a better way to use RequireJS with npm.]

[Update July 21, 2011: GitHub users arlolra and JasonGiedymin clued me in to setting up r.js as a bin file that can be installed globally, making it much easier to use. The instructions below have been updated to reflect the changes.]

As asked for on the requirejs list, the RequireJS Node adapter + optimizer is available via npm now:

npm install -g requirejs-r

This will install the r.js file as a global bin/executable that is available for any of your Node projects.

To run main.js via RequireJS in Node:

r.js main.js

To optimize a web project using RequireJS, assuming app.build.js contains your optimization profile:

r.js -o app.build.js

"requirejs-r" is a bit of a funny package name, but it comes out of the requirements to use a directory for publishing to NPM, and the r.js name. If it seems useful, I may publish other RequireJS-friendly loader plugins under the "requirejs-" prefix. I may even push require.js to npm, but I am not sure it makes sense yet. If you think it would be useful to you, feel free to leave a comment on this issue.

I'll update the requirejs.org docs if this installation option seems to work out for people.

RequireJS 0.25.0 released, AMD advancing

RequireJS 0.25.0 is available for download.

It has been a few months since the last release, longer than normal. I have had some life changes (for good reasons) and I have busy at the day job. The interest in AMD loaders has picked up, and I have done some groundwork related to that interest. More information later in this post.

I wanted RequireJS 1.0 to be this release, but decided to name it 0.25.0 since I did some changes in the interest of better compatibility with other AMD loaders. Most applications should not notice the changes, but I want to have a release to shake out any problems on the edges.

Release Highlights

The awesome part: the optimizer is now just one JS file, r.js! It also doubles as a bootstrap script that supports the full capability of AMD modules and loader plugins in Node and in Rhino.

To use the optimizer, pass the "-o" option to r.js:

    node r.js -o app.build.js

To run your AMD-based project in Node (assuming server.js is your top-level AMD module):

    node r.js server.js

Running AMD modules in Node has more information. The optimizer docs have been updated to the new optimizer syntax, and the r.js script has its own project now, to allow releases that are decoupled from require.js.

Running r.js under Rhino is still supported, but you need to fetch a couple of JAR files first.

Other highlights:

• The loader plugin API changed to allow plugins to create cross-domain-accessible resources. The main use case: you use the text plugin to dynamically load text resources, but you want to deploy your scripts and text resources to a CDN. See the text plugin's implementation of writeFile() as an example.
  
• There is a global requirejs() function object that is the same as the global require() function object. This should allow RequireJS to work better in environments like Mozilla Chromeless, which already have a built-in require() function that does not have full AMD/loader plugin capabilities.
  
• It is now possible to specify the precise version of jQuery to allow in a RequireJS context. This is useful if you know of other scripts that load different versions of jQuery on a page.

Some changes in the name of compatibility with other AMD module loaders and Node:

• The "lib" directory configuration in package support was removed. It was always very awkward to support. Node no longer supports it, and that was the extra justification I needed to remove it.
• Relative module IDs are now relative to the related module ID, not the related module's resolved path.
• includeRequire in the optimizer config was removed, Use a paths config to include require.js instead. See the optimization docs for more details.

A small change to the context-specific require() passed to a loader plugin's load() call: require.isDefined() is now require.defined() and there is require. specified().

A New Home for AMD

Part of the RequireJS release delay was because the AMD API was moved off the CommonJS wiki and a set of compatibility tests were created.

While it was useful to hash out some of the basic goals and API on the CommonJS list, support for AMD on that list did not reach consensus. However, there has been continued and increasing interest in others making AMD-compatible loaders and tools. To avoid bothering people that would rather talk about other non-module things on the CommonJS list , a new AMD-focused discussion list and wiki has been set up and there is a set of compatibility tests.

AMD is advancing

More evidence that AMD is advancing:

• Alternate AMD loaders are available. Curl continues to advance, as well as the Dojo's 1.7 loader. Loadrunner has grown some basic support.
• There are at least two "AMD lite" projects, one in Ace, and one called DefineJS, that focus on basic AMD API support for when all the modules have been optimized into one file. The goal is to use a smaller bootstrap in optimized applications instead of a full standalone loader, like RequireJS.
• Node 0.5.0 supports the simplified CommonJS wrapper version of define(). While this support is very limited (no dependency array or loader plugins), it is a start to allowing some basic low-level module sharing between Node and browser apps better. See Kris Kowal's excellent Q library as an example of a module that works in both Node and in full AMD loaders.
• Look for better opt-in support for AMD in a couple of browser-focused libraries and toolkits later this year.
  

Path to RequireJS 1.0

For RequireJS, I want to get to a 1.0 when:

• 0.25.0 has been shown to work well in existing projects.
• Do some work around require.ready: possibly using a domReady plugin/module with optional loader hooks, to help improve cross-loader compatibility.
• Try to finalize more of the loader plugin API with other AMD implementers, to make sure more code can be shared across loaders. The loader plugin API is a separate API from basic AMD support, so I may push out a 1.0 before that API is completely final, and go to a 2.0 branch for any backwards incompatible changes to the loader API.

On inventing JS module formats and script loaders

There was some recent twitter tweets over the last couple months that indicated some folks want to experiment with some JavaScript module formats that work in the browser. In addition, making a script loader is almost as popular as making your own template engine. This is my response to both of those endeavors.

AMD is Winning

If you are one of those people that think they want to invent a module format or a script loader that works well in today's browsers, let me save you some trouble: a module format has already been worked out. Check out the Asynchronous Module Definition (AMD). There is still some room for innovation though, see near the end of this post.

If your script loader does not support AMD, it will be of very limited usefulness and in a sea of competitors that will make it hard for you to gain adoption.

So, for the module syntax, that problem is solved, and the solution is AMD.

A Solved Problem

Why is it solved?

• It has been discussed for more than a year and a half by multiple people, in the Dojo world, and on the CommonJS list.
• It has been implemented more than 5 times by different people.
• It has been deployed in real projects, the BBC iPlayer being a very visible one.
• It works in the browser, web workers, Node, Rhino, Mozilla Add-On SDK-based add-ons, and can be used in traditional Mozilla add-ons (Firebug will be using it).
  
• Many AMD implementations support loader plugins. You can add modules that know how to load other things as dependencies. While not strictly part of AMD, plugin systems greatly enhance a base technology.
  

You will be facing an uphill battle trying to get your format accepted. AMD has at least a half-year head start and has been seriously battle-tested. The ECMAScript folks are considering separate, different language changes to support modules, so you have that on the event horizon. Time is really running out pushing for something new.

I do not want to harsh your mellow, and I definitely want to follow your bliss, but just know the actual cost involved, and do your research first. Make sure you understand why AMD is constructed how it is, and what it really is.

It is not the traditional CommonJS Module format, but it can support similar concepts. However, it is different, take the time to figure out why.

Here is a handy gist with a summary of the AMD API, loader plugins, and some thoughts as to how it relates to modules implemented in Node, which are more like traditional CommonJS modules. In addition, you can read the RequireJS API docs, and the AMD wrapper format for traditional CommonJS modules.

If you think something was missed, feel free to ask on the RequireJS list, or on the CommonJS list, and enlighten us all. Or just send me an email or GitHub message if you want to talk privately. However, I believe most questions will be in the minor bikeshed/personal preference arena that do not ultimately matter for adoption. In particular, do not propose anything that requires the developer to do more typing that what is in AMD already. The format needs to be usable and easy to type. Discoverability is not a goal, daily usability is, and the AMD format is easy to explain.

Requirements

This leads to a discussion of other requirements. Here is what I believe are the minimum requirements for a module format in the browser, and they well-met by AMD:

• No globals: The module format needs a way for the developer to avoid creating global variables for each module they create. The format needs to be web-scalable, which means that some sites need the ability to embed two different versions of a module on a page. This normally means embedding in different "contexts" (the different versions do not need to talk to each other, just co-exist on the page), but it is a web-scale need. Seriously. We have seen it in Dojo. jQuery has seen it, it is the reason they have noConflict(). While noConflict() is nice, there are edge cases where it falls down. Know why that happens (in particular, dynamic script loading in IE: more than one script can execute before the first script's onload event fires).
  
• Allow for globals: at the same time, you want developers to be able to do want they want, in particular, some like to modify global object prototypes. Prototype and MooTools are valid ways to build web sites. So allow for it. This is one of my concerns with the ECMAScript harmony modules experimentation, but I think they know they need to allow for this, even if within a module loader instance.
• Do not require server transforms or think that using XHR+text transform+eval() is usable across all of web development. It is not. Here are some reasons why. They are fine for your boutique rails projects, but do not assume that translates well to the web at large, for instance, to serving JS files from disk for mobile widgets.
• You will need to use a function wrapper and a way to specify dependencies before that function wrapper is executed. You will be using script elements to load content (seriously, don't use eval, that is an evolutionary dead path), and dynamic script elements will load scripts async and out of order. A function wrapper will need to be part of the format.
  
• You need a way for modules to have a name. Otherwise you cannot combine them all together for optimization purposes.
• At the same time, it is best to allow source modules (the non-optimized ones) to not be named. This makes it much easier to move code around. This is not a hard requirement though, and I think one of the easiest to try to not support, see next section.

Places for Innovation

So you are not going to come up with a better module format. I know that sounds harsh, and for some of you that will be just the thing you need to hear to try something different. But I really am trying to save you some work. Again though, go for it if it really is your bliss, just know the amount of work you are taking on.

A better use of your time might be doing the best AMD implementation/script loader. In particular, you can choose some limiting design choices to help make it easier to meet/more compact:

• Do not support the simplified wrapper for traditional CommonJS Modules.
• Only allow named modules.
• Do not support loader plugins.
• Do not worry about supporting more than one version of a module in a page. The module format allows for it, which is the most important aspect, so module authors can code for it if they wish. However, most sites can get away without needing a loader that can actually support loading multiple versions.
  

If you do those things, you still can consume the built/optimized AMD modules that someone made using another implementation, and any modules created while working with your implementation will work with other AMD implementations. And with that, you are part of a larger ecosystem.

If you make a script loader that does not understand AMD, it will be a boutique loader, and not something that fits all of front-end development, and limits the ability to share code with other JS environments. Which is fine, boutique businesses can be really satisfying, just realize you are in a boutique, and do not over-sell the loader.

FAQ

These are odds and ends that do not fit above.

Why use modules or a script loader at all? Just server-side concat all your scripts!

The goal is to establish a larger ecosystem of shared code that works without puking globals all over the place. JavaScript is also not just jQuery (although I greatly appreciate the project), so it cannot rely on a jQuery implementation.

Not all environments can use server-side script concatenation (file-served mobile widgets), and it does not allow for more nuanced optimization like loading built layers on demand after initial page load. On-demand loading is really needed for large web apps like a webmail UI.

With tools like the RequireJS optimizer, you can still get the same effect of "concat all files". If you do not want to take the hit of downloading a script loader in that scenario, there is work underway to allow a simplified stub that removes the need for a full script loader. The Ace project uses a "mini-require" after they build the code to avoid loading the RequireJS script loader.

What about YUI 3 modules?

YUI 3 got a lot of things right with their modules, in particular designing for async up front. However, I believe the way they implemented the module name-to-JS-variable-name mapping is not right. It requires knowledge of two names that allows for easy typos and creates the opportunity for a naming conflict on the Y object. For instance:

YUI().use("io-base", "my-module", function(Y) {

//Where does .on() come from? Did 'my-module' add it?
//Did 'io-base'? What if both tried to add it?
Y.on(...);

//'io-base' creates the 'io' property on Y.
//How is that known? Should it be 'ioBase'?
//What else does 'io-base' add to Y?
var request = Y.io(...);
});

In short, too much magic that requires extra documentation. YUI 3 has very nice documentation so that helps, but I do not believe that approach scales as well as AMD.

Is AMD a CommonJS specification?

No, it is a proposal, but with lots of implementations and real use. Not all participants on the CommonJS list believe it should be a elevated to "endorsed spec" status. Those participants wish browsers would grow better base technology and/or they want to maintain stricter compatibility with traditional CommonJS modules, which were not designed primarily for async loading environments like the browser.

Any browser technology change will likely be for harmony module support which operate differently than traditional CommonJS modules, and AMD works well today in all JS environments. AMD is proven, and it can work on the server side. The RequireJS Node adapter even allows using NPM-installed modules in Node.

How does AMD relate to ECMAScript harmony modules?

They are unrelated, although hopefully the AMD work and the traditional CommonJS module work will help inform the harmony effort. I believe AMD modules will be easy to convert to harmony modules, but as the harmony work is still in progress it is hard to know for sure. The return value options in AMD may be more flexible than harmony modules, and it is unclear if loader plugins will work in a harmony module world.

AMD has the benefit of working in all the JS environments now, and it will work in the future, and I do not believe the syntax gains in harmony modules are a vast improvement over the typing cost in AMD, particularly if sharp functions are supported. That said, I hope to inform the harmony effort where I can to help make it the best it can be.

Who are you?

I'm James Burke (GitHub, Twitter). I maintained the original Dojo XHR-based script loader, maintained Dojo Core, created the Dojo xdomain script element-based loader, created RequireJS (formerly RunJS), and rewrote RequireJS about three times. I have strongly advocated for browser-friendly modules on the Dojo and CommonJS lists. I like Pina Coladas and getting caught in the rain.

RequireJS+jQuery project moved, updated to jQuery 1.5.2

The sample project that demonstrates how jQuery and RequireJS can be used together has been moved to its own repository. This should make it easier to update. As proof, it now has jQuery 1.5.2, and can be downloaded here.

Also, the sample project has gone back to using an integrated RequireJS+jQuery file as the basis for the example. Keeping RequireJS and jQuery separate and using the "priority" configuration option in jQuery has some edge case considerations that required more explanation than what most people need. However, the README in the sample project repository explains how to use the priority configuration if you prefer to use that option.