A require() for jQuery

I had a fun time at the Bay Area jQuery Conference. Great people, and I learned some neat things.

In the conference wrap-up, John Resig mentioned some requirements he has for a jQuery script loader:

1) script loading must be async

2) script loading should do as much in parallel as possible. This means in particular, that it should be possible to avoid dynamic nested dependency loading.

3) it looks like a script wrapper is needed to allow #1 and #2 to work effectively, particularly for cross-domain loading. It is unfortunate, but a necessity for script loading in browsers.

I believe these requirements mesh very well with RequireJS. I will talk about how they mesh, and some other things that should be considered for any require() that might become part of jQuery.

Async Loading

As explained in the RequireJS Why page, I believe the best-performing, native browser option for async loading is dynamically created script tags. RequireJS only uses this type of script loading, no XHR.

The text plugin uses XHR in dev mode, but the optimization tool inlines the text content to avoid XHR for deployment. Also, the plugin capability in RequireJS is optional, it is possible to build RequireJS without it. That is what I do for the integrated jQuery+RequireJS build.

Parallel Loading

John mentioned that dynamic nested dependency resolution was slower and potentially a hazard for end users. Slow, because it means you need to fetch the module, wait for it to be received, then fetch its dependencies. So the module gets loaded serially relative to its dependencies. Potentially hazardous because a user may not know the loading pattern.

The optimization tool in RequireJS avoids the parallel loading for nested dependencies, by just inlining the modules together. The optimization tool can also build files into "layers" that could be loaded in parallel.

For each build layer, there is an exclude option, in which you can list a module or modules you want to exclude. exclude will also exclude their nested dependencies from the build layer.

There is an excludeShallow option if you just want specific modules to exclude, but still want their nested dependencies included in the build layer. This is a great option for making your development process fast: just excludeShallow the current module you are debugging/developing.

While dynamically loading nested dependencies can be slower than a full parallel load, what is needed is listing dependencies individually for each module. There needs to be a way to know what an individual file needs to function if the file is to be portable in any fashion. So the question is how to specify those dependencies for a given file/module.

There are schemes that list the dependencies as a separate companion file with the module, and schemes that list the dependencies in the module file. Using a separate file means the module is less portable -- more "things" need to follow the module, so it makes copy/pasting, just distributing one module more onerous.

So I prefer listing the dependencies in the file. Should the dependencies be listed in a comment or as some sort of script structure?

Comments can be nice since they can be stripped from the built/optimized layer. However, it means modules essentially need to communicate with each other through the global variable space. This ultimately does not scale -- at some point you will want to load two different versions of a module, or two modules that want to use the same global name, and you will be stuck. For that reason, I favor the way RequireJS does it:

require.def("my/module", ["dependency1"], function (dependency1) {
  //dependency1 is the module definition for "dependency1"

  //Return a value to define "my/module"
  return {
      limit: 500,
      action: function () {}
  };
});

With this model, dependency1 does not need to be global, and it allows a very terse way to reference the module. It also minifies nicely. By using string names to reference the modules and using a return value from the function, it is then possible to load two versions of module in a page. See the Multiversion Support in RequireJS for more info, and the unit tests for a working example.

This model also frees the jQuery object from namespace collisions by allowing a terse way to reference modules without needing them to hang off of the jQuery object. There are many utility functions that do not need to be on the jQuery object to be useful, and today the jQuery object itself is starting to become a global of sorts that can have name collisions.

Script Wrapper

Because async script tags are used to load modules, each script needs to be wrapped in a function wrapper, to prevent its execution before its dependencies are ready. CommonJS recognizes this concern (one of the reasons for their Transport proposals) and so does YUI3. xdomain builds for Dojo also use a script wrapper.

While it is unfortunate -- many people are not used to it -- it ends up being an advantage. Functions are JavaScript's natural module construct, and it encourages well scoped code that does not mess with the global space. For RequireJS, that wrapper is called require.def, as shown above.

Here are some other things that should be considered for a require implementation:

require as a global

I believe it makes more sense to keep require as a global, not something that is a function hanging off of the jQuery object. require can be used to load jQuery itself, and as mentioned above, it would be possible to load more than one version of jQuery if it was constructed like this.

CommonJS awareness

The CommonJS module format was not constructed for the browser, but having an awareness of their design goals and a way to support their modules in the browser will allow more code reuse. RequireJS has an adapter for the CommonJS Transport/D proposal, and it has a conversion script to change CommonJS modules into RequireJS modules.

In addition, RequireJS was constructed with many of the same design goals as CommonJS: allow modules to be enclosed/do not pollute the global space, use the "path/to/module" module identifiers, have the ability to support the module and exports variables used in CommonJS.

Browsers need more than a require API

They also need an optimization/build tool that can combine modules together. RequireJS has such a system today. It is server-independent, a command line tool. It builds up the layers as static files which can be served from anywhere.

I am more than happy to look at a runtime system that uses the optimization tool on the server. RequireJS works in Node and in Rhino. The optimization tool is written in JavaScript and uses require.js itself to build the optimization layers.

I can see using either Node or Rhino to build a run-time server tool to allow combo-loading on the fly. Using Rhino via the Java VM has an advantage because Closure Compiler or YUI Compressor could be used to minify the response, but I am open to some other minification scheme that is implemented in plain JavaScript.

Loader plugins

I have found the text plugin for RequireJS to be very useful -- it allows you to reference HTML templates on disk and edit HTML in an HTML editor vs. dealing with HTML in a string. The optimization tool is smart enough to inline that HTML during a build, so the extra network cost goes away for deployment.

In addition, Sean Vaughan and I have been talking about support for JSONP-based services and scripts that need extra setup besides just being ready on the script onload event. I can see those as easy plugins to add that open up loading Google Ajax API services on the fly.

For these reasons I have found loader plugins to be useful. They are not needed in the basic case, but they can make overall dependency management better.

script.onload

Right now RequireJS has support for knowing when a script is loaded by waiting for the script.onload event. This could be avoided by mandating that anything loaded via require() register via require.def to indicate when it is loaded.

However, by using script.onload it allows some existing scripts to be loaded without modification today, to give people time to migrate to the require.def pattern. I am open to doing a build without the script.onload support, however the amount of minified file savings will not be that great.

Explicit .js suffix

RequireJS allows two different types of strings for dependencies. Here is an example:

require(["some/module", "http://some.site.com/path/to/script.js"]);

"some/module" is transformed to "some/base/path/some/module.js", while the other one is used as-is.

The transform rules for a dependency name are as follows: if the name contains a colon before a front slash (has a protocol), starts with a front slash, or ends in .js, do not transform the name. Otherwise, transform the name to "some/base/path/some/module.js".

I believe that gives a decent compromise to short, remappable module names (by changing the baseUrl or setting a specific path via a require config call) to loading scripts that do not participate in the require.def call pattern. There is also a regexp property on require that can be changed to allow more exceptions to the rules.

However, if this was found insufficient, I am open to other rules or a different way to list dependencies. The "some/module" format was chosen to be compatible with CommonJS module names, but probably some algorithm or approach could be used to satisfy both desires.

File Size/Implementation

Right now the stock RequireJS is around 3.7KB minified and gzipped. However, there are build options that get the size down to 2.6KB minified and gzipped by removing some features:

• plugin support
• require.modify
• multiversion support (the "context" switching in RequireJS)
• DOM Ready support

I am open to getting that file size smaller based on the feature set that needs to be supported.

3 layer loading

John mentioned a typical loading scenario that might involve three sections:

1) loading core libraries from a CDN (like jQuery and maybe a require implementation)
2) loading a layer of your common app scripts
3) loading a page-specific layer

RequireJS can support this scenario like so today:

<script src="http:/some.cdn.com/jquery/1.5/require-jquery.js"></script>
<script>
require({
     baseUrl: "./scripts"
 },
 ["app/common", "app/page1"]
);
</script>

Then the optimization tool instructions would look like so:

{
 modules: [
     {
         //inside app/common.js there is a require call that
         //loads all the common modules.
         name: "app/common",
         exclude: ["jquery"]
     },
     {
         //app/page1 references jquery and app/common as a dependencies,
         //as well as page-specific modules
         name: "app/page1",

         //jquery, app/common and all their dependencies will be excluded
         exclude: ["jquery", "app/common"]
     },
     ... other pages go here following same pattern ...
 ]
}

This would result in app/common and app/page1 being loaded async in parallel. If require.js was a separate file from jquery.js, the following HTML could be used to load jQuery, app/common and app/page1 async and in parallel (the optimization instructions stay the same):

<script src="http:/some.cdn.com/jquery/1.5/require.js"></script>
<script>
require({
     baseUrl: "./scripts",
     paths: {
         "jquery": "http:/some.cdn.com/jquery/1.5/jquery"
     }
 },
 ["jquery", "app/common", "app/page1"]
);
</script>

Those configurations work today.

However, it is not quite flexible enough -- typically modules that are part of app/page1 will not want to refer to the complete "app/common" as the only dependency, but specify finer-grained dependencies, like "app/common/helper". So the above could result in a request for "app/commom/helper" from the "app/page1" script, depending on how fast "app/common" is loaded.

So I would build in support for the following:

<script src="http:/some.cdn.com/jquery/1.5/require.js"></script>
<script>
require({
     baseUrl: "./scripts",
     paths: {
         "jquery": "http:/some.cdn.com/jquery/1.5/jquery"
     },
     layers: ["jquery", "app/common", "app/page1"]
 },
 ["app/page1"]
);
</script>

Notice the new "layers" config option, and now the required modules for the page is just "app/page1". The "layers" config option would tell RequireJS to load all of those layers first, and find out what is in them before trying to fetch any other dependencies.

This would give the most flexibility in coding individual modules, but give a very clear optimization path to getting a configurable number of script layers to load async and in parallel. I will be working on this feature for RequireJS for the next release.

Summary

Hopefully I have demonstrated how RequireJS could be the require implementation for jQuery. I am very open to doing code changes to support jQuery's desires, and even if jQuery or John feel like they want to write their own implementation, hopefully we can at least agree on the same API, and maybe even still use the optimization tool in RequireJS. I am happy to help with an alternative implementation too.

I know John and the jQuery team are busy, focusing mostly on mobile and templating concerns, but hopefully they can take the above into consideration when they get to script loading.

In the meantime, I will work on the layers config option support, improving RequireJS, and keeping my jQuery fork up to date with the changes. You can try out RequireJS+jQuery today if you want to give it a spin yourself.

RequireJS+jQuery Talk

I gave a talk about RequireJS with jQuery at the jQuery Conference today. Here are the slides:

• PDF
• HTML (Warning, the inline links do not appear to work, use PDF for working links)

Thanks to the folks that came to the talk! I had a great time at the conference.

If you went to the talk, please feel free to rate the talk so I can improve for the next time.

RequireJS 0.10.0 Released, Node integration

RequireJS 0.10.0 is now available.

The big feature in this release is integration with Node. Now you can use a the same module format for both browser and server side modules. The RequireJS-Node adapter translates existing CommonJS modules on the fly, as they are loaded by the adapter, so you can continue to use server modules written in the CommonJS format for your Node projects.

The RequireJS-Node adapter is freshly baked, so there could be some rough edges with it, but it is exciting to see it work. See the docs for all the details.

0.10.0 also includes support for an excludeShallow option in the optimization tool. This will allow you to do an optimization build during development, but still excludeShallow the specific module you want to develop/debug in the browser. So you can get great debug support in the browser for just that one module, but still load the rest of your JS super-fast. No need for special server transforms.

I will be at the jQuery conference this weekend in Mountain View, CA. I will be speaking on Sunday about jQuery+RequireJS. Stop by and say hi if you are at the conference!

JavaScript object inheritance with parents

There are different ways to inherit functionality in JavaScript, including using mixins (mixing in all the properties of one object into another object) and the use of prototypes.

In Dojo, there is dojo.mixin for doing mixins, and dojo.delegate for inheriting properties via prototypes. dojo.delegate is like ECMAScript 5/Crockford's Object.create(), but with a dojo.mixin convenience call.

I really like the dojo.delegate or a Object.create+dojo.mixin combination for inheriting, but it makes it hard to call methods you override from your parent. I see this problem show up frequently with widgets, which typically inherit from each other:

var MyWidget = Object.create(BaseWidget);

//BaseWidget also defines a postCreate method,
//But we want our widget to do work too.

MyWidget.prototype.postCreate = function () {
 //Call BaseWidget's implementation
 BaseWidget.prototype.postCreate.apply(this, arguments);

 //Do MyWidget's postCreate work here.
};

Not too bad, but the BaseWidget.prototype.postCreate.apply junk is a bit much to type, and it gets a bit trickier when there are mixins that also contribute to the functionality.

In Dojo, there is dojo.declare() that helps with this by defining an "inherited" method that can be used to find the BaseWidget's postCreate:

var MyWidget = dojo.declare(BaseWidget, {
  postCreate: function () {
      //Call BaseWidget's implementation
      this.inherited("postCreate", arguments);

      //Do MyWidget's postCreate work here.
  }
});

This is an improvement as far as typing, but the implementation of dojo.declare has always scared me. My JavaScript Fu is not strong enough to follow it, and I am concerned it is actually a bit too complicated.

So here is an experiment on something simpler:

var MyWidget = object("BaseWidget", null, function (parent) {
 return {
     postCreate: function () {
         //Call BaseWidget's implementation
         parent(this, "postCreate", arguments);

         //Do MyWidget's postCreate work here.
     }
 };
});

Here is the implementation of that object function, and here are some tests. That implementation is wrapped in a RequireJS module, but it can be extracted as a standalone script.

The second argument to the object() function allows for specifying mixins.

With two mixins, mixin1 and mixin2, the parent for MyWidget would be an object that inherits from BaseWidget with mixin1 and mixin2's properties mixed in:

var MyWidget = object("BaseWidget", [mixin1, mixin2], function (parent) {
  return {
      postCreate: function () {
          //Call BaseWidget's postCreate, but if it
          //does not have a postCreate method, mixin1's
          //postCreate function will be used. If mixin1
          //does not have an implementation, then mixin2's
          //postCreate function will be used. If mixin2 does
          //not have an implementation an error is thrown.
          parent(this, "postCreate", arguments);

          //Do MyWidget's postCreate work here.
      }
  };
});

dojo.declare has the concept of calling a method called "constructor" if it is defined on the declared object, whenever a new object of the MyWidget type is created. I preserved that ability in object() but the property name for that function is "init" in the object() implementation.

The object() implementation is simpler than dojo.declare, but still gives easy access for calling a parent implementation of a function. It is not has powerful as dojo.declare -- dojo.declare has the concept of postscript and a preamble and even auto-chaining calls. However, I feel the simplified approach is better. It is clearer to follow the code, and to predict how it will behave. I also expect it to perform better.

I like the object() method because it uses closures and a function that accepts the parent function as an argument. Feels very JavaScripty. The prototype chain is a bit longer with the extra object.create() calls creating some intermediate objects, but I expect prototype walking is fast in JavaScript, particularly when you go to measure it in comparison to any DOM operation.

Are there ways in which the object() function is broken or insufficient? Is there a better way to do this? Or even a different way, something that does not rely on a parent reference?

There is traits.js, for using traits. Alex Russell experimented with a trait implementation inside dojo.delegate. Kris Zyp pointed out that Alex's implementation does not have conflict detection or method require support.

I like the idea of mixing in just part of a mixin or remapping a method to fit some other API's expectations, so I can see adding support for the remapping features, similar to what Alex does in the dojo.delegate experiment. However, I am not sure how valuable conflict detection or method require support is.

I can see in large systems it would help with detecting errors sooner, but then maybe the bigger problem is the complexity of the large system. And there is a balance to forcing strictness up front over ease of use. The trait.js syntax looks fairly wordy to me, and the extra benefit of the strictness may not be realized for most web apps.

Also, I do not see an easy way to get the parent reference. It seems like you need to remap each overridden parent function you want to call to a new property name. It seems wordy, with more properties hanging off an object. And do you need to make sure you do not pick a name that is already in use by an ancestor? Seems like it could lead to a bunch of goofy names on an object.

Reusing code effectively is an interesting topic. The traits approach is newer to me, and I keep wondering if there is a better way to do it. It has been fun to experiment with alternatives.

CommonJS Module Trade-offs

First of all: why should you care about module formats?

If you use JavaScript, particularly in the browser, more is being expected of you each day. Every site or webapp that you build will want to do more things over time, and browser engines are getting faster, making more complex, web-native experiences possible. Having modular code makes it much easier to build these experiences.

One wrinkle though, there is no standard module format for the browser. There is the very useful Module Pattern, that helps encapsulate code to define a module, but there is no standard way to indicate your module's dependencies.

I have been following some of the threads in the CommonJS mailing list about trying to come up with a require.async/ensure spec and a Transport spec. The reason those two specs are needed in addition to the basic module spec is because the CommonJS module spec decided to make some tradeoffs that were not browser-friendly.

This is my attempt to explain the trade-offs the CommonJS module spec has made, and why I believe they are not the right trade-offs. The trade-offs end up creating a bunch of extra work and gear that is needed in the browser case -- to me, the most important case to get right.

I do not expect this to influence or change the CommonJS spec -- the developers that make up most of the list seem to generally like the module format as written. At least they agreed on something. It is incredibly hard to get a group of people to code in a certain direction, and I believe they are doing it because they love coding and want to make it easier.

I want to point out the trade-offs made though, and suggest my own set of trade-offs. Hopefully by explicitly listing them out, other developers can make informed choices on what they want to use for their project.

Most importantly, just because "CommonJS" is used for the module spec, it should not be assumed that it is an optimal module spec for the browser, or that it should be the default choice for a module spec.

Disclosure: I have a horse in this race, RequireJS, and much of its design comes from a different set of tradeoffs that I will list further down. I am sure someone who prefers the CommonJS spec might have a different take on the trade-offs.

To the trade-offs:

1) No function for encapsulating a module.

A function around a module can seem like more boilerplate. Instead each module in the CommonJS spec is just a file. This means only one module per file. This is fine on the server or local disk, but not great in the browser if you want performance.

2) Referencing and loading dependencies synchronously is easier than asynchronous

In general, sync programming is easier to do. That does not work so well in the browser though.

3) exports

How do you define the module value that other modules can use? If a function was used around the module, a return value from that function could be used as the module definition. However, in the effort to avoid a function wrapper, it complicates setting up a return value. The CommonJS spec instead uses a free variable called "exports".

The value of exports is different for each module file, and it means that you can only attach properties to the exports module. Your module cannot assign a value to exports.

It means you cannot make a function as the module value. Some frameworks use constructor functions as the module values -- these will not be possible in CommonJS modules. Instead you will need to define a property on the exports object that holds the function. More typing for users of your module.

Using an exports object has an advantage: you can pass it to circular dependencies, and it reduces the probability of an error in a circular dependency case. However, it does not completely avoid circular dependency problems.

Instead, I favor these trade-offs:

1) Use a function to encapsulate the module.

This is basically the core of the previously-mentioned Module Pattern. It is in use today, it is an understood practice, and functions are at the core of JavaScript's built-in modularity.

While it is an extra function(){} to type, it is fairly standard to do this in JavaScript. It also means you can put more than one module in a file.

While you should avoid multiple modules in a file while developing, being able to concatenate a bunch of modules together for better performance in the browser is very desirable.

2) Assume async dependencies

Async performs better overall. While it may not help performance much in the server case, making sure a format performs well out of the box in the browser is very important.

This means module dependencies must be listed outside the function that defines the module, so they can be loaded before the module function is called.

3) Use return to define modules

Once a function is used to encapsulate the module, the function can return a value to define the module. No need for exports.

This fits more naturally with basic JavaScript syntax, and it allows returning functions as the module definition. Hooray!

There is a slightly higher chance of problems in circular dependency cases, but circular dependencies are rare, and usually a sign of bad design. There are valid cases for having circular dependencies, but the cases where a return value might be a problem for a circular dependency case is very small, and can be worked around.

If getting function return values means a slightly higher probability of a circular dependency error (which has a mitigation) then that is the good trade-off.

This avoids the need for the "exports" variable. This is fairly important to me, because exports has always looked odd to me, like it did not belong. It requires extra discovery to know its purpose.

Return values are more understandable, and allowing your module to return a function value, like a constructor function, seems like a basic requirement. It fits better with basic JavaScript.

4) Pass in dependencies to the module's function wrapper

This is done to decrease the amount of boilerplate needed with a function wrapped modules. If this is not done, you end up typing the dependency name twice (an opportunity for error), and it does not minify as well.

An example: let's define a module called "foo", which needs the "logger" module to work:

require.def("foo", ["logger"], function () {

    //require("logger") can be a synchronous call here, since
    //logger was specified in the dependency array outside
    //the module function
    require("logger").debug("starting foo's definition");

    //Define the foo object
    return {
        name: "foo"
    };
});

Compare with a version that passes in "logger" to the function:

require.def("foo", ["logger"], function (logger) {

    //Once "logger" module is loaded it is passed
    //to this function as the logger function arg
    logger.debug("starting foo's definition");

    //Define the foo object
    return {
        name: "foo"
    };
});

Passing in the module has some circular dependency hazards -- logger may not be defined yet if it was a circular dependency. So the first style, using require() inside the function wrapper should still be allowed. For instance, require("logger") inside a method that is created on the foo object could be used to avoid the circular dependency problem.

So again, I am making a trade-off where the more common useful case is easier to code vs increasing the probability of circular dependency issues. Circular dependencies are rare, and the above has a mitigation via the use of require("modulename").

There is another hazard that can happen with naming args in the function for each dependency. You can get an off-by-one problem:

require.def("foo", ["one", "two", "three"], function (one, three) {
    //In here, three is actually pointing to the "two" module
});

However, this is a standard coding hazard, not matching inputs args to a function. And there is mitigation, you could use require("three") inside the module if you wanted.

The convenience and less typing of having the argument be the module is useful. It also fits well with JSLint -- it can help catch spelling errors using the argument name inside the function.

5) Code the module name inside the module

To define the foo module, the name "foo" needs to be part of the module definition:

require.def("foo", ["logger"], function () {});

This is needed because we want the ability to combine multiple module definitions into one file for optimization. In addition, there is no good way to match a module definition to its name in the browser without it.

If script.onload fired exactly after the script is executed, not having the module name in the module definition might work, but this is not the case across browsers. And we still need to allow the name to be there for optimization case, where more than one module is in a file.

There is a legitimate concern that encoding the module name in the module definition makes it hard to move around code -- if you want to change the directory where the module is stored, it means touching the module source to change the names.

While that can be an issue, in Dojo we have found it is not a problem. I have not heard complaints of that specific issue. I am sure it happens, but the fix cost is not that onerous. This is not Java. And YUI 3 does something similar to Dojo, encode a name with the module definition.

I think the rate of occurrence of this issue, and the work it takes to fix are rarer and one time costs vs. forcing every browser developer taking extra, ongoing costs of using the CommonJS module format in the browser.

Conclusion

Those are the CommonJS trade-offs and my trade-offs. Some of them are not "more right" but just preferences, just like any language design. However, the lack of browser support in the basic module spec is very concerning to me.

In my eyes, the trade-offs CommonJS has made puts more work on browser developers to navigate more specs and need more gear to get it to work. Adding more specs that allow modules to be expressed in more than one way is not a good solution for me.

I see it as the CommonJS module spec making a specific bet: treating the browser as a second class module citizen will pay off in the long run and allow it to get a foothold in other environments where Ruby or Python might live.

Historically, and more importantly for the future, treating the browser as second class is a bad bet to make.

All that said, I wish the CommonJS group success, and there are lots of smart people on the list. I will try to support what I can of their specs in RequireJS, but I do feel the trade-offs in the basic module spec are not so great for browser developers.

RequireJS 0.9.0 Released

I just pushed a new release of RequireJS, 0.9.0.

The optimization tool has seen the most change in this release. It sports some CSS optimizations now and it is much more robust. It also includes command line options for optimizing just one JS file or one CSS file.

The other new feature is the support for relative module names for require.def() dependencies. So this kind of call works now:

require.def("my/project/module", ["./dependency1"], function(){});

It will load my/project/dependency1.js. This should help cut down the amount of typing for larger projects that have deep directories of modules.

This release has some backwards-incompatible changes. That was the reason for the bump to 0.9.0. The project is still not at 1.0, so backwards-incompatible changes may still be considered. I do not have any more changes like that planned, but I will be sure to give more notice in the RequireJS list before doing so in the future.

All the details are on the download page.

Raindrop has been updated to the latest RequireJS release, and it is working great. Give the new RequireJS build a spin!

RequireJS, kicking some AST

RequireJS has an optimization tool that can combine and minify your scripts. It uses Google's Closure Compiler to do the minification. Recently, but after the RequireJS 0.8.0 release, I ported over the CSS optimizations from the Dojo build system, so the optimization tool now inlines @import calls and remove comments from CSS files.

The script combining still has some rough edges though, and it was mainly due to me trying to use suboptimal regexp calls to find require() and require.def() calls in the files, so the dependencies for a script could be traced.

So I finally took the dive into Abstract Syntax Trees (ASTs) to do the work. What is an AST? An analogy that works for me: an AST is to JavaScript source as the DOM API is to HTML source. The AST has methods for walking through the nodes in the JS code structure, and you can get properties on a node.

Figuring out how to generate an AST from scratch can be a bit of work, but since I was already using Closure Compiler, I just used an AST it can generate.

Since the optimization tool for RequireJS is written in JavaScript, which makes calls into Java-land to do file access and minification calls, I wanted the same approach for working with the AST -- do my work in JavaScript, but call the Java methods for the AST walking and source transform.

My task was fairly simple -- I just wanted to find require() or require.def() calls that used strings for module names and dependencies, pull those calls out of the file, then just execute those calls to work out the dependencies.

The end result was this file:
http://github.com/jrburke/requirejs/blob/master/build/jslib/parse.js

The basic idea of the script:

//Set up shortcut to long Java package name,
//and create a Compiler instance.
var jscomp = Packages.com.google.javascript.jscomp,
  compiler = new jscomp.Compiler(),

  //The parse method returns an AST.
  //astRoot is a kind of Node for the AST.
  //Comments are not present as nodes in the AST.
  astRoot = compiler.parse(jsSourceFile),
  node = astRoot.getChildAtIndex(0);

//Use Node methods to get child nodes, and their types.
if (node.getChildAtIndex(1).getFirstChild().getType() === CALL) {
  //Convert this call node and its children to JS source.
  //This generated source does not have comments and
  //may not be space-formatted exactly the same as the input
  //source
  var codeBuilder = new jscomp.Compiler.CodeBuilder();
  compiler.toSource(codeBuilder, 1, node);

  //Return the JavaScript source.
  //Need to use String() to convert the Java String
  //to a JavaScript String.
  return String(codeBuilder.toString());
}

Thanks to the Closure Compiler team for doing the hard work and open sourcing the code. It looks like Closure Compiler deals with two AST formats -- one is perhaps an older one generated by Rhino, while the other one is a more custom one? It seems like I was getting back the Rhino-based Nodes for the methods I called.

I was tempted to try to go direct to just use Rhino for the AST, but decompiling the AST into source looked harder to do, and from what I recall, Rhino has a newer AST API in the trunk code. I believe the one in Closure Compiler is the older one? All that added up to me being wary of that path.

Most of the time spent was trying to figure out the Java invocations to get the code parsed, understand the tree structure, deal with Java-to-JavaScript translation issues and then figure out the Java invocations to convert a subtree back into source.

I am glad I finally stepped into working with a real AST. While some of the AST calls are a bit awkward (at least for me as a JavaScript person), it is a lot better than trying to use regexps for it. I still need to do more testing, but I feel more confident in the robustness of the solution now.

If you see how I can do it better, point me in the right direction!

RequireJS 0.8.0 Released

RequireJS, the next generation in script loading, now has an official release and a new web site: http://requirejs.org.

The 0.8.0 release is a formal release of the code, and it includes built versions of jQuery 1.4.2 with RequireJS already integrated.

I also updated my jQuery fork to include the latest changes -- jQuery's page load callbacks will not fire unless all scripts loaded with RequireJS have also finished loading.

I plan to do integrations with other browser toolkits, MooTools and Prototype being next on my list. I also hope the jQuery community will want to pull the changes I have in my jQuery fork into their master at some point.

If you are a team member for one of these toolkits, please let me know what I can do in RequireJS to provide the best code loading and module format for browser-based toolkits. It would be great if we can reach consensus on code loading. I am happy to make changes in RequireJS if it moves us all closer to that.

While the release version is 0.8, this code has been battle-tested in Raindrop, a sizable JavaScript-centric messaging web app. Raindrop uses a version of Dojo 1.4 that has been converted to the RequireJS module format, and all Raindrop modules are written as RequireJS modules.

Some other notes about the release:

• RequireJS 0.8.0 can run in Rhino, but its main design target is the browser.
• It comes with an optimization tool that can combine and minify your JavaScript.

Give it a spin! There is a moderated list if you have questions or need support.

RunJS is now RequireJS

As mentioned before, I considered renaming RunJS to RequireJS. I did the transition, and RequireJS is on GitHub. There is a conversion script that will convert CommonJS modules to the Transport/C proposal that works with RequireJS.

I have converted Raindrop to use a modified Dojo 1.4 that uses RequireJS instead of the normal Dojo loader, and all Raindrop modules are written in the Transport/C module format that RequireJS understands. Raindrop works, so the RequireJS code has been proven in a real project that has many modules with nested dependencies. The RequireJS code is already battle-tested.

I have opened a thread on the jQuery forum about using RequireJS for jQuery's require() needs. I can do a build with Dojo 1.4 that uses RequireJS, and any Dojo 2.0 effort is likely to use RequireJS as the module loader.

I believe RequireJS is the loader browser-based toolkits should use. At the very least, the module format and API supported by RequireJS should be used by browser-based toolkits, even if they want to build their own loader.

Next plans for RequireJS:

1) Contact the MooTools and Prototype folks to see if they want to use it. It allows loading code that does not export a module value, and has access to the global environment. They can use it to load code that augments native prototypes. RequireJS can load existing, plain JS files that do not define a module too.

2) Put up a web site with builds. While you can use RequireJS just from grabbing it from GitHub, it would be nice to have the builds of RequireJS with its different levels of functionality already built and easy to download.

2) Do a fork of Node that uses RequireJS on the server. I believe the async module format used by RequireJS is great fit for Node and its async goals.

3) See if I can do a fork of Narwhal to do the same thing.

I believe I can get RequireJS to work on the server and still support the existing CommonJS format when the server supports synchronous loading. By having native support in some server-based systems for RequireJS, it will be easier to share code with the browser.

To recap, my three main issues with using the existing CommonJS module spec, and why RequireJS exists:

1) So far the CommonJS group does not think the browser is common enough to qualify as a first class citizen in the module spec. The group is mainly concerned with environments outside the browser. As a result, the CommonJS module spec does not work well natively in the browser -- it either requires an XHR-based loader, which we have found to have problems in Dojo, or require a server-side transform process. A server-side transform process should not be required to do web development in the browser.

RequireJS uses a function wrapper around the module to avoid these problems and allow loading modules via script tags. Just save the file and hit reload in the browser.

2) There is a free variable, called "exports". It is an object. You cannot set the value of exports inside your module code, you can only add properties. This means for instance, your module cannot be a function. In Dijit, Dojo's widget system, all widgets are constructor functions. The "exports" restriction makes your APIs awkward if you want to export functions for module values. The claim is that this exports restriction helps with circular dependencies, but it only helps a little bit. To me, it is not worth slightly improving an edge case when it sacrifices a greater usefulness and simplicity in user's modules.

RequireJS and its format can handle circular dependencies just fine. In the format supported by RequireJS, you to define a module as a function. Although, you can still use exports as CommonJS uses it if you so desire.

3) The require.main property seems like a hack. It is normally used so that a module can say if (require.main === module.id) or if (require.main === module) then do some "main" work. The module format should just define an exports.main convention for indicating "main" functionality. It is less typing, and more robust, since different code entry points have a different idea of "main". For instance, an HTTP request handler likely has specific requirements on what it considers to be the "main". The top level entry point should decide what code to execute as "main", not logic inside the module.

RequireJS does not support the require.main idiom.

So I believe the path used by RequireJS is more robust overall, works better/scales better in the browser. However, I still want to provide enough support for the existing CommonJS modules in the meantime to allow more code sharing.

In the long run though, the CommonJS format as it exists today should be replaced with something better. It is troublesome that the CommonJS group is not really targeting the browser, but over time, the broader JS community will expect browser toolkits to support CommonJS specs. It does not seem right to end up with non-optimal solutions in the browser when the browser is the most common JS platform.

RunJS to RequireJS?

There was a thread that started on the CommonJS list about a transport format, something that works well in the browser via script injection. I sketched out a proposal, Transport/C, that builds on the Transport/B and Transport/A specs.

Transport/C is very similar to some basic mechanics of RunJS but uses require() as the top level function, and supports the special "module" and "exports" free variables used in the normal CommonJS module spec.

In order to prove the concept for Transport/C, I made a branch of the RunJS code, calling it RequireJS, that implements Transport/C.

It seems like it fits with the existing CommonJS module spec, but is something that works well in the browser. I also made a simple conversion script that converts traditional CommonJS modules to this format.

I am tempted to convert from RunJS to this RequireJS branch, and to start evangelizing that approach for browser toolkits. It would be great if Transport/C would also be approved as the transport format for CommonJS too.

Kris Kowal has some concerns about the *very* long-term effects of the approach. I read his comments as possibly pointing out some things that would be done differently if the primordials and e-maker type of modules were ever accepted as part of an ECMAScript standard.

As I read the primordials and e-maker strawman proposals, I think the only difference is Transport/C functions are only expected to be called once, but e-maker style would favor calling the function on every require() call. As I say in my response, I believe e-maker support, would affect regular CommonJS modules in the same way as the transport format, and it is assuming the strawmans make it in to the spec at some point, as they are specified now.

I also believe how it works in how I coded Transport/C as part of the RequireJS branch is what a normal developer would expect, and I think fits better with existing browser/script behavior, and the assumptions that go along with coding CommonJS modules today.

So I am tempted to rename the RunJS project to RequireJS and proceed with that. If you have any feedback to the contrary, please let me know. Otherwise, I will likely do the change early this week.

Script async, Raindrop and Firefox 3.6

In honor of the Firefox 3.6 release, I upgraded Raindrop to use the new async attribute for script tags.

Why is async neat? It does not block the rest of the page, and will just evaluate the script once it is retrieved. More information is in the HTML5 spec. Note that the script you add async to should NOT use document.write(), as doc.write will likely destroy your page.

Also, be aware that async is a boolean attribute, but that does not mean you should use async="true" to turn it on. The HTML5 spec on boolean attributes says that a value of empty string or a string that matches the attribute name should only be used. To avoid async, just do not include the attribute. For Raindrop, I used async="async" since that looks better to me than an empty string.

Raindrop uses RunJS for the module loader, and RunJS uses dynamically added script tags via head.appendChild(), so the modules loaded by RunJS already behave in an async manner.

However apps that want to use the raindrop front end libraries normally include a script called rdconfig.js as their own script file, and that config file does a document.write to write out the Dojo+RunJS and jQuery tags. Those Dojo+RunJS and jQuery tags now use the async attribute.

RunJS Dependency API

In my last post, I talked about a suggestion from David Ascher to try to improve the syntax of specifying dependencies when declaring a module. Here is an example of that suggestion, using run.def(), a possibly new API dedicated to just defining a module called "rdw/Message":

run.def("rdw/Message", {
rd: "rd",
dojo: "dojo",
Base: "rdw/_Base",
friendly: "rd/friendly",
hyperlink: "rd/hyperlink",
api: "rd/api",
template: "text!rdw/templates/Message!html"
}, function(R) {
//Module definition function.
//Use things like R.api and R.Base in here that map to the modules up above.
...
});

While that does make it clear what each module name's variable will be inside the function that defines the module, it has the following drawbacks:

1) It hurts minification -- having properties off the R object passed to the module function means that minification tools will not be able to minify those references as easy.

2) All the modules dependencies must be referenced via a prefix "R.". For example R.api. It is a small bit of typing and an extra property lookup. That might be seen as an advantage too -- it is clear that a symbol is a dependency because it is off the R. function.

3) Makes it hard to find typos for properties on the R. function. With the current runjs format, JSLint can actually find typos for a dependency's variable name.

4) This syntax is more verbose for JS files that do not care about scope encapsulation. For JavaScript libraries like jQuery, MooTools and Prototype, they pretty much operate in the same global scope. jQuery does have a noConflict(), but that just helps if there is only one other thing called $ in the page. MooTools and Prototype add things to global prototypes.

So for these libraries, specifying dependencies is more like just specifying script tags, and they do not need a local-scoped variable in the function module to get things defined. They would just need to do the following:

run.def("rdw/Message",
["some/module", "something/else", ...]
function() {
//This function does not need some locally scoped variables, and most
//likely, the script dependencies above may not call run.def() and define
//an object, just add things to the global space
});

So for these libraries, it would be onerous to be forced to create variable names for each of those dependencies.

This last point seems to be enough to tip the scales back to using the current model used by run. However I am aware that it is possible for the developer to not get the order or number correct, matching the dependency string with the correct variable for the function.

I am hoping using a coding standard like the following will help:

run.def("rdw/Message",
["rd", "dojo", "rdw/Base"], function(
 rd,   dojo,   Base) {
 //Define the module for rdw/Message and return it.
});

Basically, make sure all dependencies are on one line, along with the function keyword, then put the variable that matches each dependency aligned directly under the depdendency name (the example above may not be correctly aligned depending on the font or format you are viewing this message).

I purposely trimmed the list of dependencies, so I can get this example to show up in this blog. That is the down-side with this sort of code style: it can have a very long line for the dependency names.

For Raindrop, I want to try to keep local scope encapsulation, particularly since I expect some slicker extensions to it that may introduce other code into the page that may conflict. However, if I did not care to do that, I could shorten up the above example quite a bit.

So at this point, I am favoring making it easy to use RunJS for other toolkits that do not care about local scope encapsulation and detecting bad references to variables inside the module definition over avoiding errors with a mismatched function variable name to a dependency name.

It is a hard choice to make. Neither path is perfect. If you have an opinion, feel free to share it.

Raindrop + JSLint + JSDoc + RunJS

I just pushed some changes into the front-end code for Raindrop. Besides using JSLint as the code style, and JSDoc for the format of the files, it now uses a modified version of Dojo 1.4 that uses the RunJS code loader.

I am fortunate enough to work with people that will tolerate this sort of brief experimentation to help improve the state of browser code loading. Do you know Mozilla Messaging is hiring? Great people, great technology, and a great mission. OK, end of the sales speech.

The code conversion to JSLint, JSDoc and a new loader all at one time took longer than I would have liked, but I am glad I did it.

For one thing, the code looks a lot more uniform thanks to using JSLint. While I do not care for all of the rules mandated by JSLint, no one ever likes a coding standard 100%, and JSLint is widely known and a programmatic code checker.

Converting Raindrop to RunJS was really beneficial for RunJS. While RunJS has unit tests, nothing helps shake out the kinks like a large project. RunJS is now more robust because of it.

If you want to peruse the Raindrop code, you can hop to the Raindrop Mercurial source web view. That link will take you to Raindrop's client/lib/rdw directory, where we keep the UI widgets. You can click on some files to see how they look with the RunJS loader and the new formatting.

One thing that became apparent, particularly when talking to David Ascher about the code: the way the dependencies in RunJS are specified as an array, and having a separate list of function arguments that must match that array might be prone to errors.

Here is an example from rdw/Message:

run("rdw/Message",
["run", "rd", "dojo", "rdw/_Base", "rd/friendly", "rd/hyperlink", "rd/api",
"text!rdw/templates/Message!html"],
function (run, rd, dojo, Base, friendly, hyperlink, api, template) {
 ...
});

I condensed the run boilerplate to the top to three lines, otherwise, it would have been quite long vertically to express it all. But you can see the problem. Looks scary.

David and I talked about it. I talked about how YUI uses just a Y object as the only argument to the function, but I did not like how they use odd module names like "yui-anim", and then you have to know that creates a Y.anim property that you can use in your function.

So David suggested the following:

run("rdw/Message", {
 rd: "rd",
 dojo: "dojo",
 Base: "rdw/_Base",
 friendly: "rd/friendly",
 hyperlink: "rd/hyperlink",
 api: "rd/api",
 template: "text!rdw/templates/Message!html"
}, function(R) {
 //Use things like R.api and R.Base in here that map to the modules up above.
 ...
});

I like this better. It is much clearer what module gets assigned to what variable, no "off by one" errors.

The downside: it does not minify as nicely as the existing run format. However, in this case, avoiding developer errors probably trumps the minification cost, and the extra cost to type R. in front of the dependent module references. You can call it r. if you want. Hmm, I might prefer the lower-case version. Easier to type. Although the capital stands out better.

So I am very tempted to convert RunJS to use this format for listing dependencies and passing them to the function callback. I will likely do the work this weekend.

Update: I decided not to do this, favoring the original format since it has other advantages.

Another thing we talked about was the multiple invocations of run(): run() can be called with a starting string argument to define a module, but then can also be called without that starting string argument if you just want to run some code that has some dependencies.

David felt it would be clearer if those two actions were two different function names. I can see that being clearer. So what about the following for the API:

• run(), for just running code that does not define a module.
• run.def() for defining a module.
• run.mod() for a module modifier (the existing run.modify() API call)
• run.get() for getting a defined module after initial module evaluation (needed for some some dynamic cases where you do not know the module name before-hand and in circular dependency cases). This is an existing API.

So the changes are renaming run.modify() to run.mod() and moving the run() call style that defined a module to run.def().

Thanks to David for the feedback, and for the rest of the Raindrop team for being patient with me as I did some experimentation in browser module loading.

Speaking of feedback on RunJS, Rawld Gill (one of the authors of Mastering Dojo) was already working on a loader, and he did some work to convert it to use the same API as RunJS. I am still processing the results, but it is great to see alternate implementations. I am hoping we can take the best of both and make a great loader. I already changed the module evaluation algorithm to use the recursive style he suggested.

RunJS takes its roots from the cross-domain Dojo loader, and that loader was constructed when Dojo had to support Safari 2. Safari 2 has a ridiculously small call stack, so a recursive module evaluator blew up very easily. I switched to a model that traced the dependencies to work out an array of sequenced modules, then a separate loop to then call them into being. That allowed the loader to work in Safari 2, but now that we do not need to support that browser, the more natural, straight-forward recursive model can be used.

Really neat stuff! Feels like we are getting close to a robust but compact loader.

RunJS: GitHub, build options, features, file sizes

A few updates on RunJS, a JavaScript file/module loader (see the README for more documentation):

• RunJS is now on GitHub
• Plugins for RunJS are supported. i18n bundles have been pulled out as a plugin, and a new text plugin allows you to set text files (think HTML/XML/SVG files) as dependencies for a module. The plugin will use async XMLHttpRequest (XHR) to fetch those files and will pass the text of those files as an argument to a module's module definition function. The RunJS build system will then *inline* those text files with the module, so that the XHR calls be removed in deployed code, and allow cross-domain use of those text files.
  
• Any function return type is allowed from the module definition function. Before only objects and functions were allowed and functions had to be called out in a special way. Now, that special call out is removed and any return type is allowed. The cost was an extra call, run.get() that needs to be used in circular dependency cases. See the Circular Dependencies section in the README.
  
• The build system that comes with RunJS now supports build pragmas.

The build pragma support was used to build RunJS in a couple of different configurations. I am trying to get a handle on where the bulk of implementation lies, and what features add to its file size. Here is the breakdown (warning, Google Doc iframe inclusion, but interesting numbers inlined in this post after the iframe):



Let's look at the non-license sizes, since they give a better indication of code density. Google's Closure Compiler did the minification for this evaluation.

The normal config, with no plugins included (but with plugin support) is 7,970 bytes minified, 3,167 gzipped. Including both the i18n and text plugins with run.js bumps it up to 11,759 minified, 4,655 gzipped.

The interesting number for me is the version of run.js without plugin support, no run.modify, no multiversion support and no page load (run.ready/DOMContentLoaded callbacks). This version of run has just the following features:

• support for the run() module format
• nested dependency resolution
• configure paths to modules
• load just plain .js files that do not define run.js modules (scripts that do not call run(), for example jQuery, or plugins for jQuery).

That bare bones loader comes in at 5,086 minified and 2,204 gzipped. The one you should use, the one with the license, is 5,245 minified and 2,317 bytes gzipped. I need to work on the size of that license block!

That size could probably be brought down a tiny bit (probably reaching the 2,000 gzip size) if I were to really be aggressive and remove all context references, but that would be a mess to maintain and there would be no easy upgrade path to multiversion support.

I believe that is the lower limit a functional loader that does nested dependencies via run() module calls. I view run.ready/DOMContentLoaded support more of a necessity for a loader, so unless you already had an implementation for that, I suggest the version that has run.ready() support, which comes in (with license) at 5,867 minifed, 2,522 gzipped.

The nice thing about the build pragma setup for RunJS, you can upgrade run without having to change your code if you find you want more features, like plugin support, or i18n/text dependency support via plugins.

I am interested in trying to sell more front-end JavaScript toolkits on this loader. For some, I can see the bare-bones 2.3K gzipped loader a nice way to step into it, and then their users have the option to swap out a more powerful version via a different RunJS build output.

I have put up the different build outputs for 0.0.6 if you want to grab one of the minified versions and play with it. Here is the minimum set of compliance tests which use the smallest loader (no modify/plugins/page load/context support) mentioned above. See the README for documentation.

Right now I believe around 2KB gzipped is close to the lower bound for a stand-alone code loader in the browser. At least for a loader I would consider using: anything that uses XHR and eval are dead to me. Using plain script src="" tags helps the xdomain case, and just fits better with debugging. While Dojo has used an XHR-based loader for quite a while (and it will continue to be supported), it just does not work as well with the browser as a script-tag based loader. Any loader should also do nested dependency loading too -- if a module in a script has dependencies in other modules, be sure to evaluate the dependencies in the right order.

As a point of comparison, consider LABjs. I feel a kinship with the author of LABjs, Kyle Simpson, even though we have never talked. We are both focusing on efficient code loading in the browser. I recommend LABjs if it fits your style.

While LABjs does not quite do nested dependency resolution, it does something related where you can tell it to wait to load a script before continuing to load other scripts. LABjs is not trying to push a module format like run is, but targeted more at existing code that does not have the concept of a module format.

By the way, RunJS can also handle loading these types of files. Where LABjs has a wait() call for holding off loading scripts that depend on another script being loaded (like a framework), RunJS uses nested run calls.

Example from the LABjs page:

$LAB
.script("framework.js").wait()
.script("plugin.framework.js")
.script("myplugin.framework.js")
.wait(function(){
    myplugin.init();
    framework.init();
    framework.doSomething();
});

Equivalent example with RunJS:

run(["run", "framework.js"],
    function(run) {
        run("plugin.framework.js", "myplugin.framework.js"],
            function() {
                myplugin.init();
                framework.init();
                framework.doSomething();
            }
        );
    }
);

Taking the 1.0.2rc1 version of LABjs and using Closure Compiler on it (without the license) gives LABjs a size of 4,360 bytes minified and 2,170 gzipped. As a reminder, the equivalent RunJS file is 5,086 minified and 2,204 gzipped. I may be able to do better with making the structure of the RunJS code more amenable to minification, but the gzip sizes come up fairly close. I do not believe the code tricks I would do to help minification will help the gzip size any.

Both LABjs and RunJS end up around 2KB gzipped. So, about 2KB gzipped seems close to the lower limit on a standalone loader, one that uses script tags/plays nice with the browser and can do nested dependencies. I would like to be proven wrong though, and ideally by modifying RunJS to fit that lower limit. :) I am sure the code can be improved.

But remember the guidelines, no goofy XHR stuff/something that works well with the browser and can handle nested dependencies. No script tags with inlined source/eval tricks. Even though Firefox and WebKit make eval debugging easier, it is still not as nice as regular script src tags.

Irakli Gozalishvili believes web workers might help, but I do not see it. The workers are restricted to message passing, and anything interesting in a web browser will likely need to touch the DOM, so a web worker solution will just be another async-XHR-like approach, where you will need to eval the scripts or inline-script inject to get all the scripts for them to see each other and the DOM.

Irakli does have an async-XHR based loader for CommonJS modules. As of today, it comes in at 1,527 minified, 838 gzipped (license not included). But it uses XHR, so limited to the same domain as the page, and debugging support is just not as nice across browsers. It also uses CommonJS module syntax, but I have decided CommonJS modules do not play well out of the box in the browser, and I believe the format's "module", "exports", and "require.main" parts are unnecessary.

Dojo 1.4 Favorite Features

Dojo 1.4 is out! There is a metric ton of changes. Here are some of my favorite things about the release. I focus mostly on Dojo Core, and mostly in the non-animation parts of it, so my list is skewed for that focus. However, there are lots of other changes, some in the animation functionality, and in Dijit and Dojox. Check out the 1.4 release notes to get a more complete picture.

One of the things I want to do for Dojo Core is to bring the DOM APIs, particularly the methods on dojo.NodeList (the return object for dojo.query() calls, Dojo's CSS selector method) more in-line with what is available in jQuery. jQuery has demonstrated that its APIs resonate strongly with developers. Where it makes sense and fits Dojo's philosophy, we should also provide those APIs, to make it easier for developers. These Dojo 1.4 changes reflect that goal:

• dojo.ready(), just an alias for dojo.addOnLoad().
• dojo.NodeList-traverse: A helper module that adds methods to dojo.NodeList. Its goal is to bring in some methods to NodeList that exist in jQuery for DOM traversal, specifically: children, closest, parent, parents, siblings, next, nextAll, prev, prevAll, andSelf, first, last, even, odd.
• dojo.NodeList-manipulate: A helper module that adds methods to dojo.NodeList. Its goal is to bring in some methods to NodeList that exist in jQuery for DOM manipulation, specifically: innerHTML, html, text, val, append, appendTo, prepend, prependTo, after, insertAfter, before, insertBefore, remove, wrap, wrapAll, wrapInner, replaceWith, replaceAll, clone.
  
• IO pipeline topics: get notifications of IO events via dojo.subscribe/dojo.publish. Handy for putting up a generic "loading" indicator when any sort of IO call happens. These topics are not strictly how jQuery exposes this functionality, but we can leverage the power of dojo.publish/subscribe to implement this feature.

Some other new Dojo Core 1.4 features that are really sweet:

• dojo.cache(): allows you to reference external HTML files and use them as if they are strings. It is integrated into the build system, so you can avoid the XHR calls to get the external text files by just doing a build. No extra build option is needed. This is a great way to construct HTML -- by writing plain HTML instead of building awkward strings in code or using JS DOM-building calls, which can obscure what the HTML actually looks like.
  
• dojo.position(): A faster, more understandable replacement for dojo.coords(). If you were using dojo.coords() before, odds are good that you probably want to switch to dojo.position(). Douglas Hays stepped up and put in this great new method.
  
• dojo.declare(): It is faster and more robust. Many thanks to Eugene Lazutkin for doing this work. It took a lot of patience and perseverance to get this new version up to snuff and keep it backward compatible.
• dojo.hash(): An easy way to set the URL hash (fragment ID) and to watch changes to the hash. This allows you to create pages that reflect the proper state as shown by the URL in the browser. This was a contribution from community member Rob Retchless and other IBM Jazz team members.

For the build system, support was added for Google's Closure Compiler, so you can experiment with using it for minifying your code. Right now we just support the "simple" minification done by Closure Compiler, not the advanced features.

It was a little while coming, but it is great to have Dojo 1.4 out. Thanks to the community for making the toolkit better!

JavaScript module loading, the browser and CommonJS

JavaScript module syntax and loading seems like a hot topic at the moment, and here are some thoughts about how to construct module syntax and a loader, with the goal of trying to get to a more universal approach for it. This will be discussed in the context of CommonJS, but browser-based module loaders have existed for a while. All are constrained by the browser in some fashion as listed below. I have a preferred solution, also described in this post.

First, a look at module syntax.

CommonJS is an umbrella for a few different things, including a spec for a module syntax and a standard library of modules. This post is just interested in its module syntax. A simple example of CommonJS syntax, defining an "increment" module defined in increment.js:

var add = require('math').add;
exports.increment = function(val) {
       return add(val, 1);
};

How could we build a module loader with this syntax? Here are a couple of options:

1) You parse the module before executing it, looking for require calls. You make sure to fetch those modules and work out the right dependency order in which to execute the modules.

2) You just run the module and when require() is hit, do an synchronous IO operation to load the required module.

For both approaches, use a sandbox or specific context to make sure things like "exports" are defined separately for each module.

Both of those options are easy to implement on the server side since you have more control over the IO layer, and can create separate contexts for each module. However, in the browser, things are different. Creating a separate context for each module is tricky. For IO, there are two realistic approaches, and each has its difficulties:

• XMLHttpRequest (XHR)
• script tags

XHR allows us to do either approach, #1 or #2. It can get the contents of the module and parse it into a structure that pulls out the dependencies and we can sort out the right order to execute things. We could use sync XHR calls to accomplish #2, block when each require call is seen. However, sync XHR calls in the browser really hurt performance.

This is actually what the default Dojo loader has done for a very long time, and I believe some pathways in the Google's Closure library do the same thing. It is always recommended you do a custom build to combine all the modules you need into one file to cut out those XHR calls when you want to go to production.

So path #1 would make more sense with an XHR-based loader. However, for an XHR loader to work, it has to use eval() to bring the module into being. Some environments, like Adobe AIR do not allow eval(), and it makes debugging hard to do across browsers. Firefox and WebKit have a convention to allow easier eval-based debugging, but it is still not what I consider to be in keeping with traditional script loading in a browser.

Instead of eval, after the XHR call finishes its parsing and module wrapping for context, you could try to create a script tag that has a body set to the modified module source, but this really hurts debugging: if there is an error, the error line number will be some weird line in a gigantic HTML file instead of the line number of the actual module.

Dojo has a djConfig.debugAtAllCosts option that will use sync XHR to pull down all the modules, parse the for dependencies, work out the right load order, then load each module via a dynamically added script src="" tag. However, since IE and WebKit will evaluate dynamically added script tags out of DOM order -- they evaluate them in network receive order (which is nice for long-polling comet apps, but does not help module loading). So, each script tag has to be added one at a time, then wait for it to finish then add the next one. Not so speedy.

XHR is also normally limited to just accessing the same host as the web page. This makes it hard to use CDNs to load content, and get performance benefits with that approach. There is now support for xdomain XHR in most recent browsers, but IE prefers to use a non-standard XDomainRequest object, making our module loader more complicated. And xdomain XHR just plain does not work in older browsers like IE6.

So, an XHR-based loader is not so great.

Script tags are nice because they keep with the known script pathway in browsers -- easy to debug, and we can get parallel loading. However, we cannot do approach #2 in the browser: our JavaScript in the browser cannot access the module contents before they are evaluated. And since dynamically added script src="" tags via head.appendChild() are not a synchronous operation, approach #2 will not work.

So, really we need to do a variant of #1, pull out the dependencies needed by the module, then after those dependencies are loaded, execute the module. The way to do this in script: put a function wrapper around the module contents, and call a module loader function with a list of dependencies and the module function wrapper. Something like this, for a module with the name of "c" that has dependencies of "a" and "b", Here is a syntax (call it Variant A) for defining a module "c" with this approach:

loader(
       "c",
       ["a", "b"],
       function(a, b) {
              //The module definition of "c" in here.
              //return an object to define what "c" is.
              return {};
       }
);

or, another variant, call it Variant B:

loader({
       name: "c",
       dependencies: ["a", "b"],
       module: function(a, b) {
              //The module definition of "c" in here.
              //return an object to define what "c" is.
              return {};
       }
});

Ideally, we would not have to tell the loader that this structure defines module "c" (the first arg in Variant A and the name: property in Variant B) -- the loader could work this out. Unfortunately, since script tags can load asynchronously and at least IE can trigger script.onload events out of order when compared to when the script is actually evaluated, we need to keep the module name as part of the module definition. This also helps with custom builds, where you can combine a few of these module definition calls into one script.

This approach is actually what Dojo's xdomain loader has done for a very long time, but with more verbose syntax. However, it requires a custom build to convert modules into this structure. The other option is to use a server-side process to convert the modules on the fly, but I do not feel that is keeping with the simplicity of normal browser development: just open a text editor, write some script, save, reload, no extra server config/process needed, besides maybe a vanilla web server.

So, I believe that modules should be coded by the developer in this module wrapper format. YUI 3 has taken this approach, and it is the approach I have taken for RunJS too. However, YUI 3 is limited to needing some module dependency metadata files to help it out. It also uses module names that do not map to the actual module's defined name/functions.

OK, back to CommonJS.

As it stands now, I believe the CommonJS format is not suitable for modules in the browser. There have been attempts to get it to work, but the attempts either use a sync XHR loader, or a "transform-on-the-fly" server process to convert the code to a module wrapper similar to Variant B.

I would rather see a module wrapper format that works with browser natively, that can be hand-authored by developers and that will work with CommonJS modules. CommonJS started out as ServerJS. As ServerJS, the case could be made that supporting browsers may not be an aim of a ServerJS module format. However, with the name change to CommonJS, I believe supporting browsers as a first class citizen is important for CommonJS to get more traction.

So the trick is to come up with a module syntax that has a function wrapper, but is not too wordy with boilerplate. We need some boilerplate, since we need a function wrapper. I believe RunJS has the right right approach. The boilerplate is very terse, basically Variant A mentioned above:

run(
    "c",
       ["a", "b"],
       function(a, b) {
              //The module definition of "c" in here.
              //return an object to define what "c" is.
              return {};
       }
);

I can see where there is some bikeshedding on the name "run". I think script() instead of run() is a viable alternative, and I may switch to that in the near future (and rename RunJS to ScriptJS).

I have attempted to engage the CommonJS community by putting up a proposal for an Alternate Module Format.

Progress has been slow, but to be expected: the CommonJS group is trying to do lots of other things like define a standard library and build out implementations. However, I am hopeful we can get something that works for the browser front end developers.

The ideal scenario is that some variant of the above syntax is just adopted as the only CommonJS module format. That would save a lot of conversion work, and I believe it makes things much simpler for CommonJS compliant loader. Right now, for CommonJS loaders there is a concept of a require() and require.async() and having to expose Promises for the async stuff. The above format neatly avoids the issue of whether the modules are loaded async or sync and avoids any need for Promises in the module loader. I think it is fine though for modules themselves to use Promises as part of individual module APIs, but at least the loader and module syntax stays simple.

I also do not believe a "module" variable needs to be defined for each module and an exports variable is avoided by returning an object from the module function wrapper.

I can appreciate that the CommonJS folks with modules already written may not like moving to the above syntax. I think it helps in the long run if we can just have one syntax, but in the meantime, I plan on doing the following:

• Continue to engage the CommonJS community.
• build out RunJS, probably rename to ScriptJS in the near future, and use script() instead of run()
• Write a converter that converts Dojo modules to the RunJS/ScriptJS module syntax. I have something basic working, here is an example of Dojo's themeTester.html using RunJS-formatted dojo/dijit/dojox modules. That example is not bulletproof yet (I used a built version of Dojo which removes some dependency info) and i18n modules have not been converted either. RunJS also has built-in support for i18n modules.
• Convert Raindrop to use RunJS-formatted dojo and convert the Raindrop modules to that format.
• Override run.load()/script.load() in server environments so it could be used in CommonJS server implementations.
• Work on a converter for existing CommonJS modules.
• Use RunJS/ScriptJS as the module syntax for Blade and/or Dojo 2.0 efforts.

If module syntax/loading is important to you, then please join the discussion list for CommonJS, so we can sort this out. It would be great to get consensus on JavaScript module syntax and loading, and I think CommonJS is the area to do that.

I am happy to adjust some of the syntax in RunJS/ScriptJS to match some consensus, but I strongly prefer a terse format. The existing ones I have seen for server-converted CommonJS modules is too verbose for me, particularly for the common cases of defining a module with some dependencies.

Raindrop, CouchDB and data models

Raindrop uses CouchDB for data storage. We are starting to hit some tough issues with how data is stored and queried. This is my attempt to explain them. I am probably not the best to talk about these things. Mark Hammond, Raindrop's back-end lead is a better candidate for it. I am hoping by trying to write it out myself, I can get a better understanding of the issues and trade-offs. Also note that this is my opinion/view, may not be the view of my employer and work colleagues, etc...

First, what are our requirements for the data?

• Extensible Data: we want people to write extensions that extend the data.
  
• Rollback: we want it easy for people to try extensions, but this means some may not work out. We need to roll back data created by an extension by easily removing the data they create.
  
• Efficient Querying: We need to be able to efficiently query this data for UI purposes. This includes possibly filtering the data that comes back.
  
• Copies: Having copies of the data helps with two things:
  
  • Replication: beneficial when we think about a user having a Raindrop CouchDB on the client as well as the server.
  • Backup: for recovering data if something bad happens.

How Raindrop tries to meet these goals today

Extensible Data: each back-end data extension writes a new "schema" for the type of data it wants to emit. A schema for our purposes is just a type of JSON object. It has a "rd_schema_id" on it that tells us the "type" of the schema. For instance a schema object with rd_schema_id == "rd.msg.body" means that we expect it to have properties like "from", "to" and "body" on it. Details on how schemas relate to extensions:

• An extension specifies what input schema it wants to consume, and the extension is free to emit no schemas (if the input schema does not match some criteria), or one or more schemas.
• Each schema written by an extension is stamped with a property rd_schema_provider = "extension name".
• All the messages schemas are tied together via an rd_key value, a unique, per-message value. Schemas that have the same rd_key value all relate to the same message.

More info is on the Document Model page.

Rollback: Right now each schema is stored as a couch document. To roll back an extension, we just select all documents with rd_schema_provider = "extension name" that we want to remove, and remove them. As part of that action, we can re-run extensions that depended on that data to have them recalculate their values, or to just remove the schemas generated by those extensions.

Having each schema as a separate document also helps with the way CouchDB stores data -- if you make a change to a document and save it back, then it appends the new document to the end of the storage. The previous version is still in storage, but can be removed via a compaction call.

If we store all the schemas for a message in one CouchDB document, then it results in more frequent writes of larger documents to storage, making compaction much more necessary.

Efficient Querying: Querying in CouchDB means writing Views. However, a view is like a query that is run as data is written, not when the UI may actually want to retrieve the information. The views can then be very efficient and fast when actually called.

However, the down side is that you must know the query (or a pretty good idea of it) ahead of time. This is hard since we want extensible data. There may be some interesting things that need to be queried later, but adding a view after there are thousands of documents is painful: you need to wait for couch to run all the documents through the view when you create the view.

Our solution to this, started by Andrew Sutherland and refined by Mark, was to create what we call "the megaview". It essentially tries to emit every piece of interesting data in a document as a row in the view. Then, using the filtering capabilities of CouchDB when calling the view (which are cheap), we can select the documents we want to get.

Copies: While we have not actively tested it, we planned on using CouchDB's built-in replication support. This was seen as particularly valuable for master-master use cases: when I have a Raindrop CouchDB on my laptop and one in the cloud.

Problems Today

It feels like the old saying, "Features, Quality or Time, pick two", except for us it is "Extensible, Rollback, Querying or Copies, pick three". What we have now is an extensible system with rollback and copies, but the querying is really cumbersome.

One of the problems with the megaview: no way to do joins. For instance, "give me all twitter messages that have not been seen by the user". Right now, knowledge of a message being from twitter is in a different schema document than the schema document that knows if it has been seen by the user. And the structure of the megaview means we can really only select one property at a time on a schema.

So it means doing multiple megaview calls and then doing the join in application code. We recently created a server-side API layer in python to do this. So the browser only makes one call to the server API and that API layer does multiple network calls to CouchDB to get the data, then does the join merging in memory.

Possible solutions

Save all schemas for a message in one document and more CouchDB views
Saving all schemas for a message in one document makes it possible to then at least consult one document for both the "type=twtter, seen=false" sort of data, but we still cannot query that with the megaview. It most likely means using more CouchDB views to get at the data. But views are expensive to generate after data has been written. So this approach does not seem to scale for our extensible platform.

This approach means taking a bit more care on rollbacks, but it is possible. It also increases the size of data stored on disk via Couch's append-only model, and will require compaction. With our existing system, we could consider just never compacting.

This is actually the approach we are starting to take. Mark is looking at creating "summary documents" of the data, but the summary documents are based on the API entry points, and the kind of data the API wants to consume. These API entry points are very application-specific, so the summary document generation will likely operated like just another back end extension. Mark has mentioned possibly just going to one document to store all schemas for a message too.

However, what we have not sorted out how to do is an easier join model: "type=twitter and seen=false". What we really want is "type=twitter and seen=false, ordered by time with most recent first". Perhaps we can get away with a small set of CouchDB views that are very specific and that we can identify up-front. Searching on message type and being seen or unseen, ordered by time seems like a fairly generic need for a messaging system.

However, it means that the system as a whole is less extensible. Other applications on the Raindrop platform need to either use our server API model of using the megaview then doing joins in their app API code (may not be so easy to learn/perform), or tell the user to take the hit waiting for their custom views to get up to date with all the old messages.

Something that could help: Make CouchDB views less painful to create after the fact. Right now, creating a new view, then changing any document means waiting for that view to index all the documents in the couch, and it seems to take a lot of resources for this to happen. I think we would be fine with something that started with most recent documents first and worked backwards in time, using a bit more resources at first, but then tailing off and doing it in the background more, and allow the view to return data for things it has already seen.

Do not use CouchDB
It would be very hard for us to move away from CouchDB, and we would likely try to work with the CouchDB folks to make our system work best with couch and vice versa. It is helpful though to look at alternatives, and make sure we are not using a hammer for a screwdriver.

Schema-less storage is a requirement for our extensible platform. Something that handles ad-hoc queries better might be nice, since we basically are running ad-hoc queries with our API layer now, in that they have to do all the join work each time, for each request.

Dan Goldstein in the Raindrop chat mentioned MongoDB. Here is a comparison of MongoDB and CouchDB. Some things that might be useful:

• Uses update-in-place, so the file system impact/need for compaction is less if we store our schemas in one document are likely to work better.
• Queries are done at runtime. Some indexes are still helpful to set up ahead of time though.
• Has a binary format for passing data around. One of the issues we have seen is the JSON encode/decode times as data passes around through couch and to our API layer. This may be improving though.
  
• Uses language-specific drivers. While the simplicity of REST with CouchDB sounds nice, due to our data model, the megaview and now needing a server API layer means that querying the raw couch with REST calls is actually not that useful. The harder issue is trying to figure out the right queries to do and how to do the "joins" effectively in our API app code.

What we give up:
1) easy master-master replication. However, for me personally, this is not so important. In my mind, the primary use case for Raindrop is in the cloud, given that we want to support things like mobile devices and simplified systems like Chrome OS. In those cases it is not realistic to run a local couch server. So while we need backups, we probably are fine with master-slave. To support the sometimes-offline case, I think it is more likely that using HTML5 local storage is the path there. But again, that is just my opinion.

2) ad-hoc query cost may still be too high. It is nice to be able to pass back a JavaScript function to do the query work. However, it is not clear how expensive that really is. On the other hand, at least it is a formalized query language -- right now we are on the path to inventing our own with the server API with a "query language" made up of other API calls.

Persevere might be a possibility. Here is an older comparison with CouchDB. However, I have not looked in depth at it. I may ask Kris Zyp more about it and how it relates to the issues above. I have admired it from afar for a while. While it would be nice to get other features like built-in comet support, I am not sure it will address our fundamental issues any differently than say, MongoDB. It seems like an update-in-place model is used with queries run at runtime. But definitely worth more of a look.

Something else?

What did I miss? Bad formulation of the problem? Missing design solution with the tools we have now?

Blade, a JavaScript toolkit experiment

I am playing around with a different way (at least for me) to construct a JavaScript toolkit. It is called Blade, and you can follow it via the Blade GitHub repo.

I have had this on my local drive for a few weeks now, and the germ of it started with this post. I wanted to get it more polished, but best to get it up somewhere to get some feedback at least on the principles.

There is not much there now, basically a tiny amount of spaghetti code that is not really usable. However, I list out the guiding principles in the README.md, visible on the GitHub source tab.