RunJS is now RequireJS

As mentioned before, I considered renaming RunJS to RequireJS. I did the transition, and RequireJS is on GitHub. There is a conversion script that will convert CommonJS modules to the Transport/C proposal that works with RequireJS.

I have converted Raindrop to use a modified Dojo 1.4 that uses RequireJS instead of the normal Dojo loader, and all Raindrop modules are written in the Transport/C module format that RequireJS understands. Raindrop works, so the RequireJS code has been proven in a real project that has many modules with nested dependencies. The RequireJS code is already battle-tested.

I have opened a thread on the jQuery forum about using RequireJS for jQuery's require() needs. I can do a build with Dojo 1.4 that uses RequireJS, and any Dojo 2.0 effort is likely to use RequireJS as the module loader.

I believe RequireJS is the loader browser-based toolkits should use. At the very least, the module format and API supported by RequireJS should be used by browser-based toolkits, even if they want to build their own loader.

Next plans for RequireJS:

1) Contact the MooTools and Prototype folks to see if they want to use it. It allows loading code that does not export a module value, and has access to the global environment. They can use it to load code that augments native prototypes. RequireJS can load existing, plain JS files that do not define a module too.

2) Put up a web site with builds. While you can use RequireJS just from grabbing it from GitHub, it would be nice to have the builds of RequireJS with its different levels of functionality already built and easy to download.

2) Do a fork of Node that uses RequireJS on the server. I believe the async module format used by RequireJS is great fit for Node and its async goals.

3) See if I can do a fork of Narwhal to do the same thing.

I believe I can get RequireJS to work on the server and still support the existing CommonJS format when the server supports synchronous loading. By having native support in some server-based systems for RequireJS, it will be easier to share code with the browser.

To recap, my three main issues with using the existing CommonJS module spec, and why RequireJS exists:

1) So far the CommonJS group does not think the browser is common enough to qualify as a first class citizen in the module spec. The group is mainly concerned with environments outside the browser. As a result, the CommonJS module spec does not work well natively in the browser -- it either requires an XHR-based loader, which we have found to have problems in Dojo, or require a server-side transform process. A server-side transform process should not be required to do web development in the browser.

RequireJS uses a function wrapper around the module to avoid these problems and allow loading modules via script tags. Just save the file and hit reload in the browser.

2) There is a free variable, called "exports". It is an object. You cannot set the value of exports inside your module code, you can only add properties. This means for instance, your module cannot be a function. In Dijit, Dojo's widget system, all widgets are constructor functions. The "exports" restriction makes your APIs awkward if you want to export functions for module values. The claim is that this exports restriction helps with circular dependencies, but it only helps a little bit. To me, it is not worth slightly improving an edge case when it sacrifices a greater usefulness and simplicity in user's modules.

RequireJS and its format can handle circular dependencies just fine. In the format supported by RequireJS, you to define a module as a function. Although, you can still use exports as CommonJS uses it if you so desire.

3) The require.main property seems like a hack. It is normally used so that a module can say if (require.main === module.id) or if (require.main === module) then do some "main" work. The module format should just define an exports.main convention for indicating "main" functionality. It is less typing, and more robust, since different code entry points have a different idea of "main". For instance, an HTTP request handler likely has specific requirements on what it considers to be the "main". The top level entry point should decide what code to execute as "main", not logic inside the module.

RequireJS does not support the require.main idiom.

So I believe the path used by RequireJS is more robust overall, works better/scales better in the browser. However, I still want to provide enough support for the existing CommonJS modules in the meantime to allow more code sharing.

In the long run though, the CommonJS format as it exists today should be replaced with something better. It is troublesome that the CommonJS group is not really targeting the browser, but over time, the broader JS community will expect browser toolkits to support CommonJS specs. It does not seem right to end up with non-optimal solutions in the browser when the browser is the most common JS platform.

RunJS to RequireJS?

There was a thread that started on the CommonJS list about a transport format, something that works well in the browser via script injection. I sketched out a proposal, Transport/C, that builds on the Transport/B and Transport/A specs.

Transport/C is very similar to some basic mechanics of RunJS but uses require() as the top level function, and supports the special "module" and "exports" free variables used in the normal CommonJS module spec.

In order to prove the concept for Transport/C, I made a branch of the RunJS code, calling it RequireJS, that implements Transport/C.

It seems like it fits with the existing CommonJS module spec, but is something that works well in the browser. I also made a simple conversion script that converts traditional CommonJS modules to this format.

I am tempted to convert from RunJS to this RequireJS branch, and to start evangelizing that approach for browser toolkits. It would be great if Transport/C would also be approved as the transport format for CommonJS too.

Kris Kowal has some concerns about the *very* long-term effects of the approach. I read his comments as possibly pointing out some things that would be done differently if the primordials and e-maker type of modules were ever accepted as part of an ECMAScript standard.

As I read the primordials and e-maker strawman proposals, I think the only difference is Transport/C functions are only expected to be called once, but e-maker style would favor calling the function on every require() call. As I say in my response, I believe e-maker support, would affect regular CommonJS modules in the same way as the transport format, and it is assuming the strawmans make it in to the spec at some point, as they are specified now.

I also believe how it works in how I coded Transport/C as part of the RequireJS branch is what a normal developer would expect, and I think fits better with existing browser/script behavior, and the assumptions that go along with coding CommonJS modules today.

So I am tempted to rename the RunJS project to RequireJS and proceed with that. If you have any feedback to the contrary, please let me know. Otherwise, I will likely do the change early this week.

Script async, Raindrop and Firefox 3.6

In honor of the Firefox 3.6 release, I upgraded Raindrop to use the new async attribute for script tags.

Why is async neat? It does not block the rest of the page, and will just evaluate the script once it is retrieved. More information is in the HTML5 spec. Note that the script you add async to should NOT use document.write(), as doc.write will likely destroy your page.

Also, be aware that async is a boolean attribute, but that does not mean you should use async="true" to turn it on. The HTML5 spec on boolean attributes says that a value of empty string or a string that matches the attribute name should only be used. To avoid async, just do not include the attribute. For Raindrop, I used async="async" since that looks better to me than an empty string.

Raindrop uses RunJS for the module loader, and RunJS uses dynamically added script tags via head.appendChild(), so the modules loaded by RunJS already behave in an async manner.

However apps that want to use the raindrop front end libraries normally include a script called rdconfig.js as their own script file, and that config file does a document.write to write out the Dojo+RunJS and jQuery tags. Those Dojo+RunJS and jQuery tags now use the async attribute.

RunJS Dependency API

In my last post, I talked about a suggestion from David Ascher to try to improve the syntax of specifying dependencies when declaring a module. Here is an example of that suggestion, using run.def(), a possibly new API dedicated to just defining a module called "rdw/Message":

run.def("rdw/Message", {
rd: "rd",
dojo: "dojo",
Base: "rdw/_Base",
friendly: "rd/friendly",
hyperlink: "rd/hyperlink",
api: "rd/api",
template: "text!rdw/templates/Message!html"
}, function(R) {
//Module definition function.
//Use things like R.api and R.Base in here that map to the modules up above.
...
});

While that does make it clear what each module name's variable will be inside the function that defines the module, it has the following drawbacks:

1) It hurts minification -- having properties off the R object passed to the module function means that minification tools will not be able to minify those references as easy.

2) All the modules dependencies must be referenced via a prefix "R.". For example R.api. It is a small bit of typing and an extra property lookup. That might be seen as an advantage too -- it is clear that a symbol is a dependency because it is off the R. function.

3) Makes it hard to find typos for properties on the R. function. With the current runjs format, JSLint can actually find typos for a dependency's variable name.

4) This syntax is more verbose for JS files that do not care about scope encapsulation. For JavaScript libraries like jQuery, MooTools and Prototype, they pretty much operate in the same global scope. jQuery does have a noConflict(), but that just helps if there is only one other thing called $ in the page. MooTools and Prototype add things to global prototypes.

So for these libraries, specifying dependencies is more like just specifying script tags, and they do not need a local-scoped variable in the function module to get things defined. They would just need to do the following:

run.def("rdw/Message",
["some/module", "something/else", ...]
function() {
//This function does not need some locally scoped variables, and most
//likely, the script dependencies above may not call run.def() and define
//an object, just add things to the global space
});

So for these libraries, it would be onerous to be forced to create variable names for each of those dependencies.

This last point seems to be enough to tip the scales back to using the current model used by run. However I am aware that it is possible for the developer to not get the order or number correct, matching the dependency string with the correct variable for the function.

I am hoping using a coding standard like the following will help:

run.def("rdw/Message",
["rd", "dojo", "rdw/Base"], function(
 rd,   dojo,   Base) {
 //Define the module for rdw/Message and return it.
});

Basically, make sure all dependencies are on one line, along with the function keyword, then put the variable that matches each dependency aligned directly under the depdendency name (the example above may not be correctly aligned depending on the font or format you are viewing this message).

I purposely trimmed the list of dependencies, so I can get this example to show up in this blog. That is the down-side with this sort of code style: it can have a very long line for the dependency names.

For Raindrop, I want to try to keep local scope encapsulation, particularly since I expect some slicker extensions to it that may introduce other code into the page that may conflict. However, if I did not care to do that, I could shorten up the above example quite a bit.

So at this point, I am favoring making it easy to use RunJS for other toolkits that do not care about local scope encapsulation and detecting bad references to variables inside the module definition over avoiding errors with a mismatched function variable name to a dependency name.

It is a hard choice to make. Neither path is perfect. If you have an opinion, feel free to share it.

Raindrop + JSLint + JSDoc + RunJS

I just pushed some changes into the front-end code for Raindrop. Besides using JSLint as the code style, and JSDoc for the format of the files, it now uses a modified version of Dojo 1.4 that uses the RunJS code loader.

I am fortunate enough to work with people that will tolerate this sort of brief experimentation to help improve the state of browser code loading. Do you know Mozilla Messaging is hiring? Great people, great technology, and a great mission. OK, end of the sales speech.

The code conversion to JSLint, JSDoc and a new loader all at one time took longer than I would have liked, but I am glad I did it.

For one thing, the code looks a lot more uniform thanks to using JSLint. While I do not care for all of the rules mandated by JSLint, no one ever likes a coding standard 100%, and JSLint is widely known and a programmatic code checker.

Converting Raindrop to RunJS was really beneficial for RunJS. While RunJS has unit tests, nothing helps shake out the kinks like a large project. RunJS is now more robust because of it.

If you want to peruse the Raindrop code, you can hop to the Raindrop Mercurial source web view. That link will take you to Raindrop's client/lib/rdw directory, where we keep the UI widgets. You can click on some files to see how they look with the RunJS loader and the new formatting.

One thing that became apparent, particularly when talking to David Ascher about the code: the way the dependencies in RunJS are specified as an array, and having a separate list of function arguments that must match that array might be prone to errors.

Here is an example from rdw/Message:

run("rdw/Message",
["run", "rd", "dojo", "rdw/_Base", "rd/friendly", "rd/hyperlink", "rd/api",
"text!rdw/templates/Message!html"],
function (run, rd, dojo, Base, friendly, hyperlink, api, template) {
 ...
});

I condensed the run boilerplate to the top to three lines, otherwise, it would have been quite long vertically to express it all. But you can see the problem. Looks scary.

David and I talked about it. I talked about how YUI uses just a Y object as the only argument to the function, but I did not like how they use odd module names like "yui-anim", and then you have to know that creates a Y.anim property that you can use in your function.

So David suggested the following:

run("rdw/Message", {
 rd: "rd",
 dojo: "dojo",
 Base: "rdw/_Base",
 friendly: "rd/friendly",
 hyperlink: "rd/hyperlink",
 api: "rd/api",
 template: "text!rdw/templates/Message!html"
}, function(R) {
 //Use things like R.api and R.Base in here that map to the modules up above.
 ...
});

I like this better. It is much clearer what module gets assigned to what variable, no "off by one" errors.

The downside: it does not minify as nicely as the existing run format. However, in this case, avoiding developer errors probably trumps the minification cost, and the extra cost to type R. in front of the dependent module references. You can call it r. if you want. Hmm, I might prefer the lower-case version. Easier to type. Although the capital stands out better.

So I am very tempted to convert RunJS to use this format for listing dependencies and passing them to the function callback. I will likely do the work this weekend.

Update: I decided not to do this, favoring the original format since it has other advantages.

Another thing we talked about was the multiple invocations of run(): run() can be called with a starting string argument to define a module, but then can also be called without that starting string argument if you just want to run some code that has some dependencies.

David felt it would be clearer if those two actions were two different function names. I can see that being clearer. So what about the following for the API:

• run(), for just running code that does not define a module.
• run.def() for defining a module.
• run.mod() for a module modifier (the existing run.modify() API call)
• run.get() for getting a defined module after initial module evaluation (needed for some some dynamic cases where you do not know the module name before-hand and in circular dependency cases). This is an existing API.

So the changes are renaming run.modify() to run.mod() and moving the run() call style that defined a module to run.def().

Thanks to David for the feedback, and for the rest of the Raindrop team for being patient with me as I did some experimentation in browser module loading.

Speaking of feedback on RunJS, Rawld Gill (one of the authors of Mastering Dojo) was already working on a loader, and he did some work to convert it to use the same API as RunJS. I am still processing the results, but it is great to see alternate implementations. I am hoping we can take the best of both and make a great loader. I already changed the module evaluation algorithm to use the recursive style he suggested.

RunJS takes its roots from the cross-domain Dojo loader, and that loader was constructed when Dojo had to support Safari 2. Safari 2 has a ridiculously small call stack, so a recursive module evaluator blew up very easily. I switched to a model that traced the dependencies to work out an array of sequenced modules, then a separate loop to then call them into being. That allowed the loader to work in Safari 2, but now that we do not need to support that browser, the more natural, straight-forward recursive model can be used.

Really neat stuff! Feels like we are getting close to a robust but compact loader.

RunJS: GitHub, build options, features, file sizes

A few updates on RunJS, a JavaScript file/module loader (see the README for more documentation):

• RunJS is now on GitHub
• Plugins for RunJS are supported. i18n bundles have been pulled out as a plugin, and a new text plugin allows you to set text files (think HTML/XML/SVG files) as dependencies for a module. The plugin will use async XMLHttpRequest (XHR) to fetch those files and will pass the text of those files as an argument to a module's module definition function. The RunJS build system will then *inline* those text files with the module, so that the XHR calls be removed in deployed code, and allow cross-domain use of those text files.
  
• Any function return type is allowed from the module definition function. Before only objects and functions were allowed and functions had to be called out in a special way. Now, that special call out is removed and any return type is allowed. The cost was an extra call, run.get() that needs to be used in circular dependency cases. See the Circular Dependencies section in the README.
  
• The build system that comes with RunJS now supports build pragmas.

The build pragma support was used to build RunJS in a couple of different configurations. I am trying to get a handle on where the bulk of implementation lies, and what features add to its file size. Here is the breakdown (warning, Google Doc iframe inclusion, but interesting numbers inlined in this post after the iframe):



Let's look at the non-license sizes, since they give a better indication of code density. Google's Closure Compiler did the minification for this evaluation.

The normal config, with no plugins included (but with plugin support) is 7,970 bytes minified, 3,167 gzipped. Including both the i18n and text plugins with run.js bumps it up to 11,759 minified, 4,655 gzipped.

The interesting number for me is the version of run.js without plugin support, no run.modify, no multiversion support and no page load (run.ready/DOMContentLoaded callbacks). This version of run has just the following features:

• support for the run() module format
• nested dependency resolution
• configure paths to modules
• load just plain .js files that do not define run.js modules (scripts that do not call run(), for example jQuery, or plugins for jQuery).

That bare bones loader comes in at 5,086 minified and 2,204 gzipped. The one you should use, the one with the license, is 5,245 minified and 2,317 bytes gzipped. I need to work on the size of that license block!

That size could probably be brought down a tiny bit (probably reaching the 2,000 gzip size) if I were to really be aggressive and remove all context references, but that would be a mess to maintain and there would be no easy upgrade path to multiversion support.

I believe that is the lower limit a functional loader that does nested dependencies via run() module calls. I view run.ready/DOMContentLoaded support more of a necessity for a loader, so unless you already had an implementation for that, I suggest the version that has run.ready() support, which comes in (with license) at 5,867 minifed, 2,522 gzipped.

The nice thing about the build pragma setup for RunJS, you can upgrade run without having to change your code if you find you want more features, like plugin support, or i18n/text dependency support via plugins.

I am interested in trying to sell more front-end JavaScript toolkits on this loader. For some, I can see the bare-bones 2.3K gzipped loader a nice way to step into it, and then their users have the option to swap out a more powerful version via a different RunJS build output.

I have put up the different build outputs for 0.0.6 if you want to grab one of the minified versions and play with it. Here is the minimum set of compliance tests which use the smallest loader (no modify/plugins/page load/context support) mentioned above. See the README for documentation.

Right now I believe around 2KB gzipped is close to the lower bound for a stand-alone code loader in the browser. At least for a loader I would consider using: anything that uses XHR and eval are dead to me. Using plain script src="" tags helps the xdomain case, and just fits better with debugging. While Dojo has used an XHR-based loader for quite a while (and it will continue to be supported), it just does not work as well with the browser as a script-tag based loader. Any loader should also do nested dependency loading too -- if a module in a script has dependencies in other modules, be sure to evaluate the dependencies in the right order.

As a point of comparison, consider LABjs. I feel a kinship with the author of LABjs, Kyle Simpson, even though we have never talked. We are both focusing on efficient code loading in the browser. I recommend LABjs if it fits your style.

While LABjs does not quite do nested dependency resolution, it does something related where you can tell it to wait to load a script before continuing to load other scripts. LABjs is not trying to push a module format like run is, but targeted more at existing code that does not have the concept of a module format.

By the way, RunJS can also handle loading these types of files. Where LABjs has a wait() call for holding off loading scripts that depend on another script being loaded (like a framework), RunJS uses nested run calls.

Example from the LABjs page:

$LAB
.script("framework.js").wait()
.script("plugin.framework.js")
.script("myplugin.framework.js")
.wait(function(){
    myplugin.init();
    framework.init();
    framework.doSomething();
});

Equivalent example with RunJS:

run(["run", "framework.js"],
    function(run) {
        run("plugin.framework.js", "myplugin.framework.js"],
            function() {
                myplugin.init();
                framework.init();
                framework.doSomething();
            }
        );
    }
);

Taking the 1.0.2rc1 version of LABjs and using Closure Compiler on it (without the license) gives LABjs a size of 4,360 bytes minified and 2,170 gzipped. As a reminder, the equivalent RunJS file is 5,086 minified and 2,204 gzipped. I may be able to do better with making the structure of the RunJS code more amenable to minification, but the gzip sizes come up fairly close. I do not believe the code tricks I would do to help minification will help the gzip size any.

Both LABjs and RunJS end up around 2KB gzipped. So, about 2KB gzipped seems close to the lower limit on a standalone loader, one that uses script tags/plays nice with the browser and can do nested dependencies. I would like to be proven wrong though, and ideally by modifying RunJS to fit that lower limit. :) I am sure the code can be improved.

But remember the guidelines, no goofy XHR stuff/something that works well with the browser and can handle nested dependencies. No script tags with inlined source/eval tricks. Even though Firefox and WebKit make eval debugging easier, it is still not as nice as regular script src tags.

Irakli Gozalishvili believes web workers might help, but I do not see it. The workers are restricted to message passing, and anything interesting in a web browser will likely need to touch the DOM, so a web worker solution will just be another async-XHR-like approach, where you will need to eval the scripts or inline-script inject to get all the scripts for them to see each other and the DOM.

Irakli does have an async-XHR based loader for CommonJS modules. As of today, it comes in at 1,527 minified, 838 gzipped (license not included). But it uses XHR, so limited to the same domain as the page, and debugging support is just not as nice across browsers. It also uses CommonJS module syntax, but I have decided CommonJS modules do not play well out of the box in the browser, and I believe the format's "module", "exports", and "require.main" parts are unnecessary.

Dojo 1.4 Favorite Features

Dojo 1.4 is out! There is a metric ton of changes. Here are some of my favorite things about the release. I focus mostly on Dojo Core, and mostly in the non-animation parts of it, so my list is skewed for that focus. However, there are lots of other changes, some in the animation functionality, and in Dijit and Dojox. Check out the 1.4 release notes to get a more complete picture.

One of the things I want to do for Dojo Core is to bring the DOM APIs, particularly the methods on dojo.NodeList (the return object for dojo.query() calls, Dojo's CSS selector method) more in-line with what is available in jQuery. jQuery has demonstrated that its APIs resonate strongly with developers. Where it makes sense and fits Dojo's philosophy, we should also provide those APIs, to make it easier for developers. These Dojo 1.4 changes reflect that goal:

• dojo.ready(), just an alias for dojo.addOnLoad().
• dojo.NodeList-traverse: A helper module that adds methods to dojo.NodeList. Its goal is to bring in some methods to NodeList that exist in jQuery for DOM traversal, specifically: children, closest, parent, parents, siblings, next, nextAll, prev, prevAll, andSelf, first, last, even, odd.
• dojo.NodeList-manipulate: A helper module that adds methods to dojo.NodeList. Its goal is to bring in some methods to NodeList that exist in jQuery for DOM manipulation, specifically: innerHTML, html, text, val, append, appendTo, prepend, prependTo, after, insertAfter, before, insertBefore, remove, wrap, wrapAll, wrapInner, replaceWith, replaceAll, clone.
  
• IO pipeline topics: get notifications of IO events via dojo.subscribe/dojo.publish. Handy for putting up a generic "loading" indicator when any sort of IO call happens. These topics are not strictly how jQuery exposes this functionality, but we can leverage the power of dojo.publish/subscribe to implement this feature.

Some other new Dojo Core 1.4 features that are really sweet:

• dojo.cache(): allows you to reference external HTML files and use them as if they are strings. It is integrated into the build system, so you can avoid the XHR calls to get the external text files by just doing a build. No extra build option is needed. This is a great way to construct HTML -- by writing plain HTML instead of building awkward strings in code or using JS DOM-building calls, which can obscure what the HTML actually looks like.
  
• dojo.position(): A faster, more understandable replacement for dojo.coords(). If you were using dojo.coords() before, odds are good that you probably want to switch to dojo.position(). Douglas Hays stepped up and put in this great new method.
  
• dojo.declare(): It is faster and more robust. Many thanks to Eugene Lazutkin for doing this work. It took a lot of patience and perseverance to get this new version up to snuff and keep it backward compatible.
• dojo.hash(): An easy way to set the URL hash (fragment ID) and to watch changes to the hash. This allows you to create pages that reflect the proper state as shown by the URL in the browser. This was a contribution from community member Rob Retchless and other IBM Jazz team members.

For the build system, support was added for Google's Closure Compiler, so you can experiment with using it for minifying your code. Right now we just support the "simple" minification done by Closure Compiler, not the advanced features.

It was a little while coming, but it is great to have Dojo 1.4 out. Thanks to the community for making the toolkit better!

JavaScript module loading, the browser and CommonJS

JavaScript module syntax and loading seems like a hot topic at the moment, and here are some thoughts about how to construct module syntax and a loader, with the goal of trying to get to a more universal approach for it. This will be discussed in the context of CommonJS, but browser-based module loaders have existed for a while. All are constrained by the browser in some fashion as listed below. I have a preferred solution, also described in this post.

First, a look at module syntax.

CommonJS is an umbrella for a few different things, including a spec for a module syntax and a standard library of modules. This post is just interested in its module syntax. A simple example of CommonJS syntax, defining an "increment" module defined in increment.js:

var add = require('math').add;
exports.increment = function(val) {
       return add(val, 1);
};

How could we build a module loader with this syntax? Here are a couple of options:

1) You parse the module before executing it, looking for require calls. You make sure to fetch those modules and work out the right dependency order in which to execute the modules.

2) You just run the module and when require() is hit, do an synchronous IO operation to load the required module.

For both approaches, use a sandbox or specific context to make sure things like "exports" are defined separately for each module.

Both of those options are easy to implement on the server side since you have more control over the IO layer, and can create separate contexts for each module. However, in the browser, things are different. Creating a separate context for each module is tricky. For IO, there are two realistic approaches, and each has its difficulties:

• XMLHttpRequest (XHR)
• script tags

XHR allows us to do either approach, #1 or #2. It can get the contents of the module and parse it into a structure that pulls out the dependencies and we can sort out the right order to execute things. We could use sync XHR calls to accomplish #2, block when each require call is seen. However, sync XHR calls in the browser really hurt performance.

This is actually what the default Dojo loader has done for a very long time, and I believe some pathways in the Google's Closure library do the same thing. It is always recommended you do a custom build to combine all the modules you need into one file to cut out those XHR calls when you want to go to production.

So path #1 would make more sense with an XHR-based loader. However, for an XHR loader to work, it has to use eval() to bring the module into being. Some environments, like Adobe AIR do not allow eval(), and it makes debugging hard to do across browsers. Firefox and WebKit have a convention to allow easier eval-based debugging, but it is still not what I consider to be in keeping with traditional script loading in a browser.

Instead of eval, after the XHR call finishes its parsing and module wrapping for context, you could try to create a script tag that has a body set to the modified module source, but this really hurts debugging: if there is an error, the error line number will be some weird line in a gigantic HTML file instead of the line number of the actual module.

Dojo has a djConfig.debugAtAllCosts option that will use sync XHR to pull down all the modules, parse the for dependencies, work out the right load order, then load each module via a dynamically added script src="" tag. However, since IE and WebKit will evaluate dynamically added script tags out of DOM order -- they evaluate them in network receive order (which is nice for long-polling comet apps, but does not help module loading). So, each script tag has to be added one at a time, then wait for it to finish then add the next one. Not so speedy.

XHR is also normally limited to just accessing the same host as the web page. This makes it hard to use CDNs to load content, and get performance benefits with that approach. There is now support for xdomain XHR in most recent browsers, but IE prefers to use a non-standard XDomainRequest object, making our module loader more complicated. And xdomain XHR just plain does not work in older browsers like IE6.

So, an XHR-based loader is not so great.

Script tags are nice because they keep with the known script pathway in browsers -- easy to debug, and we can get parallel loading. However, we cannot do approach #2 in the browser: our JavaScript in the browser cannot access the module contents before they are evaluated. And since dynamically added script src="" tags via head.appendChild() are not a synchronous operation, approach #2 will not work.

So, really we need to do a variant of #1, pull out the dependencies needed by the module, then after those dependencies are loaded, execute the module. The way to do this in script: put a function wrapper around the module contents, and call a module loader function with a list of dependencies and the module function wrapper. Something like this, for a module with the name of "c" that has dependencies of "a" and "b", Here is a syntax (call it Variant A) for defining a module "c" with this approach:

loader(
       "c",
       ["a", "b"],
       function(a, b) {
              //The module definition of "c" in here.
              //return an object to define what "c" is.
              return {};
       }
);

or, another variant, call it Variant B:

loader({
       name: "c",
       dependencies: ["a", "b"],
       module: function(a, b) {
              //The module definition of "c" in here.
              //return an object to define what "c" is.
              return {};
       }
});

Ideally, we would not have to tell the loader that this structure defines module "c" (the first arg in Variant A and the name: property in Variant B) -- the loader could work this out. Unfortunately, since script tags can load asynchronously and at least IE can trigger script.onload events out of order when compared to when the script is actually evaluated, we need to keep the module name as part of the module definition. This also helps with custom builds, where you can combine a few of these module definition calls into one script.

This approach is actually what Dojo's xdomain loader has done for a very long time, but with more verbose syntax. However, it requires a custom build to convert modules into this structure. The other option is to use a server-side process to convert the modules on the fly, but I do not feel that is keeping with the simplicity of normal browser development: just open a text editor, write some script, save, reload, no extra server config/process needed, besides maybe a vanilla web server.

So, I believe that modules should be coded by the developer in this module wrapper format. YUI 3 has taken this approach, and it is the approach I have taken for RunJS too. However, YUI 3 is limited to needing some module dependency metadata files to help it out. It also uses module names that do not map to the actual module's defined name/functions.

OK, back to CommonJS.

As it stands now, I believe the CommonJS format is not suitable for modules in the browser. There have been attempts to get it to work, but the attempts either use a sync XHR loader, or a "transform-on-the-fly" server process to convert the code to a module wrapper similar to Variant B.

I would rather see a module wrapper format that works with browser natively, that can be hand-authored by developers and that will work with CommonJS modules. CommonJS started out as ServerJS. As ServerJS, the case could be made that supporting browsers may not be an aim of a ServerJS module format. However, with the name change to CommonJS, I believe supporting browsers as a first class citizen is important for CommonJS to get more traction.

So the trick is to come up with a module syntax that has a function wrapper, but is not too wordy with boilerplate. We need some boilerplate, since we need a function wrapper. I believe RunJS has the right right approach. The boilerplate is very terse, basically Variant A mentioned above:

run(
    "c",
       ["a", "b"],
       function(a, b) {
              //The module definition of "c" in here.
              //return an object to define what "c" is.
              return {};
       }
);

I can see where there is some bikeshedding on the name "run". I think script() instead of run() is a viable alternative, and I may switch to that in the near future (and rename RunJS to ScriptJS).

I have attempted to engage the CommonJS community by putting up a proposal for an Alternate Module Format.

Progress has been slow, but to be expected: the CommonJS group is trying to do lots of other things like define a standard library and build out implementations. However, I am hopeful we can get something that works for the browser front end developers.

The ideal scenario is that some variant of the above syntax is just adopted as the only CommonJS module format. That would save a lot of conversion work, and I believe it makes things much simpler for CommonJS compliant loader. Right now, for CommonJS loaders there is a concept of a require() and require.async() and having to expose Promises for the async stuff. The above format neatly avoids the issue of whether the modules are loaded async or sync and avoids any need for Promises in the module loader. I think it is fine though for modules themselves to use Promises as part of individual module APIs, but at least the loader and module syntax stays simple.

I also do not believe a "module" variable needs to be defined for each module and an exports variable is avoided by returning an object from the module function wrapper.

I can appreciate that the CommonJS folks with modules already written may not like moving to the above syntax. I think it helps in the long run if we can just have one syntax, but in the meantime, I plan on doing the following:

• Continue to engage the CommonJS community.
• build out RunJS, probably rename to ScriptJS in the near future, and use script() instead of run()
• Write a converter that converts Dojo modules to the RunJS/ScriptJS module syntax. I have something basic working, here is an example of Dojo's themeTester.html using RunJS-formatted dojo/dijit/dojox modules. That example is not bulletproof yet (I used a built version of Dojo which removes some dependency info) and i18n modules have not been converted either. RunJS also has built-in support for i18n modules.
• Convert Raindrop to use RunJS-formatted dojo and convert the Raindrop modules to that format.
• Override run.load()/script.load() in server environments so it could be used in CommonJS server implementations.
• Work on a converter for existing CommonJS modules.
• Use RunJS/ScriptJS as the module syntax for Blade and/or Dojo 2.0 efforts.

If module syntax/loading is important to you, then please join the discussion list for CommonJS, so we can sort this out. It would be great to get consensus on JavaScript module syntax and loading, and I think CommonJS is the area to do that.

I am happy to adjust some of the syntax in RunJS/ScriptJS to match some consensus, but I strongly prefer a terse format. The existing ones I have seen for server-converted CommonJS modules is too verbose for me, particularly for the common cases of defining a module with some dependencies.

Raindrop, CouchDB and data models

Raindrop uses CouchDB for data storage. We are starting to hit some tough issues with how data is stored and queried. This is my attempt to explain them. I am probably not the best to talk about these things. Mark Hammond, Raindrop's back-end lead is a better candidate for it. I am hoping by trying to write it out myself, I can get a better understanding of the issues and trade-offs. Also note that this is my opinion/view, may not be the view of my employer and work colleagues, etc...

First, what are our requirements for the data?

• Extensible Data: we want people to write extensions that extend the data.
  
• Rollback: we want it easy for people to try extensions, but this means some may not work out. We need to roll back data created by an extension by easily removing the data they create.
  
• Efficient Querying: We need to be able to efficiently query this data for UI purposes. This includes possibly filtering the data that comes back.
  
• Copies: Having copies of the data helps with two things:
  
  • Replication: beneficial when we think about a user having a Raindrop CouchDB on the client as well as the server.
  • Backup: for recovering data if something bad happens.

How Raindrop tries to meet these goals today

Extensible Data: each back-end data extension writes a new "schema" for the type of data it wants to emit. A schema for our purposes is just a type of JSON object. It has a "rd_schema_id" on it that tells us the "type" of the schema. For instance a schema object with rd_schema_id == "rd.msg.body" means that we expect it to have properties like "from", "to" and "body" on it. Details on how schemas relate to extensions:

• An extension specifies what input schema it wants to consume, and the extension is free to emit no schemas (if the input schema does not match some criteria), or one or more schemas.
• Each schema written by an extension is stamped with a property rd_schema_provider = "extension name".
• All the messages schemas are tied together via an rd_key value, a unique, per-message value. Schemas that have the same rd_key value all relate to the same message.

More info is on the Document Model page.

Rollback: Right now each schema is stored as a couch document. To roll back an extension, we just select all documents with rd_schema_provider = "extension name" that we want to remove, and remove them. As part of that action, we can re-run extensions that depended on that data to have them recalculate their values, or to just remove the schemas generated by those extensions.

Having each schema as a separate document also helps with the way CouchDB stores data -- if you make a change to a document and save it back, then it appends the new document to the end of the storage. The previous version is still in storage, but can be removed via a compaction call.

If we store all the schemas for a message in one CouchDB document, then it results in more frequent writes of larger documents to storage, making compaction much more necessary.

Efficient Querying: Querying in CouchDB means writing Views. However, a view is like a query that is run as data is written, not when the UI may actually want to retrieve the information. The views can then be very efficient and fast when actually called.

However, the down side is that you must know the query (or a pretty good idea of it) ahead of time. This is hard since we want extensible data. There may be some interesting things that need to be queried later, but adding a view after there are thousands of documents is painful: you need to wait for couch to run all the documents through the view when you create the view.

Our solution to this, started by Andrew Sutherland and refined by Mark, was to create what we call "the megaview". It essentially tries to emit every piece of interesting data in a document as a row in the view. Then, using the filtering capabilities of CouchDB when calling the view (which are cheap), we can select the documents we want to get.

Copies: While we have not actively tested it, we planned on using CouchDB's built-in replication support. This was seen as particularly valuable for master-master use cases: when I have a Raindrop CouchDB on my laptop and one in the cloud.

Problems Today

It feels like the old saying, "Features, Quality or Time, pick two", except for us it is "Extensible, Rollback, Querying or Copies, pick three". What we have now is an extensible system with rollback and copies, but the querying is really cumbersome.

One of the problems with the megaview: no way to do joins. For instance, "give me all twitter messages that have not been seen by the user". Right now, knowledge of a message being from twitter is in a different schema document than the schema document that knows if it has been seen by the user. And the structure of the megaview means we can really only select one property at a time on a schema.

So it means doing multiple megaview calls and then doing the join in application code. We recently created a server-side API layer in python to do this. So the browser only makes one call to the server API and that API layer does multiple network calls to CouchDB to get the data, then does the join merging in memory.

Possible solutions

Save all schemas for a message in one document and more CouchDB views
Saving all schemas for a message in one document makes it possible to then at least consult one document for both the "type=twtter, seen=false" sort of data, but we still cannot query that with the megaview. It most likely means using more CouchDB views to get at the data. But views are expensive to generate after data has been written. So this approach does not seem to scale for our extensible platform.

This approach means taking a bit more care on rollbacks, but it is possible. It also increases the size of data stored on disk via Couch's append-only model, and will require compaction. With our existing system, we could consider just never compacting.

This is actually the approach we are starting to take. Mark is looking at creating "summary documents" of the data, but the summary documents are based on the API entry points, and the kind of data the API wants to consume. These API entry points are very application-specific, so the summary document generation will likely operated like just another back end extension. Mark has mentioned possibly just going to one document to store all schemas for a message too.

However, what we have not sorted out how to do is an easier join model: "type=twitter and seen=false". What we really want is "type=twitter and seen=false, ordered by time with most recent first". Perhaps we can get away with a small set of CouchDB views that are very specific and that we can identify up-front. Searching on message type and being seen or unseen, ordered by time seems like a fairly generic need for a messaging system.

However, it means that the system as a whole is less extensible. Other applications on the Raindrop platform need to either use our server API model of using the megaview then doing joins in their app API code (may not be so easy to learn/perform), or tell the user to take the hit waiting for their custom views to get up to date with all the old messages.

Something that could help: Make CouchDB views less painful to create after the fact. Right now, creating a new view, then changing any document means waiting for that view to index all the documents in the couch, and it seems to take a lot of resources for this to happen. I think we would be fine with something that started with most recent documents first and worked backwards in time, using a bit more resources at first, but then tailing off and doing it in the background more, and allow the view to return data for things it has already seen.

Do not use CouchDB
It would be very hard for us to move away from CouchDB, and we would likely try to work with the CouchDB folks to make our system work best with couch and vice versa. It is helpful though to look at alternatives, and make sure we are not using a hammer for a screwdriver.

Schema-less storage is a requirement for our extensible platform. Something that handles ad-hoc queries better might be nice, since we basically are running ad-hoc queries with our API layer now, in that they have to do all the join work each time, for each request.

Dan Goldstein in the Raindrop chat mentioned MongoDB. Here is a comparison of MongoDB and CouchDB. Some things that might be useful:

• Uses update-in-place, so the file system impact/need for compaction is less if we store our schemas in one document are likely to work better.
• Queries are done at runtime. Some indexes are still helpful to set up ahead of time though.
• Has a binary format for passing data around. One of the issues we have seen is the JSON encode/decode times as data passes around through couch and to our API layer. This may be improving though.
  
• Uses language-specific drivers. While the simplicity of REST with CouchDB sounds nice, due to our data model, the megaview and now needing a server API layer means that querying the raw couch with REST calls is actually not that useful. The harder issue is trying to figure out the right queries to do and how to do the "joins" effectively in our API app code.

What we give up:
1) easy master-master replication. However, for me personally, this is not so important. In my mind, the primary use case for Raindrop is in the cloud, given that we want to support things like mobile devices and simplified systems like Chrome OS. In those cases it is not realistic to run a local couch server. So while we need backups, we probably are fine with master-slave. To support the sometimes-offline case, I think it is more likely that using HTML5 local storage is the path there. But again, that is just my opinion.

2) ad-hoc query cost may still be too high. It is nice to be able to pass back a JavaScript function to do the query work. However, it is not clear how expensive that really is. On the other hand, at least it is a formalized query language -- right now we are on the path to inventing our own with the server API with a "query language" made up of other API calls.

Persevere might be a possibility. Here is an older comparison with CouchDB. However, I have not looked in depth at it. I may ask Kris Zyp more about it and how it relates to the issues above. I have admired it from afar for a while. While it would be nice to get other features like built-in comet support, I am not sure it will address our fundamental issues any differently than say, MongoDB. It seems like an update-in-place model is used with queries run at runtime. But definitely worth more of a look.

Something else?

What did I miss? Bad formulation of the problem? Missing design solution with the tools we have now?

Blade, a JavaScript toolkit experiment

I am playing around with a different way (at least for me) to construct a JavaScript toolkit. It is called Blade, and you can follow it via the Blade GitHub repo.

I have had this on my local drive for a few weeks now, and the germ of it started with this post. I wanted to get it more polished, but best to get it up somewhere to get some feedback at least on the principles.

There is not much there now, basically a tiny amount of spaghetti code that is not really usable. However, I list out the guiding principles in the README.md, visible on the GitHub source tab.

Raindrop, Open Messaging for the Open Web

I work for Mozilla Messaging, and we just opened the doors on Raindrop. Raindrop is the reason I had the opportunity to move to Vancouver, BC. It has been fun building it, seeing if we could get something to work.

Raindrop is still very much an experiment and not useful for any day-to-day work. However, it has potential and we need community help to take if further.

Why I like it:

• It is driven by product design. We want an extensible platform, but a strong, simple product design will be driving much of the development.
  
• It is not trying to be a message service in itself, but collect messages from existing services.
• It is web-based: the default UI is plain HTML/JavaScript/CSS goodness.
• It is frickin awesome to be able to play with your messages: data mine them, and do interesting display things using simple script languages like JavaScript and Python.
• It is open: open source and motivated by the Mozilla Manifesto.
• The Mozilla Messaging team is talented and smart. They are motivated and care about what is best for users.
  

I am driving the front end development for Raindrop. I used Dojo to create the infrastructure for the pages, using Dijit's dijit._Widget and dijit._Templated as a base for many of the UI widgets.

Dojo's dynamic code loader and Dijit's well-defined methods on widgets have enabled the slick things we are doing with in-place extension editing and updates. JQuery is also included in the page, mostly for extension developers, so you have a choice on what to use, Dojo or JQuery.

While I expect many of the decisions I made about how the front end works might change over time, it has been a joy to make what is there so far.

There is lots to do though. If you want to get involved with the code, check the Hacking page is a good place to get started. There is a screencast of the architecture on the Raindrop home page. There is a Community page too.

RunJS updated: module modifiers and function modules

I pushed some changes to allow lazy-loaded/lazy-evaluated module modifiers and also better support for modules that just define a function. The changes are documented in the documentation page.

I am experimenting with using JSLint as a code formatter. We'll see how it goes.

All these changes bring the size up to 3.1 KB minified and gzipped. I would like to be under 3 KB, but I want to be sure the right functionality is in place first before squeezing it down.

The module modifiers are a bit of an experiment. I wanted some way to separate a bunch of bad, wordy code out for the normal cases of a module but only in bad cases load the bad code. The example I give in the documentation is a module that gets DOM node dimensions and position. In standards mode, it is fairly compact, but in quirks mode it gets uglier. So I only want to load the quirks mode code if the page is in quirks mode. I never want to develop in a quirks mode page, but for a general JavaScript library it might be important.

So I am still not sure if the module modifier approach is the right way to go, but I have used it a little bit so far on another project, and I will see how that works out.

RunJS updated: simple modules and i18n bundles

I pushed some changes to RunJS, the stand-alone JavaScript file/module loader.

Newest changes are support for simple module definitions and internationalization (i18n) bundles.

Simple module definitions are possible when there are no dependencies for the module. In that case the function wrapper for the module is not needed:

run(
    "my.simplething",
    {
        color: "red",
        size: "large"
    }
);

i18n bundle support is handy for separating out strings that might need to be translated into other languages. Quick example for a my/nls/colors.js that will define a bundle:

run(
    "my.nls.colors",
    [{
        "root": {
            "red": "red",
            "blue": "blue",
            "green": "green"
        },
        "fr-fr": "my.nls.fr-fr.colors"
    }]
);

Then define a file at my/nls/fr-fr/colors.js that has the following contents:

run(
  "my.nls.fr-fr.colors",
  {
    "red": "rouge",
    "blue": "bleu",
    "green": "vert"
  }
);

See the documentation for more information.

If you want to try out the latest code, you can use one of the following URLs to fetch the code:

• Minified version
• Full commented version

In addition to the documentation, some of the test files might be of interest to see how RunJS can be used.

RunJS is still nice and small with these new features, around 2.7KB, when minified via YUICompressor and gzipped.

RunJS

I just created what I hope to be the next generation JavaScript module loader here: RunJS. The documentation has some history and reasoning behind it.

I would like to see it used for any project that needs JS code loading, particularly since it handles dependencies, can load regular JavaScript files, handles multiple versions of modules, and is a compact 2.2KB (minified and gzipped). It is JavaScript toolkit-agnostic and has no dependencies.

Enjoy!

Chrome Frame, or Browser vs. Renderer

Chrome Frame came out yesterday. I like the conversation it is trying to start. It makes a big difference having working code to get the conversation to seriously happen. There are some mechanics of the specific approach that probably need tweaking, but the general idea is worth consideration.

The big point is about separating the innovation over organizing the user's experience of the web in general (the Browser) vs. innovation in the display of web sites (the Renderer).

The Chrome Frame approach allows browser makers to preserve their revenue models and their UI interaction models, at least in the ideal -- there are some specific issues to work out with the Chrome Frame model, like password/form data storage. But I think the direction is a good one.

It also solves the issue where a web application was developed a few years ago and is not going to be maintained any more. If the browser allows multiple rendering engines it makes it possible for those old web apps to continue to work.

Old enterprise/business applications in particular can continue to work, but we still get to move the new web experience forward.

From a browser market share perspective, this clean split of responsibility could allow a newer browser like Google Chrome to get more market share. It could use the reverse idea: embed the IE render engine in Google Chrome. Then, take that to all the IT administrators and say, here is a newer, safer, more secure browser for your company that will work on older machines, is free, and can still allow your old business web apps to work.

That may not work, since most people are attached to their browser chrome vs. the actual renderer. In those cases, the plain Chrome Frame approach of installing a plugin for another renderer works to allow better, faster web apps experiences.

One concern: user choice or control. I think it actually leads to more user control: the user gets to keep the browser chrome, their organizing model for the whole web, intact, but gets to use more of the better parts of the web. And for areas where the user does not have control now (in business environments that need old apps to work), it gives a way out to use more of the web.

Of course, there should user controls so they can set preferences and overrides for the renderers used. At a minimum, business IT groups will need it to configure the browsers to use old renderers for older in-house business web apps.

So, some tweaks to the basic mechanics I would like to see (realizing that I have no concept how hard this work would be):

1) Make sure the Renderer works well with the Browser: for instance make sure that saved passwords/form data works well no matter what Renderer is used. Make sure the split is clean.

2) Change the UA-Compatible thing to be more Renderer-based feature sets vs. browser version numbers. So, something where the developer mentions the capabilities that are desired:

<meta equiv="X-UA-Compatible" content="addEventListener=1,svg=1">

If the developer has a suggested browser renderer, it could place that at the end, sort of how CSS font names can start with generic names, then get more specific:

<meta equiv="X-UA-Compatible" content="addEventListener=1,svg=1;gecko=1.9.2">

It would be good to *not* use browser names in the tag, but rather the renderer engine names/versions. Ideally though, just list capabilities.

The above syntax is not exactly right, but just to demonstrate the idea: focus on telling the browser the capabilities the page wants, and use render engine names, not browser names.

3) Make sure the user can override the choices made by the browser. The pref control does not have to be obvious, but should be there, so the user has the final say.

In summary, as a conversation starter, I like that Chrome Frame has really tried to highlight the difference between upgrades in renderers vs. browser interface.

Developer Tools and JavaScript Syntax Checking

I was using Coda to do web development, primarily because of the simple interface, nice Panic Sans font and integrated FTP and Subversion support. I normally just use TextWrangler for Dojo Core development.

However, while doing development at the new job, FTP/Subversion support was not needed, and I have grown tired having to run my JavaScript in the browser to find basic syntax errors.

I saw David Ascher using Komodo Edit, the free editor from ActiveState, which has JavaScript syntax checking built in. Searching in the Tools, Add-Ons menu for Dojo also brought up a Dojo API Catalogs extension that allowed for autocomplete of Dojo APIs.

Komodo Edit has been mentioned in the Dojo community before as a nice option particularly for doing Dojo development, but I can get severe tunnel-vision, so it was not until recently that really tried out Komodo Edit.

After I changed the key bindings to use the normal OS X COMMAND+] and COMMAND+[ for multi-line indenting, I was ready to go!

It makes a difference having syntax checking and Dojo API autocompletion. I believe other tools, like Emacs with js2-mode, can get at least syntax checking, but I am not an Emacs user. It would be neat to see Coda (and TextWrangler) add JavaScript syntax checking support. I can see JavaScript syntax checking being a minimum requirement for JS coders going forward.

As Bespin comes along, I can see it as my next developer tool upgrade, particularly since FTP support has been implicitly needed for some of my projects. With Bespin, I might be able to edit directly on the server.

For now I am enjoying the bump in productivity with Komodo Edit. Thanks ActiveState for making a nice tool available for free!

Custom local variables using the Dojo Loader

Recently, Dion Almaer was wishing for a way to avoid typing "dojo" so many times in the Bespin code. He appreciated the namespace protection that Dojo gave, but did not like the typing tax that came with it. He talked to Alex and Pete about it. Alex suggested a way, but it is very experimental.

Here is another way that leverages the strength of the Dojo Loader.

The Dojo Loader handles the work of loading your JavaScript modules. When you do a dojo.require("foo.bar"), the loader figures out that you want to load the foo/bar.js file and loads it for you. Normally the loader uses a synchronous XMLHttpRequest (XHR) call to load the file and uses eval() to bring the code into existence.

There is an xdomain loader that does not use eval, and allows your code to be loaded from any domain (with dependencies properly loaded), but it requires a build step using the Dojo build tools.

Both versions of the loader (normal and xdomain) allow you to namespace your code -- so you can map dojo to mydojo, and even your own code to some other name. This allows you to load multiple versions of dojo and/or your code in a page. Nifty.

The Loader is able to support this scope mapping by wrapping your module in a function call like so:

(function(dojo, dijit, dojox){
 //Module code injected in here
})(dojo, dijit, dojox);

We can use this new scope created by this function to create local variables for your module.

Dojo's Loader can be modified to allow you to specify a set of local variable names that get injected with this function wrapping. And we can do this on a per-module prefix basis, so your code can have your own local variables, but some other module can specify a different set.

I did a prototype that works as follows: assume "coolio" is the namespace I use for my modules. I create a coolio.locals that has the following content:

dojo.provide("coolio.locals");

dojo.setLocalVars("coolio", {
 trim: "dojo.hitch(dojo, 'trim')",
 $: "dojo.hitch(dojo, 'query')",
 id: "dojo.hitch(dojo, 'byId')"
});

This code will create local variables called trim, $ and id that map to dojo.query, dojo.byId and dojo.trim respectively, but only for coolio* modules.

Then I have another module, called coolio.actions that uses these variables:

dojo.provide("coolio.actions");

coolio.actions = {
 init: function(){
  $("#trimButton").onclick(coolio.actions, function(evt){
   id("trimOutput").value = trim(id("trimOutput").value);
  });
 }
}

Notice that trim, id and $ were not declared in here. The final bit of magic is to use djConfig.require in the HTML page to auto-load the coolio.locals module before any other coolio code, so that the loader knows to create the local variables for any coolio.* module:

<script type="text/javascript" src="dojo/dojo.js" djConfig="require: ['coolio.locals']"><script>

See it in action with this sample page.

I created ticket #9032 to track the possibility of allowing this in the future. It also has the patch that can be applied to the Dojo source to get it to work. Or, if you are using Dojo 1.3.0, you can grab these built files with the changes: dojo.js or dojo.js.uncompressed.js

There are some caveats to this prototype:
1) We really need a build step that would inline the local variables for your build layers, so the code as-is will not work with custom build layers.
2) Does not work with xdomain builds yet, another tweak to the build system is needed for that.

If you think this might be useful for you, feel free to add your comments to the issue tracker entry or leave a comment.

Job Transition

Friday was my last day at AOL, just shy of 13 years. There were good times, bad times. Some really cool things. Neat people and teachers. A great learning experience.

I am being a bit melodramatic, but Sinead O'Connor's Thank You for Hearing Me expresses the sentiment best, including the end of the song. For work and for life.

Monday I start at Mozilla Messaging. I am really excited about the work. It is still being defined, so nothing to share yet, but I look forward to helping people take charge of their messaging.

Namespaces, Subjects and Verbs in JavaScript

It has been interesting to follow Dion Almaer's comments about using Dojo in Bespin. This post mentioning call conventions higlighted something I have wanted to talk more about.

The basic issue is namespacing and how you like to call functions. Here are some choices:

1. subject.verb()
2. namespace.verb(subject)
3. namspace(subject).verb(), which can lead to namspace(subject).verb().verb().verb()

subject.verb()
#1 is the Prototype/Ruby way. You define a verb on the object's class, or in JavaScript, on its prototype.

In this model the namespace is really the subject's namespace. There is a possibility of collision with other code that wants to use the same verb. This can be mitigated by keeping your project small and focused, and managing your dependencies. This also works well when your code is a leaf node, or one step from the leaf node -- your code is not consumed by other code, except maybe a top level web application.

The benefit is a nice call structure, and I think fits better with normal English. The subject is identified and then a verb/action is performed with that subject.

Unfortunately, in JavaScript, there can be problems adding things to basic prototypes, like Array, String (and shudder, Object). In Dojo we have had to put in some protections in our code in case the page also uses code that modifies the built-in prototypes.

I believe the situation will improve in future JavaScript releases if added properties/verbs can be marked as not enumerable. That will help a bit, but namespace collision is still an issue.

Some browsers now have native String.prototype.trim implementations and Dojo now delegates to them for Dojo's trim method. Recently, there was a bug filed for Dojo where the core issue was some other code adding a String.prototype.trim() method that did not strip out beginning whitespace.

namespace.verb(subject)
#2 takes the position that verb collision is bad and can cause hard to trace bugs, so always use your own namespace. It also feels more "functional" or procedural: use small functions that do not maintain interior state but operate on their arguments.

This has the benefit of being safer, but it can be more verbose than #1, and therefore more of a constant tax on the developer.

This is mitigated somewhat in Dojo, where you can assign Dojo to another namespace. So you could map "dojo" to "$" to cut down on the typing.

namspace(subject).verb()
#3 is a nice compromise: Define a function that wraps the real subject in an object and provides verbs to act on that subject, without directly modifying the object/class structure of the subject. jQuery took this to new heights by allowing chaining of the verbs. Similarly, dojo.query() returns a dojo.NodeList object that has chainable verbs on it.

An explicit namespace is involved, but the idea is to make it as small as possible, "$". So the overhead for having the namespace is "$()". Coupled with the chainable verbs, it can lead to short code, less of a tax on the developer.

For Dojo, I think #3 is the right approach in general: it gives nice namespace protection, but still gives short call structures.

I want to explore a dojo() function that does this verb mapping in general for all of Dojo. Dijit might be able to use a dijit() to get something similar for the verbs it exposes in its namespace.

So for instance: dojo({foo: "bar"}).clone() would act the same as dojo.clone({foo: "bar"}). Scopemap dojo to $ and you get $({foo: "bar"}).clone().

The difficult part is how to deal with verbs/methods that deal with Strings. I want dojo("div") to actually call dojo.query("div"). But how to allow dojo(" some string ").trim()?

Maybe not map dojo("string") to dojo.query("string"), but instead, allow scope mapping of "d" (or even "_"?) to dojo and another mapping of "$" to dojo.query. That would probably match best with the expectations of what $ does today in other toolkits.

I will have to do more exploration. Eugene Lazutkin's work on providing adaptAs* functions for mixing in dojo methods into an object prototype as chainable methods might point the way to do this.

Dojo 1.2 Loader and Build System

Dojo 1.2 is now available! Check out the release notes for more detailed information.

I attended the Dojo Developer Day in Boston on Sunday 9/28 and the Dojo Community Day at the Ajax Experience conference on Monday 9/29. It was really nice of the Ajax Experience folks to set up community time for the JavaScript toolkits.

During the Community Day, I gave a short presentation on some new things in Dojo 1.2 with the Dojo loader and the build system. I tried using the Google Docs presentation thing. You can see the slides here (some notes are after the slides):

customBase build option

The final example in the slides demonstrates how you can create a 5.5 KB (gzipped) dojo.js file and dynamically load it after the page loads, so if your site uses progressive enhancement, your up-front Dojo cost can be zero bytes, and just load as little as possible for the functionality that you use.

The 5.5 KB dojo.js file is possible using the "customBase" property of the dojo.js layer in a build profile. Normally we encourage you not to tamper with the contents of dojo.js (what is referred to as Dojo Base), but the customBase build option can give you more control on what parts of Dojo Base you want to load. With the customBase option, the build will go through all the JS files and automatically add dojo.require() calls for the necessary dojo._base modules.

Only use customBase if you keep all your JS code in JS modules -- good practice anyway, and keeps with progressive enhancement suggestions. Do not use customBase if you have Dojo calls inside HTML source.

Using customBase on the "dojo.js" layer with no dojo._base modules (just the loader and some bootstrap functions), the non-gzipped size of  dojo.js comes out to be 13KB (recall from above it is 5.5 KB gzipped). So it is small enough to fit under the 25KB non-gzipped size limit that the iPhone/iPod touch use to allowing caching.

You will likely need to use more of Dojo Base than what is in the 13KB non-gzipped file, but you can tune that by creating more layers and playing with the build to so that all layers are under 25KB.

djConfig.addOnLoad

Another option for use with progressively enhanced web sites, or when you want to load Dojo after page load: djConfig.addOnLoad allows you to set a function to run after Dojo loads. It is most useful when used in conjunction with djConfig.afterOnLoad (used when you manually load Dojo after page load).

Here is a complete example showing djConfig.addOnLoad, djConfig.afterOnLoad and djConfig.require.

There is a bug about using djConfig.require in conjunction with a dojo.js generated via a customBase build, so only use djConfig.require when using a complete Dojo Base file.

Thanks go to Matthew Russell for suggesting djConfig.addOnLoad for the 1.2 release.

stripConsole
There is a new build option for stripping out console method calls. stripConsole=normal strips out all console.* calls except console.error and console.warn. stripConsole=all strips all console.* calls.

Safari 3.1 sometimes has an issue with console.debug not getting defined (fixed in webkit trunk), so this build option can help avoid that issue until Safari is fixed.