Tagneto

Web dev with two turntables and a microphone

2012-01-30T13:33:00.000-08:00

There is a development style for web-based user interfaces a little up the road, from the habitations and the towns we know. It is not new or unusual, but I think it needs to be spelled out a bit since it is the "why" behind tools like RequireJS and volo, and it is what I expect to be the defacto development style for the future, particularly for mobile web-based apps:

Do not use servers to build web-based user interfaces.

What do I mean by "do not use servers"?

Servers should be restricted to providing data API services only.
A simple web server to serve static files for UI can be used, but it is only to support serving static resources that can be heavily cached. The UI serving is strongly decoupled from the API services.

There is a JavaScript corollary for this approach, and a "red pill" variant, but first more on the benefits.

Design advantages

This separation of concerns gives the following benefits:

Better scaling characteristics. Static web UI can be offloaded to Content Delivery Networks (CDNs) and heavily cached. Even if the UI is served from the same domain as the data API services, the UI can be served from separate, simpler servers that can be scaled independently from the API servers.
Forces good API design. Getting your data from a network API enforces strong constraints on what are data needs and what are UI needs.
Better offline. This is really critical for mobile UIs. You can set up the UI to use AppCache and give the user feedback if the network is not available.
Better division of labor between teams. Front end UI developers and back end data developers can have different personalities and scaling concerns. Having a strict API contract between them allows them to each focus on what they do best.

Ideal use cases

Basically, the future:

JavaScript-driven web apps. Message-based/social applications, maps, games.
Mobile web. Sites served to users on mobile devices, devices where connectivity is not always guaranteed.
Installed "web apps". Phonegap/Cordova-backed apps, or something like Mozilla Web Apps that will allow you to install your web app on an OS so that it has its own desktop icon and may launch the UI from files on the local computer/device.

Use cases that may not fit

Some content-heavy UIs, like a CMS or blog, might work well as a PHP/Rails/Django/Node app where the data (content) is injected into the page. This is primarily to satisfy search engines or deal with older browsers that, for instance, do not support the history API.

However, for more static content, like blogs, much of the content can be generated once and saved to disk, and could still fit into the "serve static resources" approach. memcached is often used to achieve this effect.

So even for these cases, see how much a data API can be used to generate static output for the UI instead of doing a runtime merge for every request. It will make it easier to deal with the request-hungry googlebots, and enable the other benefits above.

Of course this will not work for everything, but for the next project, see if it does fit.

The JavaScript corollary

Any tools you need to build the UI should be written in JavaScript.

You are already using JavaScript in the page, might as well use it for your tools. It makes it easier for front end developers to share and improve the tools.

While we may not be at this end state yet, we can now get there now with Node.

Wait, isn't Node a server? It is an event-based JavaScript runtime that happens to have good networking and file I/O support. You could use it to build an API server, but you can just treat it as a command line tool environment. That is how the RequireJS optimizer and volo use Node -- a bootstrap for single file command line tools.

By using fs.watch(), you can set up tool actions to occur on file changes, no need to run the UI through Node.

If you prefer sugar like CoffeeScript and SASS-like systems, go for it. Just use command line/watch tools to use them. And ideally the tools are written in JavaScript. If they are not, that is OK, consider JS the next time a rewrite is in order. Or better yet, create JS-backed versions of the project now and use it for future development.

You do not need to subscribe to this corollary, the big step is to just embrace serverless UI design.

The red pill

If you want to go with the "use JS for tooling", you can go a step further:

Avoid tools/dependencies/sugar to help you make UI, and embrace plain JS/CSS/HTML. It is all you really need to know. Sure, you need some tools around that to help with source code management/deployment, but those are needed in any case. You can simplify by avoiding abstractions.

Of course I still support you if you want the extra sugar or tools to make web development more palatable for yourself. I can see using tools like bicycles to help you get places. Even if using plain JS, it is helpful to use some libraries to smooth over browser differences. However it is great to keep the extra gear you need to understand and maintain to a minimum.

The tools I make will focus on getting as close to the JS/CSS/HTML bare metal as possible, but they should also work well if you want to go up the abstraction ladder.

RequireJS and AMD modules are about sticking to the JS metal but still provide a basic capability missing in JavaScript. AMD is a great "transpile" target if you want sugar on top of AMD. volo is about getting your web project set up easily with the dependencies you choose to have.

Summary

Serverless UIs supported by JS-backed tools are where it's at.

RequireJS 1.0.5 released

2012-01-27T13:04:00.000-08:00

RequireJS 1.0.5 is available.

The notable changes:

Small fix for running the optimizer on Windows with Node 0.6.8.
Fix bug with loading Dojo code from source.
New config option that allows using the RequireJS config from your main JS file as part of the optimizer options. It is mentioned in a new "common pitfalls" section in the optimizer doc.

Complete set of changes:

AMD support for Underscore and Backbone

2012-01-22T23:14:00.000-08:00

As mentioned previously, at the moment there is no support for AMD in Underscore, a decision that also affects Backbone.

However, there have been enough developers interested in using Underscore and Backbone with AMD that forks were set up under the amdjs organization that have optional define() calls in them:

https://github.com/amdjs/underscore

https://github.com/amdjs/backbone

Ryan Florence suggested the AMD community maintain these forks, and if you would like to see AMD support as part of the main repos, click on the links above and click the Watch button. But only do it if you actually use the libraries above in an AMD context.

Hopefully this will give the DocumentCloud folks some indication of how many people might benefit from having support directly in the main repos, and maybe over time AMD support can be added to the main repos.

Both forks optionally register as AMD modules but also export global objects. This is to address the difficulty Underscore saw when only doing the AMD call, which led to the AMD removal from Underscore.

If you want an easy way to fetch these AMD versions from the command line, or to have a tool to make it easy to AMD-wrap older releases of those libraries, use volo. The README's Usage section has specific instructions on how to fetch these libraries.

If you would like to try an AMD+Backbone option that does to not modify the source files, check out Tim Branyen's explorations.

AMD, the question, and a JavaScript sugar analogy

2012-01-20T11:44:00.000-08:00

I want to expand more on the feedback from Tom Dale and Patrick Mueller (see part 2 too), who do not think AMD is the answer, and Dave Geddes asking at the end of his post what was the question?

More on the question, and an analogy:

How can we get to a future where we can easily share JS code, one that works on the server and in the browser?

I have been talking about the advantages of AMD because it answers that question very well, and I have been reading people's responses to suggest CommonJS (CJS) modules as the format as impractical because it cannot work everywhere. Maybe what we can do is agree on some basics that still allow for sugar.

In other words, consider AMD the "base format" for distributing JS code, because it can work anywhere. That was its design goal. However, it is fine to use sugar on top of that, and even distribute source in that sugared form. But it would be great if the runtimes/tools you use have the capability to ingest AMD (if you take in outside code) and the tools that output your code should be be able to register as AMD modules.

An analogy:

JavaScript is winning because it is widely deployed. However, not everyone wants to program in plain JS. Some folks like to use sugar, like CoffeeScript and Stratified JS. In the end though, those sugared forms reduce to JS so that it can run in the most widely deployed environment.

AMD is to JavaScript as CJS modules/browser globals are to CoffeeScript/Stratified JS

It is fine to use sugar, it just helps if that sugar can take input/generate output that works everywhere. A bit more on why AMD can work everywhere:

Wrap It Up

Any complete JS module solution needs to have a wrapped format to allow:

1) serving optimized code, where multiple modules are combined into one file.
2) dynamically loading code from CDNs.

AMD solves this need, by allowing string IDs and a function wrapper, the shortest form is here:

define('module/id/here', function (require) {

    //Module code in here.

});

The current ES harmony module proposal may be able to get around a wrapped format for case #2 (cross domain script fetching), but they need a "wrapped" format to allow #1, and it looks like:

module moduleIdHere {

    //Module code in here. 

}

I think it is better to use string IDs for the harmony case, and harmony is by no means done, but still, the need is the same: when combining multiple modules together. they need to be demarcated and named.

Given old browsers and old code that cannot participate in harmony, I believe any harmony module format should be designed to be translatable to AMD since AMD works everywhere. At the same time, we should make converters from AMD to harmony. I can appreciate these conversions may not allow 100% conversion, but it should be a very large percentage. I see any harmony proposal as sugar given it will not be ubiquitous. Maybe in 2-5 years after ES.next comes out the story will be different.

Give Me Some Sugar

Tom and Patrick object to having to write too much boilerplate, or ceremony, to code a module, and that is great goal. It is the same reason why some people like CoffeeScript or Stratified JS. But, we need to have a common wrapped format if we want to share code in one file bundles and from CDNs.

The Middle Way

Viewed in this light, I'm hoping we can all code in our preferred sugar, but still be able to share modular code at the base level. Here are some suggestions for us to get to that goal, some for browser library authors, node library authors, JS environment authors, and AMD Toolmakers:

Browser Library Authors

Not all of these may work for you. If you cannot do the first or second one, at least consider the third option.

Consider writing your modules using AMD. The define(function(require){}) wrapper is really light on boilerplate/ceremony, particularly if you are already using a CJS-style of code. There are AMD tools you can use to help with the building of your library, and it allows others with AMD tools to easily use your code in source form.

You get a free "custom builder" capability with this approach. Consumers of your source, particularly if it consists of multiple modules, can just drop your library code in their project, and when they do a build with AMD tools, it will only pull out the modular pieces of your library that they actually use. That is awesome.

You will be using a common idiom that people understand. It will be easier to translate skills.

CJS modules have this quality too, so feel free to use them if you prefer that style. In that case, I hope you will still consider the rest of these suggestions.

Consider using AMD if you need to bundle more than one module together. Almond can be used to provide the AMD API shim. Ace's mini-require might be another option.

If you limit yourself to just define() declarations, no loader plugins or callback-style require, then you can easily make your own wrapper. If you just need a way to concatenate the modules together, but do not need the require calls after a build because you are using globals (I think Tom fits in this group) then you might be able to get by with plain anonymous function wrappings, so this item may not apply to you.

For the top-level, public API for your module, feature detect for define.amd, and register your code as an AMD module. This is particularly helpful when you have dependencies. You can also get away from using globals and allowing better named dependencies.

You can still export a global for now, even in the AMD case, because AMD is still new to some people, and as underscore discovered, it can lead to problems. I expect this type of problem will go away after AMD support has been around a while and the global export can be removed. Since modules are still a new thing to many JS programmers, there will be a transition period while developers get used to not relying completely on globals. This will be true even if/when harmony modules ship.

By calling define() an AMD consumer now has the option now to get direct references for your code under a name of their choosing, and properly state and wait for its dependencies before running with them. The umdjs/umd project has some boilerplate examples to help you get started with this item.

Node Library Authors

If you author user-land node modules for others to use, in other words, do not develop Node core, and you think your library may be usable in the browser, then:

Stick with declarative dependencies. Avoid using calculated values for dependency names:

var a = require(someCondition ? 'a' : 'a1');

Ideally node would support a callback-style require(), as AMD does, to help when you do have a calculated dependency (see JS environment authors section below), but in the meantime, the module will translate very easily to AMD with just declarative dependencies:

var a = require('a'),

    b = require('b');

These type of calculated dependencies will not work in harmony modules either, so it is good to get used to this now.

Avoid __dirname and __filename. Or at least, be aware that in AMD loaders, you have access to module.uri which is like __filename, and a __dirname can be calculated from that value. So this type of detection is a good way to go: var uri = module.uri || __filename.

Use amdefine to create a define() method in your module, and call define() to define your module. This will make it easier for browser authors to directly consume your code, and the RequireJS optimizer removes the if(){} block that uses amdefine, so browser developers will not take the hit of that code or that module.

Consider using AMD as the bundling format, if you make a bundling tool, like browserify. If you need help with this or have questions on how best to do it, feel free to contact me.

The wrapper can be as simple as define('id', function(require, exports, module){});. You may need to provide an option to wrap your output in a function and use an internal define inside that function, but call the global define to expose the public API for the bundled set of modules.

By using the amdefine module as a dependency in your package.json, this will help test a real implementation of define(), and help core Node contributors get a feel for how many modules would like to have define() as an API option directly in Node.

JS Environment Authors

Do you make an environment that can run JavaScript? Like Node, Rhino, Mozilla's Jetpack?

Support AMD APIs for easily ingesting code. You can support the sugared CJS modules as a default, but allow JS library authors to reduce their feature detection boilerplate and allow easier code transfer by allowing define()'d modules.

This does not mean that you must load modules asynchronously. The nice thing about the basic define() API is that it works well in synchronous environments too, particularly with the declarative dependencies.

For Node in particular, I know this has been a contentious thing to consider, and it can be done in phases. Not all the phases need to be done, particularly loader plugins, but each one allows easier code sharing:

Support module.uri and encourage that instead of __filename. In AMD, module.id is a module ID that is not a full path, so there is explicit module.uri for getting the module's path.
Support a simple define() call, like the one Isaac had in previously. It can still operate synchronously.
Support callback-style require([], function(){}) inside define() calls, one that fires the callback on nextTick. This is important for calculated dependencies.
Consider loader plugin support. It could be limited to loader plugins that can only be run synchronously. This one might also help with the sort of transpiler support you have in Node now.

The previously mentioned amdefine package for Node, which consists of one file, does all of the above, and I am happy to continue work on it, build up unit tests for it, do whatever code changes you think are needed that would allow something like its define() implementation to land in core.

I think Node's synchronous module behavior can be maintained, but still open up easier code sharing pathways with code written by browser-based developers.

I hope my previous posts show why Node's module system as-is is not the right general solution for module-based development in the browser. The ES Harmony module proposal set has similar high level features as AMD, like a module in a block, the equivalent of a callback-style require. The need for those things are not going away.

Plus, the more browser developers (the biggest JS audience there is) can easily transfer their knowledge/idioms/code to run under Node, the more it helps ensure Node's longevity.

Support in other JS environments is also encouraged. It can take a similar path to the one outlined for Node. Mozilla's Jetpack right now only supports a simple define() call, and that is great. It still allows for some basic code sharing. Even getting a little bit of support, doing small steps is fine.

AMD Toolmakers

For those of us that make AMD tools, there are some things we can do that can help people share their code easier. Here is what I am doing:

Create translation tools. I'm working on volo, which can do this, among other things. With volo, you can download your favorite libraries from GitHub and optionally specify AMD wrapping for the library. Right now it is set up to wrap a library that uses globals. I want to add support for converting CJS/Node modules to AMD. CJS conversion is cleaner than the globals wrapping -- the globals wrapping needs more information from the developer to work.

Consider configuration of global import and export of module values. In a runtime loader, allow the developer to configure something like "when underscore loads, grab the global _ and use it as the value for the module". And conversely, "when this AMD module loads, also assign the module export value to a global named 'foo'". This may be better served by the translation tool -- it can be more direct, precise -- but this might help for cases in which the developer does not want a tool to modify the library source. It would only help with libraries that have no dependencies though, so it may not be useful enough on its own.

It is still best for library authors to call define() themselves since they can be more precise on the dependencies and allow getting direct references instead of using globals. However, the AMD toolmakers can help bridge the gap.

Globalize All The Things

All of the above assumes you think getting local handles on other modules/scripts is better than using globals. Not all JS developers hold that position though. In particular, I think Thomas Fuchs, Jeremy Ashkenas and to some extent Scott Gonzalez have all expressed a desire for a system that does not require changing the source files from using globals.

I'm open to hearing how that might work, but I think it is unrealistic as a general approach because:

We already have a problem with globals today, where multiple third party libraries want to use the same global but at different versions. For anything larger than a simple app, it is not scalable to try to do a "noConflict()" approach on nested dependencies.

You could try to get around that by executing module sets in a "sandbox", but the tech to do that with today's browsers, iframes, is riddled with land mines and it has trouble sharing things across frames.

It is bad practice for modules to name themselves. Case in point: Zepto. Zepto is targeted as a jQuery replacement. However, to use it as-is, it means libraries like Backbone have to do manual detection for it: var $ = root.jQuery || root.Zepto. This is not scalable, and it would be to Zepto's advantage to register as an anonymous module, and then allow the end developer, not every library author, to map require('jquery') to a path that resolves to Zepto instead.

It does not map well to other environments like Node and Jetpack, which have gone away from globals.

To be clear here though -- I want a module format that allows for the use of globals, and AMD does allow that. It is just not good to enforce use of globals as the path to code sharing/use. There needs to be a way to get local references via string names that can be mapped to other providers of that functionality.

In the end, I see globals as another kind of sugar. The tricky part is that it is hard to auto-convert given the variability between what a file is called, what global(s) it can export and what dependencies it uses, and mapping those dependencies effectively to files. Systems like AMD/CJS have a much more consistent approach to this, which leads to cleaner code and more effective sharing and tooling.

Summary

Hopefully this gives some concrete ways we can work together even if you prefer some sort of sugar over plain AMD.

Supporting AMD, even just on the output/input edges, is not because you think dynamic loading of scripts is the way to go, or that you have to throw away your favorite sugar. It is about getting a base level of code understanding that allows effective sharing, tooling and reuse across all JS environments.

Reply to Tom on AMD

2012-01-16T23:02:00.000-08:00

Tom Dale had a good post about why he dislikes AMD. Please go check it out. We talked about it a bit while he was writing it. It is great he took the time to write it out. We need more of that. We are all trying to find the best path, and it is great we are talking this out.

While I appreciate the critique, what is missing is a solid alternative to AMD. In particular, "use build tools to wrap up CommonJS modules" is not a generic solution. It does not allow for generic dynamic loading of resources, particularly from CDNs. Dynamic loading is needed for big sites that still do builds but want to stage loading of the site as the user uses it.

On his concern about too much ceremony -- this is the minimum amount of ceremony:

define(function(require) {

    //Module code in here

});

By doing that, it is easier for other people to consume your module since module authors do not have to provide "built" versions of their module to use on CDNs. And if they did have to do builds, what format would they publish it in? It would need a function wrapper. Hey, AMD has that!

That simple function wrapper allows moving away from globals, although you still can use globals if you want. It allows for dynamic loading. Recall I'm talking about built layers for staged loading/performance, not "load each module with a separate HTTP request". It allows very simple module sharing now. It allows easier sharing of built files too, since AMD specifies the built file format.

With AMD, you can still use build tools if that is how you like to roll in dev. Go for it. Almond is a great AMD one for that purpose. Here is an example of using the RequireJS optimizer for runtime HTTP-served builds. And since there are multiple AMD implementations, you can choose how big of an AMD lib you want.

I'm sure Tom could adjust his 50 line loader to process the define calls, since his built code probably has function wrappers. He could even adapt his "hydrate from strings" loader to use the AMD syntax.

By using that function wrapper, others could consume his individual modules and use their own choice of AMD loader to deal with their particular loading needs.

On the "hydrate from strings" build approach: I would prefer using appcache and using built layers of AMD code instead. This gives much better overall mobile app performance since more than just JS can be cached. But that is more of a side point. Feel free to use the "hydrate from strings" approach, AMD could be used in that fashion too.

My summary: it sounds like he just wants to avoid a small function wrapping and a level of indent. And to be fair, many simple AMD introductions use the array of dependencies syntax, not the sugared form above, so I can appreciate the confusion.

But trying to avoid that function wrap+indent comes with some code sharing costs, and the alternative he mentions does not handle the breadth of JS module consumption.

Simplicity and JavaScript modules

2012-01-11T22:38:00.000-08:00

All of us are looking for simplicity, but there are different levels to simplification. This is a story of what could be considered simple for modules in JavaScript. This post was prompted by the removal the optional AMD define() call in underscore.

For a post on simplicity, it is a bit long, but I'm not a great writer, and I find I normally edit myself so much as to lose interest in posting, so then I end up not communicating. Better to start communicating even if imperfect.

I want to lay out why AMD modules are the simplest overall module solution for JavaScript at the moment, and where other approaches are not as simple as they may appear.

Script Tags

JavaScript does not have syntax for modules, but most programming languages do. "import", "require", "include" seem like popular choices.

JavaScript on the web has meant using script tags and manually ordering those script tags so that dependencies on global objects are worked out correctly. It is also important that those dependencies execute in order.

That sucks for the following reasons:

You use a separate language, HTML, to specify your JS dependencies and their order.
A script's dependencies can be unclear.
If you want to later load some code on demand, it is very difficult to work it out so that the scripts execute in order. This has gotten better over time -- newer browsers that support the async="false" attribute help with this. But older browsers still suck.

So what happens?

You constrain yourself to only creating simpler applications that can get by with concatenating all the scripts in a certain order, and depend on server tools to help you write out your HTML with the script tags and the rewrite the page after a build to not have those script tags. Think Ruby on Rails or Django.
Toolkits start building APIs to do this work for you. Dojo and YUI were some of the very first to do so. The Closure library does too, although it was more of a cousin to Dojo.

Option 1 is clearly not appropriate for the full spectrum of web development. There are HTML game engines, webmail/office suites, and then browser-based "no server" development like Phonegap-backed mobile apps.

So it is best to have some API or syntax in JavaScript to get units of code. For it to work well, we should all be using the same API. Otherwise things get messy real fast.

CommonJS

The CommonJS group tried to work out a system for doing this. It is pretty simple too:

require('some/id') to reference a bit of code. Since a string literal is used, it allows for easy static analysis of dependencies -- no more manually ordering script tags! The ID also has a convention of mapping to a partial path. So you can get by without having to specify full URLs and it opens up for ways to map an ID to a path does not fit the ID-to-path convention. This is really helpful most immediately for mock testing, but it has many other uses.
Modules do not give themselves a name, they are anonymous. This is great -- if you load require('jquery') then you do not have to have special knowledge to access some global with different spelling, like jQuery. Same for 'backbone.js' loading an object called Backbone. The end user is in control of the name.
exports is handy for circular dependencies. Yes you should avoid circular dependencies. If you have one, there is a good chance you are doing it wrong. But there are valid circular dependencies, and a script referencing system needs to support them.
module is important because modules are not named. Sometimes though you need to know the name they ended up with, and sometimes from what path, because you may have some non-JS assets you need to reference. module.id and module.uri give you that ability.

There were some wrinkles though:

They did not formalize a way to export a module value that was not just a plain object with properties. It is extremely common and useful to want to export a function as the module value. jQuery is probably the most known example. Constructor functions are another set of useful export values.
It was not so simple to get it to work in the browser.

AMD

The original participants on the CommonJS list thought it was better to blue-sky the development of a module syntax, not be held back by what might work in the browser, just what might work with existing JS syntax. The group also started off as ServerJS, so they were also in the mindset of what would work best on the server, where file I/O is cheaper/easier.

The hope was that if they worked out a module system that worked well with the existing JS syntax but had problems in the browser, hopefully they could convince browser makers to plug the holes to make it possible. In the meantime, since they were developing servers that could run JS -- they could just bundle up/transform the JS using server tools, or offer a compile step, while browsers caught up.

However, for those of us who came from Dojo, requiring a server tool or compile step to just develop in JS was a complication. I'm going to mangle Alex Russell's quote on this, but "the web already has a compile step. It's called Reload".

Why force the use of a tool to just start developing? It should be simple: just a browser and a text editor.

This was accomplished by taking the CommonJS format and allowing a function wrapper around the code:

define(function (require){

    var dep = require('dep');

    return value;

});

Or, the shorter form::

define(['dep'], function (dep) {

    return value;

});

Since a function wrapper was used, it meant:

the scripts could load in any order they want, easy to parallelize even on old browsers.
"return" could be used to return the module value, even functions.
No special tooling was needed to convert source to an acceptable browser format.
It worked in browsers, today, and would in the future.

So the story around modules gets very simple, no special tooling to start, no worrying about "I need to run a converter to share this module", no special sets of instructions for your server of choice to do conversions.

Yes, a loader library is needed, but one is needed in any case, there is no native JS syntax. That is the basic ante for any module system that scales up beyond a simple JS concatenation for a Rails application.

For some people, a function wrapper with a level of indent was not seen as simple enough. However, designing a system without it meant a bunch of complexity once you stopped looking at an individual file. A miscalculation of the overall complexity cost.

Loader Plugins

Another simplicity in AMD systems: loader plugins. Loader plugins would not be considered if you had copious, synchronous IO capabilities. However, by fully embracing the network/async loading on the web, you start to see how creating a loader plugin that treats a dependency as a simple string as useful.

Some dependencies are not static scripts, but could have more complex loading (Google Maps code) or simple HTML templates that need to be loaded for the module to be useful.

In AMD, this can look like so:

define(function (require) {

    var maps = require('googleapi!maps?sensor=false'),

        template = require('text!form.html');

    return function (data) {

        //You can  synchronously return a value

        //based on the template.

        return template.replace(...);

});

});

Compare that with a non-plugin approach. Do you want to load those two resources in parallel? That would be ideal, but then to do that, you probably need something like a promise library to help out. And now you have two problems. I mean that partly in jest -- I use a promise library for some types of code -- but it is definitely a complexity hurdle to jump.

The async networking also means your module is not completely ready until those resources are available, so your public module API now must be a callback API. For "simplicity" assume you just load the dependencies serially, to avoid some of the promise-isms.

define(function (require) {

    var googleapi = require('googleapi'),

        text = require('text');

        googleapi.fetch('maps?sensor=false', function (map) {

            text.fetch('form.html'), function (text) {

                //dependencies are now loaded.

});

});

    return function (data, callback) {

        waitForDependenciesToLoad(function (data, callback) {

            callback(template.replace(...));

});

}

});

Loader plugins simplify async development, and some of their resources, like the text templates, can be inlined in a build file with other JS modules. Loader plugins give you simplicity and speed.

If you like transpiling other languages into JS, they are great for that purpose too.

ECMAScript

The ECMAScript group wants to get modules in for the harmony effort, the next ECMAScript release. They chose to not go with the CommonJS syntax, but what did they did choose looks fairly similar on the surface. It favors the introduction of new syntax to get some advantages with compile time checks. You can check out the following for more information:

Those links have changed over time, so the rest of this feedback may not be valid in the future. As I read it today, I do not believe harmony modules give much benefit over what AMD can do now, but harmony modules do introduce more complexity, and has some unanswered questions:

It uses module Foo {} syntax for declaring an inline module, where the module ID, Foo, is a JS identifier. However dependencies can string names/URLs. How do you optimize this code for delivery in a web browser? In AMD, string names are used both for references and for the module names. This is better because it allows for loader plugin IDs that can have their output inlined in build output, and it ties module IDs more directly to the string names used in dependency names.
It does not support a loader plugin API out of the box. You can construct one by using the module_loaders API, but at that point you need to ship a .js file for the "loader", so now the developer has the same complexity cost as AMD today.
New syntax means the module support cannot be shimmed in older browser via a runtime library. It requires a compile step/server transform to work in older browsers. This is one the same problems the CommonJS format had.
It is unclear how a JS library that works in browsers without ES module support "opts in" to register as an an ES module since ES modules use new syntax. Maybe the suggestion is to use the module_loader's loader.createModule syntax? I cannot see how that fits with the compile-time export checking though. If a runtime capability check cannot be used to opt in to ES module registration, it makes it very hard to upgrade web libraries to ES module syntax. We're back to the problems that made it hard to use CommonJS syntax on the web.

The new syntax in harmony modules is used to enable some compile time checking of export names and if they are referenced correctly.

However, compile time checking to see if an export name is use correctly is a very small benefit for the end developer given the other costs above. Furthermore, I want more than just an export name/type check. I want intellisense on the arguments that can be passed to functions, general data type information and comments on usage.

Working out a comment-based system that can reflect this info into text editors will provide much more value. Since it is comment based it fits in with old browsers. I know it is harder to agree on that comment syntax (mostly because it is easy to bikeshed), but it will simplify developers' lives more, and lead to faster turn-around time on development.

If something like loader plugins are not natively supported, and the optimization story is not sorted out, it does not have an advantage for AMD today, and AMD is simpler.

However, the harmony module_loaders API is really useful. I'm not sure it needs to be as big as it currently is. I would be fine with something like Node's vm module API as a start. Basically, some container or vat I can load scripts into without interfering with other scripts. This is one area that is very hard to do on the web today.

So for me, the "simple" solution for any ES6 module-related work:

Make it a module API, not new syntax. Then we can shim it easier for older browsers, and existing libraries and capability check it and opt in. If you need a suggested API, I hear AMD is quite nice. It has a few implementations and has been used in the real world.
Support loader plugins natively. It really helps with async programming, and optimizations. If I need to ship a library to deliver the benefits that loader plugins give for async programming, then there is very little motivation for developers to switch away from AMD.
Provide something like Node's VM API, maybe with a little bit of the intrinsics stuff in the current module_loaders API.
Do not bother with compile time checking of export names. Instead put effort into a comment based system that can reflect more information for use in editors. That will save developer more time. Since it is comment based, it will work in older browsers and optimizes out cleanly.

I will post this feedback to the es-discuss group. I wanted to hone my feedback before giving it to the es-discuss group, but this will have to do, otherwise I may never give it.

Underscore and AMD

This brings us to the recent code change to remove the AMD block from underscore. The arguments for removing it are listed here, but I think the main argument is really about what is simple for Jeremy: He does not need it for the kinds of sites he builds, and by adding it, he has gotten reports of problems.

Those problem reports go away if he also exports underscore as a global, but he does not think he should have to do that if AMD is available. I do not think that is a fair bar to hold for AMD, since he would have to do the same for a harmony syntax, so his lib could be used in cases where ES loading and old style global loading is still in play.

Some feedback on his specific reasons:

Folks requesting other module formats for other loaders.

No other module format comes close to the level of support AMD has: AMD has multiple implementations, better support in other libraries (Dojo, jQuery, libraries friendly to Ender-bundling, MooTools), it is used in real sites, and has a thriving amd-implement list and thriving implementations. What else can claim all of those things? What else is being asked for inclusion?

I can appreciate it is easy for someone to come up with a loader syntax and want people to use it. I think AMD has gone the extra steps though to be considered more of a standard. But it depends on what you want to use as for standards of legitimacy.

If any library that depends on Underscore (and there are many of them: http://search.npmjs.org/) does not yet 'support' AMD, but Underscore does, things get royally screwed up.

npm is for node modules, not browser modules, so they would not have the error that was being reported for Underscore.

That said, I do believe Underscore is a common dependency for browser-based code. But for the browser, as mentioned, registering a global is perfectly acceptable for this transition period, something I believe will need to happen anyway no matter what modular format is chosen. There will always be a transition period.

In an ideal situation, libraries do not have to be modified to support a particular script loader (or group of loaders).

The point is to make developer's lives simpler. A script loader like that does nothing to help order the dependency tree correctly, or use a naming convention that can be parsed easily by tools like optimizers.

The browser globals with implicit dependencies just do not scale well past maybe 15 dependencies? Not everything fits as "lets concat all the scripts" Rails app. But I'm open to seeing a design that might scale up.

JavaScript's upcoming native module support is entirely incompatible with AMD.

They are very compatible as far as base semantics. As mentioned above, ES harmony modules are still baking, not ready for prime time. But even with that, it would be very easy to convert AMD code to harmony module code -- since the dependencies are all string names it fits in. Loader plugins would be a problem for sure, but basic modules are easy.

It is much more compatible than the current "browser globals and implicit dependencies" approach.

Loading individual modules piecemeal is a terrifically inefficient way to built a website. Because of this, there's the great RequireJS optimizer, which will turn your modules into ordinary packages.

The browser globals approach already depends on build tools, so I'm not sure why this is a knock against using AMD. A bunch of manually typed HTML script tags perform just as poorly.

Fortunately, since dependencies can be easily statically determined with AMD calls, the developer no longer has to worry about manually figuring out the load order, and there can be (and are!) many different build tools built on top of the standardized AMD API.

Summary

In the end, though I just think it boils down to Jeremy not needing this personally based on the scope of his work, and he has other things he would rather work on. It is hard to get a standard of legitimacy in this area, so it is easier to just wait it out. For the particular issue in Underscore, the simple export of a global even in the AMD case would have solved the issue, but such is life.

I'm a bit sad because Backbone and RequireJS have been a very popular combination. They fit very well together. The thought of maintaining a fork/branch is distasteful, particularly since the AMD patch for Backbone was smaller than the code it replaced.

Auto-wrapping tools are difficult to do generically given how scripts want to grab globals. The dependency name can have weirder names that do not match the file name, so it loses the nice, simple dependency parsing of AMD module IDs. It means creating a centralized list of global names that map to IDs/paths. Not very webby/distributed. Not very simple.

Oh well. It probably means AMD just needs a bit more time out in the wild, even more adoption, and we'll see how people feel in 6 months.

Another Approach?

I have heard complaints from folks on the internet about AMD from time to time, but they have not offered anything better, particularly given the simplicity tradeoffs mentioned above. I think it is just an NIH thing most of the time, or getting it mixed up with generic browser script loader.

Here is a survey of things I know about:

Ender is fine if you want to just build a file that you will not change often and use it in place of jQuery, but it really does not help with the larger site structure and loading issues, being able to dynamically load. It is not a general module system for the web. It basically is just like the CommonJS "do a build before starting development" approach. Same as SproutCore/Ember. Same complexity problems as mentioned above, and complicates individual module debugging.

Dan Webb started something with loadrunner. It supports part of AMD, and the "native format" it supported do not seem better than AMD, maybe just another function nesting and different API names.

Ext has something similar to AMD, but mixes in a particular class declaration syntax into the format, so that will not fly as a general solution.

YUI is close, but obscuring what a dependency name loads on a Y instance makes it hard to associate what came from what module, and the API to add a module relies on naming the module and putting named fields on the Y instance. This will lead to name collisions.

What else am I missing?

One thing these approaches all have in common: they get away from the browser globals and implicit dependency approach used by traditional browser scripts. If you think something using that traditional pattern is the future, please describe how it might work. Remember, requiring build step to just start developing is a complication. That does not scale well across all the types of JS development mentioned above.

While Jeremy and I were talking in the documentcloud room, he mentioned that Ryan Dahl, if he could do it over again, would prefer not to use CommonJS modules system for Node, but do something closer to how browsers load scripts in script tags.

I think I have heard that comment too, but in the context I heard it, it seemed like an off-hand comment. I'm not sure how much that was just about having to wade through CommonJS discussions or actual problems with the module API. I would love to hear more about it though. Ryan or someone how has more info on this, if you happen to see this post, please clue me in. I'm on github, the amd-implement list, or on gmail as jrburke.

I want to get to a workable, simple solution that works well in web browser too. I only do AMD because I need it and I think it is the simplest overall path for end developers. But I want to solve other higher level problems. I want something to good to win. It does not have to be AMD, but I do think it hits the right simplicity goals particularly given browser use. It will not be brain-dead simple because upgrading the web is not simple. But it does a pretty good job considering.

I feel like the folks doing AMD have done their due diligence on the matter. ES harmony may be able to do something a bit different, but the basic mechanisms will be the same: get a handle on a piece of code via a string ID and export a value. The rest starts to look like arguing paint colors.

Anyway, enough of that. Back to making things to make things better.

RequireJS 1.0.4 released

2012-01-04T21:07:00.000-08:00

RequireJS 1.0.4 is available.

The big fix for this release is making sure the config options are passed correctly to uglifyjs. There was a problem with 1.0.3 that would result in some files not being parsed correctly.

The small set of bug fixes for:

volo: A JS tool for JS projects

2012-01-03T00:27:00.000-08:00

Over the last month I have been playing with a tool to help me make JS-based projects. It is still very early in its development, but it can do a trick or two now, so it might be fun for you to try out too.

volo is a command line tool. It is written as a set of AMD modules that are built into one JS file that runs in node.

Check out the README for more info, but what I like about this project:

Just one JS file to install.
It uses GitHub as the code registry.
You can create new commands for it (using volo) and if you put your commands on GitHub it is easy for others install them.

Still lots to do and there are lots of sharp edges to it, but it already feels pretty sweet. If you would like to help out, check out the design goals and working the code. Also feel free to give some feedback on some of the more important issues I would like to sort out:

Windows support
"Short name" dependency lookups, possibly using some sort of GitHub search API.

On a side note: I have been on vacation and not checking email. If you are waiting on a reply from me, I should get to it this week.

RequireJS 1.0.3 released

2011-12-24T00:15:00.000-08:00

RequireJS 1.0.3 is available for download.

Small fixes, mostly for the optimizer. See the 1.0.3 fix list for:

almond 0.0.3 released

2011-12-07T18:52:00.001-08:00

almond is an AMD API implementation that does not do any dynamic loading -- it is best used as part of a deployment wrapper for AMD modules that are combined into one file. Version 0.0.3 is just 857 bytes when minified with Closure Compiler and gzipped.

Changes in 0.0.3:

Patrick Mueller had some suggestions on how to make it more friendly to CommonJS modules that are simply wrapped with define(function (require, exports, module){}) calls, and to allow out of order module listings in the built file.
Works properly with jQuery 1.7+ versions that optionally call define().

RequireJS 1.0.2 released

2011-11-21T23:08:00.001-08:00

RequireJS 1.0.2 is available for download.

Small fixes, see the 1.0.2 fix list for:

Why not AMD?

2011-11-15T13:14:00.001-08:00

This started as a private response to some tweets by Jeremy Ashkenas that indicated some concern around optionally registering code as an AMD JavaScript module, but since his comments were in a public space, it makes sense to post my response in a public space.

His concerns as mentioned in the tweets:

Isn't canonical in any JS runtime
Inefficient by default
Incompatible with JS.next
Known temporary solution

Twitter is bad for this kind of conversation, and Jeremy is busy with work, but hopefully he can expand on his concerns at some point if my responses below do not address them.

Isn't canonical in any JS runtime

The goal is to create a grassroots effort based on real, multiple implementations that makes it canonical. I believe it is following the way to get something standardized. There are multiple implementations (requirejs, curl, dojo and mootools), and there are multiple higher level toolkits that register with it: jQuery, Dojo and MooTools.

It is also based on real world experience supporting modular code in the browser. It is informed by Dojo's previous modular experience, and jQuery's needs for modular loading. I believe this places it in a better position than how modules in ES harmony are being designed, particularly since it can be opt-in that works with ES3 grammar.

Node adopted a variation of CommonJS modules, but Node's implementers have been very explicit about not picking up anything else from CommonJS, and they support their own non-conformant extensions, like module.exports. Node is a monoculture and very inward focused, and since it has sync IO, it is not concerned with browser needs.

Even if CommonJS were supported well in the browser today, there still needs to be a way to embed multiple CommonJS modules in a file. AMD is a great candidate for that format -- CommonJS never standardized on a "transport format" for this need.

I expand on this a bit more in the Why AMD document.

Inefficient by default

I assume this means that it allows separate modules to be loaded async. FWIW, the current state of ES harmony modules will allow separate file loading for each module. This works better for debugging.

For AMD, you can get the "one script file at the bottom of the page" loading with the 750 byte almond AMD shim and runtime http loading.

I cannot see a better solution to this issue. If you have ideas I would like to hear them.

Incompatible with JS.next

Since JS.next/ES harmony modules do not have an implementation yet, I am not sure what this means. It still needs at least one good implementation to prove out the API. I think AMD can better inform the ES harmony effort since it has real deployment by web-based JS users, the harder environment to get right.

Also, given that harmony modules uses new syntax and by default there is no global access, I think they are making it very hard to allow existing code to opt in to being a harmony module. I have given David Herman feedback to this effect.

I want to help make any harmony module implementation better based on AMD's real deployment. I go into it more in the aforementioned Why AMD page, but being able to set the exported value to a function, and the use of loader plugins to avoid nested, async callbacks for simple resources are of great value in any module system.

Known temporary solution

As you might gather from the above, I do not see it as a temporary solution. It has more legs than CommonJS modules because it works well in browsers and it can be used as the "transport format" for CommonJS modules.

In addition, unless someone can get Microsoft to adopt rapid IE updates, even for old IEs, AMD will be around for a long time. It is a great transpile target for ES harmony module code that wants to run in older browsers. I am prototyping that effort, although I need to update to a real parser instead of regexps. I have a branch of narcissus that has the core parser/lexer/definitions that will run in ES3 browsers as part of that work.

All things considered, I would rather not work on JS module formats. I want to get on to building useful things for end users. But the CommonJS effort was deficient (but a great first effort given their design goals -- AMD uses a bunch of their work), and I do not think ES harmony modules are there yet. Instead of just complaining, I worked with others to work out something better and got implementations and actual, real adoption.

I'm open to something else that has done the above, but I have not seen it yet. Feel free to offer alternatives and feedback though. I just want to get to a good solution, but I'm also tired of waiting for something magical to fix the problem.

RequireJS 1.0.1 released

2011-11-03T20:38:00.000-07:00

RequireJS 1.0.1 is available for download.

This just has three small bug fixes in it (one in require.js, two in r.js). It was prompted by the release of jQuery 1.7 and wanting to make sure people could use the "namespace" optimizer option with it, now that jQuery 1.7 registers as an AMD module.

The bug fixes were related to:

allowing full URLs for simplified CommonJS wrapped modules
AST parsing of dependencies for modules that use a variable for the factory function
catching more cases that should have the "namespace" optimizer option applied

Detailed list of changes for require.js and the r.js optimizer:

RequireJS 1.0 released

2011-10-18T15:00:00.000-07:00

Hell yeah. Get it now.

This release is basically the same as the 0.27.1 release. There was just one change, to a regexp used by the optimizer when it converts CommonJS modules to AMD modules via its -convert flag.

This release has been two years in the making. The code and APIs have changed over time based on feedback from the community, and now there are some very strong AMD loaders besides RequireJS. The AMD API and RequireJS have been proven useful and used in real projects.

I created a new Why AMD document if you want to learn more about why the AMD API exists and how you can use it.

I still plan on doing RequireJS releases and pushing modular JavaScript forward. This is not the end, but it is an important milestone. Thank you to the RequireJS and AMD communities for making this release.

Go use it already!

Use it to build apps!
Upgrade your library code to optionally call define()!
Implement AMD in your favorite JS environment!

Let's make modular JS a reality on the web today.

RequireJS 0.27.1 released

2011-10-07T17:08:00.000-07:00

RequireJS 0.27.1 is available for download.

Just a small update with some bug fixes for 0.27.0. See the 0.27.0 release for the major functionality changes over previous releases.

I will wait about a week to see if this release seems stable for a 1.0 release.

Fixes:

define(id, function () {}) where the function has require('') dependencies will now be scanned for dependencies. Allows for smaller universal module adapters.
Loader plugin that depends on a different plugin's loaded resource works as it did in 0.26.0.
Optimizer: update UglifyJS to 1.1.
Optimizer: semicolons are inserted between files if concatenating would cause errors.
Optimizer: always strip BOM files on all platforms when file transforms/concatenations are done.
Optimizer: allow override of modules used in optimizer. Example.
Optimizer: allow copying of .directories via build config option.
Optimizer: Resolving paths for .js dependencies might fail if an appDir was not part of the config.

RequireJS 0.27.0 released, 1.0 release candidate

2011-10-02T23:33:00.000-07:00

RequireJS 0.27.0 is available for download.

This is the 1.0 release candidate. Unless things go horribly awry, I expect to do the 1.0 release in about a week. So please be sure to try it out soon if you have concerns. If you need more time to evaluate it, let me know.

From the release notes:

require.ready() has been removed. In its place, use the domReady plugin. This allows better interoperability with other AMD loaders and better separation of concerns.
A new wrap config option for the optimizer is available, for wrapping built code in a function. Allows for better API hiding and tiny builds with the almond API shim.
The order plugin is improved for IE.
Loader plugins can now have dependencies and they will work in the optimizer, as long as the dependencies work in the optimizer environment (Node, Rhino).
The namespace config option for the optimizer is more robust.
Removed require.def(), use define() instead.
Removed module.setExports, use module.exports instead.

almond: a small AMD shim loader

2011-08-21T21:19:00.000-07:00

I just put up release 0.0.1 of almond, a tiny shim loader for the AMD JavaScript module API. It can be used as a replacement for RequireJS after an optimized build. It is great for standalone libs or tiny apps that want a very minimal footprint with all the benefits of AMD.

From the README:

Some developers like to use the AMD API to code modular JavaScript, but after doing an optimized build, they do not want to include a full AMD loader like RequireJS, since they do not need all that functionality. Some use cases, like mobile, are very sensitive to file sizes.

By including almond in the built file, there is no need for RequireJS. almond is 948 bytes when minified with Closure Compiler and gzipped.

Since it can support certain types of loader plugin-optimized resources, it is a great fit for a library that wants to use text templates or CoffeeScript as part of their project, but get a tiny download in one file after using the RequireJS Optimizer.

If you are building a library, the wrap=true support in the RequireJS optimizer will wrap the optimized file in a closure, so the define/require AMD API does not escape the file. Users of your optimized file will only see the global API you decide to export, not the AMD API. See the usage section below for more details.

So, you get great code cleanliness with AMD and the use of powerful loader plugins in a tiny wrapper that makes it easy for others to use your code even if they do not use AMD.

RequireJS 0.26.0 released, npm install requirejs

2011-08-17T00:07:00.000-07:00

RequireJS 0.26.0 is available for download.

The big feature is being able to npm install requirejs to allow require("requirejs"). This allows you to:

1) Load AMD modules inside node without running a bootstrap script. It also fixes some path issues using traditional Node modules alongside AMD modules. So, now you can use requirejs inside Node to load AMD modules and use loader plugins:

var requirejs = require("requirejs");



requirejs.config({

    nodeRequire: require

});



requirejs(["module/a", "module/b", "text!templates/one.html"],

 function (a,          b,          template) {

     //use a and b with the text template

 });

2) Exposes the optimizer as require("requirejs").optimize() to allow dynamic server builds for people who like to do "only one script tag before end of body tag" development. With the "excludeShallow" optimizer config, you can still debug a single module/script while having the rest combined into one script.

Pretty sweet -- using JavaScript to run a server to optimize JavaScript on the fly but still get fine-grained JavaScript debugging. JavaScript turtles all the way down (until you hit the C turtle).

You can still use the r.js script to do command-line build optimizations. If you npm install -g requirejs, then you can use r.js as an executable (the requirejs package replaces the old requirejs-r package).

More information on the Use with Node page.

The requirejs npm package includes require.js for the browser. So, for web pages, load node_modules/requirejs/require.js to use RequireJS in the browser. As always, it is easy to get require.js for the browser without using npm.

Also notable in the release: UglifyJS in the minifier is updated to 1.0.6. The upside: has() branch trimming now works with the default minifier.

Other items from the release notes:

Fixes for running under Node on Windows using the native node.exe builds that are available in the Node 0.5.x series. Now there is less of a need to use Java to drive the RequireJS Optimizer!
Configuration is now done via a require.config({}) call, to get in line with the amdjs require API. The old require({}) method works on the global require() for backwards compatibility, but the suggested API going forward is require.config({}). The API doc has been updated to show proper usage.
There is a namespace option now for builds, to allow moving require() and define() calls under a different namespace. This allows you to build an optimized file that uses RequireJS but does not interfere with any other AMD loader on the page, and you can make sure only your modules are loaded in that namespaced object.
The default error behavior when a define() factory function throws an error is to not catch it. The catching done in 0.25.0 made it more difficult to debug. However, there are some situations where catching the errors is preferred. Setting the config value catchError.define = true will switch to catching the errors and allow processing via require.onError()
Closure Compiler in the optimizer was updated. As a result, the code to invoke Closure Compiler changed, and will likely only work with the latest Closure Compiler release. You can grab a version known to work with the optimizer in the optimizer's lib/closure directory.
There is now a pragmasOnSave build option, which is used in the require-cs CoffeeScript loader plugin build profile to strip out the CoffeeScript compiler after a build. The end result: tiny build layers of converted CoffeeScript code.

RequireJS node adapter now in npm

2011-07-19T11:17:00.000-07:00

[Update August 17, 2011: This post is out of date, see the 0.26.0 release for a better way to use RequireJS with npm.]

[Update July 21, 2011: GitHub users arlolra and JasonGiedymin clued me in to setting up r.js as a bin file that can be installed globally, making it much easier to use. The instructions below have been updated to reflect the changes.]

As asked for on the requirejs list, the RequireJS Node adapter + optimizer is available via npm now:

npm install -g requirejs-r

This will install the r.js file as a global bin/executable that is available for any of your Node projects.

To run main.js via RequireJS in Node:

r.js main.js

To optimize a web project using RequireJS, assuming app.build.js contains your optimization profile:

r.js -o app.build.js

"requirejs-r" is a bit of a funny package name, but it comes out of the requirements to use a directory for publishing to NPM, and the r.js name. If it seems useful, I may publish other RequireJS-friendly loader plugins under the "requirejs-" prefix. I may even push require.js to npm, but I am not sure it makes sense yet. If you think it would be useful to you, feel free to leave a comment on this issue.

I'll update the requirejs.org docs if this installation option seems to work out for people.

RequireJS 0.25.0 released, AMD advancing

2011-07-11T14:55:00.000-07:00

RequireJS 0.25.0 is available for download.

It has been a few months since the last release, longer than normal. I have had some life changes (for good reasons) and I have busy at the day job. The interest in AMD loaders has picked up, and I have done some groundwork related to that interest. More information later in this post.

I wanted RequireJS 1.0 to be this release, but decided to name it 0.25.0 since I did some changes in the interest of better compatibility with other AMD loaders. Most applications should not notice the changes, but I want to have a release to shake out any problems on the edges.

Release Highlights

The awesome part: the optimizer is now just one JS file, r.js! It also doubles as a bootstrap script that supports the full capability of AMD modules and loader plugins in Node and in Rhino.

To use the optimizer, pass the "-o" option to r.js:

    node r.js -o app.build.js

To run your AMD-based project in Node (assuming server.js is your top-level AMD module):

    node r.js server.js

Running AMD modules in Node has more information. The optimizer docs have been updated to the new optimizer syntax, and the r.js script has its own project now, to allow releases that are decoupled from require.js.

Running r.js under Rhino is still supported, but you need to fetch a couple of JAR files first.

Other highlights:

The loader plugin API changed to allow plugins to create cross-domain-accessible resources. The main use case: you use the text plugin to dynamically load text resources, but you want to deploy your scripts and text resources to a CDN. See the text plugin's implementation of writeFile() as an example.
There is a global requirejs() function object that is the same as the global require() function object. This should allow RequireJS to work better in environments like Mozilla Chromeless, which already have a built-in require() function that does not have full AMD/loader plugin capabilities.
It is now possible to specify the precise version of jQuery to allow in a RequireJS context. This is useful if you know of other scripts that load different versions of jQuery on a page.

Some changes in the name of compatibility with other AMD module loaders and Node:

The "lib" directory configuration in package support was removed. It was always very awkward to support. Node no longer supports it, and that was the extra justification I needed to remove it.
Relative module IDs are now relative to the related module ID, not the related module's resolved path.
includeRequire in the optimizer config was removed, Use a paths config to include require.js instead. See the optimization docs for more details.

A small change to the context-specific require() passed to a loader plugin's load() call: require.isDefined() is now require.defined() and there is require. specified().

A New Home for AMD

Part of the RequireJS release delay was because the AMD API was moved off the CommonJS wiki and a set of compatibility tests were created.

While it was useful to hash out some of the basic goals and API on the CommonJS list, support for AMD on that list did not reach consensus. However, there has been continued and increasing interest in others making AMD-compatible loaders and tools. To avoid bothering people that would rather talk about other non-module things on the CommonJS list , a new AMD-focused discussion list and wiki has been set up and there is a set of compatibility tests.

AMD is advancing

More evidence that AMD is advancing:

Alternate AMD loaders are available. Curl continues to advance, as well as the Dojo's 1.7 loader. Loadrunner has grown some basic support.
There are at least two "AMD lite" projects, one in Ace, and one called DefineJS, that focus on basic AMD API support for when all the modules have been optimized into one file. The goal is to use a smaller bootstrap in optimized applications instead of a full standalone loader, like RequireJS.
Node 0.5.0 supports the simplified CommonJS wrapper version of define(). While this support is very limited (no dependency array or loader plugins), it is a start to allowing some basic low-level module sharing between Node and browser apps better. See Kris Kowal's excellent Q library as an example of a module that works in both Node and in full AMD loaders.
Look for better opt-in support for AMD in a couple of browser-focused libraries and toolkits later this year.

Path to RequireJS 1.0

For RequireJS, I want to get to a 1.0 when:

0.25.0 has been shown to work well in existing projects.
Do some work around require.ready: possibly using a domReady plugin/module with optional loader hooks, to help improve cross-loader compatibility.
Try to finalize more of the loader plugin API with other AMD implementers, to make sure more code can be shared across loaders. The loader plugin API is a separate API from basic AMD support, so I may push out a 1.0 before that API is completely final, and go to a 2.0 branch for any backwards incompatible changes to the loader API.

On inventing JS module formats and script loaders

2011-04-06T19:34:00.000-07:00

There was some recent twitter tweets over the last couple months that indicated some folks want to experiment with some JavaScript module formats that work in the browser. In addition, making a script loader is almost as popular as making your own template engine. This is my response to both of those endeavors.

AMD is Winning

If you are one of those people that think they want to invent a module format or a script loader that works well in today's browsers, let me save you some trouble: a module format has already been worked out. Check out the Asynchronous Module Definition (AMD). There is still some room for innovation though, see near the end of this post.

If your script loader does not support AMD, it will be of very limited usefulness and in a sea of competitors that will make it hard for you to gain adoption.

So, for the module syntax, that problem is solved, and the solution is AMD.

A Solved Problem

Why is it solved?

It has been discussed for more than a year and a half by multiple people, in the Dojo world, and on the CommonJS list.
It has been implemented more than 5 times by different people.
It has been deployed in real projects, the BBC iPlayer being a very visible one.
It works in the browser, web workers, Node, Rhino, Mozilla Add-On SDK-based add-ons, and can be used in traditional Mozilla add-ons (Firebug will be using it).
Many AMD implementations support loader plugins. You can add modules that know how to load other things as dependencies. While not strictly part of AMD, plugin systems greatly enhance a base technology.

You will be facing an uphill battle trying to get your format accepted. AMD has at least a half-year head start and has been seriously battle-tested. The ECMAScript folks are considering separate, different language changes to support modules, so you have that on the event horizon. Time is really running out pushing for something new.

I do not want to harsh your mellow, and I definitely want to follow your bliss, but just know the actual cost involved, and do your research first. Make sure you understand why AMD is constructed how it is, and what it really is.

It is not the traditional CommonJS Module format, but it can support similar concepts. However, it is different, take the time to figure out why.

Here is a handy gist with a summary of the AMD API, loader plugins, and some thoughts as to how it relates to modules implemented in Node, which are more like traditional CommonJS modules. In addition, you can read the RequireJS API docs, and the AMD wrapper format for traditional CommonJS modules.

If you think something was missed, feel free to ask on the RequireJS list, or on the CommonJS list, and enlighten us all. Or just send me an email or GitHub message if you want to talk privately. However, I believe most questions will be in the minor bikeshed/personal preference arena that do not ultimately matter for adoption. In particular, do not propose anything that requires the developer to do more typing that what is in AMD already. The format needs to be usable and easy to type. Discoverability is not a goal, daily usability is, and the AMD format is easy to explain.

Requirements

This leads to a discussion of other requirements. Here is what I believe are the minimum requirements for a module format in the browser, and they well-met by AMD:

No globals: The module format needs a way for the developer to avoid creating global variables for each module they create. The format needs to be web-scalable, which means that some sites need the ability to embed two different versions of a module on a page. This normally means embedding in different "contexts" (the different versions do not need to talk to each other, just co-exist on the page), but it is a web-scale need. Seriously. We have seen it in Dojo. jQuery has seen it, it is the reason they have noConflict(). While noConflict() is nice, there are edge cases where it falls down. Know why that happens (in particular, dynamic script loading in IE: more than one script can execute before the first script's onload event fires).
Allow for globals: at the same time, you want developers to be able to do want they want, in particular, some like to modify global object prototypes. Prototype and MooTools are valid ways to build web sites. So allow for it. This is one of my concerns with the ECMAScript harmony modules experimentation, but I think they know they need to allow for this, even if within a module loader instance.
Do not require server transforms or think that using XHR+text transform+eval() is usable across all of web development. It is not. Here are some reasons why. They are fine for your boutique rails projects, but do not assume that translates well to the web at large, for instance, to serving JS files from disk for mobile widgets.
You will need to use a function wrapper and a way to specify dependencies before that function wrapper is executed. You will be using script elements to load content (seriously, don't use eval, that is an evolutionary dead path), and dynamic script elements will load scripts async and out of order. A function wrapper will need to be part of the format.
You need a way for modules to have a name. Otherwise you cannot combine them all together for optimization purposes.
At the same time, it is best to allow source modules (the non-optimized ones) to not be named. This makes it much easier to move code around. This is not a hard requirement though, and I think one of the easiest to try to not support, see next section.

Places for Innovation

So you are not going to come up with a better module format. I know that sounds harsh, and for some of you that will be just the thing you need to hear to try something different. But I really am trying to save you some work. Again though, go for it if it really is your bliss, just know the amount of work you are taking on.

A better use of your time might be doing the best AMD implementation/script loader. In particular, you can choose some limiting design choices to help make it easier to meet/more compact:

Do not support the simplified wrapper for traditional CommonJS Modules.
Only allow named modules.
Do not support loader plugins.
Do not worry about supporting more than one version of a module in a page. The module format allows for it, which is the most important aspect, so module authors can code for it if they wish. However, most sites can get away without needing a loader that can actually support loading multiple versions.

If you do those things, you still can consume the built/optimized AMD modules that someone made using another implementation, and any modules created while working with your implementation will work with other AMD implementations. And with that, you are part of a larger ecosystem.

If you make a script loader that does not understand AMD, it will be a boutique loader, and not something that fits all of front-end development, and limits the ability to share code with other JS environments. Which is fine, boutique businesses can be really satisfying, just realize you are in a boutique, and do not over-sell the loader.

FAQ

These are odds and ends that do not fit above.

Why use modules or a script loader at all? Just server-side concat all your scripts!

The goal is to establish a larger ecosystem of shared code that works without puking globals all over the place. JavaScript is also not just jQuery (although I greatly appreciate the project), so it cannot rely on a jQuery implementation.

Not all environments can use server-side script concatenation (file-served mobile widgets), and it does not allow for more nuanced optimization like loading built layers on demand after initial page load. On-demand loading is really needed for large web apps like a webmail UI.

With tools like the RequireJS optimizer, you can still get the same effect of "concat all files". If you do not want to take the hit of downloading a script loader in that scenario, there is work underway to allow a simplified stub that removes the need for a full script loader. The Ace project uses a "mini-require" after they build the code to avoid loading the RequireJS script loader.

What about YUI 3 modules?

YUI 3 got a lot of things right with their modules, in particular designing for async up front. However, I believe the way they implemented the module name-to-JS-variable-name mapping is not right. It requires knowledge of two names that allows for easy typos and creates the opportunity for a naming conflict on the Y object. For instance:

YUI().use("io-base", "my-module", function(Y) {

//Where does .on() come from? Did 'my-module' add it?
//Did 'io-base'? What if both tried to add it?
Y.on(...);

//'io-base' creates the 'io' property on Y.
//How is that known? Should it be 'ioBase'?
//What else does 'io-base' add to Y?
var request = Y.io(...);
});

In short, too much magic that requires extra documentation. YUI 3 has very nice documentation so that helps, but I do not believe that approach scales as well as AMD.

Is AMD a CommonJS specification?

No, it is a proposal, but with lots of implementations and real use. Not all participants on the CommonJS list believe it should be a elevated to "endorsed spec" status. Those participants wish browsers would grow better base technology and/or they want to maintain stricter compatibility with traditional CommonJS modules, which were not designed primarily for async loading environments like the browser.

Any browser technology change will likely be for harmony module support which operate differently than traditional CommonJS modules, and AMD works well today in all JS environments. AMD is proven, and it can work on the server side. The RequireJS Node adapter even allows using NPM-installed modules in Node.

How does AMD relate to ECMAScript harmony modules?

They are unrelated, although hopefully the AMD work and the traditional CommonJS module work will help inform the harmony effort. I believe AMD modules will be easy to convert to harmony modules, but as the harmony work is still in progress it is hard to know for sure. The return value options in AMD may be more flexible than harmony modules, and it is unclear if loader plugins will work in a harmony module world.

AMD has the benefit of working in all the JS environments now, and it will work in the future, and I do not believe the syntax gains in harmony modules are a vast improvement over the typing cost in AMD, particularly if sharp functions are supported. That said, I hope to inform the harmony effort where I can to help make it the best it can be.

Who are you?

I'm James Burke (GitHub, Twitter). I maintained the original Dojo XHR-based script loader, maintained Dojo Core, created the Dojo xdomain script element-based loader, created RequireJS (formerly RunJS), and rewrote RequireJS about three times. I have strongly advocated for browser-friendly modules on the Dojo and CommonJS lists. I like Pina Coladas and getting caught in the rain.

RequireJS+jQuery project moved, updated to jQuery 1.5.2

2011-04-01T18:13:00.000-07:00

The sample project that demonstrates how jQuery and RequireJS can be used together has been moved to its own repository. This should make it easier to update. As proof, it now has jQuery 1.5.2, and can be downloaded here.

Also, the sample project has gone back to using an integrated RequireJS+jQuery file as the basis for the example. Keeping RequireJS and jQuery separate and using the "priority" configuration option in jQuery has some edge case considerations that required more explanation than what most people need. However, the README in the sample project repository explains how to use the priority configuration if you prefer to use that option.

RequireJS 0.24.0 Released

2011-03-14T01:28:00.000-07:00

RequireJS 0.24.0 is available for download.

Themes for this release: IE 9, better jQuery integration, better loader plugins, and a CoffeeScript plugin:

Support for IE 9. It has a non-conformant script loading behavior that necessitated the change. It would be ideal if IE 9 would change the behavior to be conformant.
Changes to jQuery integration:
- jQuery 1.5.1 included in sample project.
- No more bundled RequireJS and jQuery file. RequireJS includes special code to detect jQuery as a module, so the combined file is not necessary, and this approach makes it easier to swap in new versions of jQuery as they are released.
- Because of that change, the jQuery sample project uses the new priority: config approach.
allplugins-require.js has been removed. Plugins are no longer bundled with require.js or in the sample jQuery project. There are separate download links to the existing plugins on the download page. Special treatment of these plugins has also been removed from the require.js source, so those plugins behave like any other loader plugin.
There is now a CoffeScript plugin. It makes it easy to code in CoffeeScript in the browser, it can participate in the optimizer optimizations, and it works in Node and Rhino. This is the best way to do cross-environment, modular CoffeeScript.
Bug fixes.

This release is the "release candidate" for RequireJS 1.0, so please give it a whirl. There may be some small tweaks for the final release, particularly around how the optimizer is packaged, but if this release holds up well, RequireJS 1.0 should be out in about a month.

RequireJS 0.23.0 Released

2011-02-14T23:29:00.000-08:00

RequireJS 0.23.0 is available for download.

Highlights

The baseUrl defaults have changed. This will likely be the biggest impact, and may require a change in your code.
Node is now the default JS engine used for the optimizer. It is blazing fast. However, not everyone can switch to the new hotness, so it is still possible to run the optimizer in Java/Rhino. But Node is the default, so if you use build.sh/.bat and want to use Java, update to buildj.sh/.bat.
UglifyJS is the default minifier for the optimizer, since it runs in both Node and Rhino.
The Node adapter is updated and should match Node's native behavior for any npm-installed module, but see the release notes for more detail.

See the release notes for more information.

The env plugin

One of my favorite things in this release is how the optimizer can run on either Node or Rhino. The optimizer is structured as a set of RequireJS modules and a new plugin: the env loader plugin. That plugin tells RequireJS to load a particular module based on the environment that is running RequireJS.

To indicate a module dependency that varies based on the environment, modules use dependencies that look like "env!env/file", and the env loader plugin will load "node/file" if in Node or "rhino/file" if in Rhino.

While the env plugin in its current form is not ready for outside use (in particular it needs more work to be useful in an optimizer build), I believe it shows the way forward for doing environment-based branching of module loading. In particular, it can be seen as a great alternative to the concept of "overlays" that have come up in some approaches to CommonJS packages.

jQuery integration

The integrated jQuery+RequireJS file has not been updated to jQuery 1.5.0 yet. I want to wait for jQuery 1.5.1 since it may contain a fix related to jQuery.readyWait which is used by RequireJS.

Next up

With the conversion to using Node as the default optimizer engine, the code is one step closer to a 1.0 release.

I want to do another release that breaks out the text, i18n and order plugins into a separate repo or repos. Right now they have a magic status that will cause problems in the long run.

I also want to deliver the optimizer just as one script file, instead of a small collection of modules as it is now. This is possible since the optimizer is written as RequireJS modules/loader plugins, and not something that would be easy to do if they were CommonJS modules. I may punt on this though in the interest of getting to 1.0 sooner.

It may also make sense to break out the optimizer into its own repo, but again I may punt on that to reach 1.0 faster.

So hopefully just one or two more releases before going to 1.0. Depending on the timing of jQuery 1.5.1, there may be one additional release in that set.

jQuery UI as AMD modules, for use in RequireJS

2011-01-10T00:28:00.000-08:00

There was a request on the requirejs list for a conversion script that could convert jQuery UI files into a format more easily used by RequireJS. While it is possible to load custom builds of jQuery UI with RequireJS, having the jQuery UI scripts in AMD module format allows for flexible use:

Have the full range of jQuery UI available for use in different ways on different pages, but still get optimization benefits via the RequireJS optimizer.
Dynamically load parts of jQuery UI based on user actions.

The jQuery UI files already list their dependencies, but in a source comment. So it was fairly easy to whip up a command line utility (that is run via Node), to convert the files.

I created an example project too, showing how to load the Tabs widget, then when the user clicks on the second tab, dynamically load the Datepicker widget with its French localization.

The example project includes the RequireJS optimizer and a build profile. What is really neat: the unoptimized source project is about 25 HTTP requests. After the optimizer runs, it goes down to 8 requests, 4 of which are images.

There is more information on the conversion process in the README on GitHub. If the jQuery UI team is interested in supporting this more directly in their project, I'm happy to work with them to set up a define() stub/wrapper to allow the code to work even when an AMD loader like RequireJS is not in the page.

Enough talk, on the code:

RequireJS 0.22.0 Released

2011-01-05T00:39:00.000-08:00

RequireJS 0.22.0 is available for download.

The highlights:

A full plugin API that supports also including plugin loaded artifacts in optimized builds.
Robust Node support: now can use npm-installed modules, .node add-ons, and modules on require.paths.
Support for has.js optimizations in the optimizer, including dead code branch removal via Closure Compiler.
Bug Fixes from 2010-12-21 through 2011-01-05.

Thanks to Joe Parry and Francois Laberge for the Node test cases and feedback, John Hann and Kris Zyp on the plugin API.

This brings the code closer to a 1.0 release. I want the plugin API to settle a bit, see if others can build useful plugins with it, and then look at moving the optimizer from Rhino to Node. Hopefully just a couple releases away from a 1.0.

A side note: I goofed on the previous two releases, they should have been 0.20.0 and 0.21.0, but I released them as 0.2.0 and 0.2.1 respectively. This 0.22.0 release gets the version back on track.

Standards and proposals for JavaScript Modules and jQuery

2010-12-28T11:17:00.000-08:00

This commit to add registeration of jQuery as a module via the Asynchronous Module Definition (AMD) API has brought up some questions that I should have addressed previously before the commit happened. The jQuery team may decide to back out the commit, as is their prerogative, but I hope not. This post will be the justification for the AMD API in general and for its suitability for jQuery in particular.

What is the Standard?

There was a concern brought up by Kevin Smith that module loading is still up in the air, and that there is no "Standard" yet. To me a standard is something that is written down, understood by people, and has a good level of market adoption. Multiple implementations help too.

CommonJS Modules is a standard in that way. It is written down, understood by a few people, and has a a good level of market adoption via Node. There are a few implementations for other JS engines.

The CommonJS group also has a process where they have voted before to call it a Standard vs. a Proposal. Anyone can join the CommonJS discussion list, and I think it is still an ongoing discussion on how best to vote on proposals, who should get to vote, and if voting makes any sense. It is helpful to know if most of the people on the list like a particular proposal though, it gives some confidence that the proposal has been thought out.

I have a concern about the diversity of the list participants though. Most joined when it was still ServerJS, and the CommonJS modules "standard" does not work well for use in web browsers that use script src="" to load scripts. So, you could hear that "CommonJS Modules are a standard", but it does not mean it works well in all CommonJS environments, most notably, the largest distribution of a JS environment, the browser.

That is the reason the AMD proposal exists. It works well in the browser environment where module loading is by default asynchronous, and with script src="" loading (the easiest to debug and fastest way to load code), there is no control over modifying the script source before it executes. AMD is called a "proposal" in the CommonJS list because people on the CommonJS list have not voted to call it a "standard".

There is recognition on the CommonJS list that the CommonJS Modules 1.1 is not sufficient for script src="" browser loading, but there is some disagreement over how to solve it. AMD is one proposal. Kevin Smith has put together a Module Wrappings proposal, based on some discussions on the CommonJS list. The Wrappings proposal is incomplete -- there is more to it based on the list discussions. I will highlight the main difference between AMD and Wrappings below, but first, another module API contender:

ECMAScript Simple Modules. This is a "strawman", which is like a proposal in the CommonJS group: it is still something being worked on. This one is different because it is being worked on by the ECMAScript committee, and it seems like the strongest module proposal in that committee so far. If adopted, some future version of the ECMAScript (JavaScript) standard will have support for modules natively. So, any module API that used today should be aware of what is going on in the ECMAScript arena.

So, for talking about a module standard, particularly for use in the browser, these are the main contenders:

AMD
Wrappings
Simple Modules

There are other modules systems like the one used in Dojo, and the one used by YUI, but those are too tied to those existing toolkits to be broadly adopted. Dojo has started to move to AMD.

AMD and Wrappings Differences

AMD and Module Wrappings are mostly the same. There needs to be a function wrapping around the actual module, to make sure the dependencies are fetched first before executing the module. There is some bikeshedding around names that I do not think is important (define vs. module.declare).

The main difference: AMD executes the module functions for dependencies before executing the current module function, where Wrappings just makes sure the module function is available, but does not execute it until the first require("") call inside the module function asks for it.

For shorthand, I will call the AMD approach the "execution" approach, and the Wrappings approach "availability". Both relate to how dependencies are handled.

Because AMD executes the module functions of dependencies before calling the function wrapping of the current module, it can pass the dependency as an argument to the factory function. This is a side benefit of execution.

The "availability" approach was used for Wrappings to keep it most similar to how CommonJS modules deal with dependencies, but I believe it is the wrong choice for the future.

Problems with "availability" model

"Availability" is not supported in the ECMAScript Simple Modules strawman. Simple Modules uses a model that is closest to the "execution" model: you use "module" and "import" keywords to reference dependencies, but they cannot be used like require("") is used in the "availability" model. "module" and "import" are more like directives, and not something that can by dynamically assigned, as part of conditionals. Examples:

In the process of working on RequireJS and trying to translate modules written for Node to a wrapped format, I have seen the following constructs in Node modules (which assume the "availability" model):

try {
 var constants  = require("constants");
 ...
} catch (e) {
  //Older version of node here
}

and:

var foo;
if (someCondition) {
  foo = require("foo1");
} else {
  foo = require("foo2");
}

Those work in the "availability" model, but not in Simple Modules. AMD's execution model does not allow the above constructs, so modules written for AMD will be more portable to Simple Modules.

2) The "execution" model fits better for projects that use libraries like jQuery, Prototype or MooTools, where many of the modules augment other objects, and they are assumed to have already run before executing the current module function. jQuery plugins augment the jQuery object, Prototype and MooTools augment JavaScript object prototypes.

Choose AMD

AMD is the API for today that translates to the future best, and it should be the one that jQuery should support because:

There is a document that defines the API.
It is understood by a few people, and discussed in public on the CommonJS list.
It is not ratified as a "Standard" in CommonJS because there are list participants that still want to hold on to CommonJS "availability" semantics. However, that availability model is not as future proof as AMD, at least for the Simple Modules future.
AMD translates better to Simple Modules due to the execution model. It maps better to the the related Module Loaders strawman, so if Simple Modules with the Module Loaders API becomes a standard and gets implemented in browsers, code written for AMD is more likely to continue to work.
For browsers versions that probably never support Simple Modules (IE6-8, even 9?), code continues to work.
The "execution" approach for dependencies fits better with the mental model of existing browser toolkits that want to augment other objects, like jQuery plugins.
AMD is supported by a module loader (RequireJS) that has had some level of successful adoption, and it is fairly robust now, with over a year in development with frequent releases. RequireJS has an adapter that allows the same modules to be run in Node and Rhino. The Node adapter allows using existing Node modules.
There are other AMD implementations.
RequireJS in particular works hard to make sure it works well with jQuery, including integration with jQuery's readyWait to hold off DOM ready callbacks until all scripts are loaded.
AMD has more real world adoption than Wrappings. The Dojo Toolkit now supports using an AMD-compliant loader in their trunk code for Dojo Core and Dijit. There is a thriving RequireJS list with real people using it. BBC iPlayer uses it. The RequireJS implementation of the AMD API has been promoted as a useful tool in multiple jQuery-related conferences.

RequireJS 0.2.1 Released

2010-12-20T01:12:00.000-08:00

RequireJS 0.2.1 is available for download.

This is a bug fix release related to issues with the refactoring done in 0.2.0. Probably the most noticeable bug that has been fixed: the "all plugins" builds did not include the plugins.

RequireJS 0.2.0 Released

2010-12-16T00:13:00.000-08:00

RequireJS 0.2.0 is available for download. The highlights: stronger, faster, and prettier.

Refactored core.
New loader plugin API support.
Improved Rhino and Node adapters.
Bug fixes around loader plugins and the optimizer.
require.modify removed.
Removed Transport D files.

Stronger: The refactored core is more robust vs. the 0.1x releases, and the new plugin API makes it much easier to construct loader plugins. The plugin API is an implementation of Kris Zyp's plugin API proposal. The optimizer/build support is different than what he proposed, and I plan on discussing the changes on the CommonJS list where he sent the proposal. But the basic API for defining a plugin in that proposal is supported.

The Rhino and Node adapters are even better in this release. I want to improve the Node adapter in a future release by allowing the use of npm installed modules without setting up a package path config.

Faster: The refactored core responds faster to loaded modules, and it will execute a module as soon as its dependencies load. In practice you will probably not notice an appreciable speed change, but it allows the new plugin API to work well.

Prettier: The web site got a much needed facelift and a logo, thanks to the talented Andy Chung. So much better than my feeble attempt.

As part of the refactoring, I removed the Transport D files. I believe they are no longer in use by other projects. I also removed require.modify. It could not support relative path names, and was always an odd thing that did not quite fit in with the rest of the code.

The refactored code is also more strict. If you do this in main.js:

require(['foo'], function () {});

And then in foo.js, it does:

require(['bar'], function () {});

The require callback in main.js will be called before the require callback in foo.js, because foo.js did not define a module. The loader thinks it just needs execute foo.js in order to call the require callback in main.js. To make sure the require callback in main.js waits for foo's factory function to get called, change the require call in foo.js to define:

//In foo.js:
define(['bar'], function () {});

Thanks to the community for filing bugs and contributing to the discussions on the mailing list. Thanks to Ben Hockey for drilling in deep and working out how to load the Dojo trunk code with RequireJS. It prompted some great bug fixes, and a few patches from Ben.

Rawld Gill, along with Kris Zyp, has done a lot of work to convert the Dojo trunk code to use the AMD API proposal. Rawld has even done an alternative loader that implements the AMD API. Check out his work if you want a different look at an AMD implementation.

What's next? I want to move closer to a 1.0 for RequireJS, but I would like to get these things settled first:

Get feedback from the community on the loader plugin API. I need to do some better documentation for it too.
I want to do a has.js plugin generator, and I want to integrate support in the optimizer to trim if(has['test']){} if the configuration passed to the optimizer indicates that has['test'] will always be true.
Allow the optimizer to run on top of Node. Right now the optimizer runs on top of Java via Rhino, but that is increasingly becoming a liability.
Sort out full packages support. Right now RequireJS can load modules in packages, but it cannot handle two versions of the same package in a project. While this is a minority use case, I would like to have a story for it. I think it is workable in source form, but I am not sure how it will work out yet for optimized scripts that want to include modules from two different versions of the same package.

RequireJS 0.15.0 Released

2010-11-14T22:12:00.000-08:00

RequireJS 0.15.0 is available for download. This is a bug fix/robustness release:

The bundled jQuery options now use jQuery 1.4.4.
The jQuery sample project now includes the ability to use RequireJS plugins.
The jsonp! plugin has been removed, since, thanks to work by Kris Zyp, the core loader now supports loading JSONP dependencies by default. The JSONP docs have been updated accordingly.
The optimizer can now be run from any directory, not just the directory with the build profile.
r.js Node adapter is more robust, and it can handle using more Node-written modules by default now. Thanks to Francois Laberge for a great test case application that lead to improving the robustness of r.js.
Initial support for PS3 Netfront browser. Thanks to Chris Warren for investigating the load behavior of the browser. Not all tests pass, but the basic ones do.
Miscellaneous fixes, some listed in the issue tracker.

The next stage of development will likely be a refactoring of the basic code. The code has grown organically along with some changes in requirements over the course of the year, and it would be good to reorganize based on the current understanding of script loading needs. The biggest noticeable change should be a better API for writing loader plugins, but the core API for requiring and defining modules should not change.

The Open Source of Mozilla F1

2010-11-12T14:29:00.000-08:00

I am a front end developer on Mozilla F1, a sharing extension in Firefox, and this is some technical background on it. Note that any opinions in here are from my personal perspective, you should not treat this as official Mozilla policy, etc...

F1 consists of a few components:

1) Firefox browser extension
2) HTML/JS/CSS share UI served from from the F1 web server
3) API server written in Python that lives on the F1 server

The extension (#1) is responsible for the share button in the toolbar, injecting navigator.mozilla.labs.share(), and for managing a browser instance that is inserted above the main web browser area. This browser area makes a call to the F1 server to server up the main share UI (#2).

The share UI is written in pure HTML/CSS/JS. They are static files that use RequireJS to build modular JavaScript which is then optimized via the RequireJS optimizer into a single application-level file. It does not yet use appcache, but it is something I want to add. It should be straightforward. Making sure we properly version files with long cache times will also help.

The share UI uses the API server (#3) to do the actual sharing. The API server also manages the complications of the OAuth dance.

If you are curious about the implementation, all the code is open. You can download it/fork it at the Mozilla Labs F1 GitHub repository. There is a README.md file with setup instructions.

The great thing about this? You can run your own share server that you control. If you want the F1 browser extension to use your server for the share UI and API server, set the following F1 configuration in the about:config Firefox preferences:

extensions.ffshare@mozilla.org.share_url = 'http://your.server.com/share/'

Ideally that would be an https URL, but if you are setting up your own server, hopefully you know the risks involved. Be sure to restart the browser so the extension will use this new config value.

If you think this is really cool and want to help contribute code/ideas/submit pull requests, be aware that we are still in the early stages of F1. We expect to make many changes as we refine it. Mozilla is also new to GitHub, so we appreciate your patience as we try out different work flows.

I will close out this post with a couple developer bikeshed questions and answers. Again, these are my personal perspectives, and other people on the team may have different ones.

Why is the share UI served from a web server and not from the extension?

This is an experiment. More things are moving to the cloud/server, and some of the auth APIs, like OAuth, are better suited to server use. While Mozilla already runs some non-trivial server-based services, it is good to get experience with varying types of server-based services.

I like the flexibility and ease of updates a server solution provides. Also, it is easier to debug web code that is served from content space vs. browser chrome space.

It may allow us to support mobile and even other browsers easier. I played around with doing a Google Chrome extension that served up the share UI. Unfortunately, the Google Chrome extension model is not as flexible as what we can do in Firefox, so it does not really work.

In particular, I was looking at a Chrome extension Browser Action that served up the UI in an iframe in a Popup, but I could not get the popup wide enough to show the UI, and it closes as soon as an OAuth window jump occurs. We would have to do some rethinking on UI to support that, and it is not clear how beneficial that is at this early development stage.

All that said, the UI could be burned in to the extension at some point. It is still too early to tell where this will go.

Why did you use Python for the server?

Ease of setup, combined with some knowledge of real services built with it, and the mix of developer skills of the F1 team.

Ideally I would like to see the server running JavaScript via Node, but that is because I am an irrational JavaScript zealot. However, the JS libraries for Node are still young, and at the end of the day we want to solve real user problems, not debug other people's code.

That could change in the future. But for now, it is more important to get something up quickly that works well so we can test our designer's usability theories for solving real user problems.

HTML5 and Script Execution Order

2010-11-12T07:32:00.000-08:00

As the latest browser renderers, WebKit, IE9 and Firefox 4, implement HTML5 there is wording in the HTML5 spec that breaks ordered execution of dynamic script elements.

What does this mean in practice? The RequireJS order! plugin and a core feature of LABjs breaks in the latest browsers under development. More info:

When a dynamically created script element is created and appended to the DOM (via document.createElement('script'), the behavior in the current browsers differs: IE and WebKit will execute the script as soon as it is delivered from the network, where Opera and Firefox will download the script as fast as it can, but execute the scripts as they are ordered/appended to the DOM.

The Firefox/Opera behavior is desirable for making existing scripts on the web go fast since all the scripts can be downloaded in parallel, but still execute in the order you specify -- there are many scripts today that assume their implicit dependencies have already been executed before the script executes.

jQuery plugins that assume jQuery is already available in the page are a good example.

Kyle Simpson of the LABjs project figured out a nice hack for IE and Webkit to get them to execute scripts in order. He gets IE and WebKit to download the scripts first without executing them (via a non-script script type, like type="text/cache"), then adding the real scripts to the DOM in order normally means they execute in order.

However, this technique has edge cases where it can fail, in particular with poor cache headers, and the latest version of the HTML5 spec effectively disallows the hack. The code to get ordered execution right now also has some browser sniffing to get it to work, which is clearly not ideal.

The ideal case is to have a capability that can be sniffed in the browser, and something that would allow for ordered execution of dynamically added script elements without the need of browser hacks.

Kyle has been working with HTML5 spec folks to try to work out something along those lines.

However, the HTML5 spec participants are not sure how important of a use case this is to support to warrant a spec change. I do think it is important for existing scripts to get some performance benefit.

While RequireJS does not rely on this behavior for its core operation (the define() function wrapper ensures that it is not an issue, and assumes out of order script execution), the order! plugin in RequireJS does depend on this behavior.

So, if you are a RequireJS order! plugin user and you really depend on it, or if you are a LABjs user, please take a moment to let the HTML5 group know the sites you work on that would be affected if those tools no longer worked in the browsers being developed today.

Kyle Simpson has set up a wiki page to describe the problem. There is a Discussion section of the page where you can voice your feedback. Click the + in the tabs at the top to add a section where you can list how this issue would affect your sites. Be sure to leave your name in case the group would need to get more information from you.

Please take a moment if you do depend on this feature to let the HTML5 group know.

RequireJS 0.14.5 Released

2010-10-17T21:03:00.000-07:00

RequireJS 0.14.5 is available for download. Primarily a bug fix for the first item below.

Fix bug where scripts were not loaded from the correct path. Did not affect RequireJS+jQuery builds, but affected other builds. If you do not use a RequireJS+jQuery build, then it is strongly recommended that you upgrade from 0.14.4 to 0.14.5.
Added an urlArgs config option to allow for cache busting when servers/browser misbehave during development.

I apologize for the quick series of releases this week. I should be slowing them down a bit now.

RequireJS 0.14.4 Released, jQuery 1.4.3 support

2010-10-16T14:15:00.000-07:00

RequireJS 0.14.4 is available for download. This release supports jQuery 1.4.3. What does this mean?

The bundled RequireJS+jQuery file on the download page has jQuery 1.4.3 in it.
Due to a change in jQuery, there are almost no patches to jQuery in the RequireJS+jQuery file, just a convenience patch to register it as a module.
If your scripts properly use define() with RequireJS, then it is possible to load jQuery from the Google CDN. This can save you some bandwidth costs!

There are a few caveats with using RequireJS to load jQuery from the CDN:

Most jQuery plugins assume jQuery is already loaded when they execute. However, with RequireJS, jQuery could still be loading when the plugin file executes. If you plan to use jQuery plugins in your project and you really want to load jQuery from the CDN, then in your own RequireJS-based project, you should wrap each of the plugins in a define(function(){ /*plugin goes here */ }); wrapper. Or, just stick with using the combined RequireJS+jQuery file served from your server.
If you use code with define() calls in them, be sure to do the minification and combining of scripts using the RequireJS optimization tool. It will make sure the define() calls get proper scripts names to allow all the scripts to be combined together. If you use another tool to just concatenate your define()'d scripts together, it will result in errors if the define()'d modules are not named. A copy of the optimization tool is included in the jQuery+RequireJS Sample Project.
This is only recommended if you have one version of jQuery loaded in the page.

An example showing how to configure RequireJS to load jQuery from the CDN:


require({
  "paths": {
      "jquery": "http://ajax.googleapis.com/ajax/libs/jquery/1.4.3/jquery.min"
  }
});

//Now require jQuery when you want it, using 'jquery'
//as the dependency name.
//HOWEVER, note the important caveats above!

require(['jquery'], function ($) {
   //...
});

Now if only I could get RequireJS releases on a CDN, that would be even sweeter!

RequireJS 0.14.3 Released

2010-10-14T23:56:00.000-07:00

RequireJS 0.14.3 is available for download.

Support for define(). It works the same as require.def(). It is supported in order to conform with the Asynchronous Module Proposal. require.def will continue to work, but you are encouraged to gradually migrate to define() for better compatibly with other Async Module loaders.
text! plugin now works in Node.
_dirname and _filename support in the r.js Node adapter.
Bug fix for priority option.
GPL license option removed: project is now just MIT and new BSD dual-licensed, since the new BSD license is compatible with the GPL.

The big motivation was the require.def() -> define() API change. I do not expect any more top-level changes to the API now, particularly when related to the Asynchronous Module proposal. There may be some API changes around how plugins are written at some point, but those are tricky to write today, and could use an API cleanup. However, using plugins in your application code will likely stay the same.

To be clear: require.def() will continue to be supported. If you only care about loading your modules via RequireJS, you can continue to use it. However, if you think you might want to allow your code to work in other script loaders that follow the Asynchronous Module API, then you should look at switching to define().

fn with no globals

2010-10-12T11:52:00.000-07:00

This is a JavaScript sketch, please read the disclaimers.

Background

To get some decent code reuse, we need baseline support for loading code across environments. CommonJS made a good first pass with the Modules and Packages proposals. The Module one does not work well in the browser though, so I am hoping the Async Module proposal gains traction. Douglas Crockford likes async code loading too (around 39:10 in the video).

Relying on a server to generate client-friendly code is not a universal solution, and it is a particularly poor solution for doing mobile web development, where apps can be created with HTML/CSS/JS but run from the mobile device without a server.

RequireJS implements the Async Module proposal. There are some rough spots around the Packages proposal, particularly around mappings, but it is something that can be used, and I'm trying to support some version of packages with RequireJS.

The ECMAScript folks are looking at a Simple Modules spec, but it does not solve any of the problems for me. The problems have been addressed by RequireJS. It would be nice to not to have to include the code for RequireJS in an app, but it works, it is not crazy big, and it can always be pushed down to native environment support after it is broadly in use.

fn with no globals

What I would like to see instead of a Simple Modules is "fn". It is like "function" but it does not have access to the global space. You would use it just like "function" but any variables missing "var" would not be in the global space (could throw an error). If it needs a "use fn" or something like that, OK, but I am not familiar with when "use" strings are needed. Example:


fn hello() {
   //the next line would throw
   message = 'hello';

   //the next line would be an error too
   var name = window.name;

   return message;
}

That would give me one of the big things out of Simple Modules that I cannot already have via RequireJS. It would also shorten the traditionally cumbersome "function" word and encourage good coding practice since it does not have access to the global scope. For situations that need global access, fall back to good old "function".

I would not be surprised if this has already been suggested, but as mentioned in my sketches preamble, just jotting down things as they come to me without doing due diligence.

Web Workers as Browser/Server Bridges

2010-10-12T11:21:00.000-07:00

David Ascher and I had a discussion last week about server side JavaScript. He mentioned that while server side JavaScript would give some small incremental advantage, there would still be a split between what server devs needs and do and what browser devs need and do. Other people have made a similar observation.

Here is one sketch on what server side JavaScript could bring.

Server-side JavaScript

There are two ways to treat the browser and server workload split . One treats the browser as a dumb client, the other as a smart client.

1) Dumb client

Blogs, news sites, wikis. Content Management Systems (CMS). Serve HTML strings to a browser. There is likely some browser interactivity, but the sites can get by without it via progressive enhancement. The server is responsible for generating the HTML from data mashed together with templates.

This use case is well understood problem space. It is not always executed well, but it has been done many times. Having server side JavaScript will help with some sharing of code/design approaches with the browser, but that is about it. Still, for me it would be nice to have SSJS solutions for these cases.

2) Smart client

Servers as APIs/data stores. The server is doing straight-out business logic, data manipulation. The data sent to the browser is just in JSON, preferably not XML.

Server-side JavaScript is useful here given the expressiveness of the language: closures/anonymous functions for callbacks make using async easier to use, and Node's focus on an event loop is a great fit here, a more natural experience for a browser developer to help out with this area.

Using Servers as Web Workers

This an extension of #2. First, go read up on Web Workers. Short review:


var worker = new WebWorker('http://example.com/some/thing');

//Get messages from the worker
worker.onmessage = onMessageFunction;

//Get errors from the worker
worker.onerror = onErrorFunction;

//Post messages to the worker, only
//JSON-compliant messages back and forth
worker.postMessage({"msg": "hello"});

What would be ideal is that instead of running the web worker in the browser, it would do its work on the server. To bootstrap, a small JS shim could be used that runs in the browser that does the server communication to run the code on the server, and then a Web Worker environment/toolkit/library on the server needs to be created. This node-worker project might be a good place to start for the server code.

The important point: an app developer codes the logic on the server to the Web Worker environment, using postMessage to send out responses, onmessage and onerror to receive responses.

The neat thing about this approach: it can be used for simple request/response actions, or for longer term, comet-style long-lived messaging. It also could allow for actually running or mocking the server endpoint in the browser.

There needs to be some way to manage state. It would be nice to "pause" and "resume" a server-based web worker. Maybe that is just cookies, but it has to be secure to things like CSRF, and still give the user the ability to clear that state like they can clear cookies today.

The other nice thing about this model, it fits in with the event loop that JS developers use today, and it is a tightly constrained environment. No messing about with "requests" and "responses" in the traditional server sense.

There are probably concerns about what it does to a REST approach to API development. I am hoping that it just transforms the REST calls into events routed to web workers, not sure how that will shake out yet.

This is a JavaScript sketch, please read the disclaimers.

JavaScript Sketches

2010-10-12T11:01:00.000-07:00

I have a few JavaScript-related things I want to talk about. I prefer implementing something or give a proposal more consideration before talking about it. However, I find that I end up not talking about a lot of things that bounce around in my head. So I will start talking more about them, even though they are not fully formed. This blog post is to set up the context and disclaimers for those posts, so I can link back to them.

Disclaimers

I will make a bunch of declarations that I will not back up as much as they deserve. I may expand on them in later posts, but most likely I will not. The point of these sketches is to get the basic thought out instead of keeping it internal.

If I cast a negative light on your favorite project, keep in mind this is my blog, so it is naturally biased, and there always needs to be hedge bets. Until there are implementations with real world use, the future is always malleable, and often big enough to accommodate a few different views. Keep working on what you are passionate about.

I will be ruthless and what may seem like unfair in my comment management/deletion policy. You are free to say your own piece on your own blog.

Who am I?

I have been working with JavaScript since at least 1998. It was earlier than that, but that was the start of a large scale project that got real users. Since then I have used it on and off. I still need to learn more about the inner dark places of the language and its implementations, but I have used it to build some larger front-ends, particularly while at AOL: a picture service UI, a cross-browser plugin for streaming radio, a chat service UI, and helped out with a webmail service UI. I contribute to Dojo, make RequireJS, and work at Mozilla on web-based messaging services. I only want to develop for the web platform, using JavaScript. I did not study computer science, but physics, so I lack some of the CS technical underpinnings.

RequireJS 0.14.2 Released

2010-10-03T14:46:00.000-07:00

RequireJS 0.14.2 is available for download. This is a bug fix release:

Fix issue with "module" dependency not correctly assigning exported value if it is not listed as the last dependency.
Fix "packages" config option. Its behavior was fixed to match the docs.
Fix module-to-name resolution to account for package mappings.

These issues affect use cases with traditional CommonJS modules/converted modules or with CommonJS packages. If you do not deal with those use cases, there is less urgency to upgrade.

I recently started using the RequireJS code in Node beyond simple tests, working on a Node-based package tool. Once I get it up an running, I'll post more on this blog. In the meantime, there is a design sketch, and some code, but it is very rough at the moment, mostly scaffolding.

RequireJS 0.14.1 Released

2010-09-27T21:50:00.000-07:00

RequireJS 0.14.1 is now available.

I just pushed a small update to fix three issues that dealt mostly with the new shortened anonymous module syntax used to wrap traditional CommonJS modules, and the converter tool for adding in the anonymous wrapper.

If you were using the regular RequireJS module format, with the dependencies specified outside the definition function, then you likely do not need to upgrade right away.

In fact, you may want to wait a couple days to see if there are any other updates. Given the newness of the anonymous module code and more people trying it with traditional CommonJS modules, I want to push out quicker releases to give those new users the best experience. I will be sure to mention if the update is recommended for all users.

There is one fix in this release for a type of deeply cyclic/circular dependency issue, but I believe it to be a rare issue for most current users. If 0.14.0 is working for you, then no need to try the latest version right away.

RequireJS 0.14.0 Released

2010-09-26T23:29:00.000-07:00

RequireJS 0.14.0 is now available. The big changes:

Anonymous modules support, CommonJS Asynchronous Module proposal supported.
Loading modules from CommonJS packages.
Bug fixes (see commits starting from (see commits starting from 2010-09-15 through 2010-09-26))

This is a significant release. Thanks to Kris Zyp for figuring out how to make anonymous modules work with IE, and to Tom Robinson for the suggestion to use Function.prototype.toString() to make converting traditional CommonJS modules easier.

The async module format/async require that is now supported by RequireJS really feels like it is the best of both worlds: something that is close enough to traditional CommonJS modules to allow those environments to support the format, while still having something that performs well and is easy to debug in the browser. I really hope the format can be natively supported in existing CommonJS engines. Until then, RequireJS works in Rhino and has an adapter for Node.

I put up a CommonJS Notes page for people coming from a CommonJS background. Also, the API docs are updated to reflect the simpler anonymous module format, and it includes a new section about loading modules from CommonJS packages.

What is next

Working with CommonJS packages has a few manual steps: finding the package, downloading it, configuring its location. I want to work on a command line package tool that makes this easy. Hopefully it will be able to talk to a server-side package registry too, to allow simpler package lookups by name (something like npm, but something that can house modules in a format used by RequireJS). Kris Zyp has already done some work in this area, and I hope to just use it outright for RequireJS, or leverage the code.

Once that lands, then it feels like it will be time for a RequireJS 1.0 release. The code has been very usable for a few releases now, but I have kept the release numbers below 1.0 to indicate that the final mix of features were being worked out. With the changes in this release, it feels like the major format changes have landed. For those of you who have used previous RequireJS releases, your code should still work fine, and it should work as-is in future releases too.

Anonymous Module Support in RequireJS

2010-09-20T22:06:00.000-07:00

Thanks to the clever research and design feedback from Kris Zyp, I just finished committing some preliminary support for anonymous modules in RequireJS.

What are anonymous modules?

They are modules that do not declare their name as part of their definition. So instead of defining a module like so in RequireJS:

require.def('foo', ['bar'], function(bar) {});

You can now do this:

require.def(['bar'], function (bar) {});

When using RequireJS in the browser, the name of the module will be inferred by the script tag that loads it. For Rhino/Node, the module name is known at the time of the require.def call by the require() code, so those environments have an easier way to associate the module definition with the name.

Why is this important?

This allows your modules to be more portable -- if you change the directory structure of where a module is, there are fewer things that you need to change. You still may want to check module dependencies, but RequireJS now fully supports of relative module names, like "./bar" and "../bar", so by using those, it can help make your module more portable.

Requiring a module name in the module definition was also a notable objection that the CommonJS group had to the module format that RequireJS supports natively. By removing this objection, it gets easier to talk about unifying module formats across the groups.

To that end, there has been talk in the CommonJS group of an Asynchronous Module definition, something that allows the modules to work well in the browser without needing any server or client transforms. Some of the participants do not like the module format mentioned above, and prefer something that looks more like the existing CommonJS modules.

Tom Robinson put forward this suggestion:

require.def(function(require, exports, module) {
    var foo = require("foo"),
        bar = require("bar");
    exports.someProp = "value";
});

and use Function.prototype.toString() to pull out the require calls and be sure to load them before executing the module's definition function. After doing some research, it seems like this approach could work for modules in development, and optimizations could be done for deployment that would add the module name and the dependencies outside the function.

So I also put support for the above syntax into RequireJS, in an attempt to get to an async module proposal in CommonJS that works for the the people that like the old, browser-unfriendly syntax and for the people like me that prefer a browser-friendly format I can code in source.

We still need to hash out the proposal more, but I am hopeful we can find a good middle ground. I also hope the above syntax makes it easier to support setting the module export value via "return" instead of having to use "module.exports =" or "module.setExports()".

I still plan to support the syntax that RequireJS has supported in the past -- any of this new syntax will hopefully be additive.

What is the fine print?

Only one anonymous module can be in a file. This should not be a problem, since you are encouraged to only put one module in a file for your source code.

The RequireJS optimization tool can group modules together into an optimized file, and it has the smarts to also inject the module name at that time, so you get less typing and a more robust module source form, but still get the optimization benefits for deployment.

In addition to adding the module name, the RequireJS optimization tool will also pull out the dependencies that are specified using the CommonJS Asynchronous Module proposal mentioned above, and add those to the require.def call to make that form more efficient.

When will it be available?

Right now the code is in the master branch. Feel free to pull it and try it. There may be some loose ends to clean up, but there are unit tests for it, and the old unit tests pass.

This code will likely be part of a 0.14 release. I want to get in loading modules from CommonJS-formatted packages before I do the 0.14 release, so it still is probably a few weeks away. But please feel free to try out the latest code in master to get an early preview.

Again, many thanks to Kris Zyp for seeing patterns I overlooked, doing some great IE research, and for pushing for these changes.

RequireJS 0.13.0 Released

2010-09-11T00:31:00.000-07:00

RequireJS 0.13.0 is available! The changes:

module.setExports and module.exports are now supported for converted CommonJS modules.
Bug fixes (see commits starting from 2010-07-05 through 2010-09-10), in particular a fix to throw when a timeout for a script occurs. That fix should make debugging issues much easier.

Some bigger changes are in store for the next release. Kris Zyp has observed a pattern that may allow supporting "anonymous" require.def() module definitions, allowing modules to be more robust to path changes without requiring file editing. I want to get that to work, as well as support for CommonJS-style packages.

Simple Modules Feedback

2010-07-13T11:33:00.000-07:00

Executive summary, with apologies to Jay-Z: "I got 99 problems but lack of lexical scoping ain't one".

While at the Mozilla Summit, I saw Dave Herman's presentation on a Simple Modules proposal for JavaScript/ECMAScript. Dave posted the slides, and be sure to read his follow-up post. I suggest you read his slides and blog posts first for some background.

[Sidenote: I'm going to use JavaScript instead of ECMAScript in this post -- JavaScript and I go way back, before it got its colonial, skin disease-inspired name.]

Simple Modules is a strawman proposal at the moment, it is still a work in progress. Some of the more interesting parts for me, the dynamic loading, are still very rough and in a separate proposal. It sounded like Dave wants to focus on prototyping the lexical scoping and static loading bits first before proceeding further on the dynamic bits. Great idea on actual prototyping, sounds like they will leverage Narcissus for doing the prototyping.

So some of my feedback may be a bit premature, but some of it gets to why there are modules and what should be allowed as a module, so hopefully that might be useful even at this early stage.

First, some perspective on where my feedback comes from: I am a front-end developer, I do web apps in the browser. I love JavaScript, I want to use it everywhere, and I believe that it is the only language that has the potential to be used effectively anywhere.

However, that is only because JavaScript is available and works well in the browser. Any new solutions for modules should *work well* in the browser to be considered a solution. The browser environment should be treated as a first class citizen, keeping in mind the browser performance implications on any approach. This is one of my main criticisms of CommonJS modules, and the reason I write RequireJS, a module loader that works well in the browser. I also maintain Dojo's module loader and build system.

Why

Why have modules? What are they? Modules are smaller units of code that help build up larger code structures. They make programming in the large easier. They usually have specific scope, and avoid dumping properties into the global scope. Otherwise the likelihood of a name collision between two modules is very high and errors occur. So a module system has syntax to avoid polluting the global space.

There also needs to be a way for modules to reference other modules.

Simple Modules

The Simple Modules proposal outlines a Module {} block to define what looks like a JavaScript object as far as inspection (for .. in notation, dot property referencing), but is something more nuanced underneath.

Anything inside the Module {} block is not allowed to use a global object, and you cannot add/change a Module after its definition. Here is a sample module, called M, that demonstrates some of the syntax and scoped variable implications:

module M {
   //In normal code this would define a global,
   //but not inside the module declaration. This
   //is likely to be an error(?) in Simple Modules.
   foo = "bar";

   //color is only visible within module M's block
   var color = "blue";

   //Creates a publicly visible property called
   //"name" on the module.
   export name = "Module M";

   //setColor is only visible within module M's block
   function setColor() {}

   //Creates a publicly visible property called
   //"reverseName" on the module whose value
   //is a function
   export function reverseName() {}
}

You can reference/statically load other modules via load (syntax is just a placeholder, not set in stone):

module jQuery = load “jquery.js”;

The goals with this approach:

Stronger lexical scoping: no eval or with allowed in the modules, and a loaded module shares some lexical scope with the module that loaded it.
No access to a global object by default.
Hopefully better syntax for declaring modules over the existing function-based module pattern.

To the extent that modules help programming in the large, the simple modules approach do not give me anything more than I have now, and the proposal has some specific weaknesses. I can appreciate there are some juicy things in the proposal for JavaScript engine developers, but as a user of modules, I do not see it as a net advantage. Here is why:

Lexical scoping/Global access

In addition to using the function-based module pattern to avoid leaking globals, I use JSLint to avoid accessing globals and the use of eval/with. For programming in the large, JSLint helps even more because it enforces a code style that produces much more uniform code. It is built into many editors and easy to run as part of build processes.

JSLint is not perfect (I would like to see JavaScript 1.7/1.8 idioms supported, like let and for each), and you may not like some of the style choices. However, reproducible, consistent style that can be checked automatically is more important than bikeshed-based style choices. It warns of global usage, eval and with, and even helps you find unused local variables.

What is even nicer is that you can opt out of some of the JSLint choices, you can use some globals if you need to. There is some flexibility in the choices.

Functions as Modules

For #3, better syntax for module definitions, I do not see it as a net win over the function(){} module pattern, particularly how it is used for modules in RequireJS where it encourages not defining global objects.

The Simple Modules syntax does not allow exporting a function as the module definition. This is a big wart to me. Functions are first class entities in JavaScript, one of its strongest features. It is really ugly to me that I have to create a property on a module object to export a constructor function, or some module that lends itself to being a function:

module jQuery {
   export jQuery = function () {};
}

/*** In some other file ***/
module jQuery = load “jquery.js”;
//ugly
var selection = jQuery.jQuery();

//less ugly, but more typing, so still ugly
var $ = jQuery.jQuery;
var selection = $();

Again, ugly. It should be possible to set the module value to be a function. I know this makes some circular dependency cases harder to deal with, but as I outlined in the CommonJS trade-offs post, it is possible to still have circular dependencies. Even in CommonJS environments now, it is seen as useful. Node supports setting the exported value to a function via module.exports, and there is a more general CommonJS proposal for a module.setExports.

It means the developer that codes a circular dependency case needs to take some care, but it works. Coding a circular dependencies is a much rarer event than wanting to use a module that exports just a constructor function or function. The majority use case should not be punished to make a minority use case a little easier, particularly since you can still trigger errors in the minority use case. Coding a circular dependency will always require special care.

This particular point makes it hard for me to get on board with Simple Modules even in its basic lexical scoping/static loading form. I strongly urge any module proposal to make sure functions can be treated as the exported value. We have that capability today with existing module implementations, and it fits with JavaScript and the importance it places on functions.

Given the extra typing that would be needed to access functions that are exported as modules, I do not see the Simple Modules syntax a net win over the function-based module pattern, particularly as used in RequireJS.

Beyond Lexical Scoping

For programming in the large, what is really needed are more capabilities than what has been outlined so far for Simple Modules. However, making modules useful needs attention in these areas. This is the "99 problems" part:

Dynamic loading

Dynamic loading is harder to work out than static loading. If there is dynamic loading, it is unclear I would need static loading . The goal of modules is to allow programming in the large, and even for a smaller project, why do I need to learn two ways to load modules (static vs. dynamic), when one (dynamic) will do? Dynamic loading is also necessary to enable all the performance options we have today to load scripts in the browser.

There is a module loader strawman proposal that would tie into Simple Modules, but I understand it will not be nailed down more until the basic Simple Modules with static loading is worked out/prototyped.

Referring to other modules

It is unclear how a Module Resource Locator (MRL) is translated to a path to find a module. In CommonJS/RequireJS, an MRL looks like "some/module", and that MRL is used in require() calls to refer to other modules. require("some/module") translates the MRL string "some/module" to some path, "a/directory/that/has/some/module.js". That path is used to find and load the referenced module.

Looking at the Simple Modules examples, it looks like just plain URLs are used as the MRL, and those do not scale well for programming in the large. You will want to use a symbolic name for the MRL, and allow some environment config to map those symbolic names to paths. Otherwise it places too many constraints on how the code is stored. It may not even be a disk -- apparently CouchDB uses design docs to store modules.

I have seen some comments about using more symbolic names for MRLs in some of the notes around the proposals, so maybe it is planned.

In RequireJS, the symbolic name is also used in the module definition. However, since symbolic names can be mapped, they do not have to be the reverse DNS symbolic names, like "org/mozilla/foo". In fact it is encouraged to not use long names.

Distributing and sharing modules/module groups (packages)

This issue can be treated separately from a module spec, but it could affect how MRLs are mapped via a module loader. And this issue really is important for programming in the large. The solution may just be "use packages as outlined by CommonJS". While there are still some gray areas in the package-related specs for CommonJS, that could be a fine answer to the problem.

Performance in the browser

This is getting even further away from the basic Simple Modules spec, but a solution to this issue should be considered for any module solution. The browser needs to be able to deliver many modules at once to the browser in an efficient way. I have heard that Alexander Limi's Resource Packages proposal may be a way to solve this that may work with the Simple Modules approach.

A common loading pattern for web apps will be to load some base scripts from a Content Delivery Network (CDN), then have some domain-specific scripts to load. As long as this still works well with the bundling solution that is great. We already have tools today to help bundling, minifying and gzipping scripts. Any solution will have to be better than what we can do today. Resource Packages could be since it allows other things like images to be effectively bundled.

Summary

I do not feel like Simple Modules are an improvement over what can be done today. In particular, I feel RequireJS when used alongside JSLint is a compelling existing solution, and it works well, and fast, in the browser.

For the more immediate goals of Simple Modules:

the expanded, stricter lexical scoping is nice, but for a web developer, it is a slight incremental benefit if JSLint is already in use.
Not being able to set a function as the module value means the syntax is not a net win over the function-based module pattern.

The larger issues of module addressing and bundling/distribution are understandably hazy in this early stage of the strawman proposals, but they will need to be addressed as well as or better than existing solutions to gain traction.

I do not want to contribute stop energy around the proposals, I am just hoping to provide feedback to indicate what problems need to be solved better from my web developer viewpoint. I appreciate I could be wrong on some things too. I may be missing something grander or larger, but hopefully if that is the case, this feedback can indicate how to explain the proposals better.

RequireJS 0.12.0 Released

2010-07-04T22:12:00.000-07:00

RequireJS 0.12.0 is available! This release has the following enhancements:

A new plugin: order -- it ensures that scripts are fetched asynchronously and in parallel, but executed in the order specified in the call to require(). Ideal for traditional browser scripts that do not participate in modules defined via calls to require.def().
Web Worker support. RequireJS can be used in a web worker.
Multiple module names can now be mapped via the paths config option to the same URL, and that URL will only be fetched once.
Added Firefox 2 to supported browsers. Safari 3.2 also works with require().
Bug fixes.

Thanks to Alex Sexton for driving forward the order plugin. It is a great addition to the project.

Using RequireJS syntax in Jetpack Reboot

2010-05-19T13:18:00.000-07:00

I have a clone of the Jetpack SDK that has support for the require() and require.def() syntax supported by RequireJS.

Right now the syntax support is very basic. It does not support these features of RequireJS:

configuring require() by passing in a config object to it
plugins
require.modify
require.nameToUrl
require.ready (does not make sense)

But you can do the main things, like:

require(["dependency"], function (dependency){}());

and define modules via:

require.def("moduleName", ["dependency"], function (dependency){}());

It should also support CommonJS modules that were converted to RequireJS syntax via the conversion tool in RequireJS, but I have not tested it extensively.

The changes are just in one file in the sdk, securable-module.js. So you could just grab that file if you wanted to play with it. There is a sample app in the source if you want to see it in action. Also viewing the changeset shows the diff on the securable-module.js file as well as the example app source.

The full cloned repo is available via:

hg clone http://hg.mozilla.org/users/jrburke_gmail.com/jetpack-sdk-requirejs

Why do this? Because sharing code between the browser and other environments is hard with the regular CommonJS syntax. It does not work well in the browser. The browser-based CommonJS loaders that use eval() have a worse debugging experience. Starting with the RequireJS syntax makes it easy to transfer the modules for use in the web browser, and the RequireJS code works in Node and Rhino.

I would like to add support for RequireJS plugins in Jetpack. I can see the i18n plugin and text file plugin being useful for Jetpacks. That will likely take more work though. I want to see if the basic syntax support is useful first.

I ended up not using that much RequireJS code, just some argument conversions and supporting "setting the exported value". It relies on the existing Jetpack code for paths and package support.

RequireJS 0.11.0 Released

2010-05-16T21:25:00.000-07:00

RequireJS 0.11.0 is available to download! This release has the following enhancements:

There is a new priority config option to indicate priority, parallel download of build layers.
A new JSONP plugin allows you to treat any JSONP service as dependency.
require.js should be Caja-compliant. The plugins may not be, but the main require.js file passed cajoling on http://caja.appspot.com/.
Instructions and optimization support for renaming require().
There is a new RequireJS+Transport D download option that supports the CommonJS Transport D proposal. This can be useful in conjunction with the server-side Transporter project.

Thanks to Alex Sexton for outlining the Caja and require() renaming aspects and to Sean J. Vaughan for instigating the JSONP plugin.

The priority config option is the parallel download support I mentioned in the "A require() for jQuery" post. I now believe RequireJS meets all the requirements outlined in that post.

Some icing on the cake I want to pursue: a server-based service that can create optimization layers on the fly. I have all the pieces in place in the optimization tool to allow this, and I previously built a server build option for Dojo. With that, you could conceivably use the priority config support with a server that did the optimization layers on the fly:

require({
  priority: [
      "http://your.domain.com/opt?include=event,object,widget,Dialog&exclude=jquery",
      "http://your.domain.com/opt?include=page1,Tabs&exclude=jquery,event,object,widget,Dialog&exclude=jquery"
  ]
}, ["page1"]);

Or something like that. The fun part -- this server endpoint would use server-side JavaScript, since the optimization tool in RequireJS is built in JavaScript. I could use Node or something Rhino-based. It is likely to be Rhino-based since that allows the minifier, Closure Compiler, to work on the fly, since Closure Compiler is written in Java.

That server-based service will likely take a more design work and thought, but if you feel it is something necessary for your project, please let me know. Better yet, if you want to contribute to the project in this area, leave a note on the mailing list.

A require() for jQuery

2010-04-29T22:00:00.001-07:00

I had a fun time at the Bay Area jQuery Conference. Great people, and I learned some neat things.

In the conference wrap-up, John Resig mentioned some requirements he has for a jQuery script loader:

1) script loading must be async

2) script loading should do as much in parallel as possible. This means in particular, that it should be possible to avoid dynamic nested dependency loading.

3) it looks like a script wrapper is needed to allow #1 and #2 to work effectively, particularly for cross-domain loading. It is unfortunate, but a necessity for script loading in browsers.

I believe these requirements mesh very well with RequireJS. I will talk about how they mesh, and some other things that should be considered for any require() that might become part of jQuery.

Async Loading

As explained in the RequireJS Why page, I believe the best-performing, native browser option for async loading is dynamically created script tags. RequireJS only uses this type of script loading, no XHR.

The text plugin uses XHR in dev mode, but the optimization tool inlines the text content to avoid XHR for deployment. Also, the plugin capability in RequireJS is optional, it is possible to build RequireJS without it. That is what I do for the integrated jQuery+RequireJS build.

Parallel Loading

John mentioned that dynamic nested dependency resolution was slower and potentially a hazard for end users. Slow, because it means you need to fetch the module, wait for it to be received, then fetch its dependencies. So the module gets loaded serially relative to its dependencies. Potentially hazardous because a user may not know the loading pattern.

The optimization tool in RequireJS avoids the parallel loading for nested dependencies, by just inlining the modules together. The optimization tool can also build files into "layers" that could be loaded in parallel.

For each build layer, there is an exclude option, in which you can list a module or modules you want to exclude. exclude will also exclude their nested dependencies from the build layer.

There is an excludeShallow option if you just want specific modules to exclude, but still want their nested dependencies included in the build layer. This is a great option for making your development process fast: just excludeShallow the current module you are debugging/developing.

While dynamically loading nested dependencies can be slower than a full parallel load, what is needed is listing dependencies individually for each module. There needs to be a way to know what an individual file needs to function if the file is to be portable in any fashion. So the question is how to specify those dependencies for a given file/module.

There are schemes that list the dependencies as a separate companion file with the module, and schemes that list the dependencies in the module file. Using a separate file means the module is less portable -- more "things" need to follow the module, so it makes copy/pasting, just distributing one module more onerous.

So I prefer listing the dependencies in the file. Should the dependencies be listed in a comment or as some sort of script structure?

Comments can be nice since they can be stripped from the built/optimized layer. However, it means modules essentially need to communicate with each other through the global variable space. This ultimately does not scale -- at some point you will want to load two different versions of a module, or two modules that want to use the same global name, and you will be stuck. For that reason, I favor the way RequireJS does it:


require.def("my/module", ["dependency1"], function (dependency1) {
  //dependency1 is the module definition for "dependency1"

  //Return a value to define "my/module"
  return {
      limit: 500,
      action: function () {}
  };
});

With this model, dependency1 does not need to be global, and it allows a very terse way to reference the module. It also minifies nicely. By using string names to reference the modules and using a return value from the function, it is then possible to load two versions of module in a page. See the Multiversion Support in RequireJS for more info, and the unit tests for a working example.

This model also frees the jQuery object from namespace collisions by allowing a terse way to reference modules without needing them to hang off of the jQuery object. There are many utility functions that do not need to be on the jQuery object to be useful, and today the jQuery object itself is starting to become a global of sorts that can have name collisions.

Script Wrapper

Because async script tags are used to load modules, each script needs to be wrapped in a function wrapper, to prevent its execution before its dependencies are ready. CommonJS recognizes this concern (one of the reasons for their Transport proposals) and so does YUI3. xdomain builds for Dojo also use a script wrapper.

While it is unfortunate -- many people are not used to it -- it ends up being an advantage. Functions are JavaScript's natural module construct, and it encourages well scoped code that does not mess with the global space. For RequireJS, that wrapper is called require.def, as shown above.

Here are some other things that should be considered for a require implementation:

require as a global

I believe it makes more sense to keep require as a global, not something that is a function hanging off of the jQuery object. require can be used to load jQuery itself, and as mentioned above, it would be possible to load more than one version of jQuery if it was constructed like this.

CommonJS awareness

The CommonJS module format was not constructed for the browser, but having an awareness of their design goals and a way to support their modules in the browser will allow more code reuse. RequireJS has an adapter for the CommonJS Transport/D proposal, and it has a conversion script to change CommonJS modules into RequireJS modules.

In addition, RequireJS was constructed with many of the same design goals as CommonJS: allow modules to be enclosed/do not pollute the global space, use the "path/to/module" module identifiers, have the ability to support the module and exports variables used in CommonJS.

Browsers need more than a require API

They also need an optimization/build tool that can combine modules together. RequireJS has such a system today. It is server-independent, a command line tool. It builds up the layers as static files which can be served from anywhere.

I am more than happy to look at a runtime system that uses the optimization tool on the server. RequireJS works in Node and in Rhino. The optimization tool is written in JavaScript and uses require.js itself to build the optimization layers.

I can see using either Node or Rhino to build a run-time server tool to allow combo-loading on the fly. Using Rhino via the Java VM has an advantage because Closure Compiler or YUI Compressor could be used to minify the response, but I am open to some other minification scheme that is implemented in plain JavaScript.

Loader plugins

I have found the text plugin for RequireJS to be very useful -- it allows you to reference HTML templates on disk and edit HTML in an HTML editor vs. dealing with HTML in a string. The optimization tool is smart enough to inline that HTML during a build, so the extra network cost goes away for deployment.

In addition, Sean Vaughan and I have been talking about support for JSONP-based services and scripts that need extra setup besides just being ready on the script onload event. I can see those as easy plugins to add that open up loading Google Ajax API services on the fly.

For these reasons I have found loader plugins to be useful. They are not needed in the basic case, but they can make overall dependency management better.

script.onload

Right now RequireJS has support for knowing when a script is loaded by waiting for the script.onload event. This could be avoided by mandating that anything loaded via require() register via require.def to indicate when it is loaded.

However, by using script.onload it allows some existing scripts to be loaded without modification today, to give people time to migrate to the require.def pattern. I am open to doing a build without the script.onload support, however the amount of minified file savings will not be that great.

Explicit .js suffix

RequireJS allows two different types of strings for dependencies. Here is an example:

require(["some/module", "http://some.site.com/path/to/script.js"]);

"some/module" is transformed to "some/base/path/some/module.js", while the other one is used as-is.

The transform rules for a dependency name are as follows: if the name contains a colon before a front slash (has a protocol), starts with a front slash, or ends in .js, do not transform the name. Otherwise, transform the name to "some/base/path/some/module.js".

I believe that gives a decent compromise to short, remappable module names (by changing the baseUrl or setting a specific path via a require config call) to loading scripts that do not participate in the require.def call pattern. There is also a regexp property on require that can be changed to allow more exceptions to the rules.

However, if this was found insufficient, I am open to other rules or a different way to list dependencies. The "some/module" format was chosen to be compatible with CommonJS module names, but probably some algorithm or approach could be used to satisfy both desires.

File Size/Implementation

Right now the stock RequireJS is around 3.7KB minified and gzipped. However, there are build options that get the size down to 2.6KB minified and gzipped by removing some features:

plugin support
require.modify
multiversion support (the "context" switching in RequireJS)
DOM Ready support

I am open to getting that file size smaller based on the feature set that needs to be supported.

3 layer loading

John mentioned a typical loading scenario that might involve three sections:

1) loading core libraries from a CDN (like jQuery and maybe a require implementation)
2) loading a layer of your common app scripts
3) loading a page-specific layer

RequireJS can support this scenario like so today:


<script src="http:/some.cdn.com/jquery/1.5/require-jquery.js"></script>
<script>
require({
     baseUrl: "./scripts"
 },
 ["app/common", "app/page1"]
);
</script>

Then the optimization tool instructions would look like so:

{
 modules: [
     {
         //inside app/common.js there is a require call that
         //loads all the common modules.
         name: "app/common",
         exclude: ["jquery"]
     },
     {
         //app/page1 references jquery and app/common as a dependencies,
         //as well as page-specific modules
         name: "app/page1",

         //jquery, app/common and all their dependencies will be excluded
         exclude: ["jquery", "app/common"]
     },
     ... other pages go here following same pattern ...
 ]
}

This would result in app/common and app/page1 being loaded async in parallel. If require.js was a separate file from jquery.js, the following HTML could be used to load jQuery, app/common and app/page1 async and in parallel (the optimization instructions stay the same):


<script src="http:/some.cdn.com/jquery/1.5/require.js"></script>
<script>
require({
     baseUrl: "./scripts",
     paths: {
         "jquery": "http:/some.cdn.com/jquery/1.5/jquery"
     }
 },
 ["jquery", "app/common", "app/page1"]
);
</script>

Those configurations work today.

However, it is not quite flexible enough -- typically modules that are part of app/page1 will not want to refer to the complete "app/common" as the only dependency, but specify finer-grained dependencies, like "app/common/helper". So the above could result in a request for "app/commom/helper" from the "app/page1" script, depending on how fast "app/common" is loaded.

So I would build in support for the following:


<script src="http:/some.cdn.com/jquery/1.5/require.js"></script>
<script>
require({
     baseUrl: "./scripts",
     paths: {
         "jquery": "http:/some.cdn.com/jquery/1.5/jquery"
     },
     layers: ["jquery", "app/common", "app/page1"]
 },
 ["app/page1"]
);
</script>

Notice the new "layers" config option, and now the required modules for the page is just "app/page1". The "layers" config option would tell RequireJS to load all of those layers first, and find out what is in them before trying to fetch any other dependencies.

This would give the most flexibility in coding individual modules, but give a very clear optimization path to getting a configurable number of script layers to load async and in parallel. I will be working on this feature for RequireJS for the next release.

Summary

Hopefully I have demonstrated how RequireJS could be the require implementation for jQuery. I am very open to doing code changes to support jQuery's desires, and even if jQuery or John feel like they want to write their own implementation, hopefully we can at least agree on the same API, and maybe even still use the optimization tool in RequireJS. I am happy to help with an alternative implementation too.

I know John and the jQuery team are busy, focusing mostly on mobile and templating concerns, but hopefully they can take the above into consideration when they get to script loading.

In the meantime, I will work on the layers config option support, improving RequireJS, and keeping my jQuery fork up to date with the changes. You can try out RequireJS+jQuery today if you want to give it a spin yourself.

RequireJS+jQuery Talk

2010-04-25T16:26:00.000-07:00

I gave a talk about RequireJS with jQuery at the jQuery Conference today. Here are the slides:

PDF
HTML (Warning, the inline links do not appear to work, use PDF for working links)

Thanks to the folks that came to the talk! I had a great time at the conference.

If you went to the talk, please feel free to rate the talk so I can improve for the next time.

RequireJS 0.10.0 Released, Node integration

2010-04-23T11:20:00.000-07:00

RequireJS 0.10.0 is now available.

The big feature in this release is integration with Node. Now you can use a the same module format for both browser and server side modules. The RequireJS-Node adapter translates existing CommonJS modules on the fly, as they are loaded by the adapter, so you can continue to use server modules written in the CommonJS format for your Node projects.

The RequireJS-Node adapter is freshly baked, so there could be some rough edges with it, but it is exciting to see it work. See the docs for all the details.

0.10.0 also includes support for an excludeShallow option in the optimization tool. This will allow you to do an optimization build during development, but still excludeShallow the specific module you want to develop/debug in the browser. So you can get great debug support in the browser for just that one module, but still load the rest of your JS super-fast. No need for special server transforms.

I will be at the jQuery conference this weekend in Mountain View, CA. I will be speaking on Sunday about jQuery+RequireJS. Stop by and say hi if you are at the conference!

JavaScript object inheritance with parents

2010-04-13T13:58:00.000-07:00

There are different ways to inherit functionality in JavaScript, including using mixins (mixing in all the properties of one object into another object) and the use of prototypes.

In Dojo, there is dojo.mixin for doing mixins, and dojo.delegate for inheriting properties via prototypes. dojo.delegate is like ECMAScript 5/Crockford's Object.create(), but with a dojo.mixin convenience call.

I really like the dojo.delegate or a Object.create+dojo.mixin combination for inheriting, but it makes it hard to call methods you override from your parent. I see this problem show up frequently with widgets, which typically inherit from each other:


var MyWidget = Object.create(BaseWidget);

//BaseWidget also defines a postCreate method,
//But we want our widget to do work too.

MyWidget.prototype.postCreate = function () {
 //Call BaseWidget's implementation
 BaseWidget.prototype.postCreate.apply(this, arguments);

 //Do MyWidget's postCreate work here.
};

Not too bad, but the BaseWidget.prototype.postCreate.apply junk is a bit much to type, and it gets a bit trickier when there are mixins that also contribute to the functionality.

In Dojo, there is dojo.declare() that helps with this by defining an "inherited" method that can be used to find the BaseWidget's postCreate:


var MyWidget = dojo.declare(BaseWidget, {
  postCreate: function () {
      //Call BaseWidget's implementation
      this.inherited("postCreate", arguments);

      //Do MyWidget's postCreate work here.
  }
});

This is an improvement as far as typing, but the implementation of dojo.declare has always scared me. My JavaScript Fu is not strong enough to follow it, and I am concerned it is actually a bit too complicated.

So here is an experiment on something simpler:


var MyWidget = object("BaseWidget", null, function (parent) {
 return {
     postCreate: function () {
         //Call BaseWidget's implementation
         parent(this, "postCreate", arguments);

         //Do MyWidget's postCreate work here.
     }
 };
});

Here is the implementation of that object function, and here are some tests. That implementation is wrapped in a RequireJS module, but it can be extracted as a standalone script.

The second argument to the object() function allows for specifying mixins.

With two mixins, mixin1 and mixin2, the parent for MyWidget would be an object that inherits from BaseWidget with mixin1 and mixin2's properties mixed in:


var MyWidget = object("BaseWidget", [mixin1, mixin2], function (parent) {
  return {
      postCreate: function () {
          //Call BaseWidget's postCreate, but if it
          //does not have a postCreate method, mixin1's
          //postCreate function will be used. If mixin1
          //does not have an implementation, then mixin2's
          //postCreate function will be used. If mixin2 does
          //not have an implementation an error is thrown.
          parent(this, "postCreate", arguments);

          //Do MyWidget's postCreate work here.
      }
  };
});

dojo.declare has the concept of calling a method called "constructor" if it is defined on the declared object, whenever a new object of the MyWidget type is created. I preserved that ability in object() but the property name for that function is "init" in the object() implementation.

The object() implementation is simpler than dojo.declare, but still gives easy access for calling a parent implementation of a function. It is not has powerful as dojo.declare -- dojo.declare has the concept of postscript and a preamble and even auto-chaining calls. However, I feel the simplified approach is better. It is clearer to follow the code, and to predict how it will behave. I also expect it to perform better.

I like the object() method because it uses closures and a function that accepts the parent function as an argument. Feels very JavaScripty. The prototype chain is a bit longer with the extra object.create() calls creating some intermediate objects, but I expect prototype walking is fast in JavaScript, particularly when you go to measure it in comparison to any DOM operation.

Are there ways in which the object() function is broken or insufficient? Is there a better way to do this? Or even a different way, something that does not rely on a parent reference?

There is traits.js, for using traits. Alex Russell experimented with a trait implementation inside dojo.delegate. Kris Zyp pointed out that Alex's implementation does not have conflict detection or method require support.

I like the idea of mixing in just part of a mixin or remapping a method to fit some other API's expectations, so I can see adding support for the remapping features, similar to what Alex does in the dojo.delegate experiment. However, I am not sure how valuable conflict detection or method require support is.

I can see in large systems it would help with detecting errors sooner, but then maybe the bigger problem is the complexity of the large system. And there is a balance to forcing strictness up front over ease of use. The trait.js syntax looks fairly wordy to me, and the extra benefit of the strictness may not be realized for most web apps.

Also, I do not see an easy way to get the parent reference. It seems like you need to remap each overridden parent function you want to call to a new property name. It seems wordy, with more properties hanging off an object. And do you need to make sure you do not pick a name that is already in use by an ancestor? Seems like it could lead to a bunch of goofy names on an object.

Reusing code effectively is an interesting topic. The traits approach is newer to me, and I keep wondering if there is a better way to do it. It has been fun to experiment with alternatives.

CommonJS Module Trade-offs

2010-03-30T19:20:00.000-07:00

First of all: why should you care about module formats?

If you use JavaScript, particularly in the browser, more is being expected of you each day. Every site or webapp that you build will want to do more things over time, and browser engines are getting faster, making more complex, web-native experiences possible. Having modular code makes it much easier to build these experiences.

One wrinkle though, there is no standard module format for the browser. There is the very useful Module Pattern, that helps encapsulate code to define a module, but there is no standard way to indicate your module's dependencies.

I have been following some of the threads in the CommonJS mailing list about trying to come up with a require.async/ensure spec and a Transport spec. The reason those two specs are needed in addition to the basic module spec is because the CommonJS module spec decided to make some tradeoffs that were not browser-friendly.

This is my attempt to explain the trade-offs the CommonJS module spec has made, and why I believe they are not the right trade-offs. The trade-offs end up creating a bunch of extra work and gear that is needed in the browser case -- to me, the most important case to get right.

I do not expect this to influence or change the CommonJS spec -- the developers that make up most of the list seem to generally like the module format as written. At least they agreed on something. It is incredibly hard to get a group of people to code in a certain direction, and I believe they are doing it because they love coding and want to make it easier.

I want to point out the trade-offs made though, and suggest my own set of trade-offs. Hopefully by explicitly listing them out, other developers can make informed choices on what they want to use for their project.

Most importantly, just because "CommonJS" is used for the module spec, it should not be assumed that it is an optimal module spec for the browser, or that it should be the default choice for a module spec.

Disclosure: I have a horse in this race, RequireJS, and much of its design comes from a different set of tradeoffs that I will list further down. I am sure someone who prefers the CommonJS spec might have a different take on the trade-offs.

To the trade-offs:

1) No function for encapsulating a module.

A function around a module can seem like more boilerplate. Instead each module in the CommonJS spec is just a file. This means only one module per file. This is fine on the server or local disk, but not great in the browser if you want performance.

2) Referencing and loading dependencies synchronously is easier than asynchronous

In general, sync programming is easier to do. That does not work so well in the browser though.

3) exports

How do you define the module value that other modules can use? If a function was used around the module, a return value from that function could be used as the module definition. However, in the effort to avoid a function wrapper, it complicates setting up a return value. The CommonJS spec instead uses a free variable called "exports".

The value of exports is different for each module file, and it means that you can only attach properties to the exports module. Your module cannot assign a value to exports.

It means you cannot make a function as the module value. Some frameworks use constructor functions as the module values -- these will not be possible in CommonJS modules. Instead you will need to define a property on the exports object that holds the function. More typing for users of your module.

Using an exports object has an advantage: you can pass it to circular dependencies, and it reduces the probability of an error in a circular dependency case. However, it does not completely avoid circular dependency problems.

Instead, I favor these trade-offs:

1) Use a function to encapsulate the module.

This is basically the core of the previously-mentioned Module Pattern. It is in use today, it is an understood practice, and functions are at the core of JavaScript's built-in modularity.

While it is an extra function(){} to type, it is fairly standard to do this in JavaScript. It also means you can put more than one module in a file.

While you should avoid multiple modules in a file while developing, being able to concatenate a bunch of modules together for better performance in the browser is very desirable.

2) Assume async dependencies

Async performs better overall. While it may not help performance much in the server case, making sure a format performs well out of the box in the browser is very important.

This means module dependencies must be listed outside the function that defines the module, so they can be loaded before the module function is called.

3) Use return to define modules

Once a function is used to encapsulate the module, the function can return a value to define the module. No need for exports.

This fits more naturally with basic JavaScript syntax, and it allows returning functions as the module definition. Hooray!

There is a slightly higher chance of problems in circular dependency cases, but circular dependencies are rare, and usually a sign of bad design. There are valid cases for having circular dependencies, but the cases where a return value might be a problem for a circular dependency case is very small, and can be worked around.

If getting function return values means a slightly higher probability of a circular dependency error (which has a mitigation) then that is the good trade-off.

This avoids the need for the "exports" variable. This is fairly important to me, because exports has always looked odd to me, like it did not belong. It requires extra discovery to know its purpose.

Return values are more understandable, and allowing your module to return a function value, like a constructor function, seems like a basic requirement. It fits better with basic JavaScript.

4) Pass in dependencies to the module's function wrapper

This is done to decrease the amount of boilerplate needed with a function wrapped modules. If this is not done, you end up typing the dependency name twice (an opportunity for error), and it does not minify as well.

An example: let's define a module called "foo", which needs the "logger" module to work:


require.def("foo", ["logger"], function () {

    //require("logger") can be a synchronous call here, since
    //logger was specified in the dependency array outside
    //the module function
    require("logger").debug("starting foo's definition");

    //Define the foo object
    return {
        name: "foo"
    };
});

Compare with a version that passes in "logger" to the function:


require.def("foo", ["logger"], function (logger) {

    //Once "logger" module is loaded it is passed
    //to this function as the logger function arg
    logger.debug("starting foo's definition");

    //Define the foo object
    return {
        name: "foo"
    };
});

Passing in the module has some circular dependency hazards -- logger may not be defined yet if it was a circular dependency. So the first style, using require() inside the function wrapper should still be allowed. For instance, require("logger") inside a method that is created on the foo object could be used to avoid the circular dependency problem.

So again, I am making a trade-off where the more common useful case is easier to code vs increasing the probability of circular dependency issues. Circular dependencies are rare, and the above has a mitigation via the use of require("modulename").

There is another hazard that can happen with naming args in the function for each dependency. You can get an off-by-one problem:


require.def("foo", ["one", "two", "three"], function (one, three) {
    //In here, three is actually pointing to the "two" module
});

However, this is a standard coding hazard, not matching inputs args to a function. And there is mitigation, you could use require("three") inside the module if you wanted.

The convenience and less typing of having the argument be the module is useful. It also fits well with JSLint -- it can help catch spelling errors using the argument name inside the function.

5) Code the module name inside the module

To define the foo module, the name "foo" needs to be part of the module definition:


require.def("foo", ["logger"], function () {});

This is needed because we want the ability to combine multiple module definitions into one file for optimization. In addition, there is no good way to match a module definition to its name in the browser without it.

If script.onload fired exactly after the script is executed, not having the module name in the module definition might work, but this is not the case across browsers. And we still need to allow the name to be there for optimization case, where more than one module is in a file.

There is a legitimate concern that encoding the module name in the module definition makes it hard to move around code -- if you want to change the directory where the module is stored, it means touching the module source to change the names.

While that can be an issue, in Dojo we have found it is not a problem. I have not heard complaints of that specific issue. I am sure it happens, but the fix cost is not that onerous. This is not Java. And YUI 3 does something similar to Dojo, encode a name with the module definition.

I think the rate of occurrence of this issue, and the work it takes to fix are rarer and one time costs vs. forcing every browser developer taking extra, ongoing costs of using the CommonJS module format in the browser.

Conclusion

Those are the CommonJS trade-offs and my trade-offs. Some of them are not "more right" but just preferences, just like any language design. However, the lack of browser support in the basic module spec is very concerning to me.

In my eyes, the trade-offs CommonJS has made puts more work on browser developers to navigate more specs and need more gear to get it to work. Adding more specs that allow modules to be expressed in more than one way is not a good solution for me.

I see it as the CommonJS module spec making a specific bet: treating the browser as a second class module citizen will pay off in the long run and allow it to get a foothold in other environments where Ruby or Python might live.

Historically, and more importantly for the future, treating the browser as second class is a bad bet to make.

All that said, I wish the CommonJS group success, and there are lots of smart people on the list. I will try to support what I can of their specs in RequireJS, but I do feel the trade-offs in the basic module spec are not so great for browser developers.

RequireJS 0.9.0 Released

2010-03-30T15:30:00.000-07:00

I just pushed a new release of RequireJS, 0.9.0.

The optimization tool has seen the most change in this release. It sports some CSS optimizations now and it is much more robust. It also includes command line options for optimizing just one JS file or one CSS file.

The other new feature is the support for relative module names for require.def() dependencies. So this kind of call works now:

require.def("my/project/module", ["./dependency1"], function(){});

It will load my/project/dependency1.js. This should help cut down the amount of typing for larger projects that have deep directories of modules.

This release has some backwards-incompatible changes. That was the reason for the bump to 0.9.0. The project is still not at 1.0, so backwards-incompatible changes may still be considered. I do not have any more changes like that planned, but I will be sure to give more notice in the RequireJS list before doing so in the future.

All the details are on the download page.

Raindrop has been updated to the latest RequireJS release, and it is working great. Give the new RequireJS build a spin!

RequireJS, kicking some AST

2010-03-14T17:04:00.000-07:00

RequireJS has an optimization tool that can combine and minify your scripts. It uses Google's Closure Compiler to do the minification. Recently, but after the RequireJS 0.8.0 release, I ported over the CSS optimizations from the Dojo build system, so the optimization tool now inlines @import calls and remove comments from CSS files.

The script combining still has some rough edges though, and it was mainly due to me trying to use suboptimal regexp calls to find require() and require.def() calls in the files, so the dependencies for a script could be traced.

So I finally took the dive into Abstract Syntax Trees (ASTs) to do the work. What is an AST? An analogy that works for me: an AST is to JavaScript source as the DOM API is to HTML source. The AST has methods for walking through the nodes in the JS code structure, and you can get properties on a node.

Figuring out how to generate an AST from scratch can be a bit of work, but since I was already using Closure Compiler, I just used an AST it can generate.

Since the optimization tool for RequireJS is written in JavaScript, which makes calls into Java-land to do file access and minification calls, I wanted the same approach for working with the AST -- do my work in JavaScript, but call the Java methods for the AST walking and source transform.

My task was fairly simple -- I just wanted to find require() or require.def() calls that used strings for module names and dependencies, pull those calls out of the file, then just execute those calls to work out the dependencies.

The end result was this file:
http://github.com/jrburke/requirejs/blob/master/build/jslib/parse.js

The basic idea of the script:

//Set up shortcut to long Java package name,
//and create a Compiler instance.
var jscomp = Packages.com.google.javascript.jscomp,
  compiler = new jscomp.Compiler(),

  //The parse method returns an AST.
  //astRoot is a kind of Node for the AST.
  //Comments are not present as nodes in the AST.
  astRoot = compiler.parse(jsSourceFile),
  node = astRoot.getChildAtIndex(0);

//Use Node methods to get child nodes, and their types.
if (node.getChildAtIndex(1).getFirstChild().getType() === CALL) {
  //Convert this call node and its children to JS source.
  //This generated source does not have comments and
  //may not be space-formatted exactly the same as the input
  //source
  var codeBuilder = new jscomp.Compiler.CodeBuilder();
  compiler.toSource(codeBuilder, 1, node);

  //Return the JavaScript source.
  //Need to use String() to convert the Java String
  //to a JavaScript String.
  return String(codeBuilder.toString());
}

Thanks to the Closure Compiler team for doing the hard work and open sourcing the code. It looks like Closure Compiler deals with two AST formats -- one is perhaps an older one generated by Rhino, while the other one is a more custom one? It seems like I was getting back the Rhino-based Nodes for the methods I called.

I was tempted to try to go direct to just use Rhino for the AST, but decompiling the AST into source looked harder to do, and from what I recall, Rhino has a newer AST API in the trunk code. I believe the one in Closure Compiler is the older one? All that added up to me being wary of that path.

Most of the time spent was trying to figure out the Java invocations to get the code parsed, understand the tree structure, deal with Java-to-JavaScript translation issues and then figure out the Java invocations to convert a subtree back into source.

I am glad I finally stepped into working with a real AST. While some of the AST calls are a bit awkward (at least for me as a JavaScript person), it is a lot better than trying to use regexps for it. I still need to do more testing, but I feel more confident in the robustness of the solution now.

If you see how I can do it better, point me in the right direction!

RequireJS 0.8.0 Released

2010-02-18T23:28:00.000-08:00

RequireJS, the next generation in script loading, now has an official release and a new web site: http://requirejs.org.

The 0.8.0 release is a formal release of the code, and it includes built versions of jQuery 1.4.2 with RequireJS already integrated.

I also updated my jQuery fork to include the latest changes -- jQuery's page load callbacks will not fire unless all scripts loaded with RequireJS have also finished loading.

I plan to do integrations with other browser toolkits, MooTools and Prototype being next on my list. I also hope the jQuery community will want to pull the changes I have in my jQuery fork into their master at some point.

If you are a team member for one of these toolkits, please let me know what I can do in RequireJS to provide the best code loading and module format for browser-based toolkits. It would be great if we can reach consensus on code loading. I am happy to make changes in RequireJS if it moves us all closer to that.

While the release version is 0.8, this code has been battle-tested in Raindrop, a sizable JavaScript-centric messaging web app. Raindrop uses a version of Dojo 1.4 that has been converted to the RequireJS module format, and all Raindrop modules are written as RequireJS modules.

Some other notes about the release:

RequireJS 0.8.0 can run in Rhino, but its main design target is the browser.
It comes with an optimization tool that can combine and minify your JavaScript.

Give it a spin! There is a moderated list if you have questions or need support.

RunJS is now RequireJS

2010-02-09T19:15:00.000-08:00

As mentioned before, I considered renaming RunJS to RequireJS. I did the transition, and RequireJS is on GitHub. There is a conversion script that will convert CommonJS modules to the Transport/C proposal that works with RequireJS.

I have converted Raindrop to use a modified Dojo 1.4 that uses RequireJS instead of the normal Dojo loader, and all Raindrop modules are written in the Transport/C module format that RequireJS understands. Raindrop works, so the RequireJS code has been proven in a real project that has many modules with nested dependencies. The RequireJS code is already battle-tested.

I have opened a thread on the jQuery forum about using RequireJS for jQuery's require() needs. I can do a build with Dojo 1.4 that uses RequireJS, and any Dojo 2.0 effort is likely to use RequireJS as the module loader.

I believe RequireJS is the loader browser-based toolkits should use. At the very least, the module format and API supported by RequireJS should be used by browser-based toolkits, even if they want to build their own loader.

Next plans for RequireJS:

1) Contact the MooTools and Prototype folks to see if they want to use it. It allows loading code that does not export a module value, and has access to the global environment. They can use it to load code that augments native prototypes. RequireJS can load existing, plain JS files that do not define a module too.

2) Put up a web site with builds. While you can use RequireJS just from grabbing it from GitHub, it would be nice to have the builds of RequireJS with its different levels of functionality already built and easy to download.

2) Do a fork of Node that uses RequireJS on the server. I believe the async module format used by RequireJS is great fit for Node and its async goals.

3) See if I can do a fork of Narwhal to do the same thing.

I believe I can get RequireJS to work on the server and still support the existing CommonJS format when the server supports synchronous loading. By having native support in some server-based systems for RequireJS, it will be easier to share code with the browser.

To recap, my three main issues with using the existing CommonJS module spec, and why RequireJS exists:

1) So far the CommonJS group does not think the browser is common enough to qualify as a first class citizen in the module spec. The group is mainly concerned with environments outside the browser. As a result, the CommonJS module spec does not work well natively in the browser -- it either requires an XHR-based loader, which we have found to have problems in Dojo, or require a server-side transform process. A server-side transform process should not be required to do web development in the browser.

RequireJS uses a function wrapper around the module to avoid these problems and allow loading modules via script tags. Just save the file and hit reload in the browser.

2) There is a free variable, called "exports". It is an object. You cannot set the value of exports inside your module code, you can only add properties. This means for instance, your module cannot be a function. In Dijit, Dojo's widget system, all widgets are constructor functions. The "exports" restriction makes your APIs awkward if you want to export functions for module values. The claim is that this exports restriction helps with circular dependencies, but it only helps a little bit. To me, it is not worth slightly improving an edge case when it sacrifices a greater usefulness and simplicity in user's modules.

RequireJS and its format can handle circular dependencies just fine. In the format supported by RequireJS, you to define a module as a function. Although, you can still use exports as CommonJS uses it if you so desire.

3) The require.main property seems like a hack. It is normally used so that a module can say if (require.main === module.id) or if (require.main === module) then do some "main" work. The module format should just define an exports.main convention for indicating "main" functionality. It is less typing, and more robust, since different code entry points have a different idea of "main". For instance, an HTTP request handler likely has specific requirements on what it considers to be the "main". The top level entry point should decide what code to execute as "main", not logic inside the module.

RequireJS does not support the require.main idiom.

So I believe the path used by RequireJS is more robust overall, works better/scales better in the browser. However, I still want to provide enough support for the existing CommonJS modules in the meantime to allow more code sharing.

In the long run though, the CommonJS format as it exists today should be replaced with something better. It is troublesome that the CommonJS group is not really targeting the browser, but over time, the broader JS community will expect browser toolkits to support CommonJS specs. It does not seem right to end up with non-optimal solutions in the browser when the browser is the most common JS platform.

RunJS to RequireJS?

2010-01-27T22:06:00.000-08:00

There was a thread that started on the CommonJS list about a transport format, something that works well in the browser via script injection. I sketched out a proposal, Transport/C, that builds on the Transport/B and Transport/A specs.

Transport/C is very similar to some basic mechanics of RunJS but uses require() as the top level function, and supports the special "module" and "exports" free variables used in the normal CommonJS module spec.

In order to prove the concept for Transport/C, I made a branch of the RunJS code, calling it RequireJS, that implements Transport/C.

It seems like it fits with the existing CommonJS module spec, but is something that works well in the browser. I also made a simple conversion script that converts traditional CommonJS modules to this format.

I am tempted to convert from RunJS to this RequireJS branch, and to start evangelizing that approach for browser toolkits. It would be great if Transport/C would also be approved as the transport format for CommonJS too.

Kris Kowal has some concerns about the *very* long-term effects of the approach. I read his comments as possibly pointing out some things that would be done differently if the primordials and e-maker type of modules were ever accepted as part of an ECMAScript standard.

As I read the primordials and e-maker strawman proposals, I think the only difference is Transport/C functions are only expected to be called once, but e-maker style would favor calling the function on every require() call. As I say in my response, I believe e-maker support, would affect regular CommonJS modules in the same way as the transport format, and it is assuming the strawmans make it in to the spec at some point, as they are specified now.

I also believe how it works in how I coded Transport/C as part of the RequireJS branch is what a normal developer would expect, and I think fits better with existing browser/script behavior, and the assumptions that go along with coding CommonJS modules today.

So I am tempted to rename the RunJS project to RequireJS and proceed with that. If you have any feedback to the contrary, please let me know. Otherwise, I will likely do the change early this week.

Script async, Raindrop and Firefox 3.6

2010-01-21T11:25:00.000-08:00

In honor of the Firefox 3.6 release, I upgraded Raindrop to use the new async attribute for script tags.

Why is async neat? It does not block the rest of the page, and will just evaluate the script once it is retrieved. More information is in the HTML5 spec. Note that the script you add async to should NOT use document.write(), as doc.write will likely destroy your page.

Also, be aware that async is a boolean attribute, but that does not mean you should use async="true" to turn it on. The HTML5 spec on boolean attributes says that a value of empty string or a string that matches the attribute name should only be used. To avoid async, just do not include the attribute. For Raindrop, I used async="async" since that looks better to me than an empty string.

Raindrop uses RunJS for the module loader, and RunJS uses dynamically added script tags via head.appendChild(), so the modules loaded by RunJS already behave in an async manner.

However apps that want to use the raindrop front end libraries normally include a script called rdconfig.js as their own script file, and that config file does a document.write to write out the Dojo+RunJS and jQuery tags. Those Dojo+RunJS and jQuery tags now use the async attribute.

RunJS Dependency API

2010-01-09T13:54:00.000-08:00

In my last post, I talked about a suggestion from David Ascher to try to improve the syntax of specifying dependencies when declaring a module. Here is an example of that suggestion, using run.def(), a possibly new API dedicated to just defining a module called "rdw/Message":


run.def("rdw/Message", {
rd: "rd",
dojo: "dojo",
Base: "rdw/_Base",
friendly: "rd/friendly",
hyperlink: "rd/hyperlink",
api: "rd/api",
template: "text!rdw/templates/Message!html"
}, function(R) {
//Module definition function.
//Use things like R.api and R.Base in here that map to the modules up above.
...
});

While that does make it clear what each module name's variable will be inside the function that defines the module, it has the following drawbacks:

1) It hurts minification -- having properties off the R object passed to the module function means that minification tools will not be able to minify those references as easy.

2) All the modules dependencies must be referenced via a prefix "R.". For example R.api. It is a small bit of typing and an extra property lookup. That might be seen as an advantage too -- it is clear that a symbol is a dependency because it is off the R. function.

3) Makes it hard to find typos for properties on the R. function. With the current runjs format, JSLint can actually find typos for a dependency's variable name.

4) This syntax is more verbose for JS files that do not care about scope encapsulation. For JavaScript libraries like jQuery, MooTools and Prototype, they pretty much operate in the same global scope. jQuery does have a noConflict(), but that just helps if there is only one other thing called $ in the page. MooTools and Prototype add things to global prototypes.

So for these libraries, specifying dependencies is more like just specifying script tags, and they do not need a local-scoped variable in the function module to get things defined. They would just need to do the following:


run.def("rdw/Message",
["some/module", "something/else", ...]
function() {
//This function does not need some locally scoped variables, and most
//likely, the script dependencies above may not call run.def() and define
//an object, just add things to the global space
});

So for these libraries, it would be onerous to be forced to create variable names for each of those dependencies.

This last point seems to be enough to tip the scales back to using the current model used by run. However I am aware that it is possible for the developer to not get the order or number correct, matching the dependency string with the correct variable for the function.

I am hoping using a coding standard like the following will help:

run.def("rdw/Message",
["rd", "dojo", "rdw/Base"], function(
 rd,   dojo,   Base) {
 //Define the module for rdw/Message and return it.
});

Basically, make sure all dependencies are on one line, along with the function keyword, then put the variable that matches each dependency aligned directly under the depdendency name (the example above may not be correctly aligned depending on the font or format you are viewing this message).

I purposely trimmed the list of dependencies, so I can get this example to show up in this blog. That is the down-side with this sort of code style: it can have a very long line for the dependency names.

For Raindrop, I want to try to keep local scope encapsulation, particularly since I expect some slicker extensions to it that may introduce other code into the page that may conflict. However, if I did not care to do that, I could shorten up the above example quite a bit.

So at this point, I am favoring making it easy to use RunJS for other toolkits that do not care about local scope encapsulation and detecting bad references to variables inside the module definition over avoiding errors with a mismatched function variable name to a dependency name.

It is a hard choice to make. Neither path is perfect. If you have an opinion, feel free to share it.

Raindrop + JSLint + JSDoc + RunJS

2010-01-08T20:49:00.000-08:00

I just pushed some changes into the front-end code for Raindrop. Besides using JSLint as the code style, and JSDoc for the format of the files, it now uses a modified version of Dojo 1.4 that uses the RunJS code loader.

I am fortunate enough to work with people that will tolerate this sort of brief experimentation to help improve the state of browser code loading. Do you know Mozilla Messaging is hiring? Great people, great technology, and a great mission. OK, end of the sales speech.

The code conversion to JSLint, JSDoc and a new loader all at one time took longer than I would have liked, but I am glad I did it.

For one thing, the code looks a lot more uniform thanks to using JSLint. While I do not care for all of the rules mandated by JSLint, no one ever likes a coding standard 100%, and JSLint is widely known and a programmatic code checker.

Converting Raindrop to RunJS was really beneficial for RunJS. While RunJS has unit tests, nothing helps shake out the kinks like a large project. RunJS is now more robust because of it.

If you want to peruse the Raindrop code, you can hop to the Raindrop Mercurial source web view. That link will take you to Raindrop's client/lib/rdw directory, where we keep the UI widgets. You can click on some files to see how they look with the RunJS loader and the new formatting.

One thing that became apparent, particularly when talking to David Ascher about the code: the way the dependencies in RunJS are specified as an array, and having a separate list of function arguments that must match that array might be prone to errors.

Here is an example from rdw/Message:


run("rdw/Message",
["run", "rd", "dojo", "rdw/_Base", "rd/friendly", "rd/hyperlink", "rd/api",
"text!rdw/templates/Message!html"],
function (run, rd, dojo, Base, friendly, hyperlink, api, template) {
 ...
});

I condensed the run boilerplate to the top to three lines, otherwise, it would have been quite long vertically to express it all. But you can see the problem. Looks scary.

David and I talked about it. I talked about how YUI uses just a Y object as the only argument to the function, but I did not like how they use odd module names like "yui-anim", and then you have to know that creates a Y.anim property that you can use in your function.

So David suggested the following:

run("rdw/Message", {
 rd: "rd",
 dojo: "dojo",
 Base: "rdw/_Base",
 friendly: "rd/friendly",
 hyperlink: "rd/hyperlink",
 api: "rd/api",
 template: "text!rdw/templates/Message!html"
}, function(R) {
 //Use things like R.api and R.Base in here that map to the modules up above.
 ...
});

I like this better. It is much clearer what module gets assigned to what variable, no "off by one" errors.

The downside: it does not minify as nicely as the existing run format. However, in this case, avoiding developer errors probably trumps the minification cost, and the extra cost to type R. in front of the dependent module references. You can call it r. if you want. Hmm, I might prefer the lower-case version. Easier to type. Although the capital stands out better.

So I am very tempted to convert RunJS to use this format for listing dependencies and passing them to the function callback. I will likely do the work this weekend.

Update: I decided not to do this, favoring the original format since it has other advantages.

Another thing we talked about was the multiple invocations of run(): run() can be called with a starting string argument to define a module, but then can also be called without that starting string argument if you just want to run some code that has some dependencies.

David felt it would be clearer if those two actions were two different function names. I can see that being clearer. So what about the following for the API:

run(), for just running code that does not define a module.
run.def() for defining a module.
run.mod() for a module modifier (the existing run.modify() API call)
run.get() for getting a defined module after initial module evaluation (needed for some some dynamic cases where you do not know the module name before-hand and in circular dependency cases). This is an existing API.

So the changes are renaming run.modify() to run.mod() and moving the run() call style that defined a module to run.def().

Thanks to David for the feedback, and for the rest of the Raindrop team for being patient with me as I did some experimentation in browser module loading.

Speaking of feedback on RunJS, Rawld Gill (one of the authors of Mastering Dojo) was already working on a loader, and he did some work to convert it to use the same API as RunJS. I am still processing the results, but it is great to see alternate implementations. I am hoping we can take the best of both and make a great loader. I already changed the module evaluation algorithm to use the recursive style he suggested.

RunJS takes its roots from the cross-domain Dojo loader, and that loader was constructed when Dojo had to support Safari 2. Safari 2 has a ridiculously small call stack, so a recursive module evaluator blew up very easily. I switched to a model that traced the dependencies to work out an array of sequenced modules, then a separate loop to then call them into being. That allowed the loader to work in Safari 2, but now that we do not need to support that browser, the more natural, straight-forward recursive model can be used.

Really neat stuff! Feels like we are getting close to a robust but compact loader.

RunJS: GitHub, build options, features, file sizes

2009-12-25T22:35:00.000-08:00

A few updates on RunJS, a JavaScript file/module loader (see the README for more documentation):

RunJS is now on GitHub
Plugins for RunJS are supported. i18n bundles have been pulled out as a plugin, and a new text plugin allows you to set text files (think HTML/XML/SVG files) as dependencies for a module. The plugin will use async XMLHttpRequest (XHR) to fetch those files and will pass the text of those files as an argument to a module's module definition function. The RunJS build system will then *inline* those text files with the module, so that the XHR calls be removed in deployed code, and allow cross-domain use of those text files.
Any function return type is allowed from the module definition function. Before only objects and functions were allowed and functions had to be called out in a special way. Now, that special call out is removed and any return type is allowed. The cost was an extra call, run.get() that needs to be used in circular dependency cases. See the Circular Dependencies section in the README.
The build system that comes with RunJS now supports build pragmas.

The build pragma support was used to build RunJS in a couple of different configurations. I am trying to get a handle on where the bulk of implementation lies, and what features add to its file size. Here is the breakdown (warning, Google Doc iframe inclusion, but interesting numbers inlined in this post after the iframe):

Let's look at the non-license sizes, since they give a better indication of code density. Google's Closure Compiler did the minification for this evaluation.

The normal config, with no plugins included (but with plugin support) is 7,970 bytes minified, 3,167 gzipped. Including both the i18n and text plugins with run.js bumps it up to 11,759 minified, 4,655 gzipped.

The interesting number for me is the version of run.js without plugin support, no run.modify, no multiversion support and no page load (run.ready/DOMContentLoaded callbacks). This version of run has just the following features:

support for the run() module format
nested dependency resolution
configure paths to modules
load just plain .js files that do not define run.js modules (scripts that do not call run(), for example jQuery, or plugins for jQuery).

That bare bones loader comes in at 5,086 minified and 2,204 gzipped. The one you should use, the one with the license, is 5,245 minified and 2,317 bytes gzipped. I need to work on the size of that license block!

That size could probably be brought down a tiny bit (probably reaching the 2,000 gzip size) if I were to really be aggressive and remove all context references, but that would be a mess to maintain and there would be no easy upgrade path to multiversion support.

I believe that is the lower limit a functional loader that does nested dependencies via run() module calls. I view run.ready/DOMContentLoaded support more of a necessity for a loader, so unless you already had an implementation for that, I suggest the version that has run.ready() support, which comes in (with license) at 5,867 minifed, 2,522 gzipped.

The nice thing about the build pragma setup for RunJS, you can upgrade run without having to change your code if you find you want more features, like plugin support, or i18n/text dependency support via plugins.

I am interested in trying to sell more front-end JavaScript toolkits on this loader. For some, I can see the bare-bones 2.3K gzipped loader a nice way to step into it, and then their users have the option to swap out a more powerful version via a different RunJS build output.

I have put up the different build outputs for 0.0.6 if you want to grab one of the minified versions and play with it. Here is the minimum set of compliance tests which use the smallest loader (no modify/plugins/page load/context support) mentioned above. See the README for documentation.

Right now I believe around 2KB gzipped is close to the lower bound for a stand-alone code loader in the browser. At least for a loader I would consider using: anything that uses XHR and eval are dead to me. Using plain script src="" tags helps the xdomain case, and just fits better with debugging. While Dojo has used an XHR-based loader for quite a while (and it will continue to be supported), it just does not work as well with the browser as a script-tag based loader. Any loader should also do nested dependency loading too -- if a module in a script has dependencies in other modules, be sure to evaluate the dependencies in the right order.

As a point of comparison, consider LABjs. I feel a kinship with the author of LABjs, Kyle Simpson, even though we have never talked. We are both focusing on efficient code loading in the browser. I recommend LABjs if it fits your style.

While LABjs does not quite do nested dependency resolution, it does something related where you can tell it to wait to load a script before continuing to load other scripts. LABjs is not trying to push a module format like run is, but targeted more at existing code that does not have the concept of a module format.

By the way, RunJS can also handle loading these types of files. Where LABjs has a wait() call for holding off loading scripts that depend on another script being loaded (like a framework), RunJS uses nested run calls.

Example from the LABjs page:


$LAB
.script("framework.js").wait()
.script("plugin.framework.js")
.script("myplugin.framework.js")
.wait(function(){
    myplugin.init();
    framework.init();
    framework.doSomething();
});

Equivalent example with RunJS:


run(["run", "framework.js"],
    function(run) {
        run("plugin.framework.js", "myplugin.framework.js"],
            function() {
                myplugin.init();
                framework.init();
                framework.doSomething();
            }
        );
    }
);

Taking the 1.0.2rc1 version of LABjs and using Closure Compiler on it (without the license) gives LABjs a size of 4,360 bytes minified and 2,170 gzipped. As a reminder, the equivalent RunJS file is 5,086 minified and 2,204 gzipped. I may be able to do better with making the structure of the RunJS code more amenable to minification, but the gzip sizes come up fairly close. I do not believe the code tricks I would do to help minification will help the gzip size any.

Both LABjs and RunJS end up around 2KB gzipped. So, about 2KB gzipped seems close to the lower limit on a standalone loader, one that uses script tags/plays nice with the browser and can do nested dependencies. I would like to be proven wrong though, and ideally by modifying RunJS to fit that lower limit. :) I am sure the code can be improved.

But remember the guidelines, no goofy XHR stuff/something that works well with the browser and can handle nested dependencies. No script tags with inlined source/eval tricks. Even though Firefox and WebKit make eval debugging easier, it is still not as nice as regular script src tags.

Irakli Gozalishvili believes web workers might help, but I do not see it. The workers are restricted to message passing, and anything interesting in a web browser will likely need to touch the DOM, so a web worker solution will just be another async-XHR-like approach, where you will need to eval the scripts or inline-script inject to get all the scripts for them to see each other and the DOM.

Irakli does have an async-XHR based loader for CommonJS modules. As of today, it comes in at 1,527 minified, 838 gzipped (license not included). But it uses XHR, so limited to the same domain as the page, and debugging support is just not as nice across browsers. It also uses CommonJS module syntax, but I have decided CommonJS modules do not play well out of the box in the browser, and I believe the format's "module", "exports", and "require.main" parts are unnecessary.

Dojo 1.4 Favorite Features

2009-12-10T10:00:00.000-08:00

Dojo 1.4 is out! There is a metric ton of changes. Here are some of my favorite things about the release. I focus mostly on Dojo Core, and mostly in the non-animation parts of it, so my list is skewed for that focus. However, there are lots of other changes, some in the animation functionality, and in Dijit and Dojox. Check out the 1.4 release notes to get a more complete picture.

One of the things I want to do for Dojo Core is to bring the DOM APIs, particularly the methods on dojo.NodeList (the return object for dojo.query() calls, Dojo's CSS selector method) more in-line with what is available in jQuery. jQuery has demonstrated that its APIs resonate strongly with developers. Where it makes sense and fits Dojo's philosophy, we should also provide those APIs, to make it easier for developers. These Dojo 1.4 changes reflect that goal:

dojo.ready(), just an alias for dojo.addOnLoad().
dojo.NodeList-traverse: A helper module that adds methods to dojo.NodeList. Its goal is to bring in some methods to NodeList that exist in jQuery for DOM traversal, specifically: children, closest, parent, parents, siblings, next, nextAll, prev, prevAll, andSelf, first, last, even, odd.
dojo.NodeList-manipulate: A helper module that adds methods to dojo.NodeList. Its goal is to bring in some methods to NodeList that exist in jQuery for DOM manipulation, specifically: innerHTML, html, text, val, append, appendTo, prepend, prependTo, after, insertAfter, before, insertBefore, remove, wrap, wrapAll, wrapInner, replaceWith, replaceAll, clone.
IO pipeline topics: get notifications of IO events via dojo.subscribe/dojo.publish. Handy for putting up a generic "loading" indicator when any sort of IO call happens. These topics are not strictly how jQuery exposes this functionality, but we can leverage the power of dojo.publish/subscribe to implement this feature.

Some other new Dojo Core 1.4 features that are really sweet:

dojo.cache(): allows you to reference external HTML files and use them as if they are strings. It is integrated into the build system, so you can avoid the XHR calls to get the external text files by just doing a build. No extra build option is needed. This is a great way to construct HTML -- by writing plain HTML instead of building awkward strings in code or using JS DOM-building calls, which can obscure what the HTML actually looks like.
dojo.position(): A faster, more understandable replacement for dojo.coords(). If you were using dojo.coords() before, odds are good that you probably want to switch to dojo.position(). Douglas Hays stepped up and put in this great new method.
dojo.declare(): It is faster and more robust. Many thanks to Eugene Lazutkin for doing this work. It took a lot of patience and perseverance to get this new version up to snuff and keep it backward compatible.
dojo.hash(): An easy way to set the URL hash (fragment ID) and to watch changes to the hash. This allows you to create pages that reflect the proper state as shown by the URL in the browser. This was a contribution from community member Rob Retchless and other IBM Jazz team members.

For the build system, support was added for Google's Closure Compiler, so you can experiment with using it for minifying your code. Right now we just support the "simple" minification done by Closure Compiler, not the advanced features.

It was a little while coming, but it is great to have Dojo 1.4 out. Thanks to the community for making the toolkit better!

JavaScript module loading, the browser and CommonJS

2009-11-25T09:54:00.000-08:00

JavaScript module syntax and loading seems like a hot topic at the moment, and here are some thoughts about how to construct module syntax and a loader, with the goal of trying to get to a more universal approach for it. This will be discussed in the context of CommonJS, but browser-based module loaders have existed for a while. All are constrained by the browser in some fashion as listed below. I have a preferred solution, also described in this post.

First, a look at module syntax.

CommonJS is an umbrella for a few different things, including a spec for a module syntax and a standard library of modules. This post is just interested in its module syntax. A simple example of CommonJS syntax, defining an "increment" module defined in increment.js:


var add = require('math').add;
exports.increment = function(val) {
       return add(val, 1);
};

How could we build a module loader with this syntax? Here are a couple of options:

1) You parse the module before executing it, looking for require calls. You make sure to fetch those modules and work out the right dependency order in which to execute the modules.

2) You just run the module and when require() is hit, do an synchronous IO operation to load the required module.

For both approaches, use a sandbox or specific context to make sure things like "exports" are defined separately for each module.

Both of those options are easy to implement on the server side since you have more control over the IO layer, and can create separate contexts for each module. However, in the browser, things are different. Creating a separate context for each module is tricky. For IO, there are two realistic approaches, and each has its difficulties:

XMLHttpRequest (XHR)
script tags

XHR allows us to do either approach, #1 or #2. It can get the contents of the module and parse it into a structure that pulls out the dependencies and we can sort out the right order to execute things. We could use sync XHR calls to accomplish #2, block when each require call is seen. However, sync XHR calls in the browser really hurt performance.

This is actually what the default Dojo loader has done for a very long time, and I believe some pathways in the Google's Closure library do the same thing. It is always recommended you do a custom build to combine all the modules you need into one file to cut out those XHR calls when you want to go to production.

So path #1 would make more sense with an XHR-based loader. However, for an XHR loader to work, it has to use eval() to bring the module into being. Some environments, like Adobe AIR do not allow eval(), and it makes debugging hard to do across browsers. Firefox and WebKit have a convention to allow easier eval-based debugging, but it is still not what I consider to be in keeping with traditional script loading in a browser.

Instead of eval, after the XHR call finishes its parsing and module wrapping for context, you could try to create a script tag that has a body set to the modified module source, but this really hurts debugging: if there is an error, the error line number will be some weird line in a gigantic HTML file instead of the line number of the actual module.

Dojo has a djConfig.debugAtAllCosts option that will use sync XHR to pull down all the modules, parse the for dependencies, work out the right load order, then load each module via a dynamically added script src="" tag. However, since IE and WebKit will evaluate dynamically added script tags out of DOM order -- they evaluate them in network receive order (which is nice for long-polling comet apps, but does not help module loading). So, each script tag has to be added one at a time, then wait for it to finish then add the next one. Not so speedy.

XHR is also normally limited to just accessing the same host as the web page. This makes it hard to use CDNs to load content, and get performance benefits with that approach. There is now support for xdomain XHR in most recent browsers, but IE prefers to use a non-standard XDomainRequest object, making our module loader more complicated. And xdomain XHR just plain does not work in older browsers like IE6.

So, an XHR-based loader is not so great.

Script tags are nice because they keep with the known script pathway in browsers -- easy to debug, and we can get parallel loading. However, we cannot do approach #2 in the browser: our JavaScript in the browser cannot access the module contents before they are evaluated. And since dynamically added script src="" tags via head.appendChild() are not a synchronous operation, approach #2 will not work.

So, really we need to do a variant of #1, pull out the dependencies needed by the module, then after those dependencies are loaded, execute the module. The way to do this in script: put a function wrapper around the module contents, and call a module loader function with a list of dependencies and the module function wrapper. Something like this, for a module with the name of "c" that has dependencies of "a" and "b", Here is a syntax (call it Variant A) for defining a module "c" with this approach:


loader(
       "c",
       ["a", "b"],
       function(a, b) {
              //The module definition of "c" in here.
              //return an object to define what "c" is.
              return {};
       }
);

or, another variant, call it Variant B:


loader({
       name: "c",
       dependencies: ["a", "b"],
       module: function(a, b) {
              //The module definition of "c" in here.
              //return an object to define what "c" is.
              return {};
       }
});

Ideally, we would not have to tell the loader that this structure defines module "c" (the first arg in Variant A and the name: property in Variant B) -- the loader could work this out. Unfortunately, since script tags can load asynchronously and at least IE can trigger script.onload events out of order when compared to when the script is actually evaluated, we need to keep the module name as part of the module definition. This also helps with custom builds, where you can combine a few of these module definition calls into one script.

This approach is actually what Dojo's xdomain loader has done for a very long time, but with more verbose syntax. However, it requires a custom build to convert modules into this structure. The other option is to use a server-side process to convert the modules on the fly, but I do not feel that is keeping with the simplicity of normal browser development: just open a text editor, write some script, save, reload, no extra server config/process needed, besides maybe a vanilla web server.

So, I believe that modules should be coded by the developer in this module wrapper format. YUI 3 has taken this approach, and it is the approach I have taken for RunJS too. However, YUI 3 is limited to needing some module dependency metadata files to help it out. It also uses module names that do not map to the actual module's defined name/functions.

OK, back to CommonJS.

As it stands now, I believe the CommonJS format is not suitable for modules in the browser. There have been attempts to get it to work, but the attempts either use a sync XHR loader, or a "transform-on-the-fly" server process to convert the code to a module wrapper similar to Variant B.

I would rather see a module wrapper format that works with browser natively, that can be hand-authored by developers and that will work with CommonJS modules. CommonJS started out as ServerJS. As ServerJS, the case could be made that supporting browsers may not be an aim of a ServerJS module format. However, with the name change to CommonJS, I believe supporting browsers as a first class citizen is important for CommonJS to get more traction.

So the trick is to come up with a module syntax that has a function wrapper, but is not too wordy with boilerplate. We need some boilerplate, since we need a function wrapper. I believe RunJS has the right right approach. The boilerplate is very terse, basically Variant A mentioned above:


run(
    "c",
       ["a", "b"],
       function(a, b) {
              //The module definition of "c" in here.
              //return an object to define what "c" is.
              return {};
       }
);

I can see where there is some bikeshedding on the name "run". I think script() instead of run() is a viable alternative, and I may switch to that in the near future (and rename RunJS to ScriptJS).

I have attempted to engage the CommonJS community by putting up a proposal for an Alternate Module Format.

Progress has been slow, but to be expected: the CommonJS group is trying to do lots of other things like define a standard library and build out implementations. However, I am hopeful we can get something that works for the browser front end developers.

The ideal scenario is that some variant of the above syntax is just adopted as the only CommonJS module format. That would save a lot of conversion work, and I believe it makes things much simpler for CommonJS compliant loader. Right now, for CommonJS loaders there is a concept of a require() and require.async() and having to expose Promises for the async stuff. The above format neatly avoids the issue of whether the modules are loaded async or sync and avoids any need for Promises in the module loader. I think it is fine though for modules themselves to use Promises as part of individual module APIs, but at least the loader and module syntax stays simple.

I also do not believe a "module" variable needs to be defined for each module and an exports variable is avoided by returning an object from the module function wrapper.

I can appreciate that the CommonJS folks with modules already written may not like moving to the above syntax. I think it helps in the long run if we can just have one syntax, but in the meantime, I plan on doing the following:

Continue to engage the CommonJS community.
build out RunJS, probably rename to ScriptJS in the near future, and use script() instead of run()
Write a converter that converts Dojo modules to the RunJS/ScriptJS module syntax. I have something basic working, here is an example of Dojo's themeTester.html using RunJS-formatted dojo/dijit/dojox modules. That example is not bulletproof yet (I used a built version of Dojo which removes some dependency info) and i18n modules have not been converted either. RunJS also has built-in support for i18n modules.
Convert Raindrop to use RunJS-formatted dojo and convert the Raindrop modules to that format.
Override run.load()/script.load() in server environments so it could be used in CommonJS server implementations.
Work on a converter for existing CommonJS modules.
Use RunJS/ScriptJS as the module syntax for Blade and/or Dojo 2.0 efforts.

If module syntax/loading is important to you, then please join the discussion list for CommonJS, so we can sort this out. It would be great to get consensus on JavaScript module syntax and loading, and I think CommonJS is the area to do that.

I am happy to adjust some of the syntax in RunJS/ScriptJS to match some consensus, but I strongly prefer a terse format. The existing ones I have seen for server-converted CommonJS modules is too verbose for me, particularly for the common cases of defining a module with some dependencies.

Raindrop, CouchDB and data models

2009-11-20T12:04:00.000-08:00

Raindrop uses CouchDB for data storage. We are starting to hit some tough issues with how data is stored and queried. This is my attempt to explain them. I am probably not the best to talk about these things. Mark Hammond, Raindrop's back-end lead is a better candidate for it. I am hoping by trying to write it out myself, I can get a better understanding of the issues and trade-offs. Also note that this is my opinion/view, may not be the view of my employer and work colleagues, etc...

First, what are our requirements for the data?

Extensible Data: we want people to write extensions that extend the data.
Rollback: we want it easy for people to try extensions, but this means some may not work out. We need to roll back data created by an extension by easily removing the data they create.
Efficient Querying: We need to be able to efficiently query this data for UI purposes. This includes possibly filtering the data that comes back.
Copies: Having copies of the data helps with two things:
- Replication: beneficial when we think about a user having a Raindrop CouchDB on the client as well as the server.
- Backup: for recovering data if something bad happens.

How Raindrop tries to meet these goals today

Extensible Data: each back-end data extension writes a new "schema" for the type of data it wants to emit. A schema for our purposes is just a type of JSON object. It has a "rd_schema_id" on it that tells us the "type" of the schema. For instance a schema object with rd_schema_id == "rd.msg.body" means that we expect it to have properties like "from", "to" and "body" on it. Details on how schemas relate to extensions:

An extension specifies what input schema it wants to consume, and the extension is free to emit no schemas (if the input schema does not match some criteria), or one or more schemas.
Each schema written by an extension is stamped with a property rd_schema_provider = "extension name".
All the messages schemas are tied together via an rd_key value, a unique, per-message value. Schemas that have the same rd_key value all relate to the same message.

More info is on the Document Model page.

Rollback: Right now each schema is stored as a couch document. To roll back an extension, we just select all documents with rd_schema_provider = "extension name" that we want to remove, and remove them. As part of that action, we can re-run extensions that depended on that data to have them recalculate their values, or to just remove the schemas generated by those extensions.

Having each schema as a separate document also helps with the way CouchDB stores data -- if you make a change to a document and save it back, then it appends the new document to the end of the storage. The previous version is still in storage, but can be removed via a compaction call.

If we store all the schemas for a message in one CouchDB document, then it results in more frequent writes of larger documents to storage, making compaction much more necessary.

Efficient Querying: Querying in CouchDB means writing Views. However, a view is like a query that is run as data is written, not when the UI may actually want to retrieve the information. The views can then be very efficient and fast when actually called.

However, the down side is that you must know the query (or a pretty good idea of it) ahead of time. This is hard since we want extensible data. There may be some interesting things that need to be queried later, but adding a view after there are thousands of documents is painful: you need to wait for couch to run all the documents through the view when you create the view.

Our solution to this, started by Andrew Sutherland and refined by Mark, was to create what we call "the megaview". It essentially tries to emit every piece of interesting data in a document as a row in the view. Then, using the filtering capabilities of CouchDB when calling the view (which are cheap), we can select the documents we want to get.

Copies: While we have not actively tested it, we planned on using CouchDB's built-in replication support. This was seen as particularly valuable for master-master use cases: when I have a Raindrop CouchDB on my laptop and one in the cloud.

Problems Today

It feels like the old saying, "Features, Quality or Time, pick two", except for us it is "Extensible, Rollback, Querying or Copies, pick three". What we have now is an extensible system with rollback and copies, but the querying is really cumbersome.

One of the problems with the megaview: no way to do joins. For instance, "give me all twitter messages that have not been seen by the user". Right now, knowledge of a message being from twitter is in a different schema document than the schema document that knows if it has been seen by the user. And the structure of the megaview means we can really only select one property at a time on a schema.

So it means doing multiple megaview calls and then doing the join in application code. We recently created a server-side API layer in python to do this. So the browser only makes one call to the server API and that API layer does multiple network calls to CouchDB to get the data, then does the join merging in memory.

Possible solutions

Save all schemas for a message in one document and more CouchDB views
Saving all schemas for a message in one document makes it possible to then at least consult one document for both the "type=twtter, seen=false" sort of data, but we still cannot query that with the megaview. It most likely means using more CouchDB views to get at the data. But views are expensive to generate after data has been written. So this approach does not seem to scale for our extensible platform.

This approach means taking a bit more care on rollbacks, but it is possible. It also increases the size of data stored on disk via Couch's append-only model, and will require compaction. With our existing system, we could consider just never compacting.

This is actually the approach we are starting to take. Mark is looking at creating "summary documents" of the data, but the summary documents are based on the API entry points, and the kind of data the API wants to consume. These API entry points are very application-specific, so the summary document generation will likely operated like just another back end extension. Mark has mentioned possibly just going to one document to store all schemas for a message too.

However, what we have not sorted out how to do is an easier join model: "type=twitter and seen=false". What we really want is "type=twitter and seen=false, ordered by time with most recent first". Perhaps we can get away with a small set of CouchDB views that are very specific and that we can identify up-front. Searching on message type and being seen or unseen, ordered by time seems like a fairly generic need for a messaging system.

However, it means that the system as a whole is less extensible. Other applications on the Raindrop platform need to either use our server API model of using the megaview then doing joins in their app API code (may not be so easy to learn/perform), or tell the user to take the hit waiting for their custom views to get up to date with all the old messages.

Something that could help: Make CouchDB views less painful to create after the fact. Right now, creating a new view, then changing any document means waiting for that view to index all the documents in the couch, and it seems to take a lot of resources for this to happen. I think we would be fine with something that started with most recent documents first and worked backwards in time, using a bit more resources at first, but then tailing off and doing it in the background more, and allow the view to return data for things it has already seen.

Do not use CouchDB
It would be very hard for us to move away from CouchDB, and we would likely try to work with the CouchDB folks to make our system work best with couch and vice versa. It is helpful though to look at alternatives, and make sure we are not using a hammer for a screwdriver.

Schema-less storage is a requirement for our extensible platform. Something that handles ad-hoc queries better might be nice, since we basically are running ad-hoc queries with our API layer now, in that they have to do all the join work each time, for each request.

Dan Goldstein in the Raindrop chat mentioned MongoDB. Here is a comparison of MongoDB and CouchDB. Some things that might be useful:

Uses update-in-place, so the file system impact/need for compaction is less if we store our schemas in one document are likely to work better.
Queries are done at runtime. Some indexes are still helpful to set up ahead of time though.
Has a binary format for passing data around. One of the issues we have seen is the JSON encode/decode times as data passes around through couch and to our API layer. This may be improving though.
Uses language-specific drivers. While the simplicity of REST with CouchDB sounds nice, due to our data model, the megaview and now needing a server API layer means that querying the raw couch with REST calls is actually not that useful. The harder issue is trying to figure out the right queries to do and how to do the "joins" effectively in our API app code.

What we give up:
1) easy master-master replication. However, for me personally, this is not so important. In my mind, the primary use case for Raindrop is in the cloud, given that we want to support things like mobile devices and simplified systems like Chrome OS. In those cases it is not realistic to run a local couch server. So while we need backups, we probably are fine with master-slave. To support the sometimes-offline case, I think it is more likely that using HTML5 local storage is the path there. But again, that is just my opinion.

2) ad-hoc query cost may still be too high. It is nice to be able to pass back a JavaScript function to do the query work. However, it is not clear how expensive that really is. On the other hand, at least it is a formalized query language -- right now we are on the path to inventing our own with the server API with a "query language" made up of other API calls.

Persevere might be a possibility. Here is an older comparison with CouchDB. However, I have not looked in depth at it. I may ask Kris Zyp more about it and how it relates to the issues above. I have admired it from afar for a while. While it would be nice to get other features like built-in comet support, I am not sure it will address our fundamental issues any differently than say, MongoDB. It seems like an update-in-place model is used with queries run at runtime. But definitely worth more of a look.

Something else?

What did I miss? Bad formulation of the problem? Missing design solution with the tools we have now?

Blade, a JavaScript toolkit experiment

2009-10-28T22:46:00.000-07:00

I am playing around with a different way (at least for me) to construct a JavaScript toolkit. It is called Blade, and you can follow it via the Blade GitHub repo.

I have had this on my local drive for a few weeks now, and the germ of it started with this post. I wanted to get it more polished, but best to get it up somewhere to get some feedback at least on the principles.

There is not much there now, basically a tiny amount of spaghetti code that is not really usable. However, I list out the guiding principles in the README.md, visible on the GitHub source tab.

Raindrop, Open Messaging for the Open Web

2009-10-22T12:04:00.000-07:00

I work for Mozilla Messaging, and we just opened the doors on Raindrop. Raindrop is the reason I had the opportunity to move to Vancouver, BC. It has been fun building it, seeing if we could get something to work.

Raindrop is still very much an experiment and not useful for any day-to-day work. However, it has potential and we need community help to take if further.

Why I like it:

It is driven by product design. We want an extensible platform, but a strong, simple product design will be driving much of the development.
It is not trying to be a message service in itself, but collect messages from existing services.
It is web-based: the default UI is plain HTML/JavaScript/CSS goodness.
It is frickin awesome to be able to play with your messages: data mine them, and do interesting display things using simple script languages like JavaScript and Python.
It is open: open source and motivated by the Mozilla Manifesto.
The Mozilla Messaging team is talented and smart. They are motivated and care about what is best for users.

I am driving the front end development for Raindrop. I used Dojo to create the infrastructure for the pages, using Dijit's dijit._Widget and dijit._Templated as a base for many of the UI widgets.

Dojo's dynamic code loader and Dijit's well-defined methods on widgets have enabled the slick things we are doing with in-place extension editing and updates. JQuery is also included in the page, mostly for extension developers, so you have a choice on what to use, Dojo or JQuery.

While I expect many of the decisions I made about how the front end works might change over time, it has been a joy to make what is there so far.

There is lots to do though. If you want to get involved with the code, check the Hacking page is a good place to get started. There is a screencast of the architecture on the Raindrop home page. There is a Community page too.

RunJS updated: module modifiers and function modules

2009-10-18T22:40:00.000-07:00

I pushed some changes to allow lazy-loaded/lazy-evaluated module modifiers and also better support for modules that just define a function. The changes are documented in the documentation page.

I am experimenting with using JSLint as a code formatter. We'll see how it goes.

All these changes bring the size up to 3.1 KB minified and gzipped. I would like to be under 3 KB, but I want to be sure the right functionality is in place first before squeezing it down.

The module modifiers are a bit of an experiment. I wanted some way to separate a bunch of bad, wordy code out for the normal cases of a module but only in bad cases load the bad code. The example I give in the documentation is a module that gets DOM node dimensions and position. In standards mode, it is fairly compact, but in quirks mode it gets uglier. So I only want to load the quirks mode code if the page is in quirks mode. I never want to develop in a quirks mode page, but for a general JavaScript library it might be important.

So I am still not sure if the module modifier approach is the right way to go, but I have used it a little bit so far on another project, and I will see how that works out.

RunJS updated: simple modules and i18n bundles

2009-10-12T17:07:00.001-07:00

I pushed some changes to RunJS, the stand-alone JavaScript file/module loader.

Newest changes are support for simple module definitions and internationalization (i18n) bundles.

Simple module definitions are possible when there are no dependencies for the module. In that case the function wrapper for the module is not needed:


run(
    "my.simplething",
    {
        color: "red",
        size: "large"
    }
);

i18n bundle support is handy for separating out strings that might need to be translated into other languages. Quick example for a my/nls/colors.js that will define a bundle:


run(
    "my.nls.colors",
    [{
        "root": {
            "red": "red",
            "blue": "blue",
            "green": "green"
        },
        "fr-fr": "my.nls.fr-fr.colors"
    }]
);

Then define a file at my/nls/fr-fr/colors.js that has the following contents:


run(
  "my.nls.fr-fr.colors",
  {
    "red": "rouge",
    "blue": "bleu",
    "green": "vert"
  }
);

See the documentation for more information.

If you want to try out the latest code, you can use one of the following URLs to fetch the code:

In addition to the documentation, some of the test files might be of interest to see how RunJS can be used.

RunJS is still nice and small with these new features, around 2.7KB, when minified via YUICompressor and gzipped.

RunJS

2009-09-29T23:50:00.000-07:00

I just created what I hope to be the next generation JavaScript module loader here: RunJS. The documentation has some history and reasoning behind it.

I would like to see it used for any project that needs JS code loading, particularly since it handles dependencies, can load regular JavaScript files, handles multiple versions of modules, and is a compact 2.2KB (minified and gzipped). It is JavaScript toolkit-agnostic and has no dependencies.

Enjoy!

Chrome Frame, or Browser vs. Renderer

2009-09-23T11:13:00.000-07:00

Chrome Frame came out yesterday. I like the conversation it is trying to start. It makes a big difference having working code to get the conversation to seriously happen. There are some mechanics of the specific approach that probably need tweaking, but the general idea is worth consideration.

The big point is about separating the innovation over organizing the user's experience of the web in general (the Browser) vs. innovation in the display of web sites (the Renderer).

The Chrome Frame approach allows browser makers to preserve their revenue models and their UI interaction models, at least in the ideal -- there are some specific issues to work out with the Chrome Frame model, like password/form data storage. But I think the direction is a good one.

It also solves the issue where a web application was developed a few years ago and is not going to be maintained any more. If the browser allows multiple rendering engines it makes it possible for those old web apps to continue to work.

Old enterprise/business applications in particular can continue to work, but we still get to move the new web experience forward.

From a browser market share perspective, this clean split of responsibility could allow a newer browser like Google Chrome to get more market share. It could use the reverse idea: embed the IE render engine in Google Chrome. Then, take that to all the IT administrators and say, here is a newer, safer, more secure browser for your company that will work on older machines, is free, and can still allow your old business web apps to work.

That may not work, since most people are attached to their browser chrome vs. the actual renderer. In those cases, the plain Chrome Frame approach of installing a plugin for another renderer works to allow better, faster web apps experiences.

One concern: user choice or control. I think it actually leads to more user control: the user gets to keep the browser chrome, their organizing model for the whole web, intact, but gets to use more of the better parts of the web. And for areas where the user does not have control now (in business environments that need old apps to work), it gives a way out to use more of the web.

Of course, there should user controls so they can set preferences and overrides for the renderers used. At a minimum, business IT groups will need it to configure the browsers to use old renderers for older in-house business web apps.

So, some tweaks to the basic mechanics I would like to see (realizing that I have no concept how hard this work would be):

1) Make sure the Renderer works well with the Browser: for instance make sure that saved passwords/form data works well no matter what Renderer is used. Make sure the split is clean.

2) Change the UA-Compatible thing to be more Renderer-based feature sets vs. browser version numbers. So, something where the developer mentions the capabilities that are desired:

<meta equiv="X-UA-Compatible" content="addEventListener=1,svg=1">

If the developer has a suggested browser renderer, it could place that at the end, sort of how CSS font names can start with generic names, then get more specific:

<meta equiv="X-UA-Compatible" content="addEventListener=1,svg=1;gecko=1.9.2">

It would be good to *not* use browser names in the tag, but rather the renderer engine names/versions. Ideally though, just list capabilities.

The above syntax is not exactly right, but just to demonstrate the idea: focus on telling the browser the capabilities the page wants, and use render engine names, not browser names.

3) Make sure the user can override the choices made by the browser. The pref control does not have to be obvious, but should be there, so the user has the final say.

In summary, as a conversation starter, I like that Chrome Frame has really tried to highlight the difference between upgrades in renderers vs. browser interface.

Developer Tools and JavaScript Syntax Checking

2009-08-02T16:35:00.000-07:00

I was using Coda to do web development, primarily because of the simple interface, nice Panic Sans font and integrated FTP and Subversion support. I normally just use TextWrangler for Dojo Core development.

However, while doing development at the new job, FTP/Subversion support was not needed, and I have grown tired having to run my JavaScript in the browser to find basic syntax errors.

I saw David Ascher using Komodo Edit, the free editor from ActiveState, which has JavaScript syntax checking built in. Searching in the Tools, Add-Ons menu for Dojo also brought up a Dojo API Catalogs extension that allowed for autocomplete of Dojo APIs.

Komodo Edit has been mentioned in the Dojo community before as a nice option particularly for doing Dojo development, but I can get severe tunnel-vision, so it was not until recently that really tried out Komodo Edit.

After I changed the key bindings to use the normal OS X COMMAND+] and COMMAND+[ for multi-line indenting, I was ready to go!

It makes a difference having syntax checking and Dojo API autocompletion. I believe other tools, like Emacs with js2-mode, can get at least syntax checking, but I am not an Emacs user. It would be neat to see Coda (and TextWrangler) add JavaScript syntax checking support. I can see JavaScript syntax checking being a minimum requirement for JS coders going forward.

As Bespin comes along, I can see it as my next developer tool upgrade, particularly since FTP support has been implicitly needed for some of my projects. With Bespin, I might be able to edit directly on the server.

For now I am enjoying the bump in productivity with Komodo Edit. Thanks ActiveState for making a nice tool available for free!

Custom local variables using the Dojo Loader

2009-04-01T10:28:00.000-07:00

Recently, Dion Almaer was wishing for a way to avoid typing "dojo" so many times in the Bespin code. He appreciated the namespace protection that Dojo gave, but did not like the typing tax that came with it. He talked to Alex and Pete about it. Alex suggested a way, but it is very experimental.

Here is another way that leverages the strength of the Dojo Loader.

The Dojo Loader handles the work of loading your JavaScript modules. When you do a dojo.require("foo.bar"), the loader figures out that you want to load the foo/bar.js file and loads it for you. Normally the loader uses a synchronous XMLHttpRequest (XHR) call to load the file and uses eval() to bring the code into existence.

There is an xdomain loader that does not use eval, and allows your code to be loaded from any domain (with dependencies properly loaded), but it requires a build step using the Dojo build tools.

Both versions of the loader (normal and xdomain) allow you to namespace your code -- so you can map dojo to mydojo, and even your own code to some other name. This allows you to load multiple versions of dojo and/or your code in a page. Nifty.

The Loader is able to support this scope mapping by wrapping your module in a function call like so:

(function(dojo, dijit, dojox){
 //Module code injected in here
})(dojo, dijit, dojox);

We can use this new scope created by this function to create local variables for your module.

Dojo's Loader can be modified to allow you to specify a set of local variable names that get injected with this function wrapping. And we can do this on a per-module prefix basis, so your code can have your own local variables, but some other module can specify a different set.

I did a prototype that works as follows: assume "coolio" is the namespace I use for my modules. I create a coolio.locals that has the following content:

dojo.provide("coolio.locals");

dojo.setLocalVars("coolio", {
 trim: "dojo.hitch(dojo, 'trim')",
 $: "dojo.hitch(dojo, 'query')",
 id: "dojo.hitch(dojo, 'byId')"
});

This code will create local variables called trim, $ and id that map to dojo.query, dojo.byId and dojo.trim respectively, but only for coolio* modules.

Then I have another module, called coolio.actions that uses these variables:

dojo.provide("coolio.actions");

coolio.actions = {
 init: function(){
  $("#trimButton").onclick(coolio.actions, function(evt){
   id("trimOutput").value = trim(id("trimOutput").value);
  });
 }
}

Notice that trim, id and $ were not declared in here. The final bit of magic is to use djConfig.require in the HTML page to auto-load the coolio.locals module before any other coolio code, so that the loader knows to create the local variables for any coolio.* module:

<script type="text/javascript" src="dojo/dojo.js" djConfig="require: ['coolio.locals']"><script>

See it in action with this sample page.

I created ticket #9032 to track the possibility of allowing this in the future. It also has the patch that can be applied to the Dojo source to get it to work. Or, if you are using Dojo 1.3.0, you can grab these built files with the changes: dojo.js or dojo.js.uncompressed.js

There are some caveats to this prototype:
1) We really need a build step that would inline the local variables for your build layers, so the code as-is will not work with custom build layers.
2) Does not work with xdomain builds yet, another tweak to the build system is needed for that.

If you think this might be useful for you, feel free to add your comments to the issue tracker entry or leave a comment.

Job Transition

2009-03-29T10:27:00.000-07:00

Friday was my last day at AOL, just shy of 13 years. There were good times, bad times. Some really cool things. Neat people and teachers. A great learning experience.

I am being a bit melodramatic, but Sinead O'Connor's Thank You for Hearing Me expresses the sentiment best, including the end of the song. For work and for life.

Monday I start at Mozilla Messaging. I am really excited about the work. It is still being defined, so nothing to share yet, but I look forward to helping people take charge of their messaging.

Namespaces, Subjects and Verbs in JavaScript

2009-03-28T09:10:00.000-07:00

It has been interesting to follow Dion Almaer's comments about using Dojo in Bespin. This post mentioning call conventions higlighted something I have wanted to talk more about.

The basic issue is namespacing and how you like to call functions. Here are some choices:

subject.verb()
namespace.verb(subject)
namspace(subject).verb(), which can lead to namspace(subject).verb().verb().verb()

subject.verb()
#1 is the Prototype/Ruby way. You define a verb on the object's class, or in JavaScript, on its prototype.

In this model the namespace is really the subject's namespace. There is a possibility of collision with other code that wants to use the same verb. This can be mitigated by keeping your project small and focused, and managing your dependencies. This also works well when your code is a leaf node, or one step from the leaf node -- your code is not consumed by other code, except maybe a top level web application.

The benefit is a nice call structure, and I think fits better with normal English. The subject is identified and then a verb/action is performed with that subject.

Unfortunately, in JavaScript, there can be problems adding things to basic prototypes, like Array, String (and shudder, Object). In Dojo we have had to put in some protections in our code in case the page also uses code that modifies the built-in prototypes.

I believe the situation will improve in future JavaScript releases if added properties/verbs can be marked as not enumerable. That will help a bit, but namespace collision is still an issue.

Some browsers now have native String.prototype.trim implementations and Dojo now delegates to them for Dojo's trim method. Recently, there was a bug filed for Dojo where the core issue was some other code adding a String.prototype.trim() method that did not strip out beginning whitespace.

namespace.verb(subject)
#2 takes the position that verb collision is bad and can cause hard to trace bugs, so always use your own namespace. It also feels more "functional" or procedural: use small functions that do not maintain interior state but operate on their arguments.

This has the benefit of being safer, but it can be more verbose than #1, and therefore more of a constant tax on the developer.

This is mitigated somewhat in Dojo, where you can assign Dojo to another namespace. So you could map "dojo" to "$" to cut down on the typing.

namspace(subject).verb()
#3 is a nice compromise: Define a function that wraps the real subject in an object and provides verbs to act on that subject, without directly modifying the object/class structure of the subject. jQuery took this to new heights by allowing chaining of the verbs. Similarly, dojo.query() returns a dojo.NodeList object that has chainable verbs on it.

An explicit namespace is involved, but the idea is to make it as small as possible, "$". So the overhead for having the namespace is "$()". Coupled with the chainable verbs, it can lead to short code, less of a tax on the developer.

For Dojo, I think #3 is the right approach in general: it gives nice namespace protection, but still gives short call structures.

I want to explore a dojo() function that does this verb mapping in general for all of Dojo. Dijit might be able to use a dijit() to get something similar for the verbs it exposes in its namespace.

So for instance: dojo({foo: "bar"}).clone() would act the same as dojo.clone({foo: "bar"}). Scopemap dojo to $ and you get $({foo: "bar"}).clone().

The difficult part is how to deal with verbs/methods that deal with Strings. I want dojo("div") to actually call dojo.query("div"). But how to allow dojo(" some string ").trim()?

Maybe not map dojo("string") to dojo.query("string"), but instead, allow scope mapping of "d" (or even "_"?) to dojo and another mapping of "$" to dojo.query. That would probably match best with the expectations of what $ does today in other toolkits.

I will have to do more exploration. Eugene Lazutkin's work on providing adaptAs* functions for mixing in dojo methods into an object prototype as chainable methods might point the way to do this.

Dojo 1.2 Loader and Build System

2008-10-02T21:32:00.000-07:00

Dojo 1.2 is now available! Check out the release notes for more detailed information.

I attended the Dojo Developer Day in Boston on Sunday 9/28 and the Dojo Community Day at the Ajax Experience conference on Monday 9/29. It was really nice of the Ajax Experience folks to set up community time for the JavaScript toolkits.

During the Community Day, I gave a short presentation on some new things in Dojo 1.2 with the Dojo loader and the build system. I tried using the Google Docs presentation thing. You can see the slides here (some notes are after the slides):

customBase build option

The final example in the slides demonstrates how you can create a 5.5 KB (gzipped) dojo.js file and dynamically load it after the page loads, so if your site uses progressive enhancement, your up-front Dojo cost can be zero bytes, and just load as little as possible for the functionality that you use.

The 5.5 KB dojo.js file is possible using the "customBase" property of the dojo.js layer in a build profile. Normally we encourage you not to tamper with the contents of dojo.js (what is referred to as Dojo Base), but the customBase build option can give you more control on what parts of Dojo Base you want to load. With the customBase option, the build will go through all the JS files and automatically add dojo.require() calls for the necessary dojo._base modules.

Only use customBase if you keep all your JS code in JS modules -- good practice anyway, and keeps with progressive enhancement suggestions. Do not use customBase if you have Dojo calls inside HTML source.

Using customBase on the "dojo.js" layer with no dojo._base modules (just the loader and some bootstrap functions), the non-gzipped size of dojo.js comes out to be 13KB (recall from above it is 5.5 KB gzipped). So it is small enough to fit under the 25KB non-gzipped size limit that the iPhone/iPod touch use to allowing caching.

You will likely need to use more of Dojo Base than what is in the 13KB non-gzipped file, but you can tune that by creating more layers and playing with the build to so that all layers are under 25KB.

djConfig.addOnLoad

Another option for use with progressively enhanced web sites, or when you want to load Dojo after page load: djConfig.addOnLoad allows you to set a function to run after Dojo loads. It is most useful when used in conjunction with djConfig.afterOnLoad (used when you manually load Dojo after page load).

Here is a complete example showing djConfig.addOnLoad, djConfig.afterOnLoad and djConfig.require.

There is a bug about using djConfig.require in conjunction with a dojo.js generated via a customBase build, so only use djConfig.require when using a complete Dojo Base file.

Thanks go to Matthew Russell for suggesting djConfig.addOnLoad for the 1.2 release.

stripConsole
There is a new build option for stripping out console method calls. stripConsole=normal strips out all console.* calls except console.error and console.warn. stripConsole=all strips all console.* calls.

Safari 3.1 sometimes has an issue with console.debug not getting defined (fixed in webkit trunk), so this build option can help avoid that issue until Safari is fixed.

Mobile Development

2008-06-14T21:58:00.000-07:00

I was recently asked about my experiences making TinyBuddy IM. Here are my very biased thoughts about it and mobile development in general.

My Background
I like to make web products that people use. I am a front-end developer, and I love using JavaScript, HTML and CSS. I have done some C/C++ for a cross-browser streaming audio browser plugin. I recently played with Objective C as part of an iPhone SDK experiment. I did Java on the server for a few years too.

HTML, or some sort of declarative markup, is the way to code front end UI. It clearly shows the nested relationships between components. However, I liked the Interface Builder approach in the iPhone SDK, very WYSIWYG. It still some kinks to work out with the beta iPhone SDK though.

I also like trying to reach the largest possible audience with the shortest toolchain. To me, this means favoring an HTML/CSS/JavaScript UI first, and only considering other alternatives when that approach will not work. Particularly since I favor developing applications for the internet.

One Mobile Renderer to Rule Them All
I did not do mobile development until TinyBuddy IM. It just seemed too painful -- too many custom environments, too many cumbersome toolchains for native apps. And the browsers really sucked.

However, now with WebKit on the iPhone and in Android, it is tolerable. Those platforms also have a decent toolchain for "native" apps: iPhone SDK/Objective C for iPhone, and a Java environment for Android.

However, WebKit is most interesting to me: it is a modern, very capable browser that will be on two major mobile platforms. While JavaScript performance is still slow on mobile browsers (both compared to desktop browsers and native code), there is promise that it will get faster with WebKit's SquirrelFish.

For the first time, it is actually possible to consider making a powerful web app for a mobile device. You have good odds of developing one mobile app that runs on multiple devices too. Or at least have lots of code overlap.

Of course a web app is not appropriate for every app, and it does not have access to as many services as a native app, but it does have a lot. With the latest WebKit for instance, you will have local storage.

You Are Dead to Me
Not every mobile device has WebKit. What about Windows Mobile devices, BlackBerry, or all the other devices out there in the market today? Well, for me, they do not make it easy to develop. And I am lazy. I use an iPhone, so I am going to develop for what I use. However, I chose an iPhone specifically because it is easy to do development.

So the other platforms are not that important to me, and if my developer laziness is any indication, they are in for some trouble. They need to get WebKit or Gecko on their devices and use a better toolchain. Opera's browser may be an option too, but I am not clear if it is up to the capabilities of WebKit. If it is, great. I like the Opera on the Wii.

Again, it may not be reasonable to discard these other platforms, depending on your app, but I have decided the other platforms are not worth the effort, given my interests. I really want to make interesting end user products without spending time on platform minutia, and I feel like the iPhone's and Android's reach will be good enough for me.

Mobile Application Design
I approach the technical design of a mobile app by trying to see if the following approaches fit the problem:

A purely browser-based web app.
A native app that gives a "desktop" icon, and access to phone services (for instance, location services), but use a WebKit view for the main UI interface. Set up hooks in the WebKit view so it has access to the phone services.
A purely native application.

#2 has many shades to it, but I like it because it minimizes installed code. Revving a native application is more work than a server-based one.

Remember, I'm assuming this is an internet app, and you know the HTML/CSS/JavaScript trifecta already.

There are tradeoffs. Speed of loading/dependence on network, but again, I prefer to first look at a web-based solution first: maybe it makes sense for the native app to cache the HTML/JS/CSS/images locally.

Native Apps
What you want to build may not work as a web app. For those cases I still prefer iPhone or Android.

For iPhone, you have a nice dev environment with a WYSIWYG UI tool (Interface Builder). Objective C feels a lot like JavaScript to me -- some concept of a prototypical object, and runtime, dynamic dispatch where you can send messages to objects that may or may not respond to that type of message. Unfortunately you have to do the memory retain/release thing, but it is manageable. Lots of helpful videos and examples at the iPhone Developer site too.

For Android, using Java syntax and libraries are a big plus over some sort of C/C++ thing. Java can be verbose, but it is easy to program with an IDE, and hopefully they kept in the garbage collection like Java. That alone is a big win. I have not used Android yet, and I have heard of problems with trying to program in that environment (maybe they have been worked out now?), but it seems worth spending effort on that path. It seems like it has a good shot of survival over the long run.

UI Design
There is a great intro video on the iPhone Developer site about "User Interface Design for iPhone Applications". You will need a (free) Apple Developer ID to view it. But take the time to view it.

The main point: a mobile app is not a desktop app. Do not try to do too much. For TinyBuddy IM, I decided to not include buddy list management features in the UI (add/remove/move buddy). The app was designed for doing quick IMs with your buddies. Buddy list management just seemed to get in the way of that goal. Not to mention the lack of copy and paste on the phone. It just did not seem like a worthwhile activity for a mobile app.

Also, use the UI paradigms available on the phone. For native apps, Apple has a lot of built in templates for using the standard lists and tables you see throughout the iPhone UI. I used a very early version of iUI for a web toolkit that gives an approximation of the native iPhone UI. There may even be some resources in the latest iPhone SDK to make web development easier in this respect.

So, keep it simple, keep it familiar. Once you get the fundamentals down and released something, you can consider your own style (if appropriate).

Pain Points
It is not all roses. There are notable restrictions making mobile web apps. These limitations are specifically from the iPhone, but I would expect them to be roughly the same if not more oppressive in Android environments.

Resource limits.
Cache limitations (25KB max size, uncompressed, if you want it cached, 19 total cacheable items).
As mentioned above, JavaScript performs slower than desktop JavaScript or native code.
You web app is paused when it is not in the forefront (also a limit for native apps, but native apps will have a notification service you can use that will help with this).

That last one is particularly vexing for TinyBuddy IM, since it is an IM client. So a native app for the IM problem space is probably more appropriate. Or some sort of native component that ties into to the notification service? :)

TinyBuddy IM Weaknesses
I need to spend more time optimizing the TinyBuddy IM code:

I do not do any sort of "Fetch next 25" sort of thing to keep the buddy list size down.
The wiping between screens is slow.
The IM conversation window needs more polish.
Maybe some better navigation for multiple IM conversations.

So I definitely did not get it right with TinyBuddy IM. Now that there is a native IM app under development, I probably will not rev TinyBuddy IM again. However, I am amazed TinyBuddy IM works as well as it does -- I was able to be a little bit lazy, and still get something somewhat useful out. A good indicator for that platform.

Authentication
Most things I want to build require user authentication for at least some part of it. For TinyBuddy IM, that meant using AOL's OpenAuth authentication, since the Web AIM API used it. OpenAuth has a browser-based authentication system that any web page can use, and it even supports consuming some validated OpenID providers.

OpenAuth has a "clientLogin" API for desktop clients and Flash/AIR apps but I do not prefer it since it basically means the user hands over their name and password to your app, and then your app sends it to OpenAuth. While that might be nice from a usability standpoint, it bothers me personally from a security standpoint.

This is where something like OpenAuth's browser API (and in general OpenID in the browser) provides better protection. You only enter your name/password on the identity provider's web site, and a token is passed back to the web app that requires your credentials.

As an end user of this system, I can verify by the browser URL who is asking for my name/password. As a developer, I like just not having to touch passwords at all. It absolves me from the security issues with trying to manage passwords, even if it seems like I am only passing the password through to OpenAuth's clientLogin.

However, most end users are not this particular about using applications, they are happy to give their password to any site asking for it, particularly if it seems to be done in goodwill. They just want to use the app. Also, the browser based authentication can be weird -- popping a new window to get the auth credentials.

But the OpenID approach is the way of the future (going to another site to authenticate), so might as well get used to it. Particularly now that the OpenAuth authentication page has a "Remember Me" option, so the user does not have to keep entering name/password on every visit.

Have Fun!
Enough of my ramblings. The big thing for me: mobile development can be fun, particularly with the iPhone platform. Android, while not at the same level as iPhone development yet, also seems like it has promise too.

The Beauty of Dojo 1.1

2008-03-28T13:39:00.000-07:00

The Dojo JavaScript Toolkit version 1.1 is available:

Release Notes
New API tool (ability to add comments coming next week)
Dojo Book
AOL CDN info
Download Area (but try the CDN for easy use without needing to download dojo to your server. Save this sample page, and start playing.)
Follow Dojo. Subscribe to this planet feed for blogs and help sites that talk about Dojo and feeds from browser vendors.
Key Links for some demos and articles that explore Dojo.

There are lots of goodies in this release (see the release notes for full details), but here are some of the new features that resonate with me. I am primarily a Dojo Core user, not much Dijit or Dojox. For other perspectives on the new features, see the 1.1 blog announcement, and Shane O'Sullivan's blog post.

Multiversion Support
One of the slicker features (IMO, since I added it) is multiversion support: you can now run Dojo 1.1 with other versions of Dojo in the page without conflicting. You can also choose to rename dojo, dijit and dojox to other names. As proof:

A demo showing the 0.4.3 Calendar with the Dojo 1.1 Calendar
A neat dojox.gfx/charting demo showing dojo, dijit and dojox remapped to omega, omegaw and omegax, respectively

The multiversion support is great if you want to provide your own JS library, but use Dojo underneath. What is even cooler: multiversion support works from the CDN (see the source for the examples above)! So if you are thinking of migrating from an older Dojo version, you can experiment with Dojo 1.1 in a new namespace without having to download 1.1.

Load Dojo after page load
Dojo 1.1 can can be loaded after page load (after the window.onload event fires) by setting djConfig.afterOnLoad to true. This makes the initial render cost for using Dojo near zero, and it plays nice if you want to do extreme progressive enhancement. Use the new djConfig option in conjunction with djConfig.require, to load dojo along with the modules you needed dojo.required after dojo loads. See the the demo page for an example.

Adobe AIR support
Dojo now provides strong support for AIR in addition to Dojo's existing integration with Google Gears via dojox.offline.

Client-side data storage via dojox.storage
If you want client side data storage, dojox.storage gives you a few options, and auto-detects the best one. dojox.storage has been updated to allow for using Dojo Gears, HTML 5 DOM storage, Flash or AIR DB storage.

Build system CSS optimizations
The build system will now inline @import calls that are in .css files in addition to stripping comments and whitespace in .css files. See the New Build Options section.

Try it out!
If you would like to try Dojo, but are not sure how to start, you can always start small: If you just use Dojo Base (one file, dojo.xd.js, from the CDN), you get a solid JavaScript base that makes building progressively enhanced sites much more enjoyable. A rough run-down of the Dojo Base functionality can be found here. That article is for Dojo 0.9, but it still applies for 1.1. Dojo 1.1 is slightly larger than the numbers quoted for 0.9: The 1.1 dojo.xd.js is 29KB gzipped. See the 1.1 release notes for the new features that account for the size increase.

I prefer to stick with Dojo Base, with some additions from the Core modules (like dojo.io.script, which allows using JSONP APIs, like the ones provided by Web AIM). This is how I approached development for the AIM Chat web site and for the iPhone IM web app TinyBuddy IM. But given Dojo's depth, I was able to leverage the Dijit widgets to create a simple admin site for AIM Chat by adding in a few more dojo.require calls. Sweet!

I am really amazed at the amount of work that has gone into Dojo 1.1. The more I use it, the more I feel it is the complete JavaScript toolkit solution. You can do the small, quick progressively enhanced web sites using just Dojo Base from the AOL CDN, but scale all the way up to very rich experiences that use the full power of widgets, awesome data stores, offline storage, and incredible 2-D drawing capabilities (with charting!).

I feel the Dojo community is really hitting their stride now. Great job everyone!

Browser and Dojo updates on Fragment ID messaging

2008-01-31T13:15:00.000-08:00

Some browsers have changed how they deal with fragment ID messaging:

Safari 3 throws an error if a child frame tries to change a parent frame's location, but only if the parent frame is not the top-most frame. So, this test works, but this one fails. Parent frames are allowed to change a child frame's location.

Opera 9.25 has locked down cross-frame access to some frame properties. As Julien LeComte pointed out, Opera supports the HTML5 cross-document messaging API, so that is a viable alternative.

I have updated dojo.io.proxy so that it now works in these browsers: IE 6 and 7, Firefox 2 and 3beta2, Safari 3 and Opera 9.25. You can get it from the Dojo development trunk, or wait for Dojo 1.1. The new dojo.io.proxy code works with Dojo 1.0, and there are no API or usage changes.

This new code also improves the IE7 codepath: before the code used to use 3 frames to do the transport, but now it only uses 2 frames, like the rest of the browsers. I was not being very smart about the ordering of the frames in the old code.

Thanks go to Julien LeComte for pointing out the Opera cross-document API and for diagramming another way to do fragment ID messaging.

dojo.io.proxy differs from Julien's CrossFrame approach:

it allows very large messaging (it supports fragment ID chunking)
it is focused on providing a data transport -- it provides an XMLHttpRequest facade. It is not a communication transport for UI widgets.

Julien did not want to use location polling, since it increases CPU usage. dojo.io.proxy does use polling to do the communication (this allows for chunking to support large messages), but polling is active only while the message is in transport. I consider the window.location polling less resource-intensive than doing a DOM animation. CPU usage could be tuned by slightly extending polling interval if it is a concern.

dojo.io.proxy creates the frames only when the request starts, then it destroys the frames when the response finishes. This preserves the history stack in all browsers except Opera.

There is a very primitive test page if you want to see the latest code in action.

David Baron's solution for X-UA-Compatible

2008-01-30T10:00:00.000-08:00

I like David Baron's alternative to X-UA-Compatible: if the main problem is intranets, give intranet folks a way to configure IE to use the old rendering mode just for their hosts.

I'm interested if this would be enough for Microsoft. If they claim it does not cover all the cases, it would be good to get some sort of metric/percentage of how much it would cover. Seems like it would cover the vast majority case. Probably close to 100% of the cases where Microsoft's revenue is impacted.

If the claim is that they cannot "break the web" even for that small case on the internet where their revenue is not directly impacted, I wonder how important those sites are anyway. I expect there are lots of sites that were made with older browsers in mind, but the site is probably abandoned too. As long as you can sort of read the content on the page, maintaining precise layout is probably not important.

If Microsoft is still set on using a flag/switch in HTML/HTTP header, I still prefer a cultural flag and not a product version flag.

Using Aptana Jaxer for search engine friendly pages?

2008-01-26T23:09:00.000-08:00

I am interested in Aptana's Jaxer because they seemed to have gotten Gecko to work on the server. From what I have read before, the tricky part is getting Gecko to run without the need for a windowing service (like X-Windows). Maybe Jaxer still needs a windowing service, I have not looked that closely.

I am not so interested in using their runat="server" or server-proxy capabilities, but more interested in getting Jaxer to render my page to the final "onloaded" DOM that could then be sent to a search engine.

However, I have not seen an option yet to tell it "treat the whole page as runat server" or something like that.

Here is what I want to do (condensed from my previous post on searchable ajax):

Do the model/view/controller in HTML/CSS/JS. I normally want this all to run in the user's browser.
However, for search engine requests, I want to do some Apache rewrite/proxying to a server-side Gecko that would render the DOM as it would look after window.onload, then serialize that DOM as the result of the search engine's request.

I wonder if Jaxer could be used for the "server-side Gecko" part. If so, that would be sweet. That would allow my web app to be indexed by search engines properly, but still give me the development model I want. No special dev work to get search engine benefits.