On inventing JS module formats and script loaders

There was some recent twitter tweets over the last couple months that indicated some folks want to experiment with some JavaScript module formats that work in the browser. In addition, making a script loader is almost as popular as making your own template engine. This is my response to both of those endeavors.

AMD is Winning

If you are one of those people that think they want to invent a module format or a script loader that works well in today's browsers, let me save you some trouble: a module format has already been worked out. Check out the Asynchronous Module Definition (AMD). There is still some room for innovation though, see near the end of this post.

If your script loader does not support AMD, it will be of very limited usefulness and in a sea of competitors that will make it hard for you to gain adoption.

So, for the module syntax, that problem is solved, and the solution is AMD.

A Solved Problem

Why is it solved?

• It has been discussed for more than a year and a half by multiple people, in the Dojo world, and on the CommonJS list.
• It has been implemented more than 5 times by different people.
• It has been deployed in real projects, the BBC iPlayer being a very visible one.
• It works in the browser, web workers, Node, Rhino, Mozilla Add-On SDK-based add-ons, and can be used in traditional Mozilla add-ons (Firebug will be using it).
  
• Many AMD implementations support loader plugins. You can add modules that know how to load other things as dependencies. While not strictly part of AMD, plugin systems greatly enhance a base technology.
  

You will be facing an uphill battle trying to get your format accepted. AMD has at least a half-year head start and has been seriously battle-tested. The ECMAScript folks are considering separate, different language changes to support modules, so you have that on the event horizon. Time is really running out pushing for something new.

I do not want to harsh your mellow, and I definitely want to follow your bliss, but just know the actual cost involved, and do your research first. Make sure you understand why AMD is constructed how it is, and what it really is.

It is not the traditional CommonJS Module format, but it can support similar concepts. However, it is different, take the time to figure out why.

Here is a handy gist with a summary of the AMD API, loader plugins, and some thoughts as to how it relates to modules implemented in Node, which are more like traditional CommonJS modules. In addition, you can read the RequireJS API docs, and the AMD wrapper format for traditional CommonJS modules.

If you think something was missed, feel free to ask on the RequireJS list, or on the CommonJS list, and enlighten us all. Or just send me an email or GitHub message if you want to talk privately. However, I believe most questions will be in the minor bikeshed/personal preference arena that do not ultimately matter for adoption. In particular, do not propose anything that requires the developer to do more typing that what is in AMD already. The format needs to be usable and easy to type. Discoverability is not a goal, daily usability is, and the AMD format is easy to explain.

Requirements

This leads to a discussion of other requirements. Here is what I believe are the minimum requirements for a module format in the browser, and they well-met by AMD:

• No globals: The module format needs a way for the developer to avoid creating global variables for each module they create. The format needs to be web-scalable, which means that some sites need the ability to embed two different versions of a module on a page. This normally means embedding in different "contexts" (the different versions do not need to talk to each other, just co-exist on the page), but it is a web-scale need. Seriously. We have seen it in Dojo. jQuery has seen it, it is the reason they have noConflict(). While noConflict() is nice, there are edge cases where it falls down. Know why that happens (in particular, dynamic script loading in IE: more than one script can execute before the first script's onload event fires).
  
• Allow for globals: at the same time, you want developers to be able to do want they want, in particular, some like to modify global object prototypes. Prototype and MooTools are valid ways to build web sites. So allow for it. This is one of my concerns with the ECMAScript harmony modules experimentation, but I think they know they need to allow for this, even if within a module loader instance.
• Do not require server transforms or think that using XHR+text transform+eval() is usable across all of web development. It is not. Here are some reasons why. They are fine for your boutique rails projects, but do not assume that translates well to the web at large, for instance, to serving JS files from disk for mobile widgets.
• You will need to use a function wrapper and a way to specify dependencies before that function wrapper is executed. You will be using script elements to load content (seriously, don't use eval, that is an evolutionary dead path), and dynamic script elements will load scripts async and out of order. A function wrapper will need to be part of the format.
  
• You need a way for modules to have a name. Otherwise you cannot combine them all together for optimization purposes.
• At the same time, it is best to allow source modules (the non-optimized ones) to not be named. This makes it much easier to move code around. This is not a hard requirement though, and I think one of the easiest to try to not support, see next section.

Places for Innovation

So you are not going to come up with a better module format. I know that sounds harsh, and for some of you that will be just the thing you need to hear to try something different. But I really am trying to save you some work. Again though, go for it if it really is your bliss, just know the amount of work you are taking on.

A better use of your time might be doing the best AMD implementation/script loader. In particular, you can choose some limiting design choices to help make it easier to meet/more compact:

• Do not support the simplified wrapper for traditional CommonJS Modules.
• Only allow named modules.
• Do not support loader plugins.
• Do not worry about supporting more than one version of a module in a page. The module format allows for it, which is the most important aspect, so module authors can code for it if they wish. However, most sites can get away without needing a loader that can actually support loading multiple versions.
  

If you do those things, you still can consume the built/optimized AMD modules that someone made using another implementation, and any modules created while working with your implementation will work with other AMD implementations. And with that, you are part of a larger ecosystem.

If you make a script loader that does not understand AMD, it will be a boutique loader, and not something that fits all of front-end development, and limits the ability to share code with other JS environments. Which is fine, boutique businesses can be really satisfying, just realize you are in a boutique, and do not over-sell the loader.

FAQ

These are odds and ends that do not fit above.

Why use modules or a script loader at all? Just server-side concat all your scripts!

The goal is to establish a larger ecosystem of shared code that works without puking globals all over the place. JavaScript is also not just jQuery (although I greatly appreciate the project), so it cannot rely on a jQuery implementation.

Not all environments can use server-side script concatenation (file-served mobile widgets), and it does not allow for more nuanced optimization like loading built layers on demand after initial page load. On-demand loading is really needed for large web apps like a webmail UI.

With tools like the RequireJS optimizer, you can still get the same effect of "concat all files". If you do not want to take the hit of downloading a script loader in that scenario, there is work underway to allow a simplified stub that removes the need for a full script loader. The Ace project uses a "mini-require" after they build the code to avoid loading the RequireJS script loader.

What about YUI 3 modules?

YUI 3 got a lot of things right with their modules, in particular designing for async up front. However, I believe the way they implemented the module name-to-JS-variable-name mapping is not right. It requires knowledge of two names that allows for easy typos and creates the opportunity for a naming conflict on the Y object. For instance:

YUI().use("io-base", "my-module", function(Y) {

//Where does .on() come from? Did 'my-module' add it?
//Did 'io-base'? What if both tried to add it?
Y.on(...);

//'io-base' creates the 'io' property on Y.
//How is that known? Should it be 'ioBase'?
//What else does 'io-base' add to Y?
var request = Y.io(...);
});

In short, too much magic that requires extra documentation. YUI 3 has very nice documentation so that helps, but I do not believe that approach scales as well as AMD.

Is AMD a CommonJS specification?

No, it is a proposal, but with lots of implementations and real use. Not all participants on the CommonJS list believe it should be a elevated to "endorsed spec" status. Those participants wish browsers would grow better base technology and/or they want to maintain stricter compatibility with traditional CommonJS modules, which were not designed primarily for async loading environments like the browser.

Any browser technology change will likely be for harmony module support which operate differently than traditional CommonJS modules, and AMD works well today in all JS environments. AMD is proven, and it can work on the server side. The RequireJS Node adapter even allows using NPM-installed modules in Node.

How does AMD relate to ECMAScript harmony modules?

They are unrelated, although hopefully the AMD work and the traditional CommonJS module work will help inform the harmony effort. I believe AMD modules will be easy to convert to harmony modules, but as the harmony work is still in progress it is hard to know for sure. The return value options in AMD may be more flexible than harmony modules, and it is unclear if loader plugins will work in a harmony module world.

AMD has the benefit of working in all the JS environments now, and it will work in the future, and I do not believe the syntax gains in harmony modules are a vast improvement over the typing cost in AMD, particularly if sharp functions are supported. That said, I hope to inform the harmony effort where I can to help make it the best it can be.

Who are you?

I'm James Burke (GitHub, Twitter). I maintained the original Dojo XHR-based script loader, maintained Dojo Core, created the Dojo xdomain script element-based loader, created RequireJS (formerly RunJS), and rewrote RequireJS about three times. I have strongly advocated for browser-friendly modules on the Dojo and CommonJS lists. I like Pina Coladas and getting caught in the rain.

RequireJS+jQuery project moved, updated to jQuery 1.5.2

The sample project that demonstrates how jQuery and RequireJS can be used together has been moved to its own repository. This should make it easier to update. As proof, it now has jQuery 1.5.2, and can be downloaded here.

Also, the sample project has gone back to using an integrated RequireJS+jQuery file as the basis for the example. Keeping RequireJS and jQuery separate and using the "priority" configuration option in jQuery has some edge case considerations that required more explanation than what most people need. However, the README in the sample project repository explains how to use the priority configuration if you prefer to use that option.

RequireJS 0.24.0 Released

RequireJS 0.24.0 is available for download.

Themes for this release: IE 9, better jQuery integration, better loader plugins, and a CoffeeScript plugin:

• Support for IE 9. It has a non-conformant script loading behavior that necessitated the change. It would be ideal if IE 9 would change the behavior to be conformant.
• Changes to jQuery integration:
  • jQuery 1.5.1 included in sample project.
  • No more bundled RequireJS and jQuery file. RequireJS includes special code to detect jQuery as a module, so the combined file is not necessary, and this approach makes it easier to swap in new versions of jQuery as they are released.
  • Because of that change, the jQuery sample project uses the new priority: config approach.
• allplugins-require.js has been removed. Plugins are no longer bundled with require.js or in the sample jQuery project. There are separate download links to the existing plugins on the download page. Special treatment of these plugins has also been removed from the require.js source, so those plugins behave like any other loader plugin.
• There is now a CoffeScript plugin. It makes it easy to code in CoffeeScript in the browser, it can participate in the optimizer optimizations, and it works in Node and Rhino. This is the best way to do cross-environment, modular CoffeeScript.
• Bug fixes.

This release is the "release candidate" for RequireJS 1.0, so please give it a whirl. There may be some small tweaks for the final release, particularly around how the optimizer is packaged, but if this release holds up well, RequireJS 1.0 should be out in about a month.

RequireJS 0.23.0 Released

RequireJS 0.23.0 is available for download.

Highlights

• The baseUrl defaults have changed. This will likely be the biggest impact, and may require a change in your code.
• Node is now the default JS engine used for the optimizer. It is blazing fast. However, not everyone can switch to the new hotness, so it is still possible to run the optimizer in Java/Rhino. But Node is the default, so if you use build.sh/.bat and want to use Java, update to buildj.sh/.bat.
• UglifyJS is the default minifier for the optimizer, since it runs in both Node and Rhino.
• The Node adapter is updated and should match Node's native behavior for any npm-installed module, but see the release notes for more detail.
  

See the release notes for more information.

The env plugin

One of my favorite things in this release is how the optimizer can run on either Node or Rhino. The optimizer is structured as a set of RequireJS modules and a new plugin: the env loader plugin. That plugin tells RequireJS to load a particular module based on the environment that is running RequireJS.

To indicate a module dependency that varies based on the environment, modules use dependencies that look like "env!env/file", and the env loader plugin will load "node/file" if in Node or "rhino/file" if in Rhino.

While the env plugin in its current form is not ready for outside use (in particular it needs more work to be useful in an optimizer build), I believe it shows the way forward for doing environment-based branching of module loading. In particular, it can be seen as a great alternative to the concept of "overlays" that have come up in some approaches to CommonJS packages.

jQuery integration

The integrated jQuery+RequireJS file has not been updated to jQuery 1.5.0 yet. I want to wait for jQuery 1.5.1 since it may contain a fix related to jQuery.readyWait which is used by RequireJS.

Next up

With the conversion to using Node as the default optimizer engine, the code is one step closer to a 1.0 release.

I want to do another release that breaks out the text, i18n and order plugins into a separate repo or repos. Right now they have a magic status that will cause problems in the long run.

I also want to deliver the optimizer just as one script file, instead of a small collection of modules as it is now. This is possible since the optimizer is written as RequireJS modules/loader plugins, and not something that would be easy to do if they were CommonJS modules. I may punt on this though in the interest of getting to 1.0 sooner.

It may also make sense to break out the optimizer into its own repo, but again I may punt on that to reach 1.0 faster.

So hopefully just one or two more releases before going to 1.0. Depending on the timing of jQuery 1.5.1, there may be one additional release in that set.

jQuery UI as AMD modules, for use in RequireJS

There was a request on the requirejs list for a conversion script that could convert jQuery UI files into a format more easily used by RequireJS. While it is possible to load custom builds of jQuery UI with RequireJS, having the jQuery UI scripts in AMD module format allows for flexible use:

• Have the full range of jQuery UI available for use in different ways on different pages, but still get optimization benefits via the RequireJS optimizer.
• Dynamically load parts of jQuery UI based on user actions.

The jQuery UI files already list their dependencies, but in a source comment. So it was fairly easy to whip up a command line utility (that is run via Node), to convert the files.

I created an example project too, showing how to load the Tabs widget, then when the user clicks on the second tab, dynamically load the Datepicker widget with its French localization.

The example project includes the RequireJS optimizer and a build profile. What is really neat: the unoptimized source project is about 25 HTTP requests. After the optimizer runs, it goes down to 8 requests, 4 of which are images.

There is more information on the conversion process in the README on GitHub. If the jQuery UI team is interested in supporting this more directly in their project, I'm happy to work with them to set up a define() stub/wrapper to allow the code to work even when an AMD loader like RequireJS is not in the page.

Enough talk, on the code:

• jqueryui-amd on GitHub
• Example project demo (unoptimized)
• Example project demo (optimized)

RequireJS 0.22.0 Released

RequireJS 0.22.0 is available for download.

The highlights:

• A full plugin API that supports also including plugin loaded artifacts in optimized builds.
• Robust Node support: now can use npm-installed modules, .node add-ons, and modules on require.paths.
• Support for has.js optimizations in the optimizer, including dead code branch removal via Closure Compiler.
• Bug Fixes from 2010-12-21 through 2011-01-05.

Thanks to Joe Parry and Francois Laberge for the Node test cases and feedback, John Hann and Kris Zyp on the plugin API.

This brings the code closer to a 1.0 release. I want the plugin API to settle a bit, see if others can build useful plugins with it, and then look at moving the optimizer from Rhino to Node. Hopefully just a couple releases away from a 1.0.

A side note: I goofed on the previous two releases, they should have been 0.20.0 and 0.21.0, but I released them as 0.2.0 and 0.2.1 respectively. This 0.22.0 release gets the version back on track.

Standards and proposals for JavaScript Modules and jQuery

This commit to add registeration of jQuery as a module via the Asynchronous Module Definition (AMD) API has brought up some questions that I should have addressed previously before the commit happened. The jQuery team may decide to back out the commit, as is their prerogative, but I hope not. This post will be the justification for the AMD API in general and for its suitability for jQuery in particular.

What is the Standard?

There was a concern brought up by Kevin Smith that module loading is still up in the air, and that there is no "Standard" yet. To me a standard is something that is written down, understood by people, and has a good level of market adoption. Multiple implementations help too.

CommonJS Modules is a standard in that way. It is written down, understood by a few people, and has a a good level of market adoption via Node. There are a few implementations for other JS engines.

The CommonJS group also has a process where they have voted before to call it a Standard vs. a Proposal. Anyone can join the CommonJS discussion list, and I think it is still an ongoing discussion on how best to vote on proposals, who should get to vote, and if voting makes any sense. It is helpful to know if most of the people on the list like a particular proposal though, it gives some confidence that the proposal has been thought out.

I have a concern about the diversity of the list participants though. Most joined when it was still ServerJS, and the CommonJS modules "standard" does not work well for use in web browsers that use script src="" to load scripts. So, you could hear that "CommonJS Modules are a standard", but it does not mean it works well in all CommonJS environments, most notably, the largest distribution of a JS environment, the browser.

That is the reason the AMD proposal exists. It works well in the browser environment where module loading is by default asynchronous, and with script src="" loading (the easiest to debug and fastest way to load code), there is no control over modifying the script source before it executes. AMD is called a "proposal" in the CommonJS list because people on the CommonJS list have not voted to call it a "standard".

There is recognition on the CommonJS list that the CommonJS Modules 1.1 is not sufficient for script src="" browser loading, but there is some disagreement over how to solve it. AMD is one proposal. Kevin Smith has put together a Module Wrappings proposal, based on some discussions on the CommonJS list. The Wrappings proposal is incomplete -- there is more to it based on the list discussions. I will highlight the main difference between AMD and Wrappings below, but first, another module API contender:

ECMAScript Simple Modules. This is a "strawman", which is like a proposal in the CommonJS group: it is still something being worked on. This one is different because it is being worked on by the ECMAScript committee, and it seems like the strongest module proposal in that committee so far. If adopted, some future version of the ECMAScript (JavaScript) standard will have support for modules natively. So, any module API that used today should be aware of what is going on in the ECMAScript arena.

So, for talking about a module standard, particularly for use in the browser, these are the main contenders:

• AMD
• Wrappings
• Simple Modules

There are other modules systems like the one used in Dojo, and the one used by YUI, but those are too tied to those existing toolkits to be broadly adopted. Dojo has started to move to AMD.

AMD and Wrappings Differences

AMD and Module Wrappings are mostly the same. There needs to be a function wrapping around the actual module, to make sure the dependencies are fetched first before executing the module. There is some bikeshedding around names that I do not think is important (define vs. module.declare).

The main difference: AMD executes the module functions for dependencies before executing the current module function, where Wrappings just makes sure the module function is available, but does not execute it until the first require("") call inside the module function asks for it.

For shorthand, I will call the AMD approach the "execution" approach, and the Wrappings approach "availability". Both relate to how dependencies are handled.

Because AMD executes the module functions of dependencies before calling the function wrapping of the current module, it can pass the dependency as an argument to the factory function. This is a side benefit of execution.

The "availability" approach was used for Wrappings to keep it most similar to how CommonJS modules deal with dependencies, but I believe it is the wrong choice for the future.

Problems with "availability" model

"Availability" is not supported in the ECMAScript Simple Modules strawman. Simple Modules uses a model that is closest to the "execution" model: you use "module" and "import" keywords to reference dependencies, but they cannot be used like require("") is used in the "availability" model. "module" and "import" are more like directives, and not something that can by dynamically assigned, as part of conditionals. Examples:

In the process of working on RequireJS and trying to translate modules written for Node to a wrapped format, I have seen the following constructs in Node modules (which assume the "availability" model):

try {
 var constants  = require("constants");
 ...
} catch (e) {
  //Older version of node here
}

and:

var foo;
if (someCondition) {
  foo = require("foo1");
} else {
  foo = require("foo2");
}

Those work in the "availability" model, but not in Simple Modules. AMD's execution model does not allow the above constructs, so modules written for AMD will be more portable to Simple Modules.

2) The "execution" model fits better for projects that use libraries like jQuery, Prototype or MooTools, where many of the modules augment other objects, and they are assumed to have already run before executing the current module function. jQuery plugins augment the jQuery object, Prototype and MooTools augment JavaScript object prototypes.

Choose AMD

AMD is the API for today that translates to the future best, and it should be the one that jQuery should support because:

• There is a document that defines the API.
• It is understood by a few people, and discussed in public on the CommonJS list.
• It is not ratified as a "Standard" in CommonJS because there are list participants that still want to hold on to CommonJS "availability" semantics. However, that availability model is not as future proof as AMD, at least for the Simple Modules future.
• AMD translates better to Simple Modules due to the execution model. It maps better to the the related Module Loaders strawman, so if Simple Modules with the Module Loaders API becomes a standard and gets implemented in browsers, code written for AMD is more likely to continue to work.
• For browsers versions that probably never support Simple Modules (IE6-8, even 9?), code continues to work.
• The "execution" approach for dependencies fits better with the mental model of existing browser toolkits that want to augment other objects, like jQuery plugins.
• AMD is supported by a module loader (RequireJS) that has had some level of successful adoption, and it is fairly robust now, with over a year in development with frequent releases. RequireJS has an adapter that allows the same modules to be run in Node and Rhino. The Node adapter allows using existing Node modules.
  
• There are other AMD implementations.
• RequireJS in particular works hard to make sure it works well with jQuery, including integration with jQuery's readyWait to hold off DOM ready callbacks until all scripts are loaded.
• AMD has more real world adoption than Wrappings. The Dojo Toolkit now supports using an AMD-compliant loader in their trunk code for Dojo Core and Dijit. There is a thriving RequireJS list with real people using it. BBC iPlayer uses it. The RequireJS implementation of the AMD API has been promoted as a useful tool in multiple jQuery-related conferences.

RequireJS 0.2.1 Released

RequireJS 0.2.1 is available for download.

This is a bug fix release related to issues with the refactoring done in 0.2.0. Probably the most noticeable bug that has been fixed: the "all plugins" builds did not include the plugins.

RequireJS 0.2.0 Released

RequireJS 0.2.0 is available for download. The highlights: stronger, faster, and prettier.

• Refactored core.
• New loader plugin API support.
• Improved Rhino and Node adapters.
• Bug fixes around loader plugins and the optimizer.
• require.modify removed.
• Removed Transport D files.

Stronger: The refactored core is more robust vs. the 0.1x releases, and the new plugin API makes it much easier to construct loader plugins. The plugin API is an implementation of Kris Zyp's plugin API proposal. The optimizer/build support is different than what he proposed, and I plan on discussing the changes on the CommonJS list where he sent the proposal. But the basic API for defining a plugin in that proposal is supported.

The Rhino and Node adapters are even better in this release. I want to improve the Node adapter in a future release by allowing the use of npm installed modules without setting up a package path config.

Faster: The refactored core responds faster to loaded modules, and it will execute a module as soon as its dependencies load. In practice you will probably not notice an appreciable speed change, but it allows the new plugin API to work well.

Prettier: The web site got a much needed facelift and a logo, thanks to the talented Andy Chung. So much better than my feeble attempt.

As part of the refactoring, I removed the Transport D files. I believe they are no longer in use by other projects. I also removed require.modify. It could not support relative path names, and was always an odd thing that did not quite fit in with the rest of the code.

The refactored code is also more strict. If you do this in main.js:

require(['foo'], function () {});

And then in foo.js, it does:

require(['bar'], function () {});

The require callback in main.js will be called before the require callback in foo.js, because foo.js did not define a module. The loader thinks it just needs execute foo.js in order to call the require callback in main.js. To make sure the require callback in main.js waits for foo's factory function to get called, change the require call in foo.js to define:

//In foo.js:
define(['bar'], function () {});

Thanks to the community for filing bugs and contributing to the discussions on the mailing list. Thanks to Ben Hockey for drilling in deep and working out how to load the Dojo trunk code with RequireJS. It prompted some great bug fixes, and a few patches from Ben.

Rawld Gill, along with Kris Zyp, has done a lot of work to convert the Dojo trunk code to use the AMD API proposal. Rawld has even done an alternative loader that implements the AMD API. Check out his work if you want a different look at an AMD implementation.

What's next? I want to move closer to a 1.0 for RequireJS, but I would like to get these things settled first:

• Get feedback from the community on the loader plugin API. I need to do some better documentation for it too.
  
• I want to do a has.js plugin generator, and I want to integrate support in the optimizer to trim if(has['test']){} if the configuration passed to the optimizer indicates that has['test'] will always be true.
  
• Allow the optimizer to run on top of Node. Right now the optimizer runs on top of Java via Rhino, but that is increasingly becoming a liability.
  
• Sort out full packages support. Right now RequireJS can load modules in packages, but it cannot handle two versions of the same package in a project. While this is a minority use case, I would like to have a story for it. I think it is workable in source form, but I am not sure how it will work out yet for optimized scripts that want to include modules from two different versions of the same package.

RequireJS 0.15.0 Released

RequireJS 0.15.0 is available for download. This is a bug fix/robustness release:

• The bundled jQuery options now use jQuery 1.4.4.
• The jQuery sample project now includes the ability to use RequireJS plugins.
• The jsonp! plugin has been removed, since, thanks to work by Kris Zyp, the core loader now supports loading JSONP dependencies by default. The JSONP docs have been updated accordingly.
• The optimizer can now be run from any directory, not just the directory with the build profile.
• r.js Node adapter is more robust, and it can handle using more Node-written modules by default now. Thanks to Francois Laberge for a great test case application that lead to improving the robustness of r.js.
• Initial support for PS3 Netfront browser. Thanks to Chris Warren for investigating the load behavior of the browser. Not all tests pass, but the basic ones do.
• Miscellaneous fixes, some listed in the issue tracker.

The next stage of development will likely be a refactoring of the basic code. The code has grown organically along with some changes in requirements over the course of the year, and it would be good to reorganize based on the current understanding of script loading needs. The biggest noticeable change should be a better API for writing loader plugins, but the core API for requiring and defining modules should not change.

The Open Source of Mozilla F1

I am a front end developer on Mozilla F1, a sharing extension in Firefox, and this is some technical background on it. Note that any opinions in here are from my personal perspective, you should not treat this as official Mozilla policy, etc...

F1 consists of a few components:

1) Firefox browser extension
2) HTML/JS/CSS share UI served from from the F1 web server
3) API server written in Python that lives on the F1 server

The extension (#1) is responsible for the share button in the toolbar, injecting navigator.mozilla.labs.share(), and for managing a browser instance that is inserted above the main web browser area. This browser area makes a call to the F1 server to server up the main share UI (#2).

The share UI is written in pure HTML/CSS/JS. They are static files that use RequireJS to build modular JavaScript which is then optimized via the RequireJS optimizer into a single application-level file. It does not yet use appcache, but it is something I want to add. It should be straightforward. Making sure we properly version files with long cache times will also help.

The share UI uses the API server (#3) to do the actual sharing. The API server also manages the complications of the OAuth dance.

If you are curious about the implementation, all the code is open. You can download it/fork it at the Mozilla Labs F1 GitHub repository. There is a README.md file with setup instructions.

The great thing about this? You can run your own share server that you control. If you want the F1 browser extension to use your server for the share UI and API server, set the following F1 configuration in the about:config Firefox preferences:

extensions.ffshare@mozilla.org.share_url = 'http://your.server.com/share/'

Ideally that would be an https URL, but if you are setting up your own server, hopefully you know the risks involved. Be sure to restart the browser so the extension will use this new config value.

If you think this is really cool and want to help contribute code/ideas/submit pull requests, be aware that we are still in the early stages of F1. We expect to make many changes as we refine it. Mozilla is also new to GitHub, so we appreciate your patience as we try out different work flows.

I will close out this post with a couple developer bikeshed questions and answers. Again, these are my personal perspectives, and other people on the team may have different ones.

Why is the share UI served from a web server and not from the extension?

This is an experiment. More things are moving to the cloud/server, and some of the auth APIs, like OAuth, are better suited to server use. While Mozilla already runs some non-trivial server-based services, it is good to get experience with varying types of server-based services.

I like the flexibility and ease of updates a server solution provides. Also, it is easier to debug web code that is served from content space vs. browser chrome space.

It may allow us to support mobile and even other browsers easier. I played around with doing a Google Chrome extension that served up the share UI. Unfortunately, the Google Chrome extension model is not as flexible as what we can do in Firefox, so it does not really work.

In particular, I was looking at a Chrome extension Browser Action that served up the UI in an iframe in a Popup, but I could not get the popup wide enough to show the UI, and it closes as soon as an OAuth window jump occurs. We would have to do some rethinking on UI to support that, and it is not clear how beneficial that is at this early development stage.

All that said, the UI could be burned in to the extension at some point. It is still too early to tell where this will go.

Why did you use Python for the server?

Ease of setup, combined with some knowledge of real services built with it, and the mix of developer skills of the F1 team.

Ideally I would like to see the server running JavaScript via Node, but that is because I am an irrational JavaScript zealot. However, the JS libraries for Node are still young, and at the end of the day we want to solve real user problems, not debug other people's code.

That could change in the future. But for now, it is more important to get something up quickly that works well so we can test our designer's usability theories for solving real user problems.

HTML5 and Script Execution Order

As the latest browser renderers, WebKit, IE9 and Firefox 4, implement HTML5 there is wording in the HTML5 spec that breaks ordered execution of dynamic script elements.

What does this mean in practice? The RequireJS order! plugin and a core feature of LABjs breaks in the latest browsers under development. More info:

When a dynamically created script element is created and appended to the DOM (via document.createElement('script'), the behavior in the current browsers differs: IE and WebKit will execute the script as soon as it is delivered from the network, where Opera and Firefox will download the script as fast as it can, but execute the scripts as they are ordered/appended to the DOM.

The Firefox/Opera behavior is desirable for making existing scripts on the web go fast since all the scripts can be downloaded in parallel, but still execute in the order you specify -- there are many scripts today that assume their implicit dependencies have already been executed before the script executes.

jQuery plugins that assume jQuery is already available in the page are a good example.

Kyle Simpson of the LABjs project figured out a nice hack for IE and Webkit to get them to execute scripts in order. He gets IE and WebKit to download the scripts first without executing them (via a non-script script type, like type="text/cache"), then adding the real scripts to the DOM in order normally means they execute in order.

However, this technique has edge cases where it can fail, in particular with poor cache headers, and the latest version of the HTML5 spec effectively disallows the hack. The code to get ordered execution right now also has some browser sniffing to get it to work, which is clearly not ideal.

The ideal case is to have a capability that can be sniffed in the browser, and something that would allow for ordered execution of dynamically added script elements without the need of browser hacks.

Kyle has been working with HTML5 spec folks to try to work out something along those lines.

However, the HTML5 spec participants are not sure how important of a use case this is to support to warrant a spec change. I do think it is important for existing scripts to get some performance benefit.

While RequireJS does not rely on this behavior for its core operation (the define() function wrapper ensures that it is not an issue, and assumes out of order script execution), the order! plugin in RequireJS does depend on this behavior.

So, if you are a RequireJS order! plugin user and you really depend on it, or if you are a LABjs user, please take a moment to let the HTML5 group know the sites you work on that would be affected if those tools no longer worked in the browsers being developed today.

Kyle Simpson has set up a wiki page to describe the problem. There is a Discussion section of the page where you can voice your feedback. Click the + in the tabs at the top to add a section where you can list how this issue would affect your sites. Be sure to leave your name in case the group would need to get more information from you.

Please take a moment if you do depend on this feature to let the HTML5 group know.

RequireJS 0.14.5 Released

RequireJS 0.14.5 is available for download. Primarily a bug fix for the first item below.

• Fix bug where scripts were not loaded from the correct path. Did not affect RequireJS+jQuery builds, but affected other builds. If you do not use a RequireJS+jQuery build, then it is strongly recommended that you upgrade from 0.14.4 to 0.14.5.
• Added an urlArgs config option to allow for cache busting when servers/browser misbehave during development.

I apologize for the quick series of releases this week. I should be slowing them down a bit now.

RequireJS 0.14.4 Released, jQuery 1.4.3 support

RequireJS 0.14.4 is available for download. This release supports jQuery 1.4.3. What does this mean?

• The bundled RequireJS+jQuery file on the download page has jQuery 1.4.3 in it.
• Due to a change in jQuery, there are almost no patches to jQuery in the RequireJS+jQuery file, just a convenience patch to register it as a module.
  
• If your scripts properly use define() with RequireJS, then it is possible to load jQuery from the Google CDN. This can save you some bandwidth costs!

There are a few caveats with using RequireJS to load jQuery from the CDN:

• Most jQuery plugins assume jQuery is already loaded when they execute. However, with RequireJS, jQuery could still be loading when the plugin file executes. If you plan to use jQuery plugins in your project and you really want to load jQuery from the CDN, then in your own RequireJS-based project, you should wrap each of the plugins in a define(function(){ /*plugin goes here */ }); wrapper. Or, just stick with using the combined RequireJS+jQuery file served from your server.
  
• If you use code with define() calls in them, be sure to do the minification and combining of scripts using the RequireJS optimization tool. It will make sure the define() calls get proper scripts names to allow all the scripts to be combined together. If you use another tool to just concatenate your define()'d scripts together, it will result in errors if the define()'d modules are not named. A copy of the optimization tool is included in the jQuery+RequireJS Sample Project.
  
• This is only recommended if you have one version of jQuery loaded in the page.
  

An example showing how to configure RequireJS to load jQuery from the CDN:

require({
  "paths": {
      "jquery": "http://ajax.googleapis.com/ajax/libs/jquery/1.4.3/jquery.min"
  }
});

//Now require jQuery when you want it, using 'jquery'
//as the dependency name.
//HOWEVER, note the important caveats above!

require(['jquery'], function ($) {
   //...
});

Now if only I could get RequireJS releases on a CDN, that would be even sweeter!

RequireJS 0.14.3 Released

RequireJS 0.14.3 is available for download.

• Support for define(). It works the same as require.def(). It is supported in order to conform with the Asynchronous Module Proposal. require.def will continue to work, but you are encouraged to gradually migrate to define() for better compatibly with other Async Module loaders.
• text! plugin now works in Node.
• _dirname and _filename support in the r.js Node adapter.
• Bug fix for priority option.
• GPL license option removed: project is now just MIT and new BSD dual-licensed, since the new BSD license is compatible with the GPL.

The big motivation was the require.def() -> define() API change. I do not expect any more top-level changes to the API now, particularly when related to the Asynchronous Module proposal. There may be some API changes around how plugins are written at some point, but those are tricky to write today, and could use an API cleanup. However, using plugins in your application code will likely stay the same.

To be clear: require.def() will continue to be supported. If you only care about loading your modules via RequireJS, you can continue to use it. However, if you think you might want to allow your code to work in other script loaders that follow the Asynchronous Module API, then you should look at switching to define().

fn with no globals

This is a JavaScript sketch, please read the disclaimers.

Background

To get some decent code reuse, we need baseline support for loading code across environments. CommonJS made a good first pass with the Modules and Packages proposals. The Module one does not work well in the browser though, so I am hoping the Async Module proposal gains traction. Douglas Crockford likes async code loading too (around 39:10 in the video).

Relying on a server to generate client-friendly code is not a universal solution, and it is a particularly poor solution for doing mobile web development, where apps can be created with HTML/CSS/JS but run from the mobile device without a server.

RequireJS implements the Async Module proposal. There are some rough spots around the Packages proposal, particularly around mappings, but it is something that can be used, and I'm trying to support some version of packages with RequireJS.

The ECMAScript folks are looking at a Simple Modules spec, but it does not solve any of the problems for me. The problems have been addressed by RequireJS. It would be nice to not to have to include the code for RequireJS in an app, but it works, it is not crazy big, and it can always be pushed down to native environment support after it is broadly in use.

fn with no globals

What I would like to see instead of a Simple Modules is "fn". It is like "function" but it does not have access to the global space. You would use it just like "function" but any variables missing "var" would not be in the global space (could throw an error). If it needs a "use fn" or something like that, OK, but I am not familiar with when "use" strings are needed. Example:

fn hello() {
   //the next line would throw
   message = 'hello';

   //the next line would be an error too
   var name = window.name;

   return message;
}

That would give me one of the big things out of Simple Modules that I cannot already have via RequireJS. It would also shorten the traditionally cumbersome "function" word and encourage good coding practice since it does not have access to the global scope. For situations that need global access, fall back to good old "function".

I would not be surprised if this has already been suggested, but as mentioned in my sketches preamble, just jotting down things as they come to me without doing due diligence.

Web Workers as Browser/Server Bridges

David Ascher and I had a discussion last week about server side JavaScript. He mentioned that while server side JavaScript would give some small incremental advantage, there would still be a split between what server devs needs and do and what browser devs need and do. Other people have made a similar observation.

Here is one sketch on what server side JavaScript could bring.

Server-side JavaScript

There are two ways to treat the browser and server workload split . One treats the browser as a dumb client, the other as a smart client.

1) Dumb client

Blogs, news sites, wikis. Content Management Systems (CMS). Serve HTML strings to a browser. There is likely some browser interactivity, but the sites can get by without it via progressive enhancement. The server is responsible for generating the HTML from data mashed together with templates.

This use case is well understood problem space. It is not always executed well, but it has been done many times. Having server side JavaScript will help with some sharing of code/design approaches with the browser, but that is about it. Still, for me it would be nice to have SSJS solutions for these cases.

2) Smart client

Servers as APIs/data stores. The server is doing straight-out business logic, data manipulation. The data sent to the browser is just in JSON, preferably not XML.

Server-side JavaScript is useful here given the expressiveness of the language: closures/anonymous functions for callbacks make using async easier to use, and Node's focus on an event loop is a great fit here, a more natural experience for a browser developer to help out with this area.

Using Servers as Web Workers

This an extension of #2. First, go read up on Web Workers. Short review:

var worker = new WebWorker('http://example.com/some/thing');

//Get messages from the worker
worker.onmessage = onMessageFunction;

//Get errors from the worker
worker.onerror = onErrorFunction;

//Post messages to the worker, only
//JSON-compliant messages back and forth
worker.postMessage({"msg": "hello"});

What would be ideal is that instead of running the web worker in the browser, it would do its work on the server. To bootstrap, a small JS shim could be used that runs in the browser that does the server communication to run the code on the server, and then a Web Worker environment/toolkit/library on the server needs to be created. This node-worker project might be a good place to start for the server code.

The important point: an app developer codes the logic on the server to the Web Worker environment, using postMessage to send out responses, onmessage and onerror to receive responses.

The neat thing about this approach: it can be used for simple request/response actions, or for longer term, comet-style long-lived messaging. It also could allow for actually running or mocking the server endpoint in the browser.

There needs to be some way to manage state. It would be nice to "pause" and "resume" a server-based web worker. Maybe that is just cookies, but it has to be secure to things like CSRF, and still give the user the ability to clear that state like they can clear cookies today.

The other nice thing about this model, it fits in with the event loop that JS developers use today, and it is a tightly constrained environment. No messing about with "requests" and "responses" in the traditional server sense.

There are probably concerns about what it does to a REST approach to API development. I am hoping that it just transforms the REST calls into events routed to web workers, not sure how that will shake out yet.

This is a JavaScript sketch, please read the disclaimers.

JavaScript Sketches

I have a few JavaScript-related things I want to talk about. I prefer implementing something or give a proposal more consideration before talking about it. However, I find that I end up not talking about a lot of things that bounce around in my head. So I will start talking more about them, even though they are not fully formed. This blog post is to set up the context and disclaimers for those posts, so I can link back to them.

Disclaimers

I will make a bunch of declarations that I will not back up as much as they deserve. I may expand on them in later posts, but most likely I will not. The point of these sketches is to get the basic thought out instead of keeping it internal.

If I cast a negative light on your favorite project, keep in mind this is my blog, so it is naturally biased, and there always needs to be hedge bets. Until there are implementations with real world use, the future is always malleable, and often big enough to accommodate a few different views. Keep working on what you are passionate about.

I will be ruthless and what may seem like unfair in my comment management/deletion policy. You are free to say your own piece on your own blog.

Who am I?

I have been working with JavaScript since at least 1998. It was earlier than that, but that was the start of a large scale project that got real users. Since then I have used it on and off. I still need to learn more about the inner dark places of the language and its implementations, but I have used it to build some larger front-ends, particularly while at AOL: a picture service UI, a cross-browser plugin for streaming radio, a chat service UI, and helped out with a webmail service UI. I contribute to Dojo, make RequireJS, and work at Mozilla on web-based messaging services. I only want to develop for the web platform, using JavaScript. I did not study computer science, but physics, so I lack some of the CS technical underpinnings.

RequireJS 0.14.2 Released

RequireJS 0.14.2 is available for download. This is a bug fix release:

• Fix issue with "module" dependency not correctly assigning exported value if it is not listed as the last dependency.
• Fix "packages" config option. Its behavior was fixed to match the docs.
• Fix module-to-name resolution to account for package mappings.

These issues affect use cases with traditional CommonJS modules/converted modules or with CommonJS packages. If you do not deal with those use cases, there is less urgency to upgrade.

I recently started using the RequireJS code in Node beyond simple tests, working on a Node-based package tool. Once I get it up an running, I'll post more on this blog. In the meantime, there is a design sketch, and some code, but it is very rough at the moment, mostly scaffolding.

RequireJS 0.14.1 Released

RequireJS 0.14.1 is now available.

I just pushed a small update to fix three issues that dealt mostly with the new shortened anonymous module syntax used to wrap traditional CommonJS modules, and the converter tool for adding in the anonymous wrapper.

If you were using the regular RequireJS module format, with the dependencies specified outside the definition function, then you likely do not need to upgrade right away.

In fact, you may want to wait a couple days to see if there are any other updates. Given the newness of the anonymous module code and more people trying it with traditional CommonJS modules, I want to push out quicker releases to give those new users the best experience. I will be sure to mention if the update is recommended for all users.

There is one fix in this release for a type of deeply cyclic/circular dependency issue, but I believe it to be a rare issue for most current users. If 0.14.0 is working for you, then no need to try the latest version right away.