Custom local variables using the Dojo Loader

Recently, Dion Almaer was wishing for a way to avoid typing "dojo" so many times in the Bespin code. He appreciated the namespace protection that Dojo gave, but did not like the typing tax that came with it. He talked to Alex and Pete about it. Alex suggested a way, but it is very experimental.

Here is another way that leverages the strength of the Dojo Loader.

The Dojo Loader handles the work of loading your JavaScript modules. When you do a dojo.require("foo.bar"), the loader figures out that you want to load the foo/bar.js file and loads it for you. Normally the loader uses a synchronous XMLHttpRequest (XHR) call to load the file and uses eval() to bring the code into existence.

There is an xdomain loader that does not use eval, and allows your code to be loaded from any domain (with dependencies properly loaded), but it requires a build step using the Dojo build tools.

Both versions of the loader (normal and xdomain) allow you to namespace your code -- so you can map dojo to mydojo, and even your own code to some other name. This allows you to load multiple versions of dojo and/or your code in a page. Nifty.

The Loader is able to support this scope mapping by wrapping your module in a function call like so:

(function(dojo, dijit, dojox){
 //Module code injected in here
})(dojo, dijit, dojox);

We can use this new scope created by this function to create local variables for your module.

Dojo's Loader can be modified to allow you to specify a set of local variable names that get injected with this function wrapping. And we can do this on a per-module prefix basis, so your code can have your own local variables, but some other module can specify a different set.

I did a prototype that works as follows: assume "coolio" is the namespace I use for my modules. I create a coolio.locals that has the following content:

dojo.provide("coolio.locals");

dojo.setLocalVars("coolio", {
 trim: "dojo.hitch(dojo, 'trim')",
 $: "dojo.hitch(dojo, 'query')",
 id: "dojo.hitch(dojo, 'byId')"
});

This code will create local variables called trim, $ and id that map to dojo.query, dojo.byId and dojo.trim respectively, but only for coolio* modules.

Then I have another module, called coolio.actions that uses these variables:

dojo.provide("coolio.actions");

coolio.actions = {
 init: function(){
  $("#trimButton").onclick(coolio.actions, function(evt){
   id("trimOutput").value = trim(id("trimOutput").value);
  });
 }
}

Notice that trim, id and $ were not declared in here. The final bit of magic is to use djConfig.require in the HTML page to auto-load the coolio.locals module before any other coolio code, so that the loader knows to create the local variables for any coolio.* module:

<script type="text/javascript" src="dojo/dojo.js" djConfig="require: ['coolio.locals']"><script>

See it in action with this sample page.

I created ticket #9032 to track the possibility of allowing this in the future. It also has the patch that can be applied to the Dojo source to get it to work. Or, if you are using Dojo 1.3.0, you can grab these built files with the changes: dojo.js or dojo.js.uncompressed.js

There are some caveats to this prototype:
1) We really need a build step that would inline the local variables for your build layers, so the code as-is will not work with custom build layers.
2) Does not work with xdomain builds yet, another tweak to the build system is needed for that.

If you think this might be useful for you, feel free to add your comments to the issue tracker entry or leave a comment.

Job Transition

Friday was my last day at AOL, just shy of 13 years. There were good times, bad times. Some really cool things. Neat people and teachers. A great learning experience.

I am being a bit melodramatic, but Sinead O'Connor's Thank You for Hearing Me expresses the sentiment best, including the end of the song. For work and for life.

Monday I start at Mozilla Messaging. I am really excited about the work. It is still being defined, so nothing to share yet, but I look forward to helping people take charge of their messaging.

Namespaces, Subjects and Verbs in JavaScript

It has been interesting to follow Dion Almaer's comments about using Dojo in Bespin. This post mentioning call conventions higlighted something I have wanted to talk more about.

The basic issue is namespacing and how you like to call functions. Here are some choices:

1. subject.verb()
2. namespace.verb(subject)
3. namspace(subject).verb(), which can lead to namspace(subject).verb().verb().verb()

subject.verb()
#1 is the Prototype/Ruby way. You define a verb on the object's class, or in JavaScript, on its prototype.

In this model the namespace is really the subject's namespace. There is a possibility of collision with other code that wants to use the same verb. This can be mitigated by keeping your project small and focused, and managing your dependencies. This also works well when your code is a leaf node, or one step from the leaf node -- your code is not consumed by other code, except maybe a top level web application.

The benefit is a nice call structure, and I think fits better with normal English. The subject is identified and then a verb/action is performed with that subject.

Unfortunately, in JavaScript, there can be problems adding things to basic prototypes, like Array, String (and shudder, Object). In Dojo we have had to put in some protections in our code in case the page also uses code that modifies the built-in prototypes.

I believe the situation will improve in future JavaScript releases if added properties/verbs can be marked as not enumerable. That will help a bit, but namespace collision is still an issue.

Some browsers now have native String.prototype.trim implementations and Dojo now delegates to them for Dojo's trim method. Recently, there was a bug filed for Dojo where the core issue was some other code adding a String.prototype.trim() method that did not strip out beginning whitespace.

namespace.verb(subject)
#2 takes the position that verb collision is bad and can cause hard to trace bugs, so always use your own namespace. It also feels more "functional" or procedural: use small functions that do not maintain interior state but operate on their arguments.

This has the benefit of being safer, but it can be more verbose than #1, and therefore more of a constant tax on the developer.

This is mitigated somewhat in Dojo, where you can assign Dojo to another namespace. So you could map "dojo" to "$" to cut down on the typing.

namspace(subject).verb()
#3 is a nice compromise: Define a function that wraps the real subject in an object and provides verbs to act on that subject, without directly modifying the object/class structure of the subject. jQuery took this to new heights by allowing chaining of the verbs. Similarly, dojo.query() returns a dojo.NodeList object that has chainable verbs on it.

An explicit namespace is involved, but the idea is to make it as small as possible, "$". So the overhead for having the namespace is "$()". Coupled with the chainable verbs, it can lead to short code, less of a tax on the developer.

For Dojo, I think #3 is the right approach in general: it gives nice namespace protection, but still gives short call structures.

I want to explore a dojo() function that does this verb mapping in general for all of Dojo. Dijit might be able to use a dijit() to get something similar for the verbs it exposes in its namespace.

So for instance: dojo({foo: "bar"}).clone() would act the same as dojo.clone({foo: "bar"}). Scopemap dojo to $ and you get $({foo: "bar"}).clone().

The difficult part is how to deal with verbs/methods that deal with Strings. I want dojo("div") to actually call dojo.query("div"). But how to allow dojo(" some string ").trim()?

Maybe not map dojo("string") to dojo.query("string"), but instead, allow scope mapping of "d" (or even "_"?) to dojo and another mapping of "$" to dojo.query. That would probably match best with the expectations of what $ does today in other toolkits.

I will have to do more exploration. Eugene Lazutkin's work on providing adaptAs* functions for mixing in dojo methods into an object prototype as chainable methods might point the way to do this.

Dojo 1.2 Loader and Build System

Dojo 1.2 is now available! Check out the release notes for more detailed information.

I attended the Dojo Developer Day in Boston on Sunday 9/28 and the Dojo Community Day at the Ajax Experience conference on Monday 9/29. It was really nice of the Ajax Experience folks to set up community time for the JavaScript toolkits.

During the Community Day, I gave a short presentation on some new things in Dojo 1.2 with the Dojo loader and the build system. I tried using the Google Docs presentation thing. You can see the slides here (some notes are after the slides):

customBase build option

The final example in the slides demonstrates how you can create a 5.5 KB (gzipped) dojo.js file and dynamically load it after the page loads, so if your site uses progressive enhancement, your up-front Dojo cost can be zero bytes, and just load as little as possible for the functionality that you use.

The 5.5 KB dojo.js file is possible using the "customBase" property of the dojo.js layer in a build profile. Normally we encourage you not to tamper with the contents of dojo.js (what is referred to as Dojo Base), but the customBase build option can give you more control on what parts of Dojo Base you want to load. With the customBase option, the build will go through all the JS files and automatically add dojo.require() calls for the necessary dojo._base modules.

Only use customBase if you keep all your JS code in JS modules -- good practice anyway, and keeps with progressive enhancement suggestions. Do not use customBase if you have Dojo calls inside HTML source.

Using customBase on the "dojo.js" layer with no dojo._base modules (just the loader and some bootstrap functions), the non-gzipped size of  dojo.js comes out to be 13KB (recall from above it is 5.5 KB gzipped). So it is small enough to fit under the 25KB non-gzipped size limit that the iPhone/iPod touch use to allowing caching.

You will likely need to use more of Dojo Base than what is in the 13KB non-gzipped file, but you can tune that by creating more layers and playing with the build to so that all layers are under 25KB.

djConfig.addOnLoad

Another option for use with progressively enhanced web sites, or when you want to load Dojo after page load: djConfig.addOnLoad allows you to set a function to run after Dojo loads. It is most useful when used in conjunction with djConfig.afterOnLoad (used when you manually load Dojo after page load).

Here is a complete example showing djConfig.addOnLoad, djConfig.afterOnLoad and djConfig.require.

There is a bug about using djConfig.require in conjunction with a dojo.js generated via a customBase build, so only use djConfig.require when using a complete Dojo Base file.

Thanks go to Matthew Russell for suggesting djConfig.addOnLoad for the 1.2 release.

stripConsole
There is a new build option for stripping out console method calls. stripConsole=normal strips out all console.* calls except console.error and console.warn. stripConsole=all strips all console.* calls.

Safari 3.1 sometimes has an issue with console.debug not getting defined (fixed in webkit trunk), so this build option can help avoid that issue until Safari is fixed.

Mobile Development

I was recently asked about my experiences making TinyBuddy IM. Here are my very biased thoughts about it and mobile development in general.

My Background
I like to make web products that people use. I am a front-end developer, and I love using JavaScript, HTML and CSS. I have done some C/C++ for a cross-browser streaming audio browser plugin. I recently played with Objective C as part of an iPhone SDK experiment. I did Java on the server for a few years too.

HTML, or some sort of declarative markup, is the way to code front end UI. It clearly shows the nested relationships between components. However, I liked the Interface Builder approach in the iPhone SDK, very WYSIWYG. It still some kinks to work out with the beta iPhone SDK though.

I also like trying to reach the largest possible audience with the shortest toolchain. To me, this means favoring an HTML/CSS/JavaScript UI first, and only considering other alternatives when that approach will not work. Particularly since I favor developing applications for the internet.

One Mobile Renderer to Rule Them All
I did not do mobile development until TinyBuddy IM. It just seemed too painful -- too many custom environments, too many cumbersome toolchains for native apps. And the browsers really sucked.

However, now with WebKit on the iPhone and in Android, it is tolerable. Those platforms also have a decent toolchain for "native" apps: iPhone SDK/Objective C for iPhone, and a Java environment for Android.

However, WebKit is most interesting to me: it is a modern, very capable browser that will be on two major mobile platforms. While JavaScript performance is still slow on mobile browsers (both compared to desktop browsers and native code), there is promise that it will get faster with WebKit's SquirrelFish.

For the first time, it is actually possible to consider making a powerful web app for a mobile device. You have good odds of developing one mobile app that runs on multiple devices too. Or at least have lots of code overlap.

Of course a web app is not appropriate for every app, and it does not have access to as many services as a native app, but it does have a lot. With the latest WebKit for instance, you will have local storage.

You Are Dead to Me
Not every mobile device has WebKit. What about Windows Mobile devices, BlackBerry, or all the other devices out there in the market today? Well, for me, they do not make it easy to develop. And I am lazy. I use an iPhone, so I am going to develop for what I use. However, I chose an iPhone specifically because it is easy to do development.

So the other platforms are not that important to me, and if my developer laziness is any indication, they are in for some trouble. They need to get WebKit or Gecko on their devices and use a better toolchain. Opera's browser may be an option too, but I am not clear if it is up to the capabilities of WebKit. If it is, great. I like the Opera on the Wii.

Again, it may not be reasonable to discard these other platforms, depending on your app, but I have decided the other platforms are not worth the effort, given my interests. I really want to make interesting end user products without spending time on platform minutia, and I feel like the iPhone's and Android's reach will be good enough for me.

Mobile Application Design
I approach the technical design of a mobile app by trying to see if the following approaches fit the problem:

1. A purely browser-based web app.
2. A native app that gives a "desktop" icon, and access to phone services (for instance, location services), but use a WebKit view for the main UI interface. Set up hooks in the WebKit view so it has access to the phone services.
3. A purely native application.

#2 has many shades to it, but I like it because it minimizes installed code. Revving a native application is more work than a server-based one.

Remember, I'm assuming this is an internet app, and you know the HTML/CSS/JavaScript trifecta already.

There are tradeoffs. Speed of loading/dependence on network, but again, I prefer to first look at a web-based solution first: maybe it makes sense for the native app to cache the HTML/JS/CSS/images locally.

Native Apps
What you want to build may not work as a web app. For those cases I still prefer iPhone or Android.

For iPhone, you have a nice dev environment with a WYSIWYG UI tool (Interface Builder). Objective C feels a lot like JavaScript to me -- some concept of a prototypical object, and runtime, dynamic dispatch where you can send messages to objects that may or may not respond to that type of message. Unfortunately you have to do the memory retain/release thing, but it is manageable. Lots of helpful videos and examples at the iPhone Developer site too.

For Android, using Java syntax and libraries are a big plus over some sort of C/C++ thing. Java can be verbose, but it is easy to program with an IDE, and hopefully they kept in the garbage collection like Java. That alone is a big win. I have not used Android yet, and I have heard of problems with trying to program in that environment (maybe they have been worked out now?), but it seems worth spending effort on that path. It seems like it has a good shot of survival over the long run.

UI Design
There is a great intro video on the iPhone Developer site about "User Interface Design for iPhone Applications". You will need a (free) Apple Developer ID to view it. But take the time to view it.

The main point: a mobile app is not a desktop app. Do not try to do too much. For TinyBuddy IM, I decided to not include buddy list management features in the UI (add/remove/move buddy). The app was designed for doing quick IMs with your buddies. Buddy list management just seemed to get in the way of that goal. Not to mention the lack of copy and paste on the phone. It just did not seem like a worthwhile activity for a mobile app.

Also, use the UI paradigms available on the phone. For native apps, Apple has a lot of built in templates for using the standard lists and tables you see throughout the iPhone UI. I used a very early version of iUI for a web toolkit that gives an approximation of the native iPhone UI. There may even be some resources in the latest iPhone SDK to make web development easier in this respect.

So, keep it simple, keep it familiar. Once you get the fundamentals down and released something, you can consider your own style (if appropriate).

Pain Points
It is not all roses. There are notable restrictions making mobile web apps. These limitations are specifically from the iPhone, but I would expect them to be roughly the same if not more oppressive in Android environments.

• Resource limits.
• Cache limitations (25KB max size, uncompressed, if you want it cached, 19 total cacheable items).
• As mentioned above, JavaScript performs slower than desktop JavaScript or native code.
• You web app is paused when it is not in the forefront (also a limit for native apps, but native apps will have a notification service you can use that will help with this).

That last one is particularly vexing for TinyBuddy IM, since it is an IM client. So a native app for the IM problem space is probably more appropriate. Or some sort of native component that ties into to the notification service? :)

TinyBuddy IM Weaknesses
I need to spend more time optimizing the TinyBuddy IM code:

• I do not do any sort of "Fetch next 25" sort of thing to keep the buddy list size down.
• The wiping between screens is slow.
• The IM conversation window needs more polish.
• Maybe some better navigation for multiple IM conversations.

So I definitely did not get it right with TinyBuddy IM. Now that there is a native IM app under development, I probably will not rev TinyBuddy IM again. However, I am amazed TinyBuddy IM works as well as it does -- I was able to be a little bit lazy, and still get something somewhat useful out. A good indicator for that platform.

Authentication
Most things I want to build require user authentication for at least some part of it. For TinyBuddy IM, that meant using AOL's OpenAuth authentication, since the Web AIM API used it. OpenAuth has a browser-based authentication system that any web page can use, and it even supports consuming some validated OpenID providers.

OpenAuth has a "clientLogin" API for desktop clients and Flash/AIR apps but I do not prefer it since it basically means the user hands over their name and password to your app, and then your app sends it to OpenAuth. While that might be nice from a usability standpoint, it bothers me personally from a security standpoint.

This is where something like OpenAuth's browser API (and in general OpenID in the browser) provides better protection. You only enter your name/password on the identity provider's web site, and a token is passed back to the web app that requires your credentials.

As an end user of this system, I can verify by the browser URL who is asking for my name/password. As a developer, I like just not having to touch passwords at all. It absolves me from the security issues with trying to manage passwords, even if it seems like I am only passing the password through to OpenAuth's clientLogin.

However, most end users are not this particular about using applications, they are happy to give their password to any site asking for it, particularly if it seems to be done in goodwill. They just want to use the app. Also, the browser based authentication can be weird -- popping a new window to get the auth credentials.

But the OpenID approach is the way of the future (going to another site to authenticate), so might as well get used to it. Particularly now that the OpenAuth authentication page has a "Remember Me" option, so the user does not have to keep entering name/password on every visit.

Have Fun!
Enough of my ramblings. The big thing for me: mobile development can be fun, particularly with the iPhone platform. Android, while not at the same level as iPhone development yet, also seems like it has promise too.

The Beauty of Dojo 1.1

The Dojo JavaScript Toolkit version 1.1 is available:

• Release Notes
• New API tool (ability to add comments coming next week)
• Dojo Book
• AOL CDN info
  
• Download Area (but try the CDN for easy use without needing to download dojo to your server. Save this sample page, and start playing.)
• Follow Dojo. Subscribe to this planet feed for blogs and help sites that talk about Dojo and feeds from browser vendors.
• Key Links for some demos and articles that explore Dojo.
  

There are lots of goodies in this release (see the release notes for full details), but here are some of the new features that resonate with me. I am primarily a Dojo Core user, not much Dijit or Dojox. For other perspectives on the new features, see the 1.1 blog announcement, and Shane O'Sullivan's blog post.

Multiversion Support
One of the slicker features (IMO, since I added it) is multiversion support: you can now run Dojo 1.1 with other versions of Dojo in the page without conflicting. You can also choose to rename dojo, dijit and dojox to other names. As proof:

• A demo showing the 0.4.3 Calendar with the Dojo 1.1 Calendar
• A neat dojox.gfx/charting demo showing dojo, dijit and dojox remapped to omega, omegaw and omegax, respectively

The multiversion support is great if you want to provide your own JS library, but use Dojo underneath. What is even cooler: multiversion support works from the CDN (see the source for the examples above)! So if you are thinking of migrating from an older Dojo version, you can experiment with Dojo 1.1 in a new namespace without having to download 1.1.

Load Dojo after page load
Dojo 1.1 can can be loaded after page load (after the window.onload event fires) by setting djConfig.afterOnLoad to true. This makes the initial render cost for using Dojo near zero, and it plays nice if you want to do extreme progressive enhancement. Use the new djConfig option in conjunction with djConfig.require, to load dojo along with the modules you needed dojo.required after dojo loads. See the the demo page for an example.

Adobe AIR support
Dojo now provides strong support for AIR in addition to Dojo's existing integration with Google Gears via dojox.offline.

Client-side data storage via dojox.storage
If you want client side data storage, dojox.storage gives you a few options, and auto-detects the best one. dojox.storage has been updated to allow for using Dojo Gears, HTML 5 DOM storage, Flash or AIR DB storage.

Build system CSS optimizations
The build system will now inline @import calls that are in .css files in addition to stripping comments and whitespace in .css files. See the New Build Options section.

Try it out!
If you would like to try Dojo, but are not sure how to start, you can always start small: If you just use Dojo Base (one file, dojo.xd.js, from the CDN), you get a solid JavaScript base that makes building progressively enhanced sites much more enjoyable. A rough run-down of the Dojo Base functionality can be found here. That article is for Dojo 0.9, but it still applies for 1.1. Dojo 1.1 is slightly larger than the numbers quoted for 0.9: The 1.1 dojo.xd.js is 29KB gzipped. See the 1.1 release notes for the new features that account for the size increase.

I prefer to stick with Dojo Base, with some additions from the Core modules (like dojo.io.script, which allows using JSONP APIs, like the ones provided by Web AIM). This is how I approached development for the AIM Chat web site and for the iPhone IM web app TinyBuddy IM. But given Dojo's depth, I was able to leverage the Dijit widgets to create a simple admin site for AIM Chat by adding in a few more dojo.require calls. Sweet!

I am really amazed at the amount of work that has gone into Dojo 1.1. The more I use it, the more I feel it is the complete JavaScript toolkit solution. You can do the small, quick progressively enhanced web sites using just Dojo Base from the AOL CDN, but scale all the way up to very rich experiences that use the full power of widgets, awesome data stores, offline storage, and incredible 2-D drawing capabilities (with charting!).

I feel the Dojo community is really hitting their stride now. Great job everyone!

Browser and Dojo updates on Fragment ID messaging

Some browsers have changed how they deal with fragment ID messaging:

Safari 3 throws an error if a child frame tries to change a parent frame's location, but only if the parent frame is not the top-most frame. So, this test works, but this one fails. Parent frames are allowed to change a child frame's location.

Opera 9.25 has locked down cross-frame access to some frame properties. As Julien LeComte pointed out, Opera supports the HTML5 cross-document messaging API, so that is a viable alternative.

I have updated dojo.io.proxy so that it now works in these browsers: IE 6 and 7, Firefox 2 and 3beta2, Safari 3 and Opera 9.25. You can get it from the Dojo development trunk, or wait for Dojo 1.1. The new dojo.io.proxy code works with Dojo 1.0, and there are no API or usage changes.

This new code also improves the IE7 codepath: before the code used to use 3 frames to do the transport, but now it only uses 2 frames, like the rest of the browsers. I was not being very smart about the ordering of the frames in the old code.

Thanks go to Julien LeComte for pointing out the Opera cross-document API and for diagramming another way to do fragment ID messaging.

dojo.io.proxy differs from Julien's CrossFrame approach:

• it allows very large messaging (it supports fragment ID chunking)
• it is focused on providing a data transport -- it provides an XMLHttpRequest facade. It is not a communication transport for UI widgets.

Julien did not want to use location polling, since it increases CPU usage. dojo.io.proxy does use polling to do the communication (this allows for chunking to support large messages), but polling is active only while the message is in transport. I consider the window.location polling less resource-intensive than doing a DOM animation. CPU usage could be tuned by slightly extending polling interval if it is a concern.

dojo.io.proxy creates the frames only when the request starts, then it destroys the frames when the response finishes. This preserves the history stack in all browsers except Opera.

There is a very primitive test page if you want to see the latest code in action.