Raindrop, CouchDB and data models

Raindrop uses CouchDB for data storage. We are starting to hit some tough issues with how data is stored and queried. This is my attempt to explain them. I am probably not the best to talk about these things. Mark Hammond, Raindrop's back-end lead is a better candidate for it. I am hoping by trying to write it out myself, I can get a better understanding of the issues and trade-offs. Also note that this is my opinion/view, may not be the view of my employer and work colleagues, etc...

First, what are our requirements for the data?

• Extensible Data: we want people to write extensions that extend the data.
  
• Rollback: we want it easy for people to try extensions, but this means some may not work out. We need to roll back data created by an extension by easily removing the data they create.
  
• Efficient Querying: We need to be able to efficiently query this data for UI purposes. This includes possibly filtering the data that comes back.
  
• Copies: Having copies of the data helps with two things:
  
  • Replication: beneficial when we think about a user having a Raindrop CouchDB on the client as well as the server.
  • Backup: for recovering data if something bad happens.

How Raindrop tries to meet these goals today

Extensible Data: each back-end data extension writes a new "schema" for the type of data it wants to emit. A schema for our purposes is just a type of JSON object. It has a "rd_schema_id" on it that tells us the "type" of the schema. For instance a schema object with rd_schema_id == "rd.msg.body" means that we expect it to have properties like "from", "to" and "body" on it. Details on how schemas relate to extensions:

• An extension specifies what input schema it wants to consume, and the extension is free to emit no schemas (if the input schema does not match some criteria), or one or more schemas.
• Each schema written by an extension is stamped with a property rd_schema_provider = "extension name".
• All the messages schemas are tied together via an rd_key value, a unique, per-message value. Schemas that have the same rd_key value all relate to the same message.

More info is on the Document Model page.

Rollback: Right now each schema is stored as a couch document. To roll back an extension, we just select all documents with rd_schema_provider = "extension name" that we want to remove, and remove them. As part of that action, we can re-run extensions that depended on that data to have them recalculate their values, or to just remove the schemas generated by those extensions.

Having each schema as a separate document also helps with the way CouchDB stores data -- if you make a change to a document and save it back, then it appends the new document to the end of the storage. The previous version is still in storage, but can be removed via a compaction call.

If we store all the schemas for a message in one CouchDB document, then it results in more frequent writes of larger documents to storage, making compaction much more necessary.

Efficient Querying: Querying in CouchDB means writing Views. However, a view is like a query that is run as data is written, not when the UI may actually want to retrieve the information. The views can then be very efficient and fast when actually called.

However, the down side is that you must know the query (or a pretty good idea of it) ahead of time. This is hard since we want extensible data. There may be some interesting things that need to be queried later, but adding a view after there are thousands of documents is painful: you need to wait for couch to run all the documents through the view when you create the view.

Our solution to this, started by Andrew Sutherland and refined by Mark, was to create what we call "the megaview". It essentially tries to emit every piece of interesting data in a document as a row in the view. Then, using the filtering capabilities of CouchDB when calling the view (which are cheap), we can select the documents we want to get.

Copies: While we have not actively tested it, we planned on using CouchDB's built-in replication support. This was seen as particularly valuable for master-master use cases: when I have a Raindrop CouchDB on my laptop and one in the cloud.

Problems Today

It feels like the old saying, "Features, Quality or Time, pick two", except for us it is "Extensible, Rollback, Querying or Copies, pick three". What we have now is an extensible system with rollback and copies, but the querying is really cumbersome.

One of the problems with the megaview: no way to do joins. For instance, "give me all twitter messages that have not been seen by the user". Right now, knowledge of a message being from twitter is in a different schema document than the schema document that knows if it has been seen by the user. And the structure of the megaview means we can really only select one property at a time on a schema.

So it means doing multiple megaview calls and then doing the join in application code. We recently created a server-side API layer in python to do this. So the browser only makes one call to the server API and that API layer does multiple network calls to CouchDB to get the data, then does the join merging in memory.

Possible solutions

Save all schemas for a message in one document and more CouchDB views
Saving all schemas for a message in one document makes it possible to then at least consult one document for both the "type=twtter, seen=false" sort of data, but we still cannot query that with the megaview. It most likely means using more CouchDB views to get at the data. But views are expensive to generate after data has been written. So this approach does not seem to scale for our extensible platform.

This approach means taking a bit more care on rollbacks, but it is possible. It also increases the size of data stored on disk via Couch's append-only model, and will require compaction. With our existing system, we could consider just never compacting.

This is actually the approach we are starting to take. Mark is looking at creating "summary documents" of the data, but the summary documents are based on the API entry points, and the kind of data the API wants to consume. These API entry points are very application-specific, so the summary document generation will likely operated like just another back end extension. Mark has mentioned possibly just going to one document to store all schemas for a message too.

However, what we have not sorted out how to do is an easier join model: "type=twitter and seen=false". What we really want is "type=twitter and seen=false, ordered by time with most recent first". Perhaps we can get away with a small set of CouchDB views that are very specific and that we can identify up-front. Searching on message type and being seen or unseen, ordered by time seems like a fairly generic need for a messaging system.

However, it means that the system as a whole is less extensible. Other applications on the Raindrop platform need to either use our server API model of using the megaview then doing joins in their app API code (may not be so easy to learn/perform), or tell the user to take the hit waiting for their custom views to get up to date with all the old messages.

Something that could help: Make CouchDB views less painful to create after the fact. Right now, creating a new view, then changing any document means waiting for that view to index all the documents in the couch, and it seems to take a lot of resources for this to happen. I think we would be fine with something that started with most recent documents first and worked backwards in time, using a bit more resources at first, but then tailing off and doing it in the background more, and allow the view to return data for things it has already seen.

Do not use CouchDB
It would be very hard for us to move away from CouchDB, and we would likely try to work with the CouchDB folks to make our system work best with couch and vice versa. It is helpful though to look at alternatives, and make sure we are not using a hammer for a screwdriver.

Schema-less storage is a requirement for our extensible platform. Something that handles ad-hoc queries better might be nice, since we basically are running ad-hoc queries with our API layer now, in that they have to do all the join work each time, for each request.

Dan Goldstein in the Raindrop chat mentioned MongoDB. Here is a comparison of MongoDB and CouchDB. Some things that might be useful:

• Uses update-in-place, so the file system impact/need for compaction is less if we store our schemas in one document are likely to work better.
• Queries are done at runtime. Some indexes are still helpful to set up ahead of time though.
• Has a binary format for passing data around. One of the issues we have seen is the JSON encode/decode times as data passes around through couch and to our API layer. This may be improving though.
  
• Uses language-specific drivers. While the simplicity of REST with CouchDB sounds nice, due to our data model, the megaview and now needing a server API layer means that querying the raw couch with REST calls is actually not that useful. The harder issue is trying to figure out the right queries to do and how to do the "joins" effectively in our API app code.

What we give up:
1) easy master-master replication. However, for me personally, this is not so important. In my mind, the primary use case for Raindrop is in the cloud, given that we want to support things like mobile devices and simplified systems like Chrome OS. In those cases it is not realistic to run a local couch server. So while we need backups, we probably are fine with master-slave. To support the sometimes-offline case, I think it is more likely that using HTML5 local storage is the path there. But again, that is just my opinion.

2) ad-hoc query cost may still be too high. It is nice to be able to pass back a JavaScript function to do the query work. However, it is not clear how expensive that really is. On the other hand, at least it is a formalized query language -- right now we are on the path to inventing our own with the server API with a "query language" made up of other API calls.

Persevere might be a possibility. Here is an older comparison with CouchDB. However, I have not looked in depth at it. I may ask Kris Zyp more about it and how it relates to the issues above. I have admired it from afar for a while. While it would be nice to get other features like built-in comet support, I am not sure it will address our fundamental issues any differently than say, MongoDB. It seems like an update-in-place model is used with queries run at runtime. But definitely worth more of a look.

Something else?

What did I miss? Bad formulation of the problem? Missing design solution with the tools we have now?

Blade, a JavaScript toolkit experiment

I am playing around with a different way (at least for me) to construct a JavaScript toolkit. It is called Blade, and you can follow it via the Blade GitHub repo.

I have had this on my local drive for a few weeks now, and the germ of it started with this post. I wanted to get it more polished, but best to get it up somewhere to get some feedback at least on the principles.

There is not much there now, basically a tiny amount of spaghetti code that is not really usable. However, I list out the guiding principles in the README.md, visible on the GitHub source tab.

Raindrop, Open Messaging for the Open Web

I work for Mozilla Messaging, and we just opened the doors on Raindrop. Raindrop is the reason I had the opportunity to move to Vancouver, BC. It has been fun building it, seeing if we could get something to work.

Raindrop is still very much an experiment and not useful for any day-to-day work. However, it has potential and we need community help to take if further.

Why I like it:

• It is driven by product design. We want an extensible platform, but a strong, simple product design will be driving much of the development.
  
• It is not trying to be a message service in itself, but collect messages from existing services.
• It is web-based: the default UI is plain HTML/JavaScript/CSS goodness.
• It is frickin awesome to be able to play with your messages: data mine them, and do interesting display things using simple script languages like JavaScript and Python.
• It is open: open source and motivated by the Mozilla Manifesto.
• The Mozilla Messaging team is talented and smart. They are motivated and care about what is best for users.
  

I am driving the front end development for Raindrop. I used Dojo to create the infrastructure for the pages, using Dijit's dijit._Widget and dijit._Templated as a base for many of the UI widgets.

Dojo's dynamic code loader and Dijit's well-defined methods on widgets have enabled the slick things we are doing with in-place extension editing and updates. JQuery is also included in the page, mostly for extension developers, so you have a choice on what to use, Dojo or JQuery.

While I expect many of the decisions I made about how the front end works might change over time, it has been a joy to make what is there so far.

There is lots to do though. If you want to get involved with the code, check the Hacking page is a good place to get started. There is a screencast of the architecture on the Raindrop home page. There is a Community page too.

RunJS updated: module modifiers and function modules

I pushed some changes to allow lazy-loaded/lazy-evaluated module modifiers and also better support for modules that just define a function. The changes are documented in the documentation page.

I am experimenting with using JSLint as a code formatter. We'll see how it goes.

All these changes bring the size up to 3.1 KB minified and gzipped. I would like to be under 3 KB, but I want to be sure the right functionality is in place first before squeezing it down.

The module modifiers are a bit of an experiment. I wanted some way to separate a bunch of bad, wordy code out for the normal cases of a module but only in bad cases load the bad code. The example I give in the documentation is a module that gets DOM node dimensions and position. In standards mode, it is fairly compact, but in quirks mode it gets uglier. So I only want to load the quirks mode code if the page is in quirks mode. I never want to develop in a quirks mode page, but for a general JavaScript library it might be important.

So I am still not sure if the module modifier approach is the right way to go, but I have used it a little bit so far on another project, and I will see how that works out.

RunJS updated: simple modules and i18n bundles

I pushed some changes to RunJS, the stand-alone JavaScript file/module loader.

Newest changes are support for simple module definitions and internationalization (i18n) bundles.

Simple module definitions are possible when there are no dependencies for the module. In that case the function wrapper for the module is not needed:

run(
    "my.simplething",
    {
        color: "red",
        size: "large"
    }
);

i18n bundle support is handy for separating out strings that might need to be translated into other languages. Quick example for a my/nls/colors.js that will define a bundle:

run(
    "my.nls.colors",
    [{
        "root": {
            "red": "red",
            "blue": "blue",
            "green": "green"
        },
        "fr-fr": "my.nls.fr-fr.colors"
    }]
);

Then define a file at my/nls/fr-fr/colors.js that has the following contents:

run(
  "my.nls.fr-fr.colors",
  {
    "red": "rouge",
    "blue": "bleu",
    "green": "vert"
  }
);

See the documentation for more information.

If you want to try out the latest code, you can use one of the following URLs to fetch the code:

• Minified version
• Full commented version

In addition to the documentation, some of the test files might be of interest to see how RunJS can be used.

RunJS is still nice and small with these new features, around 2.7KB, when minified via YUICompressor and gzipped.

RunJS

I just created what I hope to be the next generation JavaScript module loader here: RunJS. The documentation has some history and reasoning behind it.

I would like to see it used for any project that needs JS code loading, particularly since it handles dependencies, can load regular JavaScript files, handles multiple versions of modules, and is a compact 2.2KB (minified and gzipped). It is JavaScript toolkit-agnostic and has no dependencies.

Enjoy!

Chrome Frame, or Browser vs. Renderer

Chrome Frame came out yesterday. I like the conversation it is trying to start. It makes a big difference having working code to get the conversation to seriously happen. There are some mechanics of the specific approach that probably need tweaking, but the general idea is worth consideration.

The big point is about separating the innovation over organizing the user's experience of the web in general (the Browser) vs. innovation in the display of web sites (the Renderer).

The Chrome Frame approach allows browser makers to preserve their revenue models and their UI interaction models, at least in the ideal -- there are some specific issues to work out with the Chrome Frame model, like password/form data storage. But I think the direction is a good one.

It also solves the issue where a web application was developed a few years ago and is not going to be maintained any more. If the browser allows multiple rendering engines it makes it possible for those old web apps to continue to work.

Old enterprise/business applications in particular can continue to work, but we still get to move the new web experience forward.

From a browser market share perspective, this clean split of responsibility could allow a newer browser like Google Chrome to get more market share. It could use the reverse idea: embed the IE render engine in Google Chrome. Then, take that to all the IT administrators and say, here is a newer, safer, more secure browser for your company that will work on older machines, is free, and can still allow your old business web apps to work.

That may not work, since most people are attached to their browser chrome vs. the actual renderer. In those cases, the plain Chrome Frame approach of installing a plugin for another renderer works to allow better, faster web apps experiences.

One concern: user choice or control. I think it actually leads to more user control: the user gets to keep the browser chrome, their organizing model for the whole web, intact, but gets to use more of the better parts of the web. And for areas where the user does not have control now (in business environments that need old apps to work), it gives a way out to use more of the web.

Of course, there should user controls so they can set preferences and overrides for the renderers used. At a minimum, business IT groups will need it to configure the browsers to use old renderers for older in-house business web apps.

So, some tweaks to the basic mechanics I would like to see (realizing that I have no concept how hard this work would be):

1) Make sure the Renderer works well with the Browser: for instance make sure that saved passwords/form data works well no matter what Renderer is used. Make sure the split is clean.

2) Change the UA-Compatible thing to be more Renderer-based feature sets vs. browser version numbers. So, something where the developer mentions the capabilities that are desired:

<meta equiv="X-UA-Compatible" content="addEventListener=1,svg=1">

If the developer has a suggested browser renderer, it could place that at the end, sort of how CSS font names can start with generic names, then get more specific:

<meta equiv="X-UA-Compatible" content="addEventListener=1,svg=1;gecko=1.9.2">

It would be good to *not* use browser names in the tag, but rather the renderer engine names/versions. Ideally though, just list capabilities.

The above syntax is not exactly right, but just to demonstrate the idea: focus on telling the browser the capabilities the page wants, and use render engine names, not browser names.

3) Make sure the user can override the choices made by the browser. The pref control does not have to be obvious, but should be there, so the user has the final say.

In summary, as a conversation starter, I like that Chrome Frame has really tried to highlight the difference between upgrades in renderers vs. browser interface.

Developer Tools and JavaScript Syntax Checking

I was using Coda to do web development, primarily because of the simple interface, nice Panic Sans font and integrated FTP and Subversion support. I normally just use TextWrangler for Dojo Core development.

However, while doing development at the new job, FTP/Subversion support was not needed, and I have grown tired having to run my JavaScript in the browser to find basic syntax errors.

I saw David Ascher using Komodo Edit, the free editor from ActiveState, which has JavaScript syntax checking built in. Searching in the Tools, Add-Ons menu for Dojo also brought up a Dojo API Catalogs extension that allowed for autocomplete of Dojo APIs.

Komodo Edit has been mentioned in the Dojo community before as a nice option particularly for doing Dojo development, but I can get severe tunnel-vision, so it was not until recently that really tried out Komodo Edit.

After I changed the key bindings to use the normal OS X COMMAND+] and COMMAND+[ for multi-line indenting, I was ready to go!

It makes a difference having syntax checking and Dojo API autocompletion. I believe other tools, like Emacs with js2-mode, can get at least syntax checking, but I am not an Emacs user. It would be neat to see Coda (and TextWrangler) add JavaScript syntax checking support. I can see JavaScript syntax checking being a minimum requirement for JS coders going forward.

As Bespin comes along, I can see it as my next developer tool upgrade, particularly since FTP support has been implicitly needed for some of my projects. With Bespin, I might be able to edit directly on the server.

For now I am enjoying the bump in productivity with Komodo Edit. Thanks ActiveState for making a nice tool available for free!

Custom local variables using the Dojo Loader

Recently, Dion Almaer was wishing for a way to avoid typing "dojo" so many times in the Bespin code. He appreciated the namespace protection that Dojo gave, but did not like the typing tax that came with it. He talked to Alex and Pete about it. Alex suggested a way, but it is very experimental.

Here is another way that leverages the strength of the Dojo Loader.

The Dojo Loader handles the work of loading your JavaScript modules. When you do a dojo.require("foo.bar"), the loader figures out that you want to load the foo/bar.js file and loads it for you. Normally the loader uses a synchronous XMLHttpRequest (XHR) call to load the file and uses eval() to bring the code into existence.

There is an xdomain loader that does not use eval, and allows your code to be loaded from any domain (with dependencies properly loaded), but it requires a build step using the Dojo build tools.

Both versions of the loader (normal and xdomain) allow you to namespace your code -- so you can map dojo to mydojo, and even your own code to some other name. This allows you to load multiple versions of dojo and/or your code in a page. Nifty.

The Loader is able to support this scope mapping by wrapping your module in a function call like so:

(function(dojo, dijit, dojox){
 //Module code injected in here
})(dojo, dijit, dojox);

We can use this new scope created by this function to create local variables for your module.

Dojo's Loader can be modified to allow you to specify a set of local variable names that get injected with this function wrapping. And we can do this on a per-module prefix basis, so your code can have your own local variables, but some other module can specify a different set.

I did a prototype that works as follows: assume "coolio" is the namespace I use for my modules. I create a coolio.locals that has the following content:

dojo.provide("coolio.locals");

dojo.setLocalVars("coolio", {
 trim: "dojo.hitch(dojo, 'trim')",
 $: "dojo.hitch(dojo, 'query')",
 id: "dojo.hitch(dojo, 'byId')"
});

This code will create local variables called trim, $ and id that map to dojo.query, dojo.byId and dojo.trim respectively, but only for coolio* modules.

Then I have another module, called coolio.actions that uses these variables:

dojo.provide("coolio.actions");

coolio.actions = {
 init: function(){
  $("#trimButton").onclick(coolio.actions, function(evt){
   id("trimOutput").value = trim(id("trimOutput").value);
  });
 }
}

Notice that trim, id and $ were not declared in here. The final bit of magic is to use djConfig.require in the HTML page to auto-load the coolio.locals module before any other coolio code, so that the loader knows to create the local variables for any coolio.* module:

<script type="text/javascript" src="dojo/dojo.js" djConfig="require: ['coolio.locals']"><script>

See it in action with this sample page.

I created ticket #9032 to track the possibility of allowing this in the future. It also has the patch that can be applied to the Dojo source to get it to work. Or, if you are using Dojo 1.3.0, you can grab these built files with the changes: dojo.js or dojo.js.uncompressed.js

There are some caveats to this prototype:
1) We really need a build step that would inline the local variables for your build layers, so the code as-is will not work with custom build layers.
2) Does not work with xdomain builds yet, another tweak to the build system is needed for that.

If you think this might be useful for you, feel free to add your comments to the issue tracker entry or leave a comment.

Job Transition

Friday was my last day at AOL, just shy of 13 years. There were good times, bad times. Some really cool things. Neat people and teachers. A great learning experience.

I am being a bit melodramatic, but Sinead O'Connor's Thank You for Hearing Me expresses the sentiment best, including the end of the song. For work and for life.

Monday I start at Mozilla Messaging. I am really excited about the work. It is still being defined, so nothing to share yet, but I look forward to helping people take charge of their messaging.

Namespaces, Subjects and Verbs in JavaScript

It has been interesting to follow Dion Almaer's comments about using Dojo in Bespin. This post mentioning call conventions higlighted something I have wanted to talk more about.

The basic issue is namespacing and how you like to call functions. Here are some choices:

1. subject.verb()
2. namespace.verb(subject)
3. namspace(subject).verb(), which can lead to namspace(subject).verb().verb().verb()

subject.verb()
#1 is the Prototype/Ruby way. You define a verb on the object's class, or in JavaScript, on its prototype.

In this model the namespace is really the subject's namespace. There is a possibility of collision with other code that wants to use the same verb. This can be mitigated by keeping your project small and focused, and managing your dependencies. This also works well when your code is a leaf node, or one step from the leaf node -- your code is not consumed by other code, except maybe a top level web application.

The benefit is a nice call structure, and I think fits better with normal English. The subject is identified and then a verb/action is performed with that subject.

Unfortunately, in JavaScript, there can be problems adding things to basic prototypes, like Array, String (and shudder, Object). In Dojo we have had to put in some protections in our code in case the page also uses code that modifies the built-in prototypes.

I believe the situation will improve in future JavaScript releases if added properties/verbs can be marked as not enumerable. That will help a bit, but namespace collision is still an issue.

Some browsers now have native String.prototype.trim implementations and Dojo now delegates to them for Dojo's trim method. Recently, there was a bug filed for Dojo where the core issue was some other code adding a String.prototype.trim() method that did not strip out beginning whitespace.

namespace.verb(subject)
#2 takes the position that verb collision is bad and can cause hard to trace bugs, so always use your own namespace. It also feels more "functional" or procedural: use small functions that do not maintain interior state but operate on their arguments.

This has the benefit of being safer, but it can be more verbose than #1, and therefore more of a constant tax on the developer.

This is mitigated somewhat in Dojo, where you can assign Dojo to another namespace. So you could map "dojo" to "$" to cut down on the typing.

namspace(subject).verb()
#3 is a nice compromise: Define a function that wraps the real subject in an object and provides verbs to act on that subject, without directly modifying the object/class structure of the subject. jQuery took this to new heights by allowing chaining of the verbs. Similarly, dojo.query() returns a dojo.NodeList object that has chainable verbs on it.

An explicit namespace is involved, but the idea is to make it as small as possible, "$". So the overhead for having the namespace is "$()". Coupled with the chainable verbs, it can lead to short code, less of a tax on the developer.

For Dojo, I think #3 is the right approach in general: it gives nice namespace protection, but still gives short call structures.

I want to explore a dojo() function that does this verb mapping in general for all of Dojo. Dijit might be able to use a dijit() to get something similar for the verbs it exposes in its namespace.

So for instance: dojo({foo: "bar"}).clone() would act the same as dojo.clone({foo: "bar"}). Scopemap dojo to $ and you get $({foo: "bar"}).clone().

The difficult part is how to deal with verbs/methods that deal with Strings. I want dojo("div") to actually call dojo.query("div"). But how to allow dojo(" some string ").trim()?

Maybe not map dojo("string") to dojo.query("string"), but instead, allow scope mapping of "d" (or even "_"?) to dojo and another mapping of "$" to dojo.query. That would probably match best with the expectations of what $ does today in other toolkits.

I will have to do more exploration. Eugene Lazutkin's work on providing adaptAs* functions for mixing in dojo methods into an object prototype as chainable methods might point the way to do this.

Dojo 1.2 Loader and Build System

Dojo 1.2 is now available! Check out the release notes for more detailed information.

I attended the Dojo Developer Day in Boston on Sunday 9/28 and the Dojo Community Day at the Ajax Experience conference on Monday 9/29. It was really nice of the Ajax Experience folks to set up community time for the JavaScript toolkits.

During the Community Day, I gave a short presentation on some new things in Dojo 1.2 with the Dojo loader and the build system. I tried using the Google Docs presentation thing. You can see the slides here (some notes are after the slides):

customBase build option

The final example in the slides demonstrates how you can create a 5.5 KB (gzipped) dojo.js file and dynamically load it after the page loads, so if your site uses progressive enhancement, your up-front Dojo cost can be zero bytes, and just load as little as possible for the functionality that you use.

The 5.5 KB dojo.js file is possible using the "customBase" property of the dojo.js layer in a build profile. Normally we encourage you not to tamper with the contents of dojo.js (what is referred to as Dojo Base), but the customBase build option can give you more control on what parts of Dojo Base you want to load. With the customBase option, the build will go through all the JS files and automatically add dojo.require() calls for the necessary dojo._base modules.

Only use customBase if you keep all your JS code in JS modules -- good practice anyway, and keeps with progressive enhancement suggestions. Do not use customBase if you have Dojo calls inside HTML source.

Using customBase on the "dojo.js" layer with no dojo._base modules (just the loader and some bootstrap functions), the non-gzipped size of  dojo.js comes out to be 13KB (recall from above it is 5.5 KB gzipped). So it is small enough to fit under the 25KB non-gzipped size limit that the iPhone/iPod touch use to allowing caching.

You will likely need to use more of Dojo Base than what is in the 13KB non-gzipped file, but you can tune that by creating more layers and playing with the build to so that all layers are under 25KB.

djConfig.addOnLoad

Another option for use with progressively enhanced web sites, or when you want to load Dojo after page load: djConfig.addOnLoad allows you to set a function to run after Dojo loads. It is most useful when used in conjunction with djConfig.afterOnLoad (used when you manually load Dojo after page load).

Here is a complete example showing djConfig.addOnLoad, djConfig.afterOnLoad and djConfig.require.

There is a bug about using djConfig.require in conjunction with a dojo.js generated via a customBase build, so only use djConfig.require when using a complete Dojo Base file.

Thanks go to Matthew Russell for suggesting djConfig.addOnLoad for the 1.2 release.

stripConsole
There is a new build option for stripping out console method calls. stripConsole=normal strips out all console.* calls except console.error and console.warn. stripConsole=all strips all console.* calls.

Safari 3.1 sometimes has an issue with console.debug not getting defined (fixed in webkit trunk), so this build option can help avoid that issue until Safari is fixed.

Mobile Development

I was recently asked about my experiences making TinyBuddy IM. Here are my very biased thoughts about it and mobile development in general.

My Background
I like to make web products that people use. I am a front-end developer, and I love using JavaScript, HTML and CSS. I have done some C/C++ for a cross-browser streaming audio browser plugin. I recently played with Objective C as part of an iPhone SDK experiment. I did Java on the server for a few years too.

HTML, or some sort of declarative markup, is the way to code front end UI. It clearly shows the nested relationships between components. However, I liked the Interface Builder approach in the iPhone SDK, very WYSIWYG. It still some kinks to work out with the beta iPhone SDK though.

I also like trying to reach the largest possible audience with the shortest toolchain. To me, this means favoring an HTML/CSS/JavaScript UI first, and only considering other alternatives when that approach will not work. Particularly since I favor developing applications for the internet.

One Mobile Renderer to Rule Them All
I did not do mobile development until TinyBuddy IM. It just seemed too painful -- too many custom environments, too many cumbersome toolchains for native apps. And the browsers really sucked.

However, now with WebKit on the iPhone and in Android, it is tolerable. Those platforms also have a decent toolchain for "native" apps: iPhone SDK/Objective C for iPhone, and a Java environment for Android.

However, WebKit is most interesting to me: it is a modern, very capable browser that will be on two major mobile platforms. While JavaScript performance is still slow on mobile browsers (both compared to desktop browsers and native code), there is promise that it will get faster with WebKit's SquirrelFish.

For the first time, it is actually possible to consider making a powerful web app for a mobile device. You have good odds of developing one mobile app that runs on multiple devices too. Or at least have lots of code overlap.

Of course a web app is not appropriate for every app, and it does not have access to as many services as a native app, but it does have a lot. With the latest WebKit for instance, you will have local storage.

You Are Dead to Me
Not every mobile device has WebKit. What about Windows Mobile devices, BlackBerry, or all the other devices out there in the market today? Well, for me, they do not make it easy to develop. And I am lazy. I use an iPhone, so I am going to develop for what I use. However, I chose an iPhone specifically because it is easy to do development.

So the other platforms are not that important to me, and if my developer laziness is any indication, they are in for some trouble. They need to get WebKit or Gecko on their devices and use a better toolchain. Opera's browser may be an option too, but I am not clear if it is up to the capabilities of WebKit. If it is, great. I like the Opera on the Wii.

Again, it may not be reasonable to discard these other platforms, depending on your app, but I have decided the other platforms are not worth the effort, given my interests. I really want to make interesting end user products without spending time on platform minutia, and I feel like the iPhone's and Android's reach will be good enough for me.

Mobile Application Design
I approach the technical design of a mobile app by trying to see if the following approaches fit the problem:

1. A purely browser-based web app.
2. A native app that gives a "desktop" icon, and access to phone services (for instance, location services), but use a WebKit view for the main UI interface. Set up hooks in the WebKit view so it has access to the phone services.
3. A purely native application.

#2 has many shades to it, but I like it because it minimizes installed code. Revving a native application is more work than a server-based one.

Remember, I'm assuming this is an internet app, and you know the HTML/CSS/JavaScript trifecta already.

There are tradeoffs. Speed of loading/dependence on network, but again, I prefer to first look at a web-based solution first: maybe it makes sense for the native app to cache the HTML/JS/CSS/images locally.

Native Apps
What you want to build may not work as a web app. For those cases I still prefer iPhone or Android.

For iPhone, you have a nice dev environment with a WYSIWYG UI tool (Interface Builder). Objective C feels a lot like JavaScript to me -- some concept of a prototypical object, and runtime, dynamic dispatch where you can send messages to objects that may or may not respond to that type of message. Unfortunately you have to do the memory retain/release thing, but it is manageable. Lots of helpful videos and examples at the iPhone Developer site too.

For Android, using Java syntax and libraries are a big plus over some sort of C/C++ thing. Java can be verbose, but it is easy to program with an IDE, and hopefully they kept in the garbage collection like Java. That alone is a big win. I have not used Android yet, and I have heard of problems with trying to program in that environment (maybe they have been worked out now?), but it seems worth spending effort on that path. It seems like it has a good shot of survival over the long run.

UI Design
There is a great intro video on the iPhone Developer site about "User Interface Design for iPhone Applications". You will need a (free) Apple Developer ID to view it. But take the time to view it.

The main point: a mobile app is not a desktop app. Do not try to do too much. For TinyBuddy IM, I decided to not include buddy list management features in the UI (add/remove/move buddy). The app was designed for doing quick IMs with your buddies. Buddy list management just seemed to get in the way of that goal. Not to mention the lack of copy and paste on the phone. It just did not seem like a worthwhile activity for a mobile app.

Also, use the UI paradigms available on the phone. For native apps, Apple has a lot of built in templates for using the standard lists and tables you see throughout the iPhone UI. I used a very early version of iUI for a web toolkit that gives an approximation of the native iPhone UI. There may even be some resources in the latest iPhone SDK to make web development easier in this respect.

So, keep it simple, keep it familiar. Once you get the fundamentals down and released something, you can consider your own style (if appropriate).

Pain Points
It is not all roses. There are notable restrictions making mobile web apps. These limitations are specifically from the iPhone, but I would expect them to be roughly the same if not more oppressive in Android environments.

• Resource limits.
• Cache limitations (25KB max size, uncompressed, if you want it cached, 19 total cacheable items).
• As mentioned above, JavaScript performs slower than desktop JavaScript or native code.
• You web app is paused when it is not in the forefront (also a limit for native apps, but native apps will have a notification service you can use that will help with this).

That last one is particularly vexing for TinyBuddy IM, since it is an IM client. So a native app for the IM problem space is probably more appropriate. Or some sort of native component that ties into to the notification service? :)

TinyBuddy IM Weaknesses
I need to spend more time optimizing the TinyBuddy IM code:

• I do not do any sort of "Fetch next 25" sort of thing to keep the buddy list size down.
• The wiping between screens is slow.
• The IM conversation window needs more polish.
• Maybe some better navigation for multiple IM conversations.

So I definitely did not get it right with TinyBuddy IM. Now that there is a native IM app under development, I probably will not rev TinyBuddy IM again. However, I am amazed TinyBuddy IM works as well as it does -- I was able to be a little bit lazy, and still get something somewhat useful out. A good indicator for that platform.

Authentication
Most things I want to build require user authentication for at least some part of it. For TinyBuddy IM, that meant using AOL's OpenAuth authentication, since the Web AIM API used it. OpenAuth has a browser-based authentication system that any web page can use, and it even supports consuming some validated OpenID providers.

OpenAuth has a "clientLogin" API for desktop clients and Flash/AIR apps but I do not prefer it since it basically means the user hands over their name and password to your app, and then your app sends it to OpenAuth. While that might be nice from a usability standpoint, it bothers me personally from a security standpoint.

This is where something like OpenAuth's browser API (and in general OpenID in the browser) provides better protection. You only enter your name/password on the identity provider's web site, and a token is passed back to the web app that requires your credentials.

As an end user of this system, I can verify by the browser URL who is asking for my name/password. As a developer, I like just not having to touch passwords at all. It absolves me from the security issues with trying to manage passwords, even if it seems like I am only passing the password through to OpenAuth's clientLogin.

However, most end users are not this particular about using applications, they are happy to give their password to any site asking for it, particularly if it seems to be done in goodwill. They just want to use the app. Also, the browser based authentication can be weird -- popping a new window to get the auth credentials.

But the OpenID approach is the way of the future (going to another site to authenticate), so might as well get used to it. Particularly now that the OpenAuth authentication page has a "Remember Me" option, so the user does not have to keep entering name/password on every visit.

Have Fun!
Enough of my ramblings. The big thing for me: mobile development can be fun, particularly with the iPhone platform. Android, while not at the same level as iPhone development yet, also seems like it has promise too.

The Beauty of Dojo 1.1

The Dojo JavaScript Toolkit version 1.1 is available:

• Release Notes
• New API tool (ability to add comments coming next week)
• Dojo Book
• AOL CDN info
  
• Download Area (but try the CDN for easy use without needing to download dojo to your server. Save this sample page, and start playing.)
• Follow Dojo. Subscribe to this planet feed for blogs and help sites that talk about Dojo and feeds from browser vendors.
• Key Links for some demos and articles that explore Dojo.
  

There are lots of goodies in this release (see the release notes for full details), but here are some of the new features that resonate with me. I am primarily a Dojo Core user, not much Dijit or Dojox. For other perspectives on the new features, see the 1.1 blog announcement, and Shane O'Sullivan's blog post.

Multiversion Support
One of the slicker features (IMO, since I added it) is multiversion support: you can now run Dojo 1.1 with other versions of Dojo in the page without conflicting. You can also choose to rename dojo, dijit and dojox to other names. As proof:

• A demo showing the 0.4.3 Calendar with the Dojo 1.1 Calendar
• A neat dojox.gfx/charting demo showing dojo, dijit and dojox remapped to omega, omegaw and omegax, respectively

The multiversion support is great if you want to provide your own JS library, but use Dojo underneath. What is even cooler: multiversion support works from the CDN (see the source for the examples above)! So if you are thinking of migrating from an older Dojo version, you can experiment with Dojo 1.1 in a new namespace without having to download 1.1.

Load Dojo after page load
Dojo 1.1 can can be loaded after page load (after the window.onload event fires) by setting djConfig.afterOnLoad to true. This makes the initial render cost for using Dojo near zero, and it plays nice if you want to do extreme progressive enhancement. Use the new djConfig option in conjunction with djConfig.require, to load dojo along with the modules you needed dojo.required after dojo loads. See the the demo page for an example.

Adobe AIR support
Dojo now provides strong support for AIR in addition to Dojo's existing integration with Google Gears via dojox.offline.

Client-side data storage via dojox.storage
If you want client side data storage, dojox.storage gives you a few options, and auto-detects the best one. dojox.storage has been updated to allow for using Dojo Gears, HTML 5 DOM storage, Flash or AIR DB storage.

Build system CSS optimizations
The build system will now inline @import calls that are in .css files in addition to stripping comments and whitespace in .css files. See the New Build Options section.

Try it out!
If you would like to try Dojo, but are not sure how to start, you can always start small: If you just use Dojo Base (one file, dojo.xd.js, from the CDN), you get a solid JavaScript base that makes building progressively enhanced sites much more enjoyable. A rough run-down of the Dojo Base functionality can be found here. That article is for Dojo 0.9, but it still applies for 1.1. Dojo 1.1 is slightly larger than the numbers quoted for 0.9: The 1.1 dojo.xd.js is 29KB gzipped. See the 1.1 release notes for the new features that account for the size increase.

I prefer to stick with Dojo Base, with some additions from the Core modules (like dojo.io.script, which allows using JSONP APIs, like the ones provided by Web AIM). This is how I approached development for the AIM Chat web site and for the iPhone IM web app TinyBuddy IM. But given Dojo's depth, I was able to leverage the Dijit widgets to create a simple admin site for AIM Chat by adding in a few more dojo.require calls. Sweet!

I am really amazed at the amount of work that has gone into Dojo 1.1. The more I use it, the more I feel it is the complete JavaScript toolkit solution. You can do the small, quick progressively enhanced web sites using just Dojo Base from the AOL CDN, but scale all the way up to very rich experiences that use the full power of widgets, awesome data stores, offline storage, and incredible 2-D drawing capabilities (with charting!).

I feel the Dojo community is really hitting their stride now. Great job everyone!

Browser and Dojo updates on Fragment ID messaging

Some browsers have changed how they deal with fragment ID messaging:

Safari 3 throws an error if a child frame tries to change a parent frame's location, but only if the parent frame is not the top-most frame. So, this test works, but this one fails. Parent frames are allowed to change a child frame's location.

Opera 9.25 has locked down cross-frame access to some frame properties. As Julien LeComte pointed out, Opera supports the HTML5 cross-document messaging API, so that is a viable alternative.

I have updated dojo.io.proxy so that it now works in these browsers: IE 6 and 7, Firefox 2 and 3beta2, Safari 3 and Opera 9.25. You can get it from the Dojo development trunk, or wait for Dojo 1.1. The new dojo.io.proxy code works with Dojo 1.0, and there are no API or usage changes.

This new code also improves the IE7 codepath: before the code used to use 3 frames to do the transport, but now it only uses 2 frames, like the rest of the browsers. I was not being very smart about the ordering of the frames in the old code.

Thanks go to Julien LeComte for pointing out the Opera cross-document API and for diagramming another way to do fragment ID messaging.

dojo.io.proxy differs from Julien's CrossFrame approach:

• it allows very large messaging (it supports fragment ID chunking)
• it is focused on providing a data transport -- it provides an XMLHttpRequest facade. It is not a communication transport for UI widgets.

Julien did not want to use location polling, since it increases CPU usage. dojo.io.proxy does use polling to do the communication (this allows for chunking to support large messages), but polling is active only while the message is in transport. I consider the window.location polling less resource-intensive than doing a DOM animation. CPU usage could be tuned by slightly extending polling interval if it is a concern.

dojo.io.proxy creates the frames only when the request starts, then it destroys the frames when the response finishes. This preserves the history stack in all browsers except Opera.

There is a very primitive test page if you want to see the latest code in action.

David Baron's solution for X-UA-Compatible

I like David Baron's alternative to X-UA-Compatible: if the main problem is intranets, give intranet folks a way to configure IE to use the old rendering mode just for their hosts.

I'm interested if this would be enough for Microsoft. If they claim it does not cover all the cases, it would be good to get some sort of metric/percentage of how much it would cover. Seems like it would cover the vast majority case. Probably close to 100% of the cases where Microsoft's revenue is impacted.

If the claim is that they cannot "break the web" even for that small case on the internet where their revenue is not directly impacted, I wonder how important those sites are anyway. I expect there are lots of sites that were made with older browsers in mind, but the site is probably abandoned too. As long as you can sort of read the content on the page, maintaining precise layout is probably not important.

If Microsoft is still set on using a flag/switch in HTML/HTTP header, I still prefer a cultural flag and not a product version flag.

Using Aptana Jaxer for search engine friendly pages?

I am interested in Aptana's Jaxer because they seemed to have gotten Gecko to work on the server. From what I have read before, the tricky part is getting Gecko to run without the need for a windowing service (like X-Windows). Maybe Jaxer still needs a windowing service, I have not looked that closely.

I am not so interested in using their runat="server" or server-proxy capabilities, but more interested in getting Jaxer to render my page to the final "onloaded" DOM that could then be sent to a search engine.

However, I have not seen an option yet to tell it "treat the whole page as runat server" or something like that.

Here is what I want to do (condensed from my previous post on searchable ajax):

• Do the model/view/controller in HTML/CSS/JS. I normally want this all to run in the user's browser.
• However, for search engine requests, I want to do some Apache rewrite/proxying to a server-side Gecko that would render the DOM as it would look after window.onload, then serialize that DOM as the result of the search engine's request.

I wonder if Jaxer could be used for the "server-side Gecko" part. If so, that would be sweet. That would allow my web app to be indexed by search engines properly, but still give me the development model I want. No special dev work to get search engine benefits.

X-Web-Epoch instead of X-UA-Compatible

This is in response to the X-UA-Compatible proposal from Microsoft to help IE8 and future IE browsers to know how to render web pages. There is a lot of commentary on the subject, but here are what I believe to be the original (public) sources for the proposal:

• IEBlog: Compatibility and IE8
• A List Apart: Beyond DOCTYPE: Web Standards, Forward Compatibility, and IE8

I understand and sympathize with IE's position: they need a switch to know when they can use a more standards-compliant rendering engine. There is a vast sea of font tags, tables, spacer images and document.all references inside corporate intranets, and no one is going to update that code, ever. However, any new versions of IE are expected to be able to render that sea of forgotten markup faithful to the original coder's intent. A responsibility IE gets for winning the first round of browser wars.

However, IE is getting push from today's web developers that want something easier to code against given today's suggested cultural approach to web development, namely adherence to publicly accepted web standards.

X-UA-Compatible provides a switch for IE to know what the developer wants: does the developer want the old wild west behavior, or a more standards-based behavior?

However, I do not like the X-UA-Compatible proposal because it focuses too much on specifying a rendering engine and engine version (like specifying IE=8). Actually it is even more specific than rendering engine, it is really a product version switch.

I think the switch should be more of a "culture version" switch than an explicit product version switch. Tying the switch to product versioning seems too fragile. Ideally, IE8 and IE9 will be released before there is a cultural shift in expectations over development.

It also makes it hard for me as a more standards-based developer to make a decision on X-UA-Compatible. I'll likely just use the "edge" value, but then over time, "edge" will lose its meaning and it will just mean "what web developers do today". We'll need a new switch in the future once "edge" reaches the status of today's DOCTYPE.

Maybe it makes more sense to use a culture version number instead. For the sake of argument, I'll call it X-Web-Epoch, but I am not tied to that name. The value for X-Web-Epoch would be an URL that corresponds to the epoch that is targeted. I see having two epochs right now:

• Epoch 1: The tag soup, wild west version of the web IE is expected to render, in particular inside corporate intranets.
• Epoch 2: The standards-based development of today. I would also include "the expectation that things develop and change over time" as part of that Epoch 2.

Epoch 1 is the default. It is used for web pages that do not have a X-Web-Epoch META tag or HTTP header.

So, something like this for Epoch 2:

<meta equiv="X-Web-Epoch" content="http://www.w3.org/epochs/2">

That www.w3.org URL does not actually exist, so spare the w3.org servers and do not try to actually go to it. This does not have to be the actual URL mentioned above, just some URL.

The URL should be resolvable to a real document that explains the cultural assumptions of that epoch.

The Epoch 2 document could list things like:

• Assumes public standards-based development.
• List the general set of standards.
• Maybe list things like progressive enhancement, how to design for accessibility.
• Most importantly: things will change over time as bugs are fixed to improve standards compliance. There also may be new standards adopted.
• It could have a changelog since this document is allowed to change slightly over time.

The document would hopefully set expectations accordingly, in particular that things may change slightly over time. Bonus points: the document can serve as a pointer to how best to code in that Epoch.

Ideally the definition of Epoch 2 is big enough that we might have a chance to go many browser versions without needing a new Epoch, and browsers can go a long time without needing completely new rendering engines.

In reality, cultural expectations will change over time, and we will likely need an Epoch 3 to allow the browser market leader of that time to be able to implement a new rendering engine, so they do not have spaghetti code in just one giant backwards-compatible engine. But again, there should be a longer time span vs. what might be expected with X-UA-Compatible.

It also seems like a better approach than using "edge" in X-UA-Compatible.

This would give IE the switch it needs for its new rendering engine. Other browsers of today, they are just fine moving along with what they have now. They can ignore the switch for now, until they become market leader and have to adapt to an Epoch 3. Hopefully, web developers get iteration from IE in a more timely manner, and a better baseline for development.

Defining $ for Dojo and setting a function prototype

This is a little story about defining a $ function for Dojo, and a more fundamental discussion about setting a prototype for a function.

Defining a $ function for Dojo

I generally prefer using full namespace names when using JavaScript libraries. It helps avoid confusion when passing around code or copying/pasting code that may be used in mixed-library environments. However, if you have contained environment, you may prefer extreme brevity over namespace robustness.

So, how can we define a function named $ that works with the base Dojo functionality (the stuff you get in dojo.js: DOM querying, event handling, XHR, style, basic effects, JSON, array, language and object hierarchy helpers). Ideally, $ would be mapped to dojo.query, to allow easy node selection, but then any other dojo method, like dojo.connect() would be available via $.connect().

Here is something that works:

$ = function(){
    return dojo.query.apply(dojo, arguments);
}; 
dojo.mixin($, dojo);

That does the basics: defines a function $ that returns the result of a dojo.query() call, then copies all the properties of dojo to the $ function object (via dojo.mixin()).

This allows for making these sorts of calls:

//Set on onclick listener for all divs in the body
$("div").onclick(function(){
    alert("div clicked.");
});

//Do an XMLHttpRequest POST using data from
//a form with a DOM ID of "myFormId"
$.xhrPost({
    form: $("#myFormId")[0]
});

Setting a prototype for a function

So that is nice if all I want is dojo base. However, Dojo has a neat module loader, and ideally I want to be able to do something like:

dojo.require("dojo.io.script");

Then later in the code just do a $.io.script.get(). However, this will not work if you define the $ function before doing the dojo.require() call. dojo.mixin() only copies over object properties that exist at the time of the dojo.mixin() call.

So, ideally, I want to set the prototype lookup for $ to be dojo. That way, as new things are added to the dojo object, they will be automatically available via $.

At this point I hit a wall, at least with my current understanding of JavaScript. I was hoping I could do something like this:

var Dollar = function(){
    return function(){
        return dojo.query.apply(dojo, arguments);
    };
};
Dollar.prototype = dojo;

$ = new Dollar();

In JavaScript constructor functions, you do not have to return a value. If you do not, whatever the "this" value was inside the constructor function gets returned as the result of the call the "new" call.

However, if you return a value from the constructor, that is what is returned as the result of the "new" call. This is what Dollar does above.

What I was hoping would happen is that the returned function (that calls dojo.query()) would have its __proto__ property set to Dojo, but I think the __proto__ binding happens before the constructor function is called, so that you can access methods on "this" inside the constructor.

Suppose there was a constructor function called ConstructorFunction, and you called "new ConstructorFunction()". Here is some very crude psuedo-code for what I think the "new" call is doing:

var temp = {};
temp.__proto__ = ConstructorFunction.prototype;
return ConstructorFunction.apply(temp, arguments);

If this is what is going on, it makes sense that by returning a value from the constructor function call, I lose out on the __proto__ binding.

So why not just set the $.__proto__ value directly after $ is defined? Accessing __proto__ is strongly discouraged. It is an implementation's private implementation detail, subject to change. Also, doing that only works right now for Firefox 2 and Safari 3 (failed for IE 6 and Opera 9.24). So stop thinking about it.

I also tried something like this:

Function.prototype = dojo;
$ = new Function("return dojo.query.apply(dojo, arguments);");

but that fails for what I believe to be similar reasons as my first prototype attempt.

I am very interested if someone can suggest how I can modify the $ function's prototype lookup, so that it uses the dojo object. It is neat to think about defining an object that also supports the () operator on it (with a custom prototype).

I want something that works with today's browsers, not ECMAScript 4. Although seeing an ECMAScript 4 implementation might be instructional for the future.

Ajax Experience Talk about Dojo XD Loader

I attended the Ajax Experience West conference this week, and gave a talk about Dojo's XD loader. It was my first time giving a talk in that venue. I got some good feedback from Naveed (from the dev.aol.com group): I was talking a bit too fast, and didn't make enough eye contact, focusing too much on the slides. The content was good, but I could improve on the delivery. I was suspecting as much, but it was great to get the good feedback. Apparently I'll get feedback (hopefully) from the talk evaluation sheets.

It was neat talking with one of the jQuery contributors, Yehuda Katz, who has been thinking about how to handle serving jQuery plugins. They have been considering a way to load plugins, and possibly specify dependencies. They may not need the full xd dependency mapping, but it would be neat to share some ideas in that area. In the talk, I talked about trying to modify other toolkit code from YUI, Ext and jQuery plugins to load via the dojo loader. I still might try that as an experiment.

In the presentation, I talked a bit about xd loading in general, and to address the security concerns of how to verify the code you are xd loading has not been corrupted or changed. I was thinking using digests of some kind would be good. Ideally, the browser could do it before it evaluated the JS code.

I mentioned the idea to Douglas Crockford, and he had nice idea of being able to specify more than one URL for the script, to serve as backups in case there was a failure with one of the first one failed. He thought it would make a good browser plugin. The multiple URL thing also came up as a suggestion in the Q&A part of the talk, but in the context of dojo.registerModulePath().

Initially, part of the Q&A discussion was ways to do the digest checking without needing a browser plugin, but the more I think about it, it really needs to happen by the browser, since we need to do the check before the imported script is evaluated in any way. And if we are doing xd-loading of the script, that means (at least today) that a script tag is going to be used. So we need to extend the functionality of the script tag.

Looking around at the web, it seems like (as usual) these ideas have been in the air before. I went to Andreas Andreou's cross-site js sharing post, which pointed me to this moz.dev.platform group discussion. This mozilla bug was referenced, and from there, I went to Gervase Markham's Link Fingerprints page.

The Link Fingerprints seems like a workable system. Most of the discussion has been focused on sharing of library code, but it also seems to dovetail nicely with a security aspect. I might ask the moz.dev.platform group about the status of the mozilla bug and mention my desire to have it for security concerns. I'll also ask about supporting alternate backup URLs for download. Maybe as nested script tags?

If this works out, I would feel much more comfortable strongly suggesting xd loading for folks. It would result in a safer web. Very nice.

TinyBuddy IM: Instant Messaging for iPhone

(Update 7/15/2007: I've moved the TinyBuddy IM-related info over to another blog: Tiny Notes)

Have an iPhone, but wish you could IM your buddies? Now you can, with TinyBuddy IM. It is an AIM® Enabled web-based IM tool. It works by using the Web AIM API, OpenAuth and Dojo.

The nice thing about this solution: you do not send your AIM password to me -- you are redirected to AOL's OpenAuth servers for authentication. My JavaScript only sees an auth token. Furthermore, my web page has to get your explicit consent before accessing your buddy list data and before sending the first IM or presence change.

So the price of this added security is pop-up windows. A new window will be popped so you can authenticate with the OpenAuth servers, and also when giving consent to the application to access your buddy list and IM. For the consent prompts, you can choose "Grant Always" to avoid them on subsequent logins. I think the pop-ups are worth the added security, and at least in iPhone's Safari, window popping looks neat. Unfortunately, the OpenAuth Sign In and Consent pages are made for larger windows, so you will have to double-tap zoom to read them.

Another neat feature of this web application: it is pure JavaScript, HTML and CSS. No server-side languages needed. Dojo really made it easy to do this. I used Dojo 0.4.3 because I want to reuse this code for some other projects that are on 0.4.3, but if/when I get enough time, I would like to port it over to the 0.9 code.

So give it a whirl if you like. I'm sure the code it not bulletproof, and I've noticed enough weirdness with iPhone's Safari to guarantee that I will not be able to give comprehensive support. Also, even though I'm an AOL employee, AOL does not endorse this project or have anything to do with it (but thanks to my co-workers for ideas and early testing, all done of their own accord).

Also, I'm using an OpenAuth dev key, so if there is too much usage you might see some rate limit errors, but we'll see how it goes.

Some other interesting tidbits:

• The Web AIM API is a Comet API. It uses long polling to work cross-domain in the browser. I'm using more of a short poll with pauses between the polls to hopefully smooth out network hiccups on the phone.
• Don't like the CSS? You can make your own and tell the app to use it instead. Go to the test launcher page to specify the path to your CSS. Click the Launch button, then copy launch URL. Use that URL when you want to use the application. This feature is not allowed for IE browsers given its security problems with CSS "expressions".
  
• I'm serving the code gzipped. The HTML, CSS and JavaScript combined come to about 90 KB. So it is tolerable on the EDGE network.
• Use the iPhone two finger scroll to scroll the buddy list and IM conversations.
• Typing IMs should be optimally sized for use with the virtual keyboard. Just type in the text box at the bottom of the IM window and press "Go" on the keyboard.
• I'm using Dojo Accordions for the IMs and buddy list. I like the use of space with that model and that I can show you incoming IM text if that IM AccordionPane is in the closed position.
• onbeforeunload does not seem to fire for iPhone Safari. That makes it hard to log out correctly, so to clear your OpenAuth cookies, be sure to use the Available, Sign Out menu item.

To use TinyBuddy IM, just type http://tybyim.com in the iPhone Safari browser. You can try it in other browsers, but it looks best in the iPhone Safari.

Searchable Ajax with JavaScript controllers and a headless Gecko

I'm back from a much appreciated vacation, and just wanted to jot down some thoughts that I want to expand on later. This is a continuation of how I want to build web applications on the RIA-end of the web scale, but an approach I could use for just about any data-driven web site.

I only want to use the server to serve files and to implement data storage. There should be no server-side work dealing with the UI controller. This means the "application server" part of the server (the PHP, Java, .Net, RoR part) should only be handling the API to save/modify/get the data. It should not be doing any presentation or UI flow. JavaScript, CSS and HTML should be used for the presentation and UI flow.

So, the app server implements the data API as a REST API. This gives a good boundary of what should be done in the app server. The data should use XML (preferrably ATOM) or JSON as the data markup. The only presentation part might be to apply something like an XSLT transform of the data if the GET request matches a browser or searchbot user agent. Alternatively, use the domain name of the GET request to know whether or not to apply the XSLT transform (sometimes it is nice to host the pure data API on a different domain for security and load reasons).

Progressive Enhancement should be used for adding the JavaScript controller to the page, so that hyperlinks to other resources can be followed by searchbots or non-JS enabled browsers. Editing of the resources via an HTML interface could be restricted to JavaScript-enabled browsers (mostly where a UI controller is needed anyway, and search bots only care about indexing GET resources anyway).

So that is a baseline. This baseline now cleanly separates the UI controller from the server side, and makes it less relevant what sort of app server technology you use. The UI controller is kept in the JavaScript realm, no more splitting it between JavaScript and [Struts, RoR, Django, etc...].

Now it is time to jump the shark:

Restrict the data API to its own domain, separate from the domain used for presenting the UI for the browser/search bots. For example, use api.domain.com for the data API and ui.domain.com for the browser/search bot interface.

The HTML files on ui.domain.com use Ajax-like techniques access the resources on api.domain.com to show the UI. The Ajax-based data loading would have to be a cross-domain ajax. If the API and UI domains share a common sub-domain, document.domain tricks could be used. Otherwise (and more interesting to me personally), JSONP or XMLHttpRequest IFrame Proxy could be used.

A nice benefit of this approach is that all of the UI pieces (HTML, JavaScript and CSS) are highly cacheable and servable from very different domains than the API domain.

So what about search bots? They cannot do the Ajax techniques, at least not yet. For them, in the web server config, route their user agents to a web server module that uses Gecko to fetch the page, and do the ajax loading. The HTML can send an event to Gecko (this custom, headless gecko can export a function, something like window.onAjaxFinished()) to tell it when it is done rendering. Then the Gecko module serializes the current state of the DOM and sends that HTML string back to the search bot. It could even keep a local cache if it liked, if the resources did not change that often.

This approach would allow the search bot to get a good representation of the page for their indexing, and since Progressive Enhancement was used to bind to resource hyperlinks in the document, the search bot can still navigate to other resources on the domain (and those requests would be funneled through the web server's headless Gecko module).

In this model, REST via HTTP becomes the new JDBC. And more importantly to someone like me who enjoys the front-end -- I don't need to learn about any of the app server flavors of the month to implement data access (the other app server-hugging developers on my team can do that).

So now I just need to do a custom build of a headless Gecko that I can use in a web server module. Any pointers are appreciated. Ideally someone should do this as an open source project so everyone can benefit.

IE 7 and IFrame APIs Part 2

Microsoft was kind enough to look into the issue, and here is more information.

The "Navigate sub-frames across different domains" preference is designed to address this security report:
http://secunia.com/advisories/11978/

IE 7 has shipped with automatically turning this preference on.

There is a workaround to the issue: the two iframes that do the communication must both have a parent frame from their domain loaded.

So, the original setup I use in the XHR IFrame Proxy situation looks like the following. The dashes indicate the level of nestedness (Top Window contains IFrame 1, IFrame 1 contains IFrame 2):

Top window (Domain A)
--IFrame 1 (Domain A)
----IFrame 2 (Domain B)

Where IFrame 1 and IFrame 2 communicate by setting each other's fragment identifiers (location.hash in JavaScript). When the communication is done, IFrame 1 tells the top window what the result is.

What broke was IFrame 2 (on Domain B) setting IFrame 1's location (via parent.location).

The following iframe structure works in IE7:

Top window (Domain A)
-- IFrame W (Domain B)
---- IFrame 1 (Domain A)
------ IFrame 2 (Domain B)

Notice the introduction of IFrame W. Now, apparently since IFrame W (on Domain B) owns IFrame 1, it is OK for IFrame 2 to change the location of IFrame 1.

Here is a test of this structure (thanks in large part of Microsoft).

This test uses something I personally have not done before: using window.open() to grab named iframes. Interesting technique. I'm curious to see if it is required when I patch IFrame XHR for IE 7.

So, the end result is that *another* IFrame is needed to accomplish the task. I'll be looking at modifying XHR IFrame Proxy code to use this extra frame, but only for IE 7. The down side is that now, in addition to the remote server placing xip_server.html on their server, they will have to place another HTML file too (the file for IFrame W). If the code bloat is not too much, I might consider making xip_server.html smart enough to act as IFrame W content as well as being the content for IFrame 2.

So, while it works, I still ask Microsoft to consider the following:

1) It seems like this change to address the security issue (http://secunia.com/advisories/11978/) seems like it should not affect setting parent/child frames. Firefox seems to have addressed the security issue without breaking subframe communication. I know that is not entirely a fair comparison, but it seems like the code change in IE to protect against a specific issue has too broad of an effect.

2) Even with this preference on, allow changing of fragment identifiers. It is a non-destructive change to the parent frame, and the parent frame can choose to ignore it. It is also a very solid, useful tool for cross-domain communication that works for today's browsers.
If you want to get fancy, consider supporting a security setting on the iframe, that allows Domain A to say it trusts content from Domain B to change its location. Ideally this sort of configuration of an iframe would be some sort of standardized access control scheme that other browser makers could implement if they so desired.

IE 7 Breaks IFrame APIs that use parent.location

Update (Nov. 7, 2006): Microsoft provided more information. See Part 2.

Original post (Oct. 11, 2006):

One of my co-workers was recently testing IE 7 RC1 and found that using fragment identifier messaging (FIM) with IFrames did not work. When the content in the nested IFrame tried to set parent.location, IE 7 popped a new window instead. If you have an IFrame-based API that uses parent.location, then you might be affected by this change.

I would like Microsoft to reconsider the preference change in IE 7 that causes this issue, and this post describes why. Feel free to contact Microsoft if you feel likewise. More information below.

When I investigated the issue more, I found that IE 7 only popped a new window if there was a nested IFrame. If there was just an IFrame in the top level document/browser window, it did not pop a window.

• Test Case 1 (OK: Top browser window -> IFrame)
• Test Case 2 (FAIL: Top browser window -> IFrame -> IFrame)

We contacted Microsoft through our company's Microsoft channel, and found out the following:

• There is a preference that was introduced in Windows XP SP2 that controls this behavior. You can find it by looking in the Internet Settings -> Security Tab -> Click on Custom Level button -> "Navigate sub-frames across different domains".
• With IE 6, this is set to "Enable" by default.
• When IE 7 installs, IE 7 automatically sets the preference to "Disable" by default.

I would prefer that they do not change the preference for the following reasons:

1. There is no documentation about the change, and the change was done without checking with the web developer community.
2. I think the behavior is buggy and could use refinement.
3. Fragment identifier messaging (FIM) across IFrames is the best thing we have for allowing pure browser, cross-domain web APIs.

If you feel the same, or if you feel that your own IFrame-based web API that sets parent.location might be affected, then send feedback to Microsoft. I will try to find out where we can send feedback and update this post when I have the info.

More information on why I think IE 7 should keep the preference as "Enabled":

No public documentation of discussion

Our Microsoft contact did not provide any public info indicating this was documented or discussed publicly. When I asked for a description of the change, there was no documentation that was ready for public consumption yet. There might have been some public disclosure about it, but I am not aware of any. I apologize if there was something posted. I have been subscribed to the IEBlog for a while now, but none of the previous posts triggered any warning bells for me. But I could very well have missed it.

This page describes that security preference, but reading does not seem to be very helpful (and seems to discuss talking to sub-frames and not parent frames).

I do not mean to chastise Microsoft on this point. I can appreciate with such a large project as a browser, it might not always be evident what changes can have an adverse effect on people using your product. And even if they did note it somewhere, there are lots of changes, and it could very easily get lost. It has happened to me on my software projects.

I can also see that their answer could be "why don't you just test with the IE 7 betas and release candidates? If there is an issue, let us know". That is a fair and valid position to take. But just as it might be hard to document and message all the changes for a large product like IE 7, it can be hard for web developers to stay on top of everything. I can see many not taking IE 7 testing seriously until RC 1.

And given that the parent.location works as long as you are communicating with a browser window and not an IFrame, I can see for some developers using IFrame APIs, there might not appear to be an issue.

For my case, with the XHR IFrame Proxy, I just worked out that solution July 31st. I haven't done IE 7 testing because I have been busy, and I thought the big things for that release were CSS-related changes (from a web developer POV). There did not seem to be that many JavaScript changes (besides an apparent increase in the JS engine performance). But still, I feel bad getting around to this issue this late in the IE 7 cycle.

So, ideally both of us could have caught the issue sooner and talked about it, but it is understandable that it did not happen. And I hope that since it is a preference change, that changing the IE 7 behavior back to "Enable" is hopefully an easy thing to do even this late in the IE 7 cycle. This would give more time to discuss the issue.

Behavior is buggy and could use refinement

I am sure there is a good use case to justify having this preference, and perhaps it even describes the current behavior. However, from my outsider's point of view, it seems buggy.

Why is Test Case 1 allowed (setting parent.location on the top browser window), but setting the parent.location on an IFrame is not allowed? They both seem to be "cross frame" communication, and violate whatever security issue the preference tries to address.

In addition to addressing the apparent bug with the behavior, I think setting parent.location should be allowed if the new value for parent.location is the same protocol, domain, port and path, but just differ by fragment identifier. In those cases, the parent document is not destroyed or modified, so perhaps whatever security issue is being addressed with this preference does not apply? Allowing fragment identifier changes is important for the next point...

FIM is the best thing we have for allowing pure browser, cross-domain web APIs

There are other ways to do cross-domain communication in a browser, but they all suffer from some limitation that does not make them a good general purpose, pure browser solution.

I also believe Fragement Identifier Messaging (FIM) that is used as part of the XHR IFrame Proxy is the most secure cross-domain communication option, since it requires explicit action for a server to allow the cross-domain action, and it has very fine grain control on the types
of requests that are allowed. FIM also does not automatically "execute" the response as using dynamic script tags might. The receiving frame can choose to just ignore messages if it wants.

And with the exception of the current behavior in IE 7 RC 1, FIM works in all other (major) browsers: IE 6, Firefox 1.5 and 2.0, Safari 2.0, and Opera 9.

The real solution would be a native XMLHttpRequest that can do cross-domain requests, but even when that becomes available, it will take a couple of years at the least to be widespread enough that it is actually be usable by web developers for cross-browser solutions.

In Conclusion

Microsoft, please do not change the preference to "Disable" with IE 7. Keep the default "Enable" behavior of IE 6. If at some point you want to change the preference to "Disable" by default, consider a public discourse of the implementation issues discussed here and the impact on FIM.