Tagneto

Wednesday, January 25, 2006

Clarifying Tagneto's mission

I think the description of Tagneto in the Why and What documents may be a little vague or too generic. I'm trying to clarify the mission and the message of the mission. Here goes:

Provide tools to build fast, rich web applications that don't require application server infrastructure.

Creating rich user interfaces in HTML should not require a server infrastructure. A web server is needed to serve the files and to distribute updates, but server infrastructure like an application server (whether it is in Java, Python, Ruby, Perl, PHP, .NET, whatever) should not be needed. Server infrastructure is needed for data APIs, but for the user interface, it should not be required.

For some larger applications, it is desirable to have the ability to create the user interface from component HTML pieces, and assemble them together in the page. Normally, an application server technology like JSP, ASP, PHP would be used to accomplish this task. However, since the desire is to avoid application server infrastructure, a View Assembly tool is needed that would provide that capability. Tagneto's View Assembly tool does just that, and it uses tools the UI developer already needs to know, JavaScript, XML, HTML and Entities.

The View Assembly tool also implements advanced features such as

"Aspected Oriented HTML" via overlays, allowing the matching and processing of tags across pages, and allowing processing before, after, as the first child or as the last child of the matching tag.
Chained tag handlers, for combining many small, focused tags together to accomplish something big.
The architecture is also designed to allow other scripting languages (Python, Ruby, Perl) to be used instead of JavaScript for the tag scripting (but right now only JavaScript is implemented).
The source does not have to completely valid XML.
Internationalization/Localization support.

To help with the "fast" part of "fast, rich web applications", the View Assembly tool uses a developer, compile step to generate the final UI instead of trying to do everything dynamically at runtime in JavaScript. This allows the most efficient use of processing power: use the developer's box to build things that don't depend on dynamic user input, and save only the request/user-dependent UI generation for the runtime JavaScript.
Avoiding application server infrastructure opens up the possibility to deliver the user interface from anywhere, a specialized web farm like a Content Delivery Network (CDN), or even the local disk.

However, to successfully access the data APIs, cross-domain access to the APIs via JavaScript is required. Something that is not possible with XMLHTTPRequest. Ideally, future advances of the XMLHTTPRequest object will provide that functionality, but for the browsers of today, The Dynamic Script Request API is an attempt to solve that issue.

All that said, application server infrastructure is a critical component of delivering a complete application. Hopefully it is just used for the data APIs, but the View Assembly tools should play nice with those technologies if they also need to be used to generate UI. Tagneto's View Assembly tool does play nice with the technologies, since it can be run before deploying the pages to the application server, and it is very flexible in choosing which tags to match and process.

The main goal of Tagneto is not necessarily to provide a "JavaScript library framework". Some JavaScript libraries are available with Tagneto, but in the service of the mission above. In fact, recently I have been considering providing View Assembly tool integration with some of the more well-known JavaScript frameworks, like Dojo or Prototype, with Dojo integration probably first.

Perhaps for Dojo, "View Assembly tool integration" means:

Using Dojo's string building utilties to build the JS-escaped strings for the ctrl tags. Right now the Array.append/join is used.
Supporting Dojo's connect() methods for event binding via the ctrl:listen tag.
Possibly do some compile-time "preprocessing" on Dojo widget tags to convert them to the HTML/JavaScript that will actually be used at runtime. This would avoid having to scan the HTML at runtime to find and bind the widgets.
Integrating the DSR with dojo.io.bind.

Those are just some brainstorming ideas. I'll have to investigate the Dojo code to determine feasibility. But I think one of the results of clarifying Tagneto's mission is the focus on not trying to build a JavaScript framework library. Some JS libraries will be developed as reference implementation for pieces that fulfill the mission, but ultimately, I may try to integrate those implementations into existing JS frameworks.

I'll see how it goes, but right now I like this clarification. I'll gradually work on updating the site docs to reflect it. Feedback or suggestions are welcome.

Tuesday, January 17, 2006

Using DSR to allow On-Demand JavaScript for RSS/ATOM feeds

It would be great to see data services provide a JavaScript version of their APIs to allow web developers that don't have (or want) server resources to implement data mashups. JavaScript APIs that can be used via HTML script elements allow for cross-domain data access without server intermediaries.

With the release of Tagneto 0.4.0 there is an update to the Dynamic Script Request (DSR) API, an API that makes On-Demand JavaScript via HTML script elements reliable and easy to use.

The API now uses a reserved callback function, window.onscriptload, and an event object passed to that method to communicate the success/error codes and the data associated with the response. Other browser events and the XMLHTTPRequest object were used to guide the names of the event properties.

These changes allow for data services that provide RSS/ATOM feeds (or any type of XML data feed) to provide a JavaScript response that is equivalent to the normal XML response without too much work (basically encode the XML as a JavaScript string, and put it in an event object that is passed to window.onscriptload).

The API document has more detail on how a RSS/ATOM feed might be provided as JavaScript in the Possible Applications section.

Tagneto 0.4.0 provides a Dsr.js helper script that implements the DSR API for use in a web page, and there is a DSR jar file that can be used in Java Servlet applications to send DSR responses.

Tagneto's home page uses the DSR JavaScript library and the DSR jar for the "Recent Blog Entries" section as a demonstration of the API and the implementation.

Monday, January 16, 2006

Tagneto 0.4.0 Released

The focus on 0.4.0 was on improving and simplifying the Dynamic
Script Request (DSR) API. From the Release Notes:

Significant changes to the DSR API. Decided to go with a reserved global callback handler, onscriptload, to simplify things. Other browser events and the XMLHttpRequest object were used to guide the format and names of the methods and parameters. The result is a much simpler API. A sample usage of the API for RSS and ATOM feeds is described.
The Tagneto distribution includes a server/dsr directory. This directory contains a DSR jar that helps servers implement the DSR API for use in web applications that use Java Servlets. A sample WAR file that uses the jar is also included. The JAR and WAR file should run with JRE 1.4+.
The Dsr.js file has been updated to the new API.
The Tagneto distribution includes the JavaScript libraries in a js directory at the top of the distribution.
Tagneto's home page has been updated to use the new DSR js and jar libraries for the "Recent Blog Entries" section.

Sunday, January 01, 2006

Tagneto 0.3.2 Released

The main focus of the release was the JavaScript libraries that support doing dynamic script requests.

See http://tagneto.org/ReleaseNotes.html for more information.

Thursday, December 15, 2005

Referrer test page / Authenticated script src APIs

Referrer test page

I have a test page up now to test the HTTP referrer header values:

http://tagneto.org/test/reftest

If you press the test button, it should show the date from the server and the referrer header value.

There is also another URL to try in the text box: /cgi-bin/redirect.rb. That URL will redirect to the /cgi-bin/refer.rb. I wanted to be sure a redirect did not change the referrer value.

Now I'm going to use the test page to test posting from different pages and in different scenarios to see if I can get the referrer to change to some value than the one from the actual page making the SCRIPT SRC request. If you want to try posting to http://tagneto.org/cgi-bin/refer.rb from other domains, save the index.html page from the reftest URL mentioned above, as well as the SvrScript.js file that is in that same directory.

Authenticated script src APIs
If the referrer thing holds up, then I think it will be possible to proceed with some of the ideas mentioned in this post. However, that post assumed the authorization issue concerned a developer user name using the API on his/her own page.

For the larger issue of allowing a third party web site that deals with data that needs user authentication (without exposing the auth credentials to the third party web site), what about this:

3rd party web site gets an API key. That API key is only bound for certain domains and/or URLs.
When a user uses the 3rd party web page that uses a protected data API, the data API server looks for authentication credentials in cookies that are only set to the data API server domain.
If the user is not authenticated, then respond to the web page with an error, message of "auth.needed". The 3rd party web page puts up a DIV dialog saying that authorization is needed. If the user says OK, the 3rd party calls an "authenticate" JS method that is provided by the data service script library. An argument to authenticate method would be a return URL (an URL on the 3rd party site). The authenticate method would pop a window an prompt the user for auth credentials. A new window should be used so the user can check the domain/URL of the auth credential page. After successful login, the new window sets its location to the URL passed to authenticate. That URL on the 3rd party site should just close the window and force a refresh/update of the web page that uses the data API.
If the user is authenticated, then the data service checks its own database to see if the user has explicitly granted data access to the 3rd party web page URL. If the user has not granted access yet (the data API would look at the referrer HTTP header), then data API responds to the web page with an error, "auth.needPermission". The 3rd party web page calls a "askPermission" JS method that is provided by the data service script library. askPermission would take a JS method callback as an argument. This method creates an IFRAME dialog in the page that asks if the user wants to give permission to this website to access the data. There could be a "Remember this answer" checkbox too, if the data API wanted to allow that. If the user says OK, then askPermission notifies the callback of the answer. The callback could then update the page accordingly (probably by re-calling the data API).

More experiments to do, but it seems promising.

Monday, December 12, 2005

Ctrl.js event listening updated

I was reading more on the web about registering event listeners in JavaScript. Mainly:

http://www.quirksmode.org/blog/archives/2005/10/_and_the_winner_1.html
(including comments)
http://dean.edwards.name/weblog/2005/10/add-event2/
http://www.quirksmode.org/js/introevents.html
http://www.dustindiaz.com/rock-solid-addevent/

I reworked the event listening methods in Ctrl.js to do the following:

modify the events that are used in MSIE to look more like the standard events.
have "this" point to the element that has the event listener.
allow the ability for any JavaScript object to have listeners.
allow for cleaning up to avoid memory leaks for individual DOM elements (and optionally their children). Ctrl does register a global cleanup function on document unload, but it also surfaces a removeListeners() method for cases when there are multiple nodes attached/detached within the web page's lifetime, and memory leaks want to be contained as nodes are detached before the page is unloaded. (Even does cleanup for Mozilla because of bug: https://bugzilla.mozilla.org/show_bug.cgi?id=241518

I also renamed the methods, and now there is only one "register event listener" method, Ctrl.listen().

The changes are in CVS but not visible on the web site yet. Still need to round out a little more testing.

Friday, December 02, 2005

Security and the Dynamic Script Request API

Possible Security Issue

Dynamically adding SCRIPT SRC tags to pull in data as JavaScript is nice since it gets around cross-domain issues, but it could also be problematic if you are providing user data, data that required the user to authenticate in some way.

If the User A authenticates for using some JavaScript data APIs via SCRIPT SRC tags, it is possible that Hacker B could guide User A to Hacker B's page that includes a data source that uses User A's credentials.

User A authentication could have been through setting cookies that will travel to the data API domain. The API could also require that User A have an "API key" to use the API (User A registers with the data service provider, and receives a API key text string that must be passed to any data calls. Google Maps is an example).

Hacker B could find out User A's data API key, and if the API required authentication cookies to be set, Hacker B just needs to be sure User A authenticates before coming to Hacker B's page.

Protection Measures

Data Service Registration

The user must register for an API key. To obtain the API key, the user must authenticate with the data service provider to prove their identity.
As part of this registration, the user must specify which domains are allows to use the API key, and possibly which user names are authorized to use the API Key.
Since the user is authenticated, that user name can use the API. Any other user names that are added to the API key must receive an email notification. The other user names must authenticate and grant permission to be on the allowed users for the API key.

Script SRC requests

In addition to sending the API key, there must be a timestamp as part of the request. These parameters should be querystring parameters (like dataapi.com?key=xyz&stamp=55959833920)

Data Service Server

The server for the data service should not allow caching of the data request. They should set the appropriate HTTP headers to expire the results of the request immediately, to avoid Hacker B taking advantage of the browser cache (that is why query string params are used on the URL too).
The server will reject any request that does not have a registered domain in the Referrer HTTP header.
The data service domain should only have data services. It should not allow any user-generated web pages under that domain. Ideally it should be unrelated to any domains that host user web page content -- no subdomains should match to prevent document.domain tricks from working.

Test scenarios

Even with these measures in place, are there other holes? I want to make some tests to verify the following:

Does authenticating with the data service pass the cookies along properly for SCRIPT SRC requests?
Does the Referrer field get set correctly for SCRIPT SRC requests? In all browsers?
How can the Referrer field be spoofed? Use XMLHTTPRequest. But hopefully since only data services are allowed on the data service domain that can't be used. Hopefully the document.domain protection above will help too. Is there some way a server proxy running on Hacker B's site be used? Hopefully the User's authentication cookies can't travel to that domain.
Even if the server sets expiration headers, will the browser still cache? Hopefully since a timestamp is sent and the URL uses query string params, Hacker B's URL won't be the same anyway, and therefore a different browser cache entry?

These are just first thoughts. I haven't checked yet to see if someone else has already worked this out. And it would be good to do some testing too, think of other scenarios that might break.

Saturday, November 26, 2005

Tagneto 0.3.1 Released

Tagneto 0.3.1 was released yesterday. It is available for download from the SourceForge site.

Release Notes:

Added source examples to website/documentation for ease of reference.
Dynamic Script Request API document and SvrScript implementation of the API.
Broke up Srvr.js into 3 files: SvrScript.js, SvrReq.js and SvrFrame.js.
No more warnings when building in Eclipse. Most warnings were from autogenerated JAXB files.
Fixed tagneto/src/test/RunTest.bat and RunTest.sh.
Added ant lib to make it easy to build tagneto directly from cvs source checkout in Eclipse.
Added notes on how to build Tagneto on the Where page.

Monday, November 21, 2005

Dynamic Script Request API

I updated the Tagneto website with a document describing the Dynamic Script Request API, and Tagneto's support of it through the Srvr.js library. The goal of the API is to make it possible to reliably use dynamically added SCRIPT SRC elements to load data and UI. It also makes it possible to send up more data to the server via multipart requests. This should make it easier to access web APIs across server domains (something XMLHTTPRequest normally cannot do). The limitation being the cross server web API must return JavaScript.

This document is another revision of the API first described in this post.

The changes to Srvr.js to support this API are not in the 0.3.0 release of Tagneto. Either grab from CVS or a later version of Tagneto (if available).

Wednesday, November 16, 2005

Using On-Demand JavaScript for Everything

I really like using On-Demand JavaScript (dynamically creating a SCRIPT tag and attaching to the HEAD) for loading new data or UI, particularly since it gets around the domain restrictions of XMLHTTPRequest. There is also no need to deal with messy XML to JavaScript transformations.

The challenges:

GET URL length restrictions.
Data has to be JavaScript.
Knowing when script load is complete.

Tagneto can help deal with these issues. The approach and issues are slightly different for retrieving UI (View) vs. Data (Model):

On Demand UI

1. GET URL length restrictions

This normally shouldn't be an issue for retrieving UI, particularly if the UI does not depend on particular request or user data (see Why document). If it is an issue, please see the approach mapped out below in On Demand Data.

2. Data has to be JavaScript

It is particularly tricky to encode HTML as JavaScript. Tagneto's ctrl: tags make this easier. An example, to create a JavaScript function that returns an HTML string based on some input parameters.

Sample.js:

<ctrl:function name="getHelloWorld" parameters="name, planet">
<b>Hello &ctrl:out-value:planet;, this is <ctrl:out value="name/>.</b>
</ctrl:function>

After running this file through Tagneto's view assembly tool, Sample.js will look something like this (the actual result is a little more complex and efficient than string concatenations::

function getHelloWorld(name, planet)
{
return '<b>Hello ' + planet + ', this is ' + name + .</b>';
}

Now Sample.js is ready to be dynamically included in the page.

3. Knowing when script load is complete

(Update 1/21/2006: the original post referred to a Srvr JS library. That library was renamed/changed and now is the Dsr library. This post was changed to reference the new library.)

The script can be dynamically added to the page using a method in Dsr.js:

Dsr.sendAndPoll('Sample.js', 'getHelloWorld', sampleLoadedListener);

where sampleLoadedListener is an object that could look like this:

var sampleLoadedListener = {
          onLoad:    function()
                       {
                       document.getElementById('outDiv').innerHTML = getHelloWorld();
                       },
          onError:   function(status, statusText, response)
                       {
                       alert('status: ' + status + ', error: ' + response);
                       },
          onTimeout: function() { alert('timeout'); },
          timeout:   30
          };

sampleLoadedListener.onLoad will be called once getHelloWorld is detected as being defined.

On Demand Data

This section is bit more experimental. It is just a first thought, probably needs more work. I think I'll need to make changes to Dsr to have it play nice with this model, perhaps even provide convenience method to handle multi part URLs.

1. GET URL length restrictions

(UPDATE 1/21/2006: The API described below has been significantly changed. See the Dynamic Script Request API for the new version. The Dsr.js library now implements the DSR API.)

It seems like URLs can only be at maximum around 1KB, so this can be problematic if you need to send a large amount back to the server. What about using multiple requests (parts) to post the data back? The server would collect the parts, and on the final part, do the action. Some spec is needed for GET parameters on the part URLs:

?part=currentPartNumber.totalParts
&succes=methodName
&amp;amp;amp;amp;amp;error=methodName
&partComplete=methodName
[&actual data to give to server]

An example URL (broken into multiple lines for readability):

?part=1.4
&success=myDataLoaded
&error=myError
&partComplete=myPartComplete
&name=foo
&quantity=3
&....

1.4 means this is part 1 of a 4 part request.

myDataLoaded is a JavaScript function already defined in the page, and it takes one parameter, the actual JavaScript response data from the server after the server finishes collecting all the parts and processing the requests.

myError is a JavaScript function already defined in the page that would take perhaps an error object/message as the only parameter.

partComplete is a JavaScript function already defined in the page that would take the part number that was just completed by the server. The server's response would call this method as the very last line in the response.

As the JavaScript gets notification of a successful part being processed
(via the partComplete function), it would attach the next part as a SCRIPT element to the page.

The method callback parameters only need to be sent up as part of the first part of the total request.

2. Data has to be JavaScript

Frameworks like DWR or JSON (in the Java world) may be able to help with dynamic data (based on request or user data). If it is static data, custom tags using Tagneto's org.tagnetic.core.tags.define.DefineInclude tag handler or the view:xmldatasource tag could be used to transform XML data into JavaScript.

3. Knowing when script load is complete

Completion is known by the calling of the success/partComplete/error functions. Srvr.Script should be changed to allow the checkString to be optional, and to provide a wrapper around this approach.

CVS source update to make building easier

I updated the CVS source last night to make it easier for people who try to build the source from a CVS checkout. Changes:

No missing dependencies
tagneto/src/test/RunTests.bat should now run all the tests (still have to check RunTests.sh)
Fixed warnings when building in Eclipse (warnings mostly from the autogenerated JAXB code that is used to parse tagneticconfig files).

You will only see these changes if you get the source from CVS.