Referrer test page / Authenticated script src APIs

Referrer test page

I have a test page up now to test the HTTP referrer header values:

http://tagneto.org/test/reftest

If you press the test button, it should show the date from the server and the referrer header value.

There is also another URL to try in the text box: /cgi-bin/redirect.rb. That URL will redirect to the /cgi-bin/refer.rb. I wanted to be sure a redirect did not change the referrer value.

Now I'm going to use the test page to test posting from different pages and in different scenarios to see if I can get the referrer to change to some value than the one from the actual page making the SCRIPT SRC request. If you want to try posting to http://tagneto.org/cgi-bin/refer.rb from other domains, save the index.html page from the reftest URL mentioned above, as well as the SvrScript.js file that is in that same directory.

Authenticated script src APIs
If the referrer thing holds up, then I think it will be possible to proceed with some of the ideas mentioned in this post. However, that post assumed the authorization issue concerned a developer user name using the API on his/her own page.

For the larger issue of allowing a third party web site that deals with data that needs user authentication (without exposing the auth credentials to the third party web site), what about this:

• 3rd party web site gets an API key. That API key is only bound for certain domains and/or URLs.
• When a user uses the 3rd party web page that uses a protected data API, the data API server looks for authentication credentials in cookies that are only set to the data API server domain.
• If the user is not authenticated, then respond to the web page with an error, message of "auth.needed". The 3rd party web page puts up a DIV dialog saying that authorization is needed. If the user says OK, the 3rd party calls an "authenticate" JS method that is provided by the data service script library. An argument to authenticate method would be a return URL (an URL on the 3rd party site). The authenticate method would pop a window an prompt the user for auth credentials. A new window should be used so the user can check the domain/URL of the auth credential page. After successful login, the new window sets its location to the URL passed to authenticate. That URL on the 3rd party site should just close the window and force a refresh/update of the web page that uses the data API.
• If the user is authenticated, then the data service checks its own database to see if the user has explicitly granted data access to the 3rd party web page URL. If the user has not granted access yet (the data API would look at the referrer HTTP header), then data API responds to the web page with an error, "auth.needPermission". The 3rd party web page calls a "askPermission" JS method that is provided by the data service script library. askPermission would take a JS method callback as an argument. This method creates an IFRAME dialog in the page that asks if the user wants to give permission to this website to access the data. There could be a "Remember this answer" checkbox too, if the data API wanted to allow that. If the user says OK, then askPermission notifies the callback of the answer. The callback could then update the page accordingly (probably by re-calling the data API).

More experiments to do, but it seems promising.

Ctrl.js event listening updated

I was reading more on the web about registering event listeners in JavaScript. Mainly:

• http://www.quirksmode.org/blog/archives/2005/10/_and_the_winner_1.html
  (including comments)
• http://dean.edwards.name/weblog/2005/10/add-event2/
• http://www.quirksmode.org/js/introevents.html
  
• http://www.dustindiaz.com/rock-solid-addevent/

I reworked the event listening methods in Ctrl.js to do the following:

• modify the events that are used in MSIE to look more like the standard events.
• have "this" point to the element that has the event listener.
• allow the ability for any JavaScript object to have listeners.
• allow for cleaning up to avoid memory leaks for individual DOM elements (and optionally their children). Ctrl does register a global cleanup function on document unload, but it also surfaces a removeListeners() method for cases when there are multiple nodes attached/detached within the web page's lifetime, and memory leaks want to be contained as nodes are detached before the page is unloaded. (Even does cleanup for Mozilla because of bug: https://bugzilla.mozilla.org/show_bug.cgi?id=241518

I also renamed the methods, and now there is only one "register event listener" method, Ctrl.listen().

The changes are in CVS but not visible on the web site yet. Still need to round out a little more testing.

Security and the Dynamic Script Request API

Possible Security Issue

Dynamically adding SCRIPT SRC tags to pull in data as JavaScript is nice since it gets around cross-domain issues, but it could also be problematic if you are providing user data, data that required the user to authenticate in some way.

If the User A authenticates for using some JavaScript data APIs via SCRIPT SRC tags, it is possible that Hacker B could guide User A to Hacker B's page that includes a data source that uses User A's credentials.

User A authentication could have been through setting cookies that will travel to the data API domain. The API could also require that User A have an "API key" to use the API (User A registers with the data service provider, and receives a API key text string that must be passed to any data calls. Google Maps is an example).

Hacker B could find out User A's data API key, and if the API required authentication cookies to be set, Hacker B just needs to be sure User A authenticates before coming to Hacker B's page.

Protection Measures

Data Service Registration

• The user must register for an API key. To obtain the API key, the user must authenticate with the data service provider to prove their identity.
• As part of this registration, the user must specify which domains are allows to use the API key, and possibly which user names are authorized to use the API Key.
• Since the user is authenticated, that user name can use the API. Any other user names that are added to the API key must receive an email notification. The other user names must authenticate and grant permission to be on the allowed users for the API key.

Script SRC requests

• In addition to sending the API key, there must be a timestamp as part of the request. These parameters should be querystring parameters (like dataapi.com?key=xyz&stamp=55959833920)

Data Service Server

• The server for the data service should not allow caching of the data request. They should set the appropriate HTTP headers to expire the results of the request immediately, to avoid Hacker B taking advantage of the browser cache (that is why query string params are used on the URL too).
• The server will reject any request that does not have a registered domain in the Referrer HTTP header.
• The data service domain should only have data services. It should not allow any user-generated web pages under that domain. Ideally it should be unrelated to any domains that host user web page content -- no subdomains should match to prevent document.domain tricks from working.
  

Test scenarios

Even with these measures in place, are there other holes? I want to make some tests to verify the following:

• Does authenticating with the data service pass the cookies along properly for SCRIPT SRC requests?
• Does the Referrer field get set correctly for SCRIPT SRC requests? In all browsers?
  
• How can the Referrer field be spoofed? Use XMLHTTPRequest. But hopefully since only data services are allowed on the data service domain that can't be used. Hopefully the document.domain protection above will help too. Is there some way a server proxy running on Hacker B's site be used? Hopefully the User's authentication cookies can't travel to that domain.
• Even if the server sets expiration headers, will the browser still cache? Hopefully since a timestamp is sent and the URL uses query string params, Hacker B's URL won't be the same anyway, and therefore a different browser cache entry?

These are just first thoughts. I haven't checked yet to see if someone else has already worked this out. And it would be good to do some testing too, think of other scenarios that might break.

Tagneto 0.3.1 Released

Tagneto 0.3.1 was released yesterday. It is available for download from the SourceForge site.

Release Notes:

• Added source examples to website/documentation for ease of reference.
• Dynamic Script Request API document and SvrScript implementation of the API.
• Broke up Srvr.js into 3 files: SvrScript.js, SvrReq.js and SvrFrame.js.
• No more warnings when building in Eclipse. Most warnings were from autogenerated JAXB files.
• Fixed tagneto/src/test/RunTest.bat and RunTest.sh.
• Added ant lib to make it easy to build tagneto directly from cvs source checkout in Eclipse.
• Added notes on how to build Tagneto on the Where page.

Dynamic Script Request API

I updated the Tagneto website with a document describing the Dynamic Script Request API, and Tagneto's support of it through the Srvr.js library. The goal of the API is to make it possible to reliably use dynamically added SCRIPT SRC elements to load data and UI. It also makes it possible to send up more data to the server via multipart requests. This should make it easier to access web APIs across server domains (something XMLHTTPRequest normally cannot do). The limitation being the cross server web API must return JavaScript.

This document is another revision of the API first described in this post.

The changes to Srvr.js to support this API are not in the 0.3.0 release of Tagneto. Either grab from CVS or a later version of Tagneto (if available).

Using On-Demand JavaScript for Everything

I really like using On-Demand JavaScript (dynamically creating a SCRIPT tag and attaching to the HEAD) for loading new data or UI, particularly since it gets around the domain restrictions of XMLHTTPRequest. There is also no need to deal with messy XML to JavaScript transformations.

The challenges:

1. GET URL length restrictions.
2. Data has to be JavaScript.
3. Knowing when script load is complete.

Tagneto can help deal with these issues. The approach and issues are slightly different for retrieving UI (View) vs. Data (Model):

On Demand UI

1. GET URL length restrictions

This normally shouldn't be an issue for retrieving UI, particularly if the UI does not depend on particular request or user data (see Why document). If it is an issue, please see the approach mapped out below in On Demand Data.

2. Data has to be JavaScript

It is particularly tricky to encode HTML as JavaScript. Tagneto's ctrl: tags make this easier. An example, to create a JavaScript function that returns an HTML string based on some input parameters.

Sample.js:

<ctrl:function name="getHelloWorld" parameters="name, planet">
<b>Hello &ctrl:out-value:planet;, this is <ctrl:out value="name/>.</b>
</ctrl:function>

After running this file through Tagneto's view assembly tool, Sample.js will look something like this (the actual result is a little more complex and efficient than string concatenations::

function getHelloWorld(name, planet)
{
return '<b>Hello ' + planet + ', this is ' + name + .</b>';
}

Now Sample.js is ready to be dynamically included in the page.

3. Knowing when script load is complete

(Update 1/21/2006: the original post referred to a Srvr JS library. That library was renamed/changed and now is the Dsr library. This post was changed to reference the new library.)

The script can be dynamically added to the page using a method in Dsr.js:

Dsr.sendAndPoll('Sample.js', 'getHelloWorld', sampleLoadedListener);

where sampleLoadedListener is an object that could look like this:

var sampleLoadedListener = {
          onLoad:    function()
                       {
                       document.getElementById('outDiv').innerHTML = getHelloWorld();
                       },
          onError:   function(status, statusText, response)
                       {
                       alert('status: ' + status + ', error: ' + response);
                       },
          onTimeout: function() { alert('timeout'); },
          timeout:   30
          };

sampleLoadedListener.onLoad will be called once getHelloWorld is detected as being defined.

On Demand Data

This section is bit more experimental. It is just a first thought, probably needs more work. I think I'll need to make changes to Dsr to have it play nice with this model, perhaps even provide convenience method to handle multi part URLs.

1. GET URL length restrictions

(UPDATE 1/21/2006: The API described below has been significantly changed. See the Dynamic Script Request API for the new version. The Dsr.js library now implements the DSR API.)

It seems like URLs can only be at maximum around 1KB, so this can be problematic if you need to send a large amount back to the server. What about using multiple requests (parts) to post the data back? The server would collect the parts, and on the final part, do the action. Some spec is needed for GET parameters on the part URLs:

?part=currentPartNumber.totalParts
&succes=methodName
&error=methodName
&partComplete=methodName
[&actual data to give to server]

An example URL (broken into multiple lines for readability):

?part=1.4
&success=myDataLoaded
&error=myError
&partComplete=myPartComplete
&name=foo
&quantity=3
&....

1.4 means this is part 1 of a 4 part request.

myDataLoaded is a JavaScript function already defined in the page, and it takes one parameter, the actual JavaScript response data from the server after the server finishes collecting all the parts and processing the requests.

myError is a JavaScript function already defined in the page that would take perhaps an error object/message as the only parameter.

partComplete is a JavaScript function already defined in the page that would take the part number that was just completed by the server. The server's response would call this method as the very last line in the response.

As the JavaScript gets notification of a successful part being processed
(via the partComplete function), it would attach the next part as a SCRIPT element to the page.

The method callback parameters only need to be sent up as part of the first part of the total request.

2. Data has to be JavaScript

Frameworks like DWR or JSON (in the Java world) may be able to help with dynamic data (based on request or user data). If it is static data, custom tags using Tagneto's org.tagnetic.core.tags.define.DefineInclude tag handler or the view:xmldatasource tag could be used to transform XML data into JavaScript.

3. Knowing when script load is complete

Completion is known by the calling of the success/partComplete/error functions. Srvr.Script should be changed to allow the checkString to be optional, and to provide a wrapper around this approach.

CVS source update to make building easier

I updated the CVS source last night to make it easier for people who try to build the source from a CVS checkout. Changes:

• No missing dependencies
• tagneto/src/test/RunTests.bat should now run all the tests (still have to check RunTests.sh)
• Fixed warnings when building in Eclipse (warnings mostly from the autogenerated JAXB code that is used to parse tagneticconfig files).

You will only see these changes if you get the source from CVS.

http://tagneto.org/who

Entry for allowing comments about the following page:

http://tagneto.org/who

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/when

Entry for allowing comments about the following page:

http://tagneto.org/when

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/how/reference/js

Entry for allowing comments about the following page:

http://tagneto.org/how/reference/js

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/how/reference/tagneticconfig/

Entry for allowing comments about the following page:

http://tagneto.org/how/reference/tagneticconfig/

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/how/reference/tags/CtrlTags.html

Entry for allowing comments about the following page:

http://tagneto.org/how/reference/tags/CtrlTags.html

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/how/reference/tags/HtmlTags.html

Entry for allowing comments about the following page:

http://tagneto.org/how/reference/tags/HtmlTags.html

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/how/reference/js/DynamicScriptRequest.html

Entry for allowing comments about the following page:

http://tagneto.org/how/reference/js/DynamicScriptRequest.html

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/how/reference/tags/ViewTags.html

Entry for allowing comments about the following page:

http://tagneto.org/how/reference/tags/ViewTags.html

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/how/quickstart/AlternateSyntax.html

Entry for allowing comments about the following page:

http://tagneto.org/how/quickstart/AlternateSyntax.html

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/how/quickstart/Overlays.html

Entry for allowing comments about the following page:

http://tagneto.org/how/quickstart/Overlays.html

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/how/quickstart/CustomTag.html

Entry for allowing comments about the following page:

http://tagneto.org/how/quickstart/CustomTag.html

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/how/quickstart/Xml.html

Entry for allowing comments about the following page:

http://tagneto.org/how/quickstart/Xml.html

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/how/quickstart/Html.html

Entry for allowing comments about the following page:

http://tagneto.org/how/quickstart/Html.html

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/how

Entry for allowing comments about the following page:

http://tagneto.org/how

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/where

Entry for allowing comments about the following page:

http://tagneto.org/where

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/why

Entry for allowing comments about the following page:

http://tagneto.org/why

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/what

Entry for allowing comments about the following page:

http://tagneto.org/what

Comments may be modified/purged periodically as the content of the page referenced above changes.

Comments for http://tagneto.org/

Entry for allowing comments about the following page:

http://tagneto.org/

Comments may be modified/purged periodically as the content of the page referenced above changes.

Tagneto 0.3.0 Released

I just released version 0.3.0 of Tagneto. From tagneto.org:

Tagneto is a web developer tool and JavaScript libraries to aid MVC development of XML user interfaces, with HTML web applications (DHTML, AJAX, RIA, Web 2.0, etc...) being the primary target. It is available under the GNU Lesser General Public License (LGPL).

Tagneto includes a "View Assembly" web developer tool written in Java (1.4+ supported). It does not require Java to run on a web server or to be installed on the end user's computer. The Java tool is an HTML/XML parser that allows the web developer to assemble the HTML/XML source into the final set of pages that will be used as the web application. The source does not have to be valid HTML or XML, but the tags that Tagneto handles do have to be well-formed, as defined by one of the supported syntaxes. Similar technologies to the View Assembly tool would be JSP/ASP/PHP. Read the Why document to understand the motivation for Tagneto and how it is different (it will also work with JSP/ASP/PHP).

Tagneto also comes with some JavaScript (JS) libraries that aid building the Controller in JavaScript ("JavaScript" is used on this web site, even though ECMAScript is the proper name). The helper JS libraries are not required to use the View assembly part of the tool.

Tagneto was designed to support internationalization (I18N) and localization (L10N), both in the output pages and in the syntax and wording of the actual Tagneto tags. However, the I18N and L10N support has the least amount of testing, so you may find issues.

For more information on Tagneto features (like non-invasive source configuration, overlays, custom tags), please read the What page.