Thursday, May 31, 2012

Package Management for Front End JavaScript

I have been working on volo, which is a real, working attempt at providing a solution for front end Javascript package management.

It seems like more people are starting to look at this, and here are my suggestions on what the solution should look like. I do not expect everyone to use volo, but if we all agreed to some basics, then it will allow some easier interop (see the end of this post).

Resist the temptation to make your own registry

One of the harder parts is to agree on a registry for code. With volo I have gone with using GitHub because:
  • It already exists and works/scales.
  • It has social/feedback tools and search.
  • Used by many JS libraries already, it is already part of developer workflow.
I suggest starting with GitHub as the registry. They have a nice API that volo uses to get tags and files out of a repo.

It would be easy to replicate a stand-alone server that has the same API later if it seemed like GitHub as the registry did not make sense, but by using GitHub and its API, some of the mundane bikeshedding over API and registry construction goes away.

The "owner/repo" IDs with github, vs just the "repo" single namespace that is in something like npm, is a distinct benefit to have for a registry. Forks should be possible, with the default search on just "repo" giving the most popular version, which is usually the original repo. Given GitHub's social tools, they have a way to measure popularity, and it seems to work out pretty well. 

Easy convention

Try to find a convention so that configuration is not required. The basic convention that volo uses: it checks for any explicit configuration (see below) but if none is found, it pulls down the zipball of a version tag, and if there is one JS file at the top level of the zipball, that is the script that is installed.

There is a bit more to the convention, but this basic convention works for many JS libraries. It helps encourage providing small libraries that can be composed together well with other libraries in other repos.

Easy configuration

The above convention does not work for every project. Some folks do not want to host their code on github, and some projects need to do a "build" step to deliver the final code, and like hosting that built code outside the git repository. So there needs to be a way to configure what to download for a dependency.

volo uses a package.json property, volo.url, to find it. Example for jQuery:

{
    "volo": {
        "url": "http://code.jquery.com/jquery-{version}.js"
    }
}

It supports {version} substitution with a version value from a version tag.

More is documented in the package.json page. volo also understands a volo.archive, but I want to reduce that config to just volo.url, and have it do content type detection to know if it is a single JS file or an archive zip/tarball.

Do not require server devs to change

volo has a "shims" repo it checks if a library does not have the package.json "volo" property it is looking for, so it makes it easy to bootstrap new libraries into the system without requiring the library author to do anything.

Of course it is best and more distributed if the library author supports the package.json property/properties directly, but for now it has been easy to consume scripts without getting complete buy-in from library authors.

Let's coordinate

I would love it if there were other tools besides volo for this functionality, as long as we agreed to the above GitHub bootstrapping and some package.json properties we can all read and understand.

In particular, I do not want these properties under a "volo" name in the package.json. I am doing that for now just so I do not claim a more generic name without any agreement with others. What is a name we can use instead? "frontend"? "browser"? I'm up for a more generic name so that we can open up the client tool building space.

I can be reached on gmail at jrburke. If you want a public space to talk, there is the volo list, but I am happy to talk on another list if that is preferable.


4 comments:

Yarden said...

Hmmm, a very interesting concept. I'm looking at having to do the same sort of thing. I have a front-end framework, and need to be able to support third-party plugins. I can try to lend what help I can, but I'm by no means an expert.

James Burke said...

Yarden: If you like, you can start by just trying out volo, if it feels right for fetching dependencies, and give feedback on the volojs list.

Brian C said...

I wrote a little script called Grabass (https://github.com/brian-c/grabass) to download and install remote assets, and it's been working really well. You can install it with `gem install grabass`.

It's absolutely unopinionated in that it cares neither what kind of files you get (scripts, fonts, images, etc. are all fair game) nor what you do with them. It also has no concept of a repo, so you're not dependent on the resources' creators to make them available in someone else's dedicated repo. There's might also be some security benefit to downloading straight from the source.

Configuration is in a relatively simple JSON file. Simply map a source (a remote .js or .zip file or git repo), to an optional selection from that source (i.e. "lib/*.js") and a destination (like "src/scripts"). There's an example at https://github.com/brian-c/grabass/blob/master/example.json

I'd be very curious to get your opinion.

James Burke said...

Brian C: that seems very flexible, although I would like something that understood github IDs. For instance, being able to specify 'owner/repo/tag' as a shorthand for the dependency and version, then have

So in your example.json, be able to do {"jquery/jquery/1.7.1", "temp/jquery.js"} and have that github ID resolved to the file download location.