volo, a command line tool to create web projects, add front end dependencies and automate tasks, is now at 0.2.3. Get it via npm:
npm install -g volo
The complete set of changes are here. The notable ones:
1. semver ranges can be used with dependencies now. For example:
volo add requirejs/~2
will download the latest 2.x version of jrburke/requirejs from GitHub.
2. volo add mentions possible incompatibilities after doing a nested dependency install, and gives commands for manually choosing one of the other versions.
3. volo now has a simple web site that hopefully gives a better idea intro to volo. Many thanks go to James Long for starting the structure and helping to refine its message. There is more to do for the site, but it is already much better than what was there before.
Showing posts with label volo. Show all posts
Showing posts with label volo. Show all posts
Wednesday, August 29, 2012
Wednesday, July 25, 2012
On client components for web apps
This is a response to a blog post by TJ Holowaychuk about browser-based components for web applications, and Isaac's notes on TJ's post.
I am going to try to make this brief because I get tired of people in the Node community wanting to apply the same patterns from Node in the browser. I feel like I say these things on a periodic basis, but human communication is hard, and I certainly could do better. But I also want to get back to just making things. So this will be terser than I normally would like.
Web components
I suggest TJ look at volo, my attempt in this space. It does lots of what he describes already, and it can even be used as a module in another command line tool. We use volo for some things in Mozilla already.
volo uses GitHub as the component registry. It does so without the downsides that TJ mentions.
Specifically, volo uses the GitHub HTTP API to get version tags, do registry searches. I grabs .zip snapshots for a given version/github commit/branch, so the command line tool (the consumer) does not need use git. Git is not necessary on the client side.
This means the downloaded code is smaller -- no need to get a full repo and all of its commits.
volo also understands dependencies via the shorter "owner/repo/tag" IDs instead of the full github URLs.
It has a "shim" repo that means it can support installing components without needing the author of the component to conform to some new publishing system. Since it allows {version} replacement in URLs, the registry setup just needs to be done once. From then on, normal best-practice versioning via git tags is enough.
Some other notes in this post.
Base module format
Node bros, the AMD trolling is getting tiresome. Node's module system is woefully under-specified for web-based loading. While you can limp along with browserify, there are still these issues:
* For builds you need a wrapped format. For CDN deployment you need a wrapped format. browserify uses a wrapped format. AMD anyone? For that reason alone, AMD will never go away. Get used to it already.
* Web code needs a callback-style require for on-demand loading.
* Browserify's uses of file paths for IDs is awful for mixed local and CDN-based loading. Module IDs need to stay in ID format, not translated to a specific file path.
* Loader plugins reduce the need for callback-style APIs, and callback pyramid of doom, or inside-out callback hell, or the need for promise based programs. This more than makes up for the extra level of indent in AMD.
Loader plugins solve the translation issues TJ talks about, and they can participate in optimization builds, meaning templating engines can inline the JS function form of the template. Ditto for language transpilers like CoffeeScript.
By doing this:
define(function (require) {
//node module code in here.
//Return module value instead
//of needing `module.exports`
});
you have an AMD module.
Quit dismissing AMD for surface issues. AMD avoids mandating translation layers that lead to more things for the developer to understand and fix, and more process for the user to go through to deploy code. It is a net win when the source file works when deployed anywhere, without requiring specialized builds/converters.
Even if you want to personally use Node style and always do builds before loading in the browser, AMD is a great target for the built, wrapped format. You can even use the requirejs optimizer to do this, with the cjsTranslate option.
The universal module boilerplate gets simpler when Node supports AMD's define along with Node's existing module format. If you want to help improve the ugliness, start there.
AMD comes from real world exprience in Dojo with trying to deploy an unwrapped module format that depended on XHR+eval in dev and a wrapped format for builds. Yes, you can something to to work but the second order translation and support costs are not worth it. Some environments disallow eval. CORS configuration is awkward, and potentially hazardous if your API is on the same domain and CORS is done incorrectly.
The simplicity of the complete module lifecycle is worth the function wrapping. Quit looking just at what you type once, and consider the complete code lifecycle, and how much time could be wasted there.
npm's registry as the component registry
The implied rules with npm and node's module behavior are not good fits for front end web development:
By using GitHub, it comes with user auth handled, private repos, and robust social tools that will not be matched by a something like npm because the financial incentives are not there. Plus developers already use it. For simple open source sharing, make it easy without introducing more things in the middle.
Github as the registry is not perfect, and we still need some standalone servers that can be run inside corporations/for mirrors, but I would model those standalone servers on the github API. At least the default case of a public repo can be bootstrapped very quickly.
I am going to try to make this brief because I get tired of people in the Node community wanting to apply the same patterns from Node in the browser. I feel like I say these things on a periodic basis, but human communication is hard, and I certainly could do better. But I also want to get back to just making things. So this will be terser than I normally would like.
Web components
I suggest TJ look at volo, my attempt in this space. It does lots of what he describes already, and it can even be used as a module in another command line tool. We use volo for some things in Mozilla already.
volo uses GitHub as the component registry. It does so without the downsides that TJ mentions.
Specifically, volo uses the GitHub HTTP API to get version tags, do registry searches. I grabs .zip snapshots for a given version/github commit/branch, so the command line tool (the consumer) does not need use git. Git is not necessary on the client side.
This means the downloaded code is smaller -- no need to get a full repo and all of its commits.
volo also understands dependencies via the shorter "owner/repo/tag" IDs instead of the full github URLs.
It has a "shim" repo that means it can support installing components without needing the author of the component to conform to some new publishing system. Since it allows {version} replacement in URLs, the registry setup just needs to be done once. From then on, normal best-practice versioning via git tags is enough.
Some other notes in this post.
Base module format
Node bros, the AMD trolling is getting tiresome. Node's module system is woefully under-specified for web-based loading. While you can limp along with browserify, there are still these issues:
* For builds you need a wrapped format. For CDN deployment you need a wrapped format. browserify uses a wrapped format. AMD anyone? For that reason alone, AMD will never go away. Get used to it already.
* Web code needs a callback-style require for on-demand loading.
* Browserify's uses of file paths for IDs is awful for mixed local and CDN-based loading. Module IDs need to stay in ID format, not translated to a specific file path.
* Loader plugins reduce the need for callback-style APIs, and callback pyramid of doom, or inside-out callback hell, or the need for promise based programs. This more than makes up for the extra level of indent in AMD.
Loader plugins solve the translation issues TJ talks about, and they can participate in optimization builds, meaning templating engines can inline the JS function form of the template. Ditto for language transpilers like CoffeeScript.
By doing this:
define(function (require) {
//node module code in here.
//Return module value instead
//of needing `module.exports`
});
you have an AMD module.
Quit dismissing AMD for surface issues. AMD avoids mandating translation layers that lead to more things for the developer to understand and fix, and more process for the user to go through to deploy code. It is a net win when the source file works when deployed anywhere, without requiring specialized builds/converters.
Even if you want to personally use Node style and always do builds before loading in the browser, AMD is a great target for the built, wrapped format. You can even use the requirejs optimizer to do this, with the cjsTranslate option.
The universal module boilerplate gets simpler when Node supports AMD's define along with Node's existing module format. If you want to help improve the ugliness, start there.
AMD comes from real world exprience in Dojo with trying to deploy an unwrapped module format that depended on XHR+eval in dev and a wrapped format for builds. Yes, you can something to to work but the second order translation and support costs are not worth it. Some environments disallow eval. CORS configuration is awkward, and potentially hazardous if your API is on the same domain and CORS is done incorrectly.
The simplicity of the complete module lifecycle is worth the function wrapping. Quit looking just at what you type once, and consider the complete code lifecycle, and how much time could be wasted there.
npm's registry as the component registry
The implied rules with npm and node's module behavior are not good fits for front end web development:
- Forcing a directory structure is complicating project layout and loading for web-based projects. It should be possible to publish and install single JS libraries as single files. volo can do this.
- Related: the "index.js" convention is awful for web development and debugging. Debugging 'jquery.js' instead of trying to find 'jquery/index.js' in the web tools? No thank you.
- npm's registry namespace is already polluted. Check searches for 'jquery'. Maybe that just means having a separate npm registry-based registry for client code. But if there needs to be a separate repo, might as well use one that can adapt better to front end development. Like single JS/CSS file installs without extra Java-esque directory structures and metadata debris on the file system.
By using GitHub, it comes with user auth handled, private repos, and robust social tools that will not be matched by a something like npm because the financial incentives are not there. Plus developers already use it. For simple open source sharing, make it easy without introducing more things in the middle.
Github as the registry is not perfect, and we still need some standalone servers that can be run inside corporations/for mirrors, but I would model those standalone servers on the github API. At least the default case of a public repo can be bootstrapped very quickly.
Saturday, July 07, 2012
volo 0.2.2 released
volo 0.2.2, a JS package manager and project automator, has been released. To install/update:
Here is a list of changes. Probably the most notable one:
volo create will run npm if there is a package.json in the downloaded project template with a dependencies property, and if there is not an existing node_modules directory.
This makes it easier to share volo commands between projects and their volofiles.
Example: the create-responsive-template now uses these separate volo commands in its volofile:
npm install -g volo
Here is a list of changes. Probably the most notable one:
volo create will run npm if there is a package.json in the downloaded project template with a dependencies property, and if there is not an existing node_modules directory.
This makes it easier to share volo commands between projects and their volofiles.
Example: the create-responsive-template now uses these separate volo commands in its volofile:
- appcache: generates an appcache manifest for a project.
- ghdeploy: deploys a directory of code to github pages.
Wednesday, June 20, 2012
volo 0.2.1 released, using volo as a library
volo 0.2.1 has been released. This is a bug fix release, if you are using 0.2.0, you are encouraged to upgrade. Here is the list of fixes. To install:
npm install -g volo
Also new: the ghvolo project. It shows how to use volo as a library to just resolve dependency IDs to github-hosted resources. This is useful if you are building a command line interface for fetching front end dependencies and you want to use the same github resolution logic as volo, but without using volo to do the actual installation of dependencies.
ghvolo is a command line tool that supports "search" and "resolve". Nothing is installed by using ghvolo. It just uses volo as library to resolve IDs to some JSON data. See the README for more information and some sample commands.
npm install -g volo
Also new: the ghvolo project. It shows how to use volo as a library to just resolve dependency IDs to github-hosted resources. This is useful if you are building a command line interface for fetching front end dependencies and you want to use the same github resolution logic as volo, but without using volo to do the actual installation of dependencies.
ghvolo is a command line tool that supports "search" and "resolve". Nothing is installed by using ghvolo. It just uses volo as library to resolve IDs to some JSON data. See the README for more information and some sample commands.
Thursday, May 31, 2012
Package Management for Front End JavaScript
I have been working on volo, which is a real, working attempt at providing a solution for front end Javascript package management.
It seems like more people are starting to look at this, and here are my suggestions on what the solution should look like. I do not expect everyone to use volo, but if we all agreed to some basics, then it will allow some easier interop (see the end of this post).
Resist the temptation to make your own registry
One of the harder parts is to agree on a registry for code. With volo I have gone with using GitHub because:
It would be easy to replicate a stand-alone server that has the same API later if it seemed like GitHub as the registry did not make sense, but by using GitHub and its API, some of the mundane bikeshedding over API and registry construction goes away.
The "owner/repo" IDs with github, vs just the "repo" single namespace that is in something like npm, is a distinct benefit to have for a registry. Forks should be possible, with the default search on just "repo" giving the most popular version, which is usually the original repo. Given GitHub's social tools, they have a way to measure popularity, and it seems to work out pretty well.
Easy convention
Try to find a convention so that configuration is not required. The basic convention that volo uses: it checks for any explicit configuration (see below) but if none is found, it pulls down the zipball of a version tag, and if there is one JS file at the top level of the zipball, that is the script that is installed.
There is a bit more to the convention, but this basic convention works for many JS libraries. It helps encourage providing small libraries that can be composed together well with other libraries in other repos.
Easy configuration
The above convention does not work for every project. Some folks do not want to host their code on github, and some projects need to do a "build" step to deliver the final code, and like hosting that built code outside the git repository. So there needs to be a way to configure what to download for a dependency.
volo uses a package.json property, volo.url, to find it. Example for jQuery:
{
"volo": {
"url": "http://code.jquery.com/jquery-{version}.js"
}
}
It supports {version} substitution with a version value from a version tag.
More is documented in the package.json page. volo also understands a volo.archive, but I want to reduce that config to just volo.url, and have it do content type detection to know if it is a single JS file or an archive zip/tarball.
Do not require server devs to change
volo has a "shims" repo it checks if a library does not have the package.json "volo" property it is looking for, so it makes it easy to bootstrap new libraries into the system without requiring the library author to do anything.
Of course it is best and more distributed if the library author supports the package.json property/properties directly, but for now it has been easy to consume scripts without getting complete buy-in from library authors.
Let's coordinate
I would love it if there were other tools besides volo for this functionality, as long as we agreed to the above GitHub bootstrapping and some package.json properties we can all read and understand.
In particular, I do not want these properties under a "volo" name in the package.json. I am doing that for now just so I do not claim a more generic name without any agreement with others. What is a name we can use instead? "frontend"? "browser"? I'm up for a more generic name so that we can open up the client tool building space.
I can be reached on gmail at jrburke. If you want a public space to talk, there is the volo list, but I am happy to talk on another list if that is preferable.
It seems like more people are starting to look at this, and here are my suggestions on what the solution should look like. I do not expect everyone to use volo, but if we all agreed to some basics, then it will allow some easier interop (see the end of this post).
Resist the temptation to make your own registry
One of the harder parts is to agree on a registry for code. With volo I have gone with using GitHub because:
- It already exists and works/scales.
- It has social/feedback tools and search.
- Used by many JS libraries already, it is already part of developer workflow.
It would be easy to replicate a stand-alone server that has the same API later if it seemed like GitHub as the registry did not make sense, but by using GitHub and its API, some of the mundane bikeshedding over API and registry construction goes away.
The "owner/repo" IDs with github, vs just the "repo" single namespace that is in something like npm, is a distinct benefit to have for a registry. Forks should be possible, with the default search on just "repo" giving the most popular version, which is usually the original repo. Given GitHub's social tools, they have a way to measure popularity, and it seems to work out pretty well.
Easy convention
Try to find a convention so that configuration is not required. The basic convention that volo uses: it checks for any explicit configuration (see below) but if none is found, it pulls down the zipball of a version tag, and if there is one JS file at the top level of the zipball, that is the script that is installed.
There is a bit more to the convention, but this basic convention works for many JS libraries. It helps encourage providing small libraries that can be composed together well with other libraries in other repos.
Easy configuration
The above convention does not work for every project. Some folks do not want to host their code on github, and some projects need to do a "build" step to deliver the final code, and like hosting that built code outside the git repository. So there needs to be a way to configure what to download for a dependency.
volo uses a package.json property, volo.url, to find it. Example for jQuery:
{
"volo": {
"url": "http://code.jquery.com/jquery-{version}.js"
}
}
It supports {version} substitution with a version value from a version tag.
More is documented in the package.json page. volo also understands a volo.archive, but I want to reduce that config to just volo.url, and have it do content type detection to know if it is a single JS file or an archive zip/tarball.
Do not require server devs to change
volo has a "shims" repo it checks if a library does not have the package.json "volo" property it is looking for, so it makes it easy to bootstrap new libraries into the system without requiring the library author to do anything.
Of course it is best and more distributed if the library author supports the package.json property/properties directly, but for now it has been easy to consume scripts without getting complete buy-in from library authors.
Let's coordinate
I would love it if there were other tools besides volo for this functionality, as long as we agreed to the above GitHub bootstrapping and some package.json properties we can all read and understand.
In particular, I do not want these properties under a "volo" name in the package.json. I am doing that for now just so I do not claim a more generic name without any agreement with others. What is a name we can use instead? "frontend"? "browser"? I'm up for a more generic name so that we can open up the client tool building space.
I can be reached on gmail at jrburke. If you want a public space to talk, there is the volo list, but I am happy to talk on another list if that is preferable.
Thursday, May 03, 2012
Web app template update, now with GitHub auth
This is a quick update about the latest version of the web app template I have been working on.
You can check out some background on this app template from the previous post.
The big change since that last post: Dan Mosedale had some good feedback about treating GitHub Pages as a deploy target, so I changed around the template to do do that.
As part of these changes, volo has a new release 0.1.2 that has support for doing GitHub authorization to do OAuth-based API calls with GitHub. The app template uses this to create a GitHub repo on the fly for GitHub pages deployment.
The nice thing about this setup is that the app creation is simpler (no more questions), and you can decide later to deploy to GitHub pages without needing to make the decision up front:
See the video for a more complete walkthrough, but basically to get started now it just looks like this:
volo ghdeploy (copies the built contents and pushes them to GitHub Pages)
Useful links from the video:
You can check out some background on this app template from the previous post.
The big change since that last post: Dan Mosedale had some good feedback about treating GitHub Pages as a deploy target, so I changed around the template to do do that.
As part of these changes, volo has a new release 0.1.2 that has support for doing GitHub authorization to do OAuth-based API calls with GitHub. The app template uses this to create a GitHub repo on the fly for GitHub pages deployment.
The nice thing about this setup is that the app creation is simpler (no more questions), and you can decide later to deploy to GitHub pages without needing to make the decision up front:
See the video for a more complete walkthrough, but basically to get started now it just looks like this:
volo create myproject volojs/create-responsive-template
cd myproject
volo appcache (generates the appcache-enabled build)volo ghdeploy (copies the built contents and pushes them to GitHub Pages)
Useful links from the video:
- create-responsive-template - the web app template.
- jrburke/gaia-devserver - easy way to serve the Boot to Gecko Gaia UI in Nightly Firefox on your Desktop.
- Getting Started with the Mozilla Web Apps API - for installing apps into Gaia.
Friday, March 09, 2012
A dependency manager for web projects
The problem
Effective, modular development usually comes with a dependency management tool. Think easy_install for python, ruby gems for ruby, npm for Node.
Front-end web development does not have this kind of tool. Traditionally it has been ad-hoc script management done by the developer. However, now that modular development is available now, and will become more prevalent as ECMAScript committee solidifies native module support, front end developers should have a dependency management tool.
volo, a solution
volo is a concrete attempt at this solution. While I do not believe it is fully formed yet, it is already useful, and by having something real to start from, it will better inform any coordinated effort.
volo does not have to be the only dependency management tool. Hopefully we can use it as way to discuss common approaches that we can all share.
There have been some efforts in this space, in particular npm, cpm and bpm. I go into why those were not used for volo in a Prior art page.
The main point of departure is the choice of registry. volo uses GitHub.
GitHub as the registry
One of the difficulties with running a dependency management tool is having an effective way to find dependencies. The JavaScript community has really embraced GitHub, and I think it has all of the right pieces to serve as a registry:
Conventions
volo uses some conventions that are used by some JS projects already, to make it easier to bootstrap. For other projects, usually one package.json property is enough for it to be used by volo.
Check out the Library best practices document for the conventions and configuration used by volo. volo has some project-specific overrides it uses during this bootstrap phase, to simulate how those projects would behave if they used the package.json property that mapped to their development style.
Try it Out
So try it out. It is just one JS file that runs in Node, so it is easy to get rid of it if you do not like it.
volo's 0.1.0 release has search built in, so it is fun just to try the following kinds of commands to see what it finds on GitHub:
Next Steps
While volo is useful already, it is certainly not done. In particular,
If you have a specific code issue, feel free to open a volo issue.
Effective, modular development usually comes with a dependency management tool. Think easy_install for python, ruby gems for ruby, npm for Node.
Front-end web development does not have this kind of tool. Traditionally it has been ad-hoc script management done by the developer. However, now that modular development is available now, and will become more prevalent as ECMAScript committee solidifies native module support, front end developers should have a dependency management tool.
volo, a solution
volo is a concrete attempt at this solution. While I do not believe it is fully formed yet, it is already useful, and by having something real to start from, it will better inform any coordinated effort.
volo does not have to be the only dependency management tool. Hopefully we can use it as way to discuss common approaches that we can all share.
There have been some efforts in this space, in particular npm, cpm and bpm. I go into why those were not used for volo in a Prior art page.
The main point of departure is the choice of registry. volo uses GitHub.
GitHub as the registry
One of the difficulties with running a dependency management tool is having an effective way to find dependencies. The JavaScript community has really embraced GitHub, and I think it has all of the right pieces to serve as a registry:
- Allows forks. Namespacing is possible via the use of the owner's name, not just a project name.
- Has a search function that can give owner/repo values for a given dependency name.
- Encourages open source and social communities around the source.
- Supports tags for versions.
- GitHub has a great simple API for getting versions/repo info.
- Provides zipballs/tarballs of tags/branches.
- Dependencies can be pulled not only by version tag but by branch name.
- Provides a Downloads area for built code.
Conventions
volo uses some conventions that are used by some JS projects already, to make it easier to bootstrap. For other projects, usually one package.json property is enough for it to be used by volo.
Check out the Library best practices document for the conventions and configuration used by volo. volo has some project-specific overrides it uses during this bootstrap phase, to simulate how those projects would behave if they used the package.json property that mapped to their development style.
Try it Out
So try it out. It is just one JS file that runs in Node, so it is easy to get rid of it if you do not like it.
volo's 0.1.0 release has search built in, so it is fun just to try the following kinds of commands to see what it finds on GitHub:
> volo search jquery
> volo search backbone
> volo add jquery
> volo add dojo
> volo add dijit
> volo add ember
Next Steps
While volo is useful already, it is certainly not done. In particular,
- Do the library best practices make sense? Also, there is info the rules "volo add" uses. I am hopeful volo allows for a reasonable default convention but still give enough flexibility via a couple package.json properties, including a way to embed a /*package.json*/ comment for single file JS libraries.
- volo does not install nested dependencies yet. It is doable, just needs a bit more coding.
- A private/local server that responds to the GitHub APIs used by volo would be useful. Not all code can be placed on GitHub or a public URL, and sometimes even for a developer, having a local cache server is useful. I'm hoping this server can proxy to GitHub for common public libraries and cache them, and then provide a mechanism for private libraries to live in that server.
- volo does not support .tar.gz files yet, but it might be nice if it did. zip files were just easier to get working cross-platform.
- Allow access to private GitHub repos by using OAuth with GitHub.
- Allow a "volo.versions" section in a package.json, to allow listing versions without having to use GitHub tags. This could be useful for libraries that do not want to host their code in GitHub.
If you have a specific code issue, feel free to open a volo issue.
Monday, February 20, 2012
Template for responsive mobile, offline web apps
It can be difficult to create a web app that is responsive to different screen resolutions and works well in mobile environments or environments with spotty network availability.
In order to make it easier, Jason Strimpel and I created a project template, create-responsive-template, that takes care of a few things for you:
Here is a video that shows how you can use the template:
The next steps I would like to see for this project template:
In order to make it easier, Jason Strimpel and I created a project template, create-responsive-template, that takes care of a few things for you:
- Uses Twitter Bootstrap for a baseline style and responsive UI
- A couple of small JS helpers for AppCache and network events
- A build step that optimizes JS and CSS and generates the AppCache manifest for you
Here is a video that shows how you can use the template:
The next steps I would like to see for this project template:
- An IndexedDB data layer: A JS helper library that looks like the IndexedDB API, but will use an API shim over Web SQL if IndexedDB is not available. The shim does not have to implement 100% of the IndexedDB API, but something that allows a good start to local data storage. I am looking at Lawnchair at the moment, but open to other ideas.
- An easy way to generate app store manifests for Mozilla Web Apps and Google Chrome Store.
Subscribe to:
Posts (Atom)