Shifter, fast YUI module building

By YUI TeamAugust 27th, 2012

Our build system has been on my mind lately, our old ant system was just getting too slow to keep up with the changes I was making and it was making it very difficult for our Automatic Travis Pull Request builds to test incoming changes properly if it took so long to build our modules.

One night a week or so back I couldn’t sleep, so I built Shifter. Shifter is a command line tool built in Node.js using the Gear.js build system. Gear.js was chosen as the base system because it’s what powers Mojito Shaker plus it’s small and good at what it does.

Why It’s Faster

Shifter get’s a lot of its speed from streaming changes throughout the build process. Our old builder would write files into a temp directory, then perform its tasks on them (compress, filter, concat, etc). This made it inherently slow to begin with. Shifter processes all its files as strings and only writes to disk when it has completed its job. Shifter replaces our need to use Ant, but we still have a Java dependency (for YUI Compressor and YUI Test Coverage). However, these Java dependencies have been streamlined so they don’t use file access either. Both YUI Compressor and YUI Test Coverage have npm packages that allow them to be required as a Node.js module and used asynchronously. YUI Test Coverage was updated to allow you to pipe a file into the jar and print the results to standard out similar to the way YUI Compressor handles it. This allows the Node.js module to spawn the Java process under the hood while piping the files into the jar and not writing files to disk.

The other way that Shifter optimizes builds is that it doesn’t do anything twice if it doesn’t have to. With the old builder, modules would get built once for debug, raw and minified files, then again for rollups and again for coverage files. This wasted a lot of cycles, but it was more of a limitation of ant (XML is not a coding language). Shifter deals with rollups a little differently. Instead of building the files again, the default action for a rollup is merging several prebuilt modules into one file and stamping it with the rollup module stamp (the YUI.add wrapper.

Depending on your system, you should see build times improved anywhere from 30% to 1000%!

Installation

Installation is simple, npm -g install shifter, it’s been tested on OSX, Linux and Windows as long as you are running a recent Node.js build.

When To Use It

We are recommending anyone that builds a YUI module (for themselves or as a contribution) should switch to using Shifter. When you run Shifter on your module the first time, it will parse your ant build files for you and create a build.json file to use in their place. Once you have verified that your module builds properly, we are recommending that you remove your ant files and forget about them.

Gallery Builds

For now, please do not remove your ant files from your Gallery modules. The current Gallery build is being rewritten, once that is complete it will have support for Shifter builds (actually it will only use Shifter). For now, feel free to use it to build your Gallery modules but remember that if you submit a CDN request you must have your ant files in your repository.

Docs and Stuff

Of course it has docs, how else are you going to know how to if it’s a feature or a bug! I’ve also documented the relevant keys from the build.json file to help when you are building your own.

And as always, Pull Requests are always welome!

5 Comments

  1. This is awesome, and greatly increases development speed. Thanks for your hard work.

    When will we start to see build.json be the standard in the yui3 repo? Additionally, is there an ETA for the gallery rewrite to be completed?

    I worry if I’m using Shifter locally, I will not properly test my gallery modules and they will break under ant :)

  2. You should start seeing those files being removed from the YUI repo starting today ;)

    As for the Gallery rewrite, I’ve been working on it for about a month now but I don’t have an ETA yet ;)

  3. WOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO!!!

    That is all. =D

  4. I echo Derek’s comment. So awesome. :D

  5. [...] files from our 3.x branch, and rebuilt all of our modules using the latest version of Shifter, our new build tool, which uses UglifyJS by [...]