YUI and Travis sitting in a git-tree
By Dav GlassMay 11th, 2012
Over the last few weeks, I have spent a great deal of time getting YUI’s core tests executing on Travis CI. As of today, every push to our YUI 3 repo on GitHub results in over 6,000 (6,053 to be exact) unit tests being executed and logged. These tests include 1,130 of our core JavaScript-only unit tests executing natively inside of Node.js on versions 0.4.x and 0.6.x (with 0.7.x support to be added soon). We follow that with our full unit test suite (4,923 tests), running with my command-line YUITest/PhantomJS wrapper, Grover.
Today I want to tell you a little more about how and why we are doing this.
Why?
For the longest time, our tests and builds were a mystery to our users. They all happened behind closed doors and nobody really knew what we were doing. Over the last few years, we started adding more and more of our tests to our source tree, so that people can see what we are actively testing. This helped a little, but it didn’t show that we run every test on every build and that if a test fails, we don’t push that build to GitHub.
Then along came Travis CI, the open source continuous integration platform that hooks directly into GitHub. I played around with it for a day and immediately began moving our tests around so we could use it. We want to be more transparent in our processes and allow the public to see what we test and how we do it.
One thing we want to add to this process is access to our current code coverage report. Currently, we run a code coverage report daily from tests executed in FF12, Chrome-latest and IE8, but none of our users can see that we are at ~80% covered:
We are working very hard to rectify this issue and hopefully get this data out in the open for everyone to see.
How?
I had to make minimal changes to our core test suites in order to get them to run under Node.js in Travis CI. By “core test suites”, I mean any YUI module that can execute without the need for a working DOM. This includes, but is not limited to: YUI Core, Loader, YQL, Y.Array, Y.Object, etc. These modules are all perfectly usable inside of Node.js without modification.
Let’s look at the YQL tests as an example.
All I had to do was create a Node.js wrapper similar to the standard test wrapper we use in a browser and include the exact same test that’s executed in the browser.
Here is the YQL module’s wrapper:
Now, this same test module can be executed in a browser and in Node.js without modification!
Caveats?
In order for these tests to run natively in Node.js, they need to not interact with the DOM. For example, Y.Array consists of helper methods for dealing with Arrays in JavaScript. But there are a few tests that include dealing with DOM elements to ensure that the helper methods return the right values. For these tests, I had to move the DOM-related code into a new test and add that test to the list of ignored tests when Y.UA.nodejs is detected. This way, such tests are ignored in Node.js, but still run in the browser.
Here’s an example:
As you can see, it’s relatively easy to make our tests run in both environments to ensure that our code is stable and fully functional when used in Node.js as it is in the browser.
What is Grover?
Grover is a command-line tool that allows you to execute YUITest-based tests in PhantomJS. PhantomJS is a headless Webkit instance that allows you to render an HTML page without a GUI present. So Grover closes the gap on this and allows you to run our unit tests from the command-line inside of a CI system like Travis.
Grover is free and available via: npm install -g grover (You must have the PhantomJS binary installed before using Grover.)
How do we see all this?
Travis CI provides a full report of previous builds, as well as an up-to-date status information. Below are the links to our current projects hosted on Travis CI:
Here is a small snippet of what our build output looks like (from the build history for the yui3 repo):
What else does it do?
We are trying out the new Travis/GitHub Pull Request feature on all of our projects. This means that whenever a developer submits a Pull Request to us, Travis will automatically pull their code, merge it into master (on their server) and run our full unit test suite against it. Their “Travis Bot” will automatically post a comment back to the Pull Request telling the developer whether or not their patch passed its tests.
Here’s an example of a Pull Request passing:
And one where it fails:
What’s next
We plan on adding support for executing our tests with Yeti as soon as it becomes stable enough to run on each build. We are also looking into deploying our code coverage numbers as well. Other than that, feel free to tell us in what other ways we can be more open than we are now.
I, for one, am very happy with all of these new features and I hope you are too!
May 11, 2012 at 4:11 pm
This looks awesome Dav!
I cant helper wonder about the overlap between Yeti and Grover. Can you speak more about that?
May 11, 2012 at 7:37 pm
Automatic pull request testing is soo cool! Congratz!
May 13, 2012 at 1:36 pm
Wow, great work, Dave!
And the fact that the developer’s pull request will be automatically verified, is awesome!
But what if developer has updated some test too? This may happen often, especially on bug resolving…
May 13, 2012 at 9:21 pm
Good stuff. You might be able to test your DOM-dependent code using mock objects or something like Jasmine.
May 14, 2012 at 8:03 am
@dmtrinh — Theoretically that is possible, but given the large test suite we have it would take much longer than just running the tests against an actual DOM (PhantomJS)
May 14, 2012 at 8:05 am
@Antonio Morandi — The Pull Requests are not “automatically merged into master”, they are merged “on-the-fly” on the test machine and the tests are run against it.
Our developers will still engage with the Pull Request and vet it before it’s actually merged into our source tree.
This is just a “Hey, this change passed all tests. Can you look at pulling this in now”
May 14, 2012 at 8:06 am
@Simon — Grover is just a wrapper around PhantomJS so we can run DOM based tests from a CLI without a third party browser that must be up on each build.
I can’t say for sure, but I would assume that Grover’s wrapper will get integrated into Yeti in the future once a few more of Yeti’s features are ironed out.
June 15, 2012 at 11:11 pm
Any thoughts or plans to integrate this with Arrow https://github.com/arrowfmk/arrow and run tests against a real browser using selenium2/webdriver?
September 1, 2012 at 8:05 am
The test relocation effort has caused the link to YQL tests to point to nothingness.
September 4, 2012 at 8:36 am
@jshirley Thanks, link updated.