How to Write End-to-End Software Tests Like a BOSS!

I’m a big fan of writing tests — especially unit tests — but sometimes there’s a limit to how much you can test with them.

Unit tests confirm that each component is fully functioning on its own, but in the end, you definitely need a few “end-to-end” tests as well to check that all of the different components of your system are communicating with each other properly.

Unit tests — check!

I recently started to write functional tests at Logz.io, and I knew that I was going to use Selenium because of its community support as well as my prior positive experiences — I used it a lot in the past with C# and Java drivers.

However, there were various problems with those drivers, and I found myself writing a bunch of helper methods to get around those issues (especially when polling for various actions to happen). In addition, it always felt a little weird writing functional tests for a website in C#/Java because your “API” to communicate with the web app is the HTML/CSS/JS. You sometimes find yourself injecting JavaScript code to the browser via C#, and it’s never a comfortable thing to do.

So, this time, I started searching for a JavaScript framework.

I admit that I didn’t do a thorough investigation into the various frameworks out there, so I won’t go into comparisons of them but rather give just a quick comparison that will focus on the parts in which I was interested. I chose to go with Intern. It’s less popular than Protractor or Nightwatch.js, but it really does an amazing job.

Here are some of Intern’s really appealing strengths:

It runs both unit tests and functional tests. This lets me keep all of my tests under one runner to minimize complexity.
It has parallelism built into it. Just configure how many tests should run in parallel, and it works!
It supports AMD natively.
It’s really easy to write your own reporter and extend the framework.

Writing the tests

One of the most important things to me is readability, especially when it comes to tests. Anyone who reads your tests should be able to understand everything about the expected behavior of your system.

That is why I like when my tests look like this:

bdd.it('saves with correct credentials', function() {
    return archivingPage.clearForm()
        .then(() => archivingPage.enterBucket(testBucket))
        .then(() => archivingPage.enterAccessKey(testAccessKey))
        .then(() => archivingPage.enterSecretKey(testSecretKey))
        .then(() => archivingPage.clickSave())
        .then(() => archivingPage.hasSuccessMessage())
        .then((success) => {
            expect(success).to.equal(true);
        })
        .catch(common.handleError(this));
});

You can practically read it just like a story!

Abstracting the details

Of course, I abstract all of the “dirty work” in my “page objects.” I like to hide it there for a few reasons. First, simply looking at the test can reveal what it shows without needing to know the details of how the page or component is built. Second, this builds the code out of many small, reusable bits.

If I decide to change the HTML/CSS of the “archivingPage” in this example, I’ll have only one place to change the code inside the object itself, and multiple tests relying on this component will continue to work without any changes.

Page Objects

So how are my “page objects” built?

Here’s a look at the archivingPage from the test above:

'use strict';

define(function(require) {

   function ArchivingPage(remote) {
       this.remote = remote;
   }

   ArchivingPage.prototype = {
       constructor: ArchivingPage,

       enterBucket: function(bucket) {
           return this.remote
               .findByCssSelector('.s3-archive-section .bucket')
               .click().type(bucket).end();
       },
       enterAccessKey: function(accessKey) {
           return this.remote
               .findByCssSelector('.s3-archive-section .access-key')
               .click().type(accessKey).end();
       },
       clickSave: function() {
           return this.remote
               .findByCssSelector('.s3-archive-section .actions-section .save')
               .click().end();
       },
       hasSuccessMessage: function() {
           return this.remote
               .findByCssSelector('.status.success')
               .isDisplayed();
       },
       clearForm: function() {
           return this.remote
               .findByCssSelector('.s3-archive-section .bucket')
               .clearValue().end()
               .findByCssSelector('.s3-archive-section .access-key')
               .clearValue().end()
               .findByCssSelector('.s3-archive-section .secret-key')
               .clearValue().end()
           );
       }

   };

   return ArchivingPage;

});

Polling

One of the really nice things about the Intern’s framework is that polling is built into most of the functions. This is really helpful because many applications (like ours) are single-page applications, and many user actions cause Ajax requests that subsequently cause UI changes.

Promises

Another really nice feature is that most methods return promises that can be chained. Take another look at the original test above, and you will see that I call ‘then()’ on each method’s result and pass it a callback.

I’m running this on Node.js, so I’m using ES6 callbacks freely here — it makes the code much more concise and readable.

Screenshots

I run my tests constantly (after each commit), and we obviously don’t sit around making sure they don’t break. This is why it’s very crucial to collect as much data as you can when a test breaks. When that occurs, the biggest help here is an actual screenshot of the browser.

This can easily be done with Intern. If you look at the original test that I had I posted, you’ll see the “catch” clause. This calls a method named ‘common.handleError,’ which returns a method that takes a screenshot of the browser and saves it. (This is not a special feature of the framework but a feature of Selenium. So, it’s possible with other frameworks as well.)

handleError: function (testObj) {
       var self = this;
       var outDir = path.resolve(localConfig.outputDir);

       return function (reason) {
           var now = Date.now();
           var filename = [testName, now].join('_') + '.png';

           return self
               .remote.takeScreenshot()
               .then(function (screenshotData) {
                   fs.writeFileSync(path.resolve(outDir, filename), screenshotData);
               })
               .finally(function () {
                   throw new Error(reason);
               });
       };
}

Continuous Deployment

Running these tests constantly is an important issue for us. Intern integrates well with Jenkins, which we use as our build server. Our Jenkins server runs all tests in a Docker container that runs Selenium and retrieves the output along with screenshots saved from Docker that we can view later.

A final note: Intern also integrates with other tools such as Travis CI, Bamboo, CodeShip, and TeamCity.