Automated Testing in Agile Software Projects

Introduction

Recently I finished a new feature of my Customfield Editor Plugin for Atlassian JIRA®. Normally the plugin provides mostly REST API Features which are already heavily tested by an REST-API-Testuite. The new feature on the other hand involves more UX and I wanted to have it tested automatically.

Then further questions arouse - Why should we test? Who should test? What to test? When to test? And how to test?.

Being on other Projects and asking for end-to-end tests mostly there were none, they were poorly maintained or didn’t even run successfully. This brings the Testing Pyramid to mind and the reality of most projects test coverage versus the desired state.

That being said we want to dive into each question and answer them in relation to agile methodologies like Scrum. That means setting the need to test software in relation to the Manifesto for Agile Software Development and the Twelve Principles of Agile Software

TL;DR

• If you do only care for the actual implementation of my end-to-end test you can jump to the bottom to the ‘how’ part.

Why should we have automated tests?

Firstly we should look at the agile aspect. When we think of ‘Working software over comprehensive documentation’ then I come to the result that well written tests that actually work, that are well maintained and bring constant stability into the project should count to the ‘Working Software’ part. To think of automated tests as part of the working software makes automated testing as important as delivered features in an agile context. I don’t say that you can write only tests for three sprints in a row and not deliver business value but that automated tests should not be seen as some optional thing you can do but as crucial part of the Definition-Of-Done.

Secondly let’s have a look at the monetary aspect. The above paragraph might be enough to convince agile thinking people but as you might know there are still some people around that think in numbers. So let’s try to convince them too. From my experience I can say that I have been to many projects and the ones with a well maintained amount of automated tests had the following benefits. Faster time-to-market since releases can be rolled our faster because your tests assure you that the software works. Faster reaction to change since you can implement new features faster when the processes are already there to develop and test new features. And in my opinion the biggest factor is that software developers have a less painful work days since they do not need to bugfix bugs in the middle of the night after a not-so-successful release rollout which leads to more motivation and better software which should lead to more business value. Also bringing new members into a project is simpler. New developers can run the tests and understand the project faster which means that the learning curve is steeper and they can contribute to the business value faster.

Lastly just some commentary. For me the real question is: ‘Would you buy a car which just in theory should be able to drive but no one actually tested that?’ Sure the factory tells you that of the 1000 cars they produced they tested one and that could drive. But would that really convince you? Bring that common sense into your project and just write tests. Your life will be much easier on the long run.

Who should test?

Now that we have heard about the ‘why’ let’s talk about the ‘who’. In agile projects we have a very clear separation of responsibilities. In Scrum we have the ‘Team’ which is fully responsible for the ‘how’ part regarding used technology. The ‘Scrum Master’ is fully responsible for the ‘process’. And the ‘Product Owner’ is responsible for the ‘what’. That being overly simplified we can say it even more simpler: ‘The Product-Owner says WHAT he wants. The Team decides HOW to do it. And the Scrum Master makes sure everything runs smooth.’. So in theory the responsibility for the testing sits in the ‘Team’ and somewhere around the ‘Scrum Master’.

But in reality that is not enough. If you just do it that way you might just end up with poorly maintained tests no one really cares about. So rather than explaining what could go wrong I will give you an outline of how I see the ideal agile responsibilities regarding automated tests.

What should the Scrum-Master do?

• Encourage the Team to write automated tests.
• Encourage the Team to use a test-driven software development approach.
• Encourage the Team to include automated tests as part of the Definition-of-Done.
• Encourage the Product Owner to care for automated tests.
• Constantly annoy people with the above points in good way. :)
• Don’t be fooled by numbers like ‘100% test coverage’.
• Do not enforce numbers but instead build up actual understanding for the need of tests.

What should the Product-Owner do?

• Demand the Team to show the actual Test-Run or Test-Results of the automated Testsuite during the Sprint-Review.
• Be happy about the Test-Results and show that to the Team.
• If someone presents you a done feature ask ‘That looks awesome! Can I see the automated testresults for that? Did you write end-to-end tests?’.
• Pursue a common line with the Scrum Master regarding automated tests.
• Always plan in time in mind that even if a feature is small it needs time for the tests to be implemented.
• Understand that the process needs time to fully work and that test-driven development is a long time investment which makes your software still work in one year.

What should the Team do?

• Try to build an understanding that a feature is not done when it is implemented.
• Use the Retrospectives to come to a mutual understanding of what and how you want to test.
• Create a Definition-of-Done you can all be comfortable with.
• Maybe you work on Stories always in pairs of two. One implements the feature. The other one implements the test.
• Setup a full continuous integration system that runs the complete testsuite on every code push on every branch.
• Integrate early. Integrate often.
• Talk to each other and constantly improve the Testsuite.
• If you have don’t have full-stack-developers let the backend-engineer and the frontend-engineer talk about how a feature can be implemented so that it can be tested easily.
• Be proud of your Testsuite. Show it to everyone.

What to test?

Alright. I have talked waaaaay too much about theoretical stuff so let’s just have a look at the new feature I implemented and inspect what to do from there on out. The feature is called ‘#11 Hide administrator Sidebar-Navigation Links from Non-Administrator Users’ and it basically should do the following:

• My JIRA Plugin has a sidebar navigation on the left. In this sidebar navigation reside four links of which three are only useful for JIRA Administrators.
• The feature I implemented let’s you change a setting on the newly created Settings-Page to hide these three links from non-administrators.

You can see a preview of the feature below. On the left side I am in a Browser-Session logged into JIRA as administrator. On the right I am logged in as a non-administrator user. The GIF shows the complete feature in use.

So now we know what feature we want to test but what parts on which layers of the App do we test? That depends on how you implement your features and how your application is structured. In my case:

• The Sidebar-Navigation is rendered server-side with Velocity and Soy.
• There is a tiny Soy-Template that accepts a Boolean to hide/show the admin links in the sidebar
• The AdminSettingsHelper.java can be wired via Spring into the WebWorkAction (Controller) so that the view (Velocity/Soy) has access to the Admin-Settings which provides a getter getHideAdminLinksForNonAdminUsers().
• The Settings-Page is rendered via JavaScript also using Soy-Templates in JS.
• The actual Settings are loaded and saved via REST-API. There is a GET and POST Endpoint for saving and loading the JSON-Settings-Object.
• When clicking the actual setting on the Settings-Page a POST request is sent to the save-endpoint and the new setting is saved.

So that is how the feature is implemented. And I decided to test it this way:

• Create atomic tests in REST-API Testsuite for loading and saving admin settings. With Cornercases like non-admin user should not be able to save admin-setting a.s.o. (This is not part of this blogpost).
• Create some UnitTests and IntegrationTests for the AdminSettingsHelper.java which will run when running the Atlassian SDK with ‘atlas-run’ and ‘atlas-integration-test’. (This is not part of this blogpost).
• Create an End-To-End Test to test the complete workflow the GIF above shows. In the ‘How to test?’ section you will see more details about it.

What rules to follow? What about agile?

• E2E-Tests should be written corresponding to User-Stories. Usually User-Stories already should be structured to define atomic use-cases.
• E2E-Tests should test cornercases as well. Usually User-Stories contain Acceptance-Criteria which should lead to corner cases.
• Try to write tests on all layers of the app following the Testing Pyramid.
• Keep the e2e tests as small as possible since they are the first to break when the app is e.g. redesigned.
• Write Unit-Tests, Integration-Tests and API Tests and put a lot of effort into the corner-cases such as empty request-bodys, security violations a.s.o

When to test?

As already written above you should run your tests on every code change in every branch. Therefore it might be a very good idea to setup CI-Server like Jenkins or Bamboo. And configure them to run your tests on code changes.

How to test?

Now it finally comes down to the ‘how’ part. I will only write about the End-to-End test here, since Unit-Tests and REST-API Tests are not the scope of this blog-post (I might write about those some time).

You already saw the Workflow-GIF above and what steps our End-To-End Test will test is:

• Login as admin
• Change settings to show all nav links
• Logout
• Login as user
• Check that all 4 links are there
• Logout
• Login as admin
• Change settings to hide admin nav links
• Logout
• Login as user
• Check that only 1 user-link is there and the three admin links are hidden
• Logout

We use a new testing tool called CodeceptJS which is really nice and leads to understandable test code. Let’s get started to set up CodeceptJS for our End-to-End Test.

You need Node.js installed and have a Project initialized with ‘npm init’. First we install a bunch of tools we need to write some runner to start the tests

shell:npm install --save-dev shelljs yargs kill3k babel-cli babel-core babel-loader babel-preset-es2015

Now you are free to use any of the popular drivers (see CodeceptJS Doc) an I decided to go with Nightmare instead of Selenium WebDriver.

shell:npm install --save-dev codeceptjs nightmare nightmare-upload

After that we install PhantomJS which will server as a replacement for the Selenium Server and test the installation.

shell:npm install --save-dev phantomjs-prebuilt

‍

shell:./node_modules/phantomjs-prebuilt/bin/phantomjs --version

Should output 2.1.1.

Now we want to configure CodeceptJS. You can do this with the init command, so check out the Quickstart Guide.

shell:./node_modules/codeceptjs/bin/codecept.js init

This will create this codecept.json config (if you select options accordingly).

github:dd0487dbcfddc97cbb4249b5ec358ed1

Add the script to package.json so that we can run npm run test:e2e later

github:882817ff63297457170021f4f9578751

Create a ‘run-e2e-test.js’ file and put the code there which will start the PhantomJS Selenium-Server and run the CodeceptJS tests when we execute ‘npm run test:e2e’.

github:6b46a86b748a6dade7da6529a7378761

Since our JS-Code is ES6 already we need a .babelrc file so also create that one

github:53c1e3d779598f22e06a25c9a5a01632

Now we change the ‘tests/end-to-end/codecept-js-steps-file.js’ file an define some Actors for login and logout. Actors are basically things you need more than once and you can use them anywhere in your tests. E.g. the ‘loginToJIRA’ actor can be called from the test with I.loginToJIRA(‘admin’, ‘admin’); which logs in the user admin with password admin.

github:5827571d659bcd408c2b1a3d875b0f09

Now we create the actual End-To-End Test and create the ‘tests/end-to-end/admin-settings_test.js’ file and paste the following.

github:6eaf3d48a074f7a7d7284acc8ef01277

You should be able to understand the test just by reading it. I used some pretty long ids in my HTML code that are self explanatory so that I know what ‘I.click(‘#cep-admin-settings—navigation-links-form—hide-from-non-admins-radio-true’);’ means. My used ids explain this way:

• ‘cep-admin-settings’ = Element is on the settings Page
• ‘navigation-links-form’ = In the Navigation Links Form
• ‘hide-from-non-admins-radio-true’ = The Setting that sets the hideFromNonAdmins Boolean to true when clicked.

Ok now we run the npm run test:e2e and have of course JIRA already running via ‘atlas-run’.

shell:npm run test:e2e

(see gif: https://media.comsysto.com/images/2016-10-31-end-to-end-testing/codecept-run.gif)

‍

If you need help with JIRA, Confluence or any other Atlassian product we offer Atlassian Experts Services. We are experienced in the Setup of Atlassian Products in your own datacenter and on AWS. We also offer consulting via webmeeting worldwide. Feel free to contact us via the contact form below.

‍