Does your test suite launch your application or expect it to be running already? This question came up while working on a client project; I launched the Selenium-based test suite and everything failed ... no requests got processed and I spent some time tracking down why the test suite was failing to launch the application.
I chatted about this with my client ... and found out that the Selenium test suite expects the application to already be running. It starts up Selenium Server, sure, but the application-under-test should already be running.
And suddenly, I realized I had a Big Blind Spot, dating back to when I first started using Selenium. For years, my testing pattern was that the test suite would start up the application, run the Selenium tests, then shut it down ... and this makes perfect sense for testing the Tapestry framework, where the test suite starts and stops a number of different mini-applications (and has to run headless on a continuous integration server).
But an application is different, and there's a lot of advantages to testing against the running application ... including being able to quickly and easily reproduce any failures in the running application. Also, without the time to start and stop the application, the test suite runs a bit faster.
Certainly, the command line build needs to be able to start and stop the application for headless and hands-off testing. However, the test suite as run from inside your IDE ... well, maybe not.
So ... how do you handle this on your projects?
It's less effort to require the application to be running. It's more dangerous in my opinion. I'm a big fan of a single command from a fresh checkout that works. And works the same for a developer as it does for a CI build.
ReplyDeleteIt takes more effort to have the tests spin up the app. It means that external portions of the app need to be faked out. It means that your app needs to support a light-weight database. It means effort around (re)creating the schema. Etc.
There are lots of benefits though too. Who says the whole app needs to be started? What if some tests are only interested in functional tests against one layer (i.e. the UI)? The layer beneath can be faked out. More effort again, but it ensures a clean separation between UI and the service layer.
Used such pattern on my earlier non-tapestry projects:
ReplyDeletestart application->execute selenium tests->stop application
Everything was configured in maven pom. Application was started with jetty plugin. So it makes possible to test the whole web application without need to restart container.
Also there was another pattern on one project:
application build->deploy->tests build->run tests
In this way application and its tests was separated into different projects.
Our test suite is in a separate (Eclipse, Maven) project and expects the application to run already.
ReplyDeleteIt's easier that way for devs because they can run tests (or part of the tests) to the in-IDE running instance. And it's easier for the QA people because they can run the tests against our QA instance.
Our previous acceptance test used to start the app instance embedded and also use some services directly. We changed that because it grew bigger, was hard to debug and QA could not see what was going on.
Our current test is built with cucumber-jvm and Selenium Web Driver.
A very good source of ideas and best practices for us was Continuous Delivery
If one is using TestNG then it's easy to add method that checks is application already running and if not then it will start jetty + app.
ReplyDeleteAs for myself: anything that developers wrote has to start application.
Anything that QA writes runs against some instance of application (for most of the time it's nightly
I'd kind of assume (in this scenario) that the command-line build would spin up the app as well, but the in-IDE test suite would expect the app to be already running.
ReplyDelete