In my previous 2 posts I had a look at 2 Integration Testing tools – Fitnesse and Selenium.  In this post I’m going to take a look at the new Microsoft Web Testing tool – Lightweight Test Automation Framework.

The Lightweight Test Automation Framework is in a similar mould to Selenium – it is designed only for testing web applications.  Unlike Selenium there is no IDE for recording tests – you add a single reference to your project and use a special ‘testing page’ for executing your tests.

Example

I’m going to use the same example as in my last post – a very simple website that calculates the user’s age based on their year of birth and the current year.

To get started you need to download the Lightweight Test Automation Framework.  Since it’s still Beta you will need to compile the source code and reference the generated assembly.  You will also need to copy the ‘Test’ folder from the sample website (included with the source code as well as a standalone download).

To get the framework working with ASP .Net MVC you will also need to add the following line to your routing.

routes.IgnoreRoute("Test/{resource}.axd/{*pathInfo}");

Now to write the actual test.  The testing API is quite well-designed and intuitive.  There are a couple of methods which would make it a little easier, but more on that later.

[WebTestClass]
public class AgeCalculatorFixture
{
    [WebTestMethod]
    public void ShouldBeAbleToCalculateAge()
    {
        var page = new HtmlPage("/");

        page.Elements.Find("currentYear").SetText("2010");
        page.Elements.Find("birthYear").SetText("1983");
        page.Elements.Find("calculateButton").Click(WaitFor.Postback);

        var result = page.Elements.Find("result").GetAttributes()["value"];

        Assert.AreEqual("27", result);
    }
}

I particularly like the syntax for waiting for a postback.  To run the test you need to navigate to /Test – the testing framework will pick up any tests and display them in a grid where you can select the ones you wish to run.

Unfortunately the testing interface is rather cumbersome – the tree structure is not that easy to work with and for some reason the buttons jumped around when I selected/deselected tests.  Despite this the interface is functional and the tests execute quickly.  When comparing the execution times to Selenium the Lightweight Test Automation Framework is definitely faster (1 second compared to 7 for Selenium).  However Selenium starts the browser before running the tests so the time difference is probably very little when taking this into account.  (Selenium would probably be faster in the long run since tests can be run in parallel)

Things I like about the Lightweight Test Automation Framework

• Multiple browser support.  You can use just about any browser.
• Feedback while the tests are running.  As you can see in the screenshot above, the screen is split into 2 panes – the tests on the left and the website on the right.  As the tests are executing you can see the values being set in the browser window on the right.
• The API.  The API seems pretty well designed and is generally intuitive to work with.  I haven’t tried everything, but most of the functionality you would expect is easy to find.  There is even support for executing scripts and navigating to other pages.  The only thing missing that I picked up is the lack of a ‘Value’ property on the HtmlElementAttributeReader class.

Things I dislike about the Lightweight Test Automation Framework

• The name.  Writing ‘Lightweight Test Automation Framework’ all the time gets really annoying really quickly.  Please choose a real name.
• Debugging.  According to the documentation/release notes, failed tests should be marked in red and when you click on them you should see a stack trace of the failed test.  This isn’t working – failed tests are not marked in red and there’s no stack trace when you click on them.
• Testing page.  The testing page itself could definitely be implemented better – adding a ‘select all’ button and making the layout look like it took more than 2 minutes to implement would help.
• Test output.  Apart from the debugging functionality not working, the output is rather poor.  Instead of saying ‘Test run finished with 0 failure(s)’ rather say ‘Test passed’.  Also, the list of command being execute (GetPageDom,Click,FillTextBox..) isn’t very helpful unless we can actually see the specifics (GetPageDom – currentYear,Click – submitButton…).
• Hosting the tests in the website.  I usually keep my tests in a separate project – ASP .Net MVC even automatically creates a .Testing project.  With the Lightweight Test Automation Framework we need to keep the testing pages in our website and the samples include the tests themselves as part of the website (I moved my tests to another assembly, but then you need to reference the testing assembly from within your website).  I know you could exclude these pages when deploying, but it still seems rather messy.
• No automated build support.  This is probably the most glaring issue – there is no easy way to include the tests as part of an automated build process.

How to improve the Lightweight Test Automation Framework

Since this project is still in the beta phase, I thought it would be worthwhile to make some suggestions on how I think the project should go forward.  (One of the developers also asked me for suggestions)

• Get automated build support.  ASAP.  I can’t really see this being used until we can truly automate the entire process.  Almost every web testing framework has support for an automated build process.  I realise that there is probably some obscure way of adding this to a build process, but it needs to be a built-in feature.
• Take a look at the competition.  There are quite a few quality web testing frameworks out there – Selenium, WatiN, Windmill, etc.  For the Lightweight Test Automation Framework to be a viable alternative to these frameworks there needs to be some clear benefit.  If I had to justify switching from Selenium to this framework to my manager, how would I convince him?
• Give better feedback while running the tests.  As I mentioned earlier, the testing page is split into 2 panes allowing you to see the test being executed on the right.  While this seems pretty neat at first, the tests execute so quickly that, unless you’re running Netscape 1.4, you’re not going to see it happening.  The only benefit the testing page could possibly offer is allowing you to step through the test and see the different steps happening on the page.  Unless this functionality is implemented the testing page will be a drawback, not an advantage.
• Multiple browser support.  While the Lightweight Test Automation Framework does allow you to use any browser you choose, we need the ability to be able to execute the tests in any browser automatically.  (This ties in with the automated build support)  For example, developers should be able to specify that the tests should be run in IE, Firefox, Safari and Chrome and see the results for each browser.

Summary

The Lightweight Test Automation Framework is a beta project aimed at automated web application testing.  While some developers might choose this as an alternative to similar frameworks, at the moment there is no clear benefit except for a reasonably neat API. 

It will be interesting to see how this project evolves, since the roadmap specifies that one of the next steps for future releases is “gather feedback from the community to better understand Web automation user requirements”.

Share on: Twitter, Facebook, Google+ or use the Permalink