What is PNUnit?

PNUnit is an extension to the NUnit framework we have developed to test our software. The "P" stands for "Parallel" and basically means that you can run several tests at the same time on different machines.

The PNUnit development

Our team is developing a new Software Configuration Management system. During the last year, starting from scratch, we have developed a new SCM server and some client tools (GUI, command line and IDE integrations). Almost everything has been written in C# and since the beginning we wanted to make it run on non-windows platforms using Mono.

One of our main concerns has always been testing. We have written unit tests with NUnit since the project start up, but after a couple of sprints we needed to test the server's behaviour under heavy load, and under certain race conditions. We have evaluated several commercial tools (some of them we have been working with in previous projects with very good results) to automate client actions (both on the GUI and on the command line) performing several operations.

To really test our system we needed to simulate hundreds of clients against the same server. Unfortunately, the tools we could afford didn't work on both Windows and Linux platforms. Then we thought the NUnit platform could be the basis to distribute our tests over the network and reproduce real scenarios. We could write tests in NUnit automating our client command line tool (both directly writing tests using our classes our launching the tool as a separate process).

When we started, NUnit lacked the following: manage test launching on remote machines, gather the results, and test synchronization between different machines to create more complex scenarios.

The PNUnit structure

The scripts with the test configuration are read by the launcher, which is also responsible of creating the Runner. The Runner will launch the tests in the specified machines, and it provides the synchronization mechanism and the collection of results. The agent is able to launch tests remotely, by creating instances of PNUnitAgent. Every time the PNUnitAgent receive a test to run creates an instance of PNUnitTestRunner, and finally the PNUnitTestRunner will run the test, gather the results and send them to the Runner.

A very basic test

Let's check the result of a sum operations using a PNUnit test. We suppose we have a function called add, which is able to do the sum of two numbers. The code of the test itself (included in the tests.zip file below) is:

[TestFixture]
public class Testing
{
[Test]
public void EqualTo19()
{
Assert.AreEqual(19, Cmp.Add(15,4));
}
...
}

Notice that the structure is the same as NUnit. We generate the .dll file (TestLibraries.dll in this case) and we put it into the directory with all the PNUnit files included in the test.zip file. He have to create a script with the configuration of the test. The following is the test.conf file.

<TestGroup>
<ParallelTests>

<ParallelTest>
<Name>Testing</Name>
<Tests>
<TestConf>
<Name>Testing</Name>
<Assembly>TestLibraries.dll</Assembly>
<TestToRun>TestLibraries.Testing.EqualTo19</TestToRun>
<Machine>localhost:8080</Machine>
<TestParams>
<string>..\server</string> <!-- server dir -->
<string></string> <!-- database server -->
<string></string><!-- conn string -->
</TestParams>
</TestConf>

</Tests>
</ParallelTest>

</ParallelTests>
</TestGroup>

In Assembly he have to put the name of the .dll file with the tests. In TestToRun we have to specify the individual test we want to run. In Machine we put the ip address (localhost for local testing) and the port. In agent.conf it is stated that the port, 8080, so we can change it in both files and test will run properly too.

Then, we have to start the agent. In the command line:

C:\pnunit>start agent agent.conf

And to launch the test:

C:\pnunit>launcher test.conf

A synchronized test

One advantage of PNUnit is that it is possible to synchronize several tests. We will run a very simple one, just with a client and a server, waiting the second one for the client. The code of this test is:

[Test]
public void Server()
{
PNUnitServices.Get().InitBarrier("BARRIER");
PNUnitServices.Get().WriteLine("Server started");

Thread.Sleep(10000);

PNUnitServices.Get().EnterBarrier("BARRIER");
Assert.IsTrue(false, "The test failed");
}

[Test]
public void Client()
{
PNUnitServices.Get().WriteLine("The client should wait until the server starts");
PNUnitServices.Get().InitBarrier("BARRIER");

PNUnitServices.Get().EnterBarrier("BARRIER");

Console.WriteLine("Server should be started now");
Assert.IsTrue(true, "The test failed");

}

The structure of the test.conf file is the same as in the previous case. The only differences is that in the new tests, in TestToRun, we write:

<TestToRun>TestLibraries.Testing.Server</TestToRun>

<TestToRun>TestLibraries.Testing.Client</TestToRun>

The synchronization mechanism

The way PNUnit provide synchronization facilities is by means of barriers. The tests waits until all clients hit the barrier, and then they can pass.

How do we use it

We are already using PNUnit for more than six months. We have "smoke tests" that each developer runs locally when he finishes a tasks (together with normal NUnit tests). Smoke tests don't take a long time to run and rapidly notify developers about "broken things". They have proved to be very useful over the months. We also have a bigger test battery that is run on each release. Those tests cover merge functionality too. During the release process both smoke and merge tests run on different platform combinations: Windows-Windows, Linux-Windows and so on. All of our supported platforms are covered this way. In the following video there are six machines running this smoke tests from different platforms.

Then we have load and stress tests. Those are a different kind of test suite specifically written to be scalable. Using the config file we can set up very different load tests scenarios, configuring both the platforms and the number of machines involved. We are currently using a cluster to test our system under heavy load.

What you can do with PNUnit

If you find it useful you can probably use it as we do. We have subtly modified the NUnit framework, and we are obtaining good results out of it, so we thought sharing it would be a good idea.

Downloads

PNUnit and the tests above, binaries (257 KB): tests.zip

PNUnit and tests sources (195 KB): pnunit.zip