Sunday, May 15, 2011

Reporting only Failed Test in GoogleTest

GoogleTest with GoogleMock (gtest and gmock) is an outstanding platform for testing C++ code.  At work, we use gtest and gmock for unit test as well as integration style test.  On one of out projects, we are up to over 600 unit tests.  This is great for code coverage, but can generate very verbose output.  When adding new tests, and a failure occurs, the explanation of the failed will scroll by to quickly to read and may be lost if your console buffer is not large enough.  It would be really nice to only output a summary of the executed tests and any failed tests. 
The default output printer in gtest is an object named PrettyUnitTestResultPrinter that extends the TestEventListener class.  TestEventListener objects can hook into the test framework and handle various events as tests are executed. You can learn about implementing TestEventListener classes here.  The documentation even shows you how to implement a minimalist printer, but without the functionality of the default printer.  The built in PrettyUnitTestResultPrinter does a nice job of reporting test results.  The results are even colored green for success and red for failure.  Being an open source project, you could just copy the implementation of PrettyUnitTestResultPrinter and modify it to your needs.  I did not want to take this approach as it would be messy to extract this class and all of its dependencies and cause more maintenance. 
Instead I created a TestEventListenerProxy class that implements all of the methods in TestEventListener and forwards the calls to a TestEventListener object.  This class will take ownership of the pointer passed in to the constructor (deleting it in the destructor).  Thus if I can get a pointer to a TestEventListener object, I can wrap it and override methods.  See the code below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23




class TestEventListenerProxy : public TestEventListener 
{
public:
explicit TestEventListenerProxy(TestEventListener* event_listener);
virtual ~TestEventListenerProxy();

virtual void OnTestProgramStart(const UnitTest& unit_test);
virtual void OnTestIterationStart(const UnitTest& unit_test, int iteration);
virtual void OnEnvironmentsSetUpStart(const UnitTest& unit_test);
virtual void OnEnvironmentsSetUpEnd(const UnitTest& unit_test);
virtual void OnTestCaseStart(const TestCase& test_case);
virtual void OnTestStart(const TestInfo& test_info);
virtual void OnTestPartResult(const TestPartResult& result);
virtual void OnTestEnd(const TestInfo& test_info);
virtual void OnTestCaseEnd(const TestCase& test_case);
virtual void OnEnvironmentsTearDownStart(const UnitTest& unit_test);
virtual void OnEnvironmentsTearDownEnd(const UnitTest& unit_test);
virtual void OnTestIterationEnd(const UnitTest& unit_test, int iteration);
virtual void OnTestProgramEnd(const UnitTest& unit_test);

protected:
TestEventListener* listener;
};

Then from this class I created the CaseSummaryAndFailurePrinter class.  This class will override some of the TestEventListener methods with empty implementations to effectively remove the printing functionality of those methods.  Then, in the OnTestEnd method, the default printer is only called in the event that the test failed.  This class will then only print the overall summaries, case summaries and failed tests.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17




class CaseSummaryAndFailurePrinter : public TestEventListenerProxy
{
public:
explicit CaseSummaryAndFailurePrinter(TestEventListener* default_printer)
    : TestEventListenerProxy(default_printer)
{
}

virtual void OnEnvironmentsTearDownStart(const UnitTest& /*unit_test*/) { }
virtual void OnEnvironmentsSetUpStart(const UnitTest& /*unit_test*/) { }
virtual void OnTestStart(const TestInfo& /*test_info*/) { }

virtual void OnTestEnd(const TestInfo& test_info) {
    if (test_info.result()->Failed())
        listener->OnTestEnd(test_info);
    }
};


I use the CaseSummaryAndFailurePrinter for integration style tests.  These tests may take a while to run and having the case summaries printed allows you to see progress.  Test cases are basically the tests defined in one C++ class.

Finally, for unit tests I use the SummaryAndFailurePrinter class.  This class extends CaseSummaryAndFailurePrinter and overrides the OnTestCaseStart and OnTestCaseEnd methods to remove the printing of the test case information.  Thus for my 600+ unit test I see only the summary of the number of tests that were executed, and any that may have failed.  This removes the verbosity of the default printer in gtest while still getting its functionality when needed.



1
2
3
4
5
6
7
8
9
10
11




class SummaryAndFailurePrinter : public CaseSummaryAndFailurePrinter
{
public:
explicit SummaryAndFailurePrinter(TestEventListener* default_printer)
    : CaseSummaryAndFailurePrinter(default_printer)
{
}

virtual void OnTestCaseStart(const TestCase& /*test_case*/) { }
virtual void OnTestCaseEnd(const TestCase& /*test_case*/) { }
};


The only thing left is to replaces the default printer in gtest with out wrapper.  This can be accomplished in your main method with the following code:



1
2
3
4
5
6
7
8
9
10




testing::InitGoogleTest(&argc, argv);
testing::InitGoogleMock(&argc, argv);

testing::TestEventListeners& listeners =
testing::UnitTest::GetInstance()->listeners();

auto default_printer = listeners.Release(listeners.default_result_printer());
listeners.Append(new SummaryAndFailurePrinter(default_printer));

return RUN_ALL_TESTS();


As shown above, we can tell the TestEventListeners container to release ownership of the default printer.  We can then wrap this default printer with a SummaryAndFailerPrinter and add that object to the TestEventListeners container.

The more I use gtest and gmock the more impressed I am.  This is by far the best C++ testing framework that I have ever seen.  Great job google people!

3 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. This is hugely helpful! This is just the bit I've found missing from the gtest framework.

    ReplyDelete
  3. Your article came helpful in writing my own listener, thank you janglin.

    ReplyDelete