Opened 6 years ago

Closed 3 years ago

#7419 closed Bugs (obsolete)

Support multiple calls to framework::init() allowing wrappers to support running tests using test tools in full systems

Reported by: Jamie Allsop <ja11sop@…> Owned by: Gennadiy Rozental
Milestone: Boost 1.59.0 Component: test
Version: Boost Development Trunk Severity: Problem
Keywords: Cc:


Boost.Test is designed on the assumption that a special main() function will be provided to ensure the test framework is initialised properly and that the function needed to add test suites for execution is properly provided. This works well for simple test programs.

However another important use case we have and I'm sure others too is that we also wish to run our production systems 'as-if' in production but have tests execute when run in a 'testing' mode.

This has the benefit of allowing full use of all the Boost.Test tool facilities and makes test points easily recognisable. Additionally it also means the test out analysis tools that we use for Boost.Test in our unit testing can be used without modification. Hence on a coding level we use the same test code as for unit tests and we then benefit from being able to directly interpret the results, just as we do for normal unit tests.

In order to support this within the Boost.Test framework (we use the unit-test framework for this as we simply require access to the tools and reporting facilities) we must make it possible to write a test runner wrapper than can ultimately call boost::unit_test::framework::init() more than once.

A minimal test runner class might look like this (taken from a real code but stripped down so will not work as is):

namespace test {

class runner

    using master_test_suite_t  = boost::unit_test::master_test_suite_t;
    using test_suite_t         = boost::unit_test::test_suite;
    using call_add_tests_t     = boost::function<void( master_test_suite_t& )>;


    void store_properties( /* properties from commandline or config file */ )
        auto& Runner = instance();
        Runner.store_arguments( /* properties */ );

    int run( const std::string& Title, const call_add_tests_t& Callback )
        auto& Runner = instance();

        Runner.set_title( Title );
        Runner.set_add_tests_callback( Callback );

        boost::unit_test::init_unit_test_func init_func = &runner::init_unit_test;
        boost::unit_test::init_unit_test_func init_func = &runner::init_unit_test_suite;
        return boost::unit_test::unit_test_main( init_func, Runner.argc(), Runner.argv() );



    static runner& instance()
        static runner Runner;
        return Runner;


    bool init_unit_test()
        init_unit_test_suite( 0, 0 );
        return true;

    boost::unit_test::test_suite* init_unit_test_suite( int argc, char* argv[] )
        auto AddTests = runner::instance().get_add_tests_callback();
        if( AddTests )
            AddTests( boost::unit_test::framework::master_test_suite() );
        return 0;


    void store_arguments( /* properties */  )
        ArgumentStrings_ = /* copy of Boost.Test relevant properties */
        Arguments_ = /* pointers to actual strings in ArgumentStrings_ */


    int argc() const
        return ArgumentStrings_.size();

    char** argv() const
        return const_cast<char**>( &Arguments_[0] );

    void set_title( const std::string& Title )
        if( ArgumentStrings_.empty() )
            ArgumentStrings_.push_back( Title );
            Arguments_.push_back( ArgumentStrings_[0].c_str() );
            Arguments_.push_back( 0 );
            ArgumentStrings_[0] = Title;
            Arguments_[0] = ArgumentStrings_[0].c_str();

    void set_add_tests_callback( const call_add_tests_t& Callback )
        Callback_ = Callback;

    const call_add_tests_t& get_add_tests_callback()
        return Callback_;


    call_add_tests_t            Callback_;
    std::vector<std::string>    ArgumentStrings_;
    std::vector<const char*>    Arguments_;

} // test

We would use a runner like this as follows:

    // Implement a function to add tests to the Master Test Suite
    // Note this can be a member function and simply call
    // another member to collate the test suites
    void add_tests( test::runner::master_test_suite_t& MasterSuite )
        MasterSuite.add( scenario_test_suite() );

    // A
    test::runner::test_suite_t* scenario_test_suite()
        auto* test_suite = BOOST_TEST_SUITE( Name_.c_str() );

        test_suite->add( BOOST_TEST_CASE( boost::bind( &self_t::validate_incoming_messages, this->shared_from_this() ) ) );
        test_suite->add( BOOST_TEST_CASE( boost::bind( &self_t::validate_outgoing_messages, this->shared_from_this() ) ) );
        test_suite->add( BOOST_TEST_CASE( boost::bind( &self_t::validate_data_messages,     this->shared_from_this() ) ) );

        return test_suite;

    // Calling test() actually causes the tests to be executed
    // We assume in this example that the object that
    // test() belongs to is a shared_ptr
    void test()
        test::runner::run( Name_, boost::bind( &self_t::add_tests, this->shared_from_this(), _1 ) );

The attached patch makes the above code possible by doing 2 things:

  1. If the macro BOOST_TEST_USE_QUALIFIED_COMMANDLINE_ARGUMENTS is defined then the Boost.Test command line arguments are scoped using boost.test.* as a prefix. This means commandline arguments used to control tests will not interfere with command line arguments used to control the code being tested. --log_level is an excellent example of why you might need to do that. --boost.test.log_level is much clearer and will not clash.
  1. Calls to boost::unit_test::framework::init() will occur each time run() is called. However with the patch each call resets the framework to a consistent initial state and the tests proceed as expected as if the framework had been called once from main().

We are using this patch with boost 1.48 right now (and so it is very slightly different due to the changes between 1.48 and trunk) and it allows us to run our main processes in a 'test' mode generating full test result output with timings that are then sent to our build server. It works very well, has no impact on normal unit test usage and is a very small change for a very big gain.

This patch along with the patches attached to these tickets:

Boost.Test, since boost 1.48 is using the deprecated Boost.Timer class - it should be updated to use the new class
Test Units (Cases and Suites) in Boost.Test do not capture __FILE__ and __LINE__ at declaration point making it impossible to provide source file linking using external test management tools
Detailed test status is not available in the Boost.Test log (status, assertions, passed) and so live test case status cannot be tracked

...combine to significantly improve the value of Boost.Test in terms of integration with thirdparty analysis tools and integration with other code libraries.

Given the small changes that are made I'd like to see them hopefully make it into the next boost release.

Attachments (1)

boost_test_support_multiple_test_runs.diff (4.9 KB) - added by Jamie Allsop <ja11sop@…> 6 years ago.

Download all attachments as: .zip

Change History (2)

Changed 6 years ago by Jamie Allsop <ja11sop@…>

comment:1 Changed 3 years ago by Gennadiy Rozental

Milestone: To Be DeterminedBoost 1.59.0
Resolution: obsolete
Status: newclosed

Please open new ticket against trunk version of Boost.Test with detailed description of what you are trying to achieve

Modify Ticket

Change Properties
Set your email in Preferences
as closed The owner will remain Gennadiy Rozental.
The resolution will be deleted.

Add Comment

E-mail address and name can be saved in the Preferences.

Note: See TracTickets for help on using tickets.