Review Board 1.7.16


Add an option to the test suite that makes it believe that the version of Asterisk under test is some other version

Review Request #2202 - Created Nov. 20, 2012 and submitted

Matt Jordan
testsuite
Reviewers
asterisk-dev
testsuite
At first glance, this patch sounds stupid at best, harmful at worst.  However, there is a purpose to it...

The Asterisk Test Suite has to "understand" Asterisk version numbers.  It uses the version reported by the installed Asterisk to determine if a test can run.  In general, this is used so that tests written against Asterisk 10 don't execute when the version of Asterisk is 1.8 (and in other similar cases).  This mechanism works great for Asterisk version branches and tags (minus all of the usual shenanigans that ensue given all of the ways in which we've changed the Asterisk versioning scheme).

Trunk gets a little funky.  We treat trunk as being greater then, well, everything.  Fair enough.

Team branches are worse.  What is mjordan-cool-branch?  How do you know that branch was based off of 1.8, 10, 11, or trunk?  Inspecting subversion properties for the information gets complicated, and puts some constraints on the Test Suite that we're trying to avoid.  Currently, we treat team branches as being equivalent to trunk.  That works for any team branch that is being used to develop new features against trunk.

This doesn't work if the team branch is actually related to 1.8, or some other version of Asterisk.  If you install a team branch that is based on 1.8, all tests will be executed - including those that require features developed in later versions of Asterisk.  Those tests will fail spectacularly, making it very difficult to tell if your branch has legitimate issues.

Rather then trying to manipulate the version parsing schemes (again), this patch adds an option to the test configurations that allow them to override the Asterisk version under test.  In the top level test-config.yaml, define:

properties:
  forced-version: 1.8.11.1

Or some other version you'd like to make the Test Suite think its running against.
Used the forced-version to make the Test Suite think it was running against 1.8.0.0.  Checked out a team branch based on trunk, compiled, installed, and attempted to run a ConfBridge test (confbridge_marked).  Test refused to run, because it thought it was running against 1.8.  Removed the forced-version; test ran.

Diff revision 2

This is not the most recent revision of the diff. The latest diff is revision 3. See what's changed.

1 2 3
1 2 3

  1. /asterisk/trunk/runtests.py: Loading...
  2. /asterisk/trunk/test-config.yaml: Loading...
  3. /asterisk/trunk/lib/python/asterisk/TestConfig.py: Loading...
/asterisk/trunk/runtests.py
Revision 3508 New Change
[20] 28 lines
[+20]
29

    
   
29

   
30
TESTS_CONFIG = "tests.yaml"
30
TESTS_CONFIG = "tests.yaml"
31
TEST_RESULTS = "asterisk-test-suite-report.xml"
31
TEST_RESULTS = "asterisk-test-suite-report.xml"
32

    
   
32

   
33
class TestRun:
33
class TestRun:
34
    def __init__(self, test_name, ast_version, options):
34

   

    
   
35
    def __init__(self, test_name, ast_version, options, global_config=None):
35
        self.can_run = False
36
        self.can_run = False
36
        self.did_run = False
37
        self.did_run = False
37
        self.time = 0.0
38
        self.time = 0.0
38
        self.test_name = test_name
39
        self.test_name = test_name
39
        self.ast_version = ast_version
40
        self.ast_version = ast_version
40
        self.options = options
41
        self.options = options
41
        self.test_config = TestConfig(test_name)
42
        self.test_config = TestConfig(test_name, global_config)
42
        self.failure_message = ""
43
        self.failure_message = ""
43
        self.__check_can_run(ast_version)
44
        self.__check_can_run(ast_version)
44
        self.stdout = ""
45
        self.stdout = ""
45

    
   
46

   
46
    def run(self):
47
    def run(self):
[+20] [20] 153 lines
[+20] [+] def _parse_test_yaml(self, test_dir, ast_version):
200
                if val == "test":
201
                if val == "test":
201
                    # If we specified a subset of tests, there's no point loading the others.
202
                    # If we specified a subset of tests, there's no point loading the others.
202
                    if self.options.test and not (path + '/').startswith(self.options.test):
203
                    if self.options.test and not (path + '/').startswith(self.options.test):
203
                        continue
204
                        continue
204

    
   
205

   
205
                    tests.append(TestRun(path, ast_version, self.options))
206
                    tests.append(TestRun(path, ast_version, self.options, self.global_config))
206
                elif val == "dir":
207
                elif val == "dir":
207
                    tests += self._parse_test_yaml(path, ast_version)
208
                    tests += self._parse_test_yaml(path, ast_version)
208

    
   
209

   
209
        return tests
210
        return tests
210

    
   
211

   
[+20] [20] 53 lines
[+20] [+] def run(self):
264
            if t.can_run is False:
265
            if t.can_run is False:
265
                if t.test_config.skip is not None:
266
                if t.test_config.skip is not None:
266
                    print "--> %s ... skipped '%s'" % (t.test_name, t.test_config.skip)
267
                    print "--> %s ... skipped '%s'" % (t.test_name, t.test_config.skip)
267
                    continue
268
                    continue
268
                print "--> Cannot run test '%s'" % t.test_name
269
                print "--> Cannot run test '%s'" % t.test_name

    
   
270
                if t.test_config.forced_version is not None:

    
   
271
                    print "--- --> Forced Asterisk Version: %s" % \

    
   
272
                        (str(t.test_config.forced_version))
269
                print "--- --> Minimum Version: %s (%s)" % \
273
                print "--- --> Minimum Version: %s (%s)" % \
270
                    (str(t.test_config.minversion), str(t.test_config.minversion_check))
274
                    (str(t.test_config.minversion), str(t.test_config.minversion_check))
271
                if t.test_config.maxversion is not None:
275
                if t.test_config.maxversion is not None:
272
                    print "--- --> Maximum Version: %s (%s)" % \
276
                    print "--- --> Maximum Version: %s (%s)" % \
273
                        (str(t.test_config.maxversion), str(t.test_config.maxversion_check))
277
                        (str(t.test_config.maxversion), str(t.test_config.maxversion_check))
[+20] [20] 205 lines
/asterisk/trunk/test-config.yaml
Revision 3508 New Change
 
/asterisk/trunk/lib/python/asterisk/TestConfig.py
Revision 3508 New Change
 
  1. /asterisk/trunk/runtests.py: Loading...
  2. /asterisk/trunk/test-config.yaml: Loading...
  3. /asterisk/trunk/lib/python/asterisk/TestConfig.py: Loading...

https://reviewboard.asterisk.org/ runs on a server provided by Digium, Inc. and uses bandwidth donated to the open source Asterisk community by API Digital Communications in Huntsville, AL USA.
Please report problems with this site to asteriskteam@digium.com.