Review Board 1.7.16


Add an option to the test suite that makes it believe that the version of Asterisk under test is some other version

Review Request #2202 - Created Nov. 20, 2012 and submitted

Matt Jordan
testsuite
Reviewers
asterisk-dev
testsuite
At first glance, this patch sounds stupid at best, harmful at worst.  However, there is a purpose to it...

The Asterisk Test Suite has to "understand" Asterisk version numbers.  It uses the version reported by the installed Asterisk to determine if a test can run.  In general, this is used so that tests written against Asterisk 10 don't execute when the version of Asterisk is 1.8 (and in other similar cases).  This mechanism works great for Asterisk version branches and tags (minus all of the usual shenanigans that ensue given all of the ways in which we've changed the Asterisk versioning scheme).

Trunk gets a little funky.  We treat trunk as being greater then, well, everything.  Fair enough.

Team branches are worse.  What is mjordan-cool-branch?  How do you know that branch was based off of 1.8, 10, 11, or trunk?  Inspecting subversion properties for the information gets complicated, and puts some constraints on the Test Suite that we're trying to avoid.  Currently, we treat team branches as being equivalent to trunk.  That works for any team branch that is being used to develop new features against trunk.

This doesn't work if the team branch is actually related to 1.8, or some other version of Asterisk.  If you install a team branch that is based on 1.8, all tests will be executed - including those that require features developed in later versions of Asterisk.  Those tests will fail spectacularly, making it very difficult to tell if your branch has legitimate issues.

Rather then trying to manipulate the version parsing schemes (again), this patch adds an option to the test configurations that allow them to override the Asterisk version under test.  In the top level test-config.yaml, define:

properties:
  forced-version: 1.8.11.1

Or some other version you'd like to make the Test Suite think its running against.
Used the forced-version to make the Test Suite think it was running against 1.8.0.0.  Checked out a team branch based on trunk, compiled, installed, and attempted to run a ConfBridge test (confbridge_marked).  Test refused to run, because it thought it was running against 1.8.  Removed the forced-version; test ran.

Changes between revision 1 and 2

1 2 3
1 2 3

  1. /asterisk/trunk/runtests.py: Loading...
/asterisk/trunk/runtests.py
Diff Revision 1 Diff Revision 2
[20] 29 lines
[+20]
30
TESTS_CONFIG = "tests.yaml"
30
TESTS_CONFIG = "tests.yaml"
31
TEST_RESULTS = "asterisk-test-suite-report.xml"
31
TEST_RESULTS = "asterisk-test-suite-report.xml"
32

    
   
32

   
33
class TestRun:
33
class TestRun:
34

    
   
34

   
35
    __global_config = TestConfig(os.getcwd())
35
    def __init__(self, test_name, ast_version, options, global_config=None):
36

    
   

   
37
    def __init__(self, test_name, ast_version, options):

   
38
        self.can_run = False
36
        self.can_run = False
39
        self.did_run = False
37
        self.did_run = False
40
        self.time = 0.0
38
        self.time = 0.0
41
        self.test_name = test_name
39
        self.test_name = test_name
42
        self.ast_version = ast_version
40
        self.ast_version = ast_version
43
        self.options = options
41
        self.options = options
44
        self.test_config = TestConfig(test_name, TestRun.__global_config)
42
        self.test_config = TestConfig(test_name, global_config)
45
        self.failure_message = ""
43
        self.failure_message = ""
46
        self.__check_can_run(ast_version)
44
        self.__check_can_run(ast_version)
47
        self.stdout = ""
45
        self.stdout = ""
48

    
   
46

   
49
    def run(self):
47
    def run(self):
[+20] [20] 153 lines
[+20] [+] def _parse_test_yaml(self, test_dir, ast_version):
203
                if val == "test":
201
                if val == "test":
204
                    # If we specified a subset of tests, there's no point loading the others.
202
                    # If we specified a subset of tests, there's no point loading the others.
205
                    if self.options.test and not (path + '/').startswith(self.options.test):
203
                    if self.options.test and not (path + '/').startswith(self.options.test):
206
                        continue
204
                        continue
207

    
   
205

   
208
                    tests.append(TestRun(path, ast_version, self.options))
206
                    tests.append(TestRun(path, ast_version, self.options, self.global_config))
209
                elif val == "dir":
207
                elif val == "dir":
210
                    tests += self._parse_test_yaml(path, ast_version)
208
                    tests += self._parse_test_yaml(path, ast_version)
211

    
   
209

   
212
        return tests
210
        return tests
213

    
   
211

   
[+20] [20] 271 lines
  1. /asterisk/trunk/runtests.py: Loading...

https://reviewboard.asterisk.org/ runs on a server provided by Digium, Inc. and uses bandwidth donated to the open source Asterisk community by API Digital Communications in Huntsville, AL USA.
Please report problems with this site to asteriskteam@digium.com.