Review Board 1.7.16

Various unit test API updates

Review Request #493 - Created Feb. 8, 2010 and submitted

Russell Bryant
This is a bit of a mix of changes to the test API that were a part of what I worked on during part of a long flight.  Here is a breakdown of changes included here:

1) It occurred to me that the difference in usage between the error ast_str and the ast_test_update_status() usage has turned out to be a bit ambiguous in practice.  In a lot of cases, the same message was being sent to both.  In other cases, it was only sent to one or the other.  My opinion now is that in every case, I think it makes sense to do both; we should output it to the CLI as well as save it off for logging purposes.

This change results in most of the changes in this diff, since it required changes to all existing unit tests.  It also allowed for some simplifications of unit test API implementation code.

2) Update ast_test_status_update() to include the file, function, and line number for the code providing the update.

3) There are some formatting tweaks here and there.  Hopefully they aren't too distracting for code review purposes.  Reviewboard's diff viewer seems to do a pretty good job of pointing out when something is a whitespace change.

4) I moved the md5_test and sha1_test into the test_utils module.  It seemed like a better approach since these tests are so tiny.

5) I changed the number of nodes used in heap_test_2 from 1 million to 100 thousand.  The only reason for this was to reduce the time it took for this test to run.

6) Remove an unused function prototype that was at the bottom of utils.h.

7) Simplify test_insert() using the LIST_INSERT_SORTALPHA() macro.  The one minor difference in behavior is that it no longer checks for a test registered with the same name.

8) Expand the code in test_alloc() to provide specific error messages for each failure case, to clearly inform developers if they forget to set the name, summary, description, etc.

9) Tweak the output of the "test show registered" CLI command.  I swapped the name and category to have the category first.  It seemed more natural since that is the sort key.

10) Don't output the status ast_str in the "test show results" CLI command.  This is going to tend to be pretty verbose, so just leave that for the detailed test logs (test generate results).

I have run the tests in the test suite to make sure they all still pass.  I have also run through the CLI commands to verify that I get the expected output.
Posted (Feb. 9, 2010, 5:25 a.m.)
Great work, I like this! Something definitely needed to be done to simplify this and I believe this is the right approach.  This is one of those tweaks that need to be made early on before things just get way out of control.  The only thing I'm debating about from a design standpoint is whether or not we should distinguish between errors and updates.  I agree that duplicate messages should go away.  There is absolutely no reason to send the same message twice, but would it be helpful to be able to easily identify error messages from other messages if we are going to allow everything to go to the same place?
  1. I understand the concern.  I'm thinking that the errors may be easier to digest if surrounded by the status that led up to them.  Also, I suspect it's only useful to review the log on failed test cases.  That is the only time the output is included in the XML logs.  We could do the same for the text output, as well.
/trunk/include/asterisk/test.h (Diff revision 1)
Should we const the ast_test in these macros? I guess since we don't define the ast_test in the header the test functions don't actually know what an ast_test is, but it would be nice if they couldn't modify it at all.
  1. The only usage of the argument requires a non const reference to the ast_test.  It is passed to the status update function, which will modify the ast_str status log.
/trunk/main/test.c (Diff revision 1)
This is going to contain a log of updates as well as errors.  Will that make this field more verbose than necessary?
  1. It will certainly become more verbose, but I think it's useful to have a full log of the test.  Perhaps we should only include it if the test failed, like is done in the XML case?
/trunk/tests/test_utils.c (Diff revision 1)
All these double \n or log strings beginning with a \n need to change to just being the log sting ending with a single \n.  This was my code, and it is dumb. We probably don't even need all these status updates on success to begin with.  If you don't change this I will afterwards. 
  1. I think I removed the double messages already.  As for extra \ns, I'll take a look for them to include a fix in my commit.  I don't mind.
Ship it!
Posted (Feb. 9, 2010, 6:54 a.m.)
I agree, since the logs are specific for each test and are only reported on failure it makes sense to include all status updates.  I really like this, Thanks Russell!
Ship it!
Posted (Feb. 9, 2010, 6:54 a.m.)
I agree, since the logs are specific for each test and are only reported on failure it makes sense to include all status updates.  I really like this, Thanks Russell! runs on a server provided by Digium, Inc. and uses bandwidth donated to the open source Asterisk community by API Digital Communications in Huntsville, AL USA.
Please report problems with this site to