Current state (2012-01-13)

Autotest server

The autotest server verifies basic checks and all JUnit tests, see the details at ​http://dev.geogebra.org/trac/wiki/ServerLinz2#Details.

Deployment server

Some important checks are not included in the autotest process yet, but they are built into the build-pack-installer script on the deployment server. These checks are:

• Packing errors
• JAR signature test
• Installer creation tests for Windows and Mac
• Missing files for webstart
• Rsync problems with www.geogebra.org

Common checks (also built into autotest):

• Eclipse compilation errors

Others? Please describe!

To be written.

Unit Testing

The GeoGebraTest Project includes JUnit Tests for both geogebra commands in general and GeoGebra CAS in special.

EasyB testing see Testing

Testing during Deployment

Ongoing Automatic Testing

• dedicated server
• pre-build each revision in trunk and ggb40 and store (just need unpacked debug jars)
• also have last ggb32 revision (copy JAR files)
• web interface / command line
• upload ggb file (optional: specify starting/ending revision)
• report back problem revision
• web interface -> popup trac page for revision

• database? results for 18,000 ggb files tested
• web interface: query database eg recent changes, files that don't complete loading, change in file loading time?
• initially test against eg every 100th revision so that it doesn't take 1 year to become useful
• record time to load file: then we can monitor if there are significant changes
• test ggbMobile too?

Problems:

• MathPiper, Reduce giving different outputs -> use CAS to check
• (-x)+3 or -x+3, 2 x or 2x -> use CAS to check

Details

The current code is available at ​http://dev.geogebra.org/trac/browser/branches/regression. The ​README file will give some instructions on the usage. The ​tests directory contains a few GGB files with their output for the makeregr script (which is, in fact the output for java -jar geogebra.jar --regressionFile=SOMEGGBFILE.GGB).

Filters

Due to some slightly different behavior of 3.2 and 4.0, it is important to filter the output somehow. It could be done in GeoGebra, but also directly at the output state. It seems the whitespaces should be removed from the output everywhere. Also "(x)" should be deleted. The same for the "(", ")" and "*" characters. "-0.0" should be substituted to "0.0" everywhere. For this purpose we use the filter script.

Of course other filters may be used, but we currently do not have enough experience to know the best filter for our needs. To run the regression test suite one may use the runtests command with the ./filter parameter. On using a different filter, the parameter must be substituted by the execution path of the filter script (or other executable).

Analyzing the results

There are some basic tools provided for investigating the results. They can be run after running a full regression test (runtests).

• ​create-error-report: Creates a directory with error report, containing the erroneous GGBs, their required output and the diffs. This directory can be easily zipped and transferred. Such an example directory can be seen for ​version 3.9.294.0.
• showdiffs: Very basic tool, just prints all differences.
• worst-tests: Creates a list for those GGB files which give different filtered output for 3.2 and 4.0. Such lists exist for ​3.9.307.0 and ​3.9.337.0. Note that the lists are not alphabetically sorted (yet).

Challenges (or criticism)

Expressions seem to be quite difficult to check due to the MathPiper/MPReduce change from 3.2 to 4.0.

Required software to create a (virtual) machine for continuous testing

• Linux (the scripts require Unix shell scripting)
• sed, grep, tail, find (usually present in Linux by default)
• timeout
• Java Runtime Environment (version 6 is suggested from package, e.g. sun-java6-jre on Debian/Ubuntu) for running the tests
• Subversion client (to download latest regression sources)

For automated compilation of the latest development/beta JARs

• Ant
• Java 5 JDK (sun-java5-jdk on Debian/Ubuntu)