Running Automated Test Suite

From FirebugWiki

(Difference between revisions)
Jump to: navigation, search
(Prerequisities)
m (Fixed test list URL)
 
(12 intermediate revisions not shown)
Line 1: Line 1:
-
This page explains how to run Firebug automated tests. All information on this page are related to Firebug 1.5.
+
This page explains how to run automated tests for Firebug.
 +
== Prerequisits ==
-
See also:
+
# Install [http://getfirebug.com/releases/firebug/ Firebug]
-
 
+
# Install [http://getfirebug.com/releases/fbtest/ FBTest]
-
* [http://getfirebug.com/wiki/index.php/Firebug_Automated_Test_API Firebug Automated Test API]
+
-
 
+
-
 
+
-
== Prerequisities ==
+
-
 
+
-
# Install [http://getfirebug.com/releases/firebug/1.6X/ Firebug 1.6] (the latest version recommended)
+
-
# Install [http://getfirebug.com/releases/fbtest/1.6/ FBTest 1.6] (the latest version recommended)
+
== Running Tests ==
== Running Tests ==
-
# Use Firebug->Open Test Console menu to open test console window.
+
# Use Firebug->Open Test Console menu to open test console window
-
# Put ''http://getfirebug.com/tests/content/testlists/firebug1.5.html'' URL into the URL bar.
+
# Put the URL ''http://getfirebug.com/tests/head/firebug.html'' for the most current test list.<br/>Older test lists can be accessed via ''http://getfirebug.com/tests/x/firebug.html'' into the URL bar (x stands for the version of Firebug, e.g. "1.9")
-
# Press Run All button on the toolbar.
+
# Press the <code>Run All</code> button on the toolbar
-
 
+
That's it!
That's it!
Line 23: Line 16:
You should see how all tests displayed in the test console are executed step by step. The browser window should automatically load test pages and automate specific steps needed for test execution.
You should see how all tests displayed in the test console are executed step by step. The browser window should automatically load test pages and automate specific steps needed for test execution.
-
Note: The FBTest extension will be part of the Firebug 1.5X at some point (development version). This means that you need to install it only if you want to test Firebug 1.5 (end user version). This page will be updated as soon as this happens.
+
In order to test Firebug you need to install [[FBTest]] and load the related test suite.
 +
''Note: The [[FBTest]] extension will be part of the Firebug development version at some point. This means that you'll need to install it only if you want to test the Firebug end user version. This page will be updated as soon as this happens.''
== Test Console UI ==
== Test Console UI ==
Line 35: Line 29:
* '''Run All''' Use this button to run all tests from the current test suite.
* '''Run All''' Use this button to run all tests from the current test suite.
* '''Stop Use''' this button to stop running tests.
* '''Stop Use''' this button to stop running tests.
-
* '''Fail Halt''' Use this option (two state button) to automatically stop test execution when an error occurs (you need [http://getfirebug.com/wiki/index.php/Chromebug_User_Guide Chromebug 1.5] installed). In such a case Chromebug will automatically switch into the debug mode showing the source line where the error happend. You can continue debugging the test code or put a breakpoint and run the test again.
+
* '''Fail Halt''' Use this option (two state button) to automatically stop test execution when an error occurs (you need [[Chromebug]] installed). In such a case Chromebug will automatically switch into the debug mode showing the source line where the error happend. You can continue debugging the test code or put a breakpoint and run the test again.
* '''Refresh''' Reloads the current test list.
* '''Refresh''' Reloads the current test list.
* '''No Timeout''' Use this option to disable timeout on running test. Useful when debugging in Chromebug.
* '''No Timeout''' Use this option to disable timeout on running test. Useful when debugging in Chromebug.
-
 
There is also an URL bar that allows to load a test setup file. This file specifies all info necessary to execute the tests. Such a file is included in Firebug distribution and so, if you have Firebug installed you can always load: chrome://firebug/content/testList.html (the default Firebug test setup file).
There is also an URL bar that allows to load a test setup file. This file specifies all info necessary to execute the tests. Such a file is included in Firebug distribution and so, if you have Firebug installed you can always load: chrome://firebug/content/testList.html (the default Firebug test setup file).
-
 
== Export Errors ==
== Export Errors ==
Line 47: Line 39:
[[File:Fb-test-copy-all-errors.png|center]]
[[File:Fb-test-copy-all-errors.png|center]]
 +
 +
== Submit Test Results ==
 +
Firebug test console also allows submitting your test results to an online database. This DB is used mainly by [http://getfirebug.com/wiki/index.php/Firebug_Test_Bot testbot], but can contains also results from users who manually run automated test suite. This DB is consequently [http://getfirebug.com/testresults monitored] by Firebug team members to check whether test are running on various configurations.
 +
 +
[[File:Fbtest-submit-test-results.png|center]]
 +
 +
''Note that submitted into contains list of all installed extensions in the current Firefox profile.''
== Running Tests From The Command Line ==
== Running Tests From The Command Line ==
It's also possible to automatically run all tests (or one specific test) from the command line when launching Firefox. The syntax of the parameter is as follows:
It's also possible to automatically run all tests (or one specific test) from the command line when launching Firefox. The syntax of the parameter is as follows:
-
<pre>-runFBTests <test-setup-file-uri>#<test-driver-relative-path></pre>
+
<source lang="bash">
 +
-runFBTests <test-setup-file-uri>#<test-driver-relative-path>
 +
</source>
Let's see some examples.
Let's see some examples.
Line 57: Line 58:
Run all default tests afer Firefox is started using the default user profile
Run all default tests afer Firefox is started using the default user profile
-
<pre>firefox -runFBTests chrome://firebug/content/testList.html</pre>
+
<source lang="bash">
 +
firefox -runFBTests http://getfirebug.com/tests/content/testlists/firebug1.7.html
 +
</source>
Run all default tests after Firefox is started using the dev user profile.
Run all default tests after Firefox is started using the dev user profile.
-
<pre>firefox -dev -runFBTests chrome://firebug/content/testList.html</pre>
+
<source lang="bash">
 +
firefox -dev -runFBTests http://getfirebug.com/tests/content/testlists/firebug1.7.html
 +
</source>
-
Load chrome://firebug/content/testList.html test setup file and run net/1468/issue1468.js test after Firefox is started using the dev user profile.
+
Load http://getfirebug.com/tests/content/testlists/firebug1.7.html test setup file and run net/1468/issue1468.js test after Firefox is started using the dev user profile.
-
 
+
-
<pre>firefox -dev -runFBTests chrome://firebug/content/testList.html#net/1468/issue1468.js</pre>
+
 +
<source lang="bash">
 +
firefox -dev -runFBTests http://getfirebug.com/tests/content/testlists/firebug1.7.html#net/1468/issue1468.js
 +
</source>
== Terminology ==
== Terminology ==
Line 74: Line 80:
* Test Driver (*.js) This is the automation part. Piece of javascript that loads the test page and executes all steps automatically.
* Test Driver (*.js) This is the automation part. Piece of javascript that loads the test page and executes all steps automatically.
-
 
Finally, tests can be combined together.
Finally, tests can be combined together.
Line 81: Line 86:
== Test List File Structure ==
== Test List File Structure ==
 +
The test list file must be implemented as a regular page. Here is an example of such a file.
The test list file must be implemented as a regular page. Here is an example of such a file.
-
<pre code="xml">
+
<source lang="xml">
<html>
<html>
-
<head></head>
+
  <head>
-
<body>
+
    <title>Firebug Test File</title>
-
<script type="text/javascript">
+
  </head>
-
var baseURI = "https://getfirebug.com/content/";
+
  <body>
-
var serverURI = "https://getfirebug.com/content/";
+
    <script type="text/javascript">
-
var testList = [
+
      var baseURI = "https://getfirebug.com/content/";
-
    {
+
      var serverURI = "https://getfirebug.com/content/";
-
        group: "Group1 Title",
+
      var testList = [
-
        uri: "group1/testDriver1.js",
+
          {
-
        desc: "This is a description of test 1",
+
              group: "Group1 Title",
-
        testPage: "group1/testCase1.html"
+
              uri: "group1/testDriver1.js",
-
    }
+
              desc: "This is a description of test 1",
-
];
+
              testPage: "group1/testCase1.html"
-
</script>
+
          }
-
</body>
+
      ];
 +
    </script>
 +
  </body>
</html>
</html>
-
</pre>
+
</source>
If you load such a file into the test console, you should see one expandable group with one test.
If you load such a file into the test console, you should see one expandable group with one test.
Line 107: Line 115:
There are following variables defined in the file.
There are following variables defined in the file.
-
* baseURI Root directory with all test-drivers.
+
* '''baseURI:''' Root directory with all test-drivers
-
* serverURI Root directory with all test-cases
+
* '''serverURI:''' Root directory with all test-cases
-
* testList List of tests.
+
* '''testList:''' List of tests
Each test in the array has following properties.
Each test in the array has following properties.
-
* group Name of the parent group where the test belongs to.
+
* '''group:''' Name of the parent group where the test belongs to
-
* uri Relative path of the test driver  
+
* '''uri:''' Relative path of the test driver  
-
* desc Short summary of the test (displayed in the UI)
+
* '''desc:''' Short summary of the test (displayed in the UI)
-
* testPage: relative path to the test case file.
+
* '''testPage:''' Relative path to the test case file
-
 
+
In this particular case, both test drivers and test cases are located at https://getfirebug.com/content/ and the directory structure under this root folder would look like as follows:
In this particular case, both test drivers and test cases are located at https://getfirebug.com/content/ and the directory structure under this root folder would look like as follows:
Line 127: Line 134:
         testCase1.js
         testCase1.js
</pre>
</pre>
 +
 +
== Swarm Testing Procedure ==
 +
 +
Swarm testing is a semi-automatic test procedure for multiple Firebug extensions. 
 +
 +
=== Swarm Testing Setup ===
 +
You need:
 +
* Firefox (duh!)
 +
* [http://ant.apache.org/manual/index.html Ant, an Apache build tool]
 +
* [http://code.google.com/p/fbug/source/checkout The build scripts]
 +
** You should only need fbug/extensions/swarm/branches/swarm1.6/swarms
 +
** create a local.properties file in the swarms/ folder and define the path to your firefox and the swarm server:
 +
<source lang="properties">
 +
firefox=C:/mozilla/firefox/3.6/firefox/firefox.exe
 +
#testListServer=file://c:/Users/johnjbarton/firebug/fbug
 +
swarmServer=file://c:/Users/johnjbarton/firebug/fbug/extensions/swarm/branches/swarm1.6
 +
</source>
 +
 +
*If you are testing a previous defined swarm, comment out or do not set the swarmServer property. That will use getfirebug.com.
 +
*If you are testing a local new swarm, set the swarmServer to a folder name or a URL if you have a local server.
 +
*Optional, you can set the test list server local for changing tests.
 +
 +
=== Testing Process ===
 +
 +
# Run ant with build.xml from the swarms directory (default target is 'test')
 +
# The build script will create a new profile, open the swarm page, and prompt you with a list of extensions to be installed
 +
# Click "Install Now", Firefox will begin downloading extensions
 +
# When Firefox says "Restart", do not do this. Just exit Firefox.
 +
# The build script will immediately run Firefox again, this time with the automatic testing running.
 +
# About half the time Firefox crashes before the tests complete. Just re-run the ant command again.
 +
# A successful run takes about 4 minutes on my computer; Firefox will be going wild.
 +
# When the tests complete, Firefox will ask you to verify that you want to exit. Go to the FBTest window, right click and select "Submit Test Results".
 +
# The save operation will open a new page, copy its URL from the location bar
 +
# If you are defining a new swarm, paste the test result URL in to the bottom of the page around the Tested With words.
 +
# Exit Firefox. It will crash most of the time on exit.
 +
# When you are all done, run the ant build.xml with target "clean". That will erase the temporary profile used for testing.
 +
 +
== See also ==
 +
* [[Running Tests From Extensions]]
 +
* [[Firebug Test Bot]]
 +
* [[Firebug Automated Test API]]
 +
* [[Releasing FBTest]]

Latest revision as of 06:31, 6 June 2012

This page explains how to run automated tests for Firebug.

Contents

[edit] Prerequisits

  1. Install Firebug
  2. Install FBTest

[edit] Running Tests

  1. Use Firebug->Open Test Console menu to open test console window
  2. Put the URL http://getfirebug.com/tests/head/firebug.html for the most current test list.
    Older test lists can be accessed via http://getfirebug.com/tests/x/firebug.html into the URL bar (x stands for the version of Firebug, e.g. "1.9")
  3. Press the Run All button on the toolbar

That's it!

You should see how all tests displayed in the test console are executed step by step. The browser window should automatically load test pages and automate specific steps needed for test execution.

In order to test Firebug you need to install FBTest and load the related test suite.

Note: The FBTest extension will be part of the Firebug development version at some point. This means that you'll need to install it only if you want to test the Firebug end user version. This page will be updated as soon as this happens.

[edit] Test Console UI

The main purpose of the test console is displaying and executing tests in a test suite. First of all let's see a screenshot of the console. Note that all tests are divided into groups.

Fbtest-ui.png

You can see a toolbar with following actions:

  • Run All Use this button to run all tests from the current test suite.
  • Stop Use this button to stop running tests.
  • Fail Halt Use this option (two state button) to automatically stop test execution when an error occurs (you need Chromebug installed). In such a case Chromebug will automatically switch into the debug mode showing the source line where the error happend. You can continue debugging the test code or put a breakpoint and run the test again.
  • Refresh Reloads the current test list.
  • No Timeout Use this option to disable timeout on running test. Useful when debugging in Chromebug.

There is also an URL bar that allows to load a test setup file. This file specifies all info necessary to execute the tests. Such a file is included in Firebug distribution and so, if you have Firebug installed you can always load: chrome://firebug/content/testList.html (the default Firebug test setup file).

[edit] Export Errors

In order to export all reported failures, just right-click within the Test Console window content and pick Copy All Errors from the context menu. An error report will be copied into the clipboard together with information about current Firebug & Firefox versions, current test-list URL and also version and name of the underlying OS.

Fb-test-copy-all-errors.png

[edit] Submit Test Results

Firebug test console also allows submitting your test results to an online database. This DB is used mainly by testbot, but can contains also results from users who manually run automated test suite. This DB is consequently monitored by Firebug team members to check whether test are running on various configurations.

Fbtest-submit-test-results.png

Note that submitted into contains list of all installed extensions in the current Firefox profile.

[edit] Running Tests From The Command Line

It's also possible to automatically run all tests (or one specific test) from the command line when launching Firefox. The syntax of the parameter is as follows:

-runFBTests <test-setup-file-uri>#<test-driver-relative-path>

Let's see some examples.

Run all default tests afer Firefox is started using the default user profile

firefox -runFBTests http://getfirebug.com/tests/content/testlists/firebug1.7.html

Run all default tests after Firefox is started using the dev user profile.

firefox -dev -runFBTests http://getfirebug.com/tests/content/testlists/firebug1.7.html

Load http://getfirebug.com/tests/content/testlists/firebug1.7.html test setup file and run net/1468/issue1468.js test after Firefox is started using the dev user profile.

firefox -dev -runFBTests http://getfirebug.com/tests/content/testlists/firebug1.7.html#net/1468/issue1468.js

[edit] Terminology

Each automated test is typically composed two parts.

  • Test Page (*.html) This is a regular web page (often with AJAX code) that represents specific scenario being tested. Since (at least in case of Firebug tests) these pages are available online, the tester can simply load it in the browser (no need for FBTest extension), manually follow described scenario and execute the test without any automation.
  • Test Driver (*.js) This is the automation part. Piece of javascript that loads the test page and executes all steps automatically.

Finally, tests can be combined together.

  • Test List (*.html) This file is loaded into the Test Console and specifies neccessary information (URLs) about test cases and test drivers locations. It also contains a javascript inline defintion of test drivers or includes it from an external *.js file using the <script> tag.

[edit] Test List File Structure

The test list file must be implemented as a regular page. Here is an example of such a file.

<html>
  <head>
    <title>Firebug Test File</title>
  </head>
  <body>
    <script type="text/javascript">
      var baseURI = "https://getfirebug.com/content/";
      var serverURI = "https://getfirebug.com/content/";
      var testList = [
          {
              group: "Group1 Title",
              uri: "group1/testDriver1.js",
              desc: "This is a description of test 1",
              testPage: "group1/testCase1.html"
          }
      ];
    </script>
  </body>
</html>

If you load such a file into the test console, you should see one expandable group with one test.

There are following variables defined in the file.

  • baseURI: Root directory with all test-drivers
  • serverURI: Root directory with all test-cases
  • testList: List of tests

Each test in the array has following properties.

  • group: Name of the parent group where the test belongs to
  • uri: Relative path of the test driver
  • desc: Short summary of the test (displayed in the UI)
  • testPage: Relative path to the test case file

In this particular case, both test drivers and test cases are located at https://getfirebug.com/content/ and the directory structure under this root folder would look like as follows:

<root>
    group1
        testDriver1.js
        testCase1.js

[edit] Swarm Testing Procedure

Swarm testing is a semi-automatic test procedure for multiple Firebug extensions.

[edit] Swarm Testing Setup

You need:

  • Firefox (duh!)
  • Ant, an Apache build tool
  • The build scripts
    • You should only need fbug/extensions/swarm/branches/swarm1.6/swarms
    • create a local.properties file in the swarms/ folder and define the path to your firefox and the swarm server:
firefox=C:/mozilla/firefox/3.6/firefox/firefox.exe
#testListServer=file://c:/Users/johnjbarton/firebug/fbug
swarmServer=file://c:/Users/johnjbarton/firebug/fbug/extensions/swarm/branches/swarm1.6
  • If you are testing a previous defined swarm, comment out or do not set the swarmServer property. That will use getfirebug.com.
  • If you are testing a local new swarm, set the swarmServer to a folder name or a URL if you have a local server.
  • Optional, you can set the test list server local for changing tests.

[edit] Testing Process

  1. Run ant with build.xml from the swarms directory (default target is 'test')
  2. The build script will create a new profile, open the swarm page, and prompt you with a list of extensions to be installed
  3. Click "Install Now", Firefox will begin downloading extensions
  4. When Firefox says "Restart", do not do this. Just exit Firefox.
  5. The build script will immediately run Firefox again, this time with the automatic testing running.
  6. About half the time Firefox crashes before the tests complete. Just re-run the ant command again.
  7. A successful run takes about 4 minutes on my computer; Firefox will be going wild.
  8. When the tests complete, Firefox will ask you to verify that you want to exit. Go to the FBTest window, right click and select "Submit Test Results".
  9. The save operation will open a new page, copy its URL from the location bar
  10. If you are defining a new swarm, paste the test result URL in to the bottom of the page around the Tested With words.
  11. Exit Firefox. It will crash most of the time on exit.
  12. When you are all done, run the ant build.xml with target "clean". That will erase the temporary profile used for testing.

[edit] See also

Personal tools