Firebug Test Bot
Firebug test-bot support for scheduled running of Firebug test suite.
See how you can install testbot on your machine. I have been installing it on Windows, but it's also possible to install on other OSes.
- You need Python virtual environment
- In case it doesn't come within the Python package you also need Python setuptools
- You might also need to install lxml (tested with lxml-2.3.6.win32-py2.7.exe). See also Known Issues on this page.
- You might want to add following to your PATH:
C:/Python27/;C:/Python27/Scripts/(related to Python 2.7)
- In case you don't wont to install from the source, you can just (assuming easy_install is in your path):
However, it is recommended to install from source
- Download Testbot source from GIT repository:
git clone http://github.com/mozilla/Firebug-Test-Runner.git
- Install from the source:
cd Firebug-Test-Runner python setup.py install
- Using testbot:
- Update tests from the GIT:
git pull origin master python setup.py install
Command Line Arguments
-h, --help show this help message and exit -p PROFILE, --profile=PROFILE The profile to use when running Firefox -b BINARY, --binary=BINARY The binary path to use. If unspecified appropriate binaries will be downloaded automatically -s SERVERPATH, --serverpath=SERVERPATH The http server containing the firebug tests -c COUCHURI, --couch=COUCHURI URI to couchdb server for log information -d DATABASENAME, --database=DATABASENAME Database name to keep log information -t TESTLIST, --testlist=TESTLIST Url to the testlist to use --interval=WAITTIME Number of hours to wait between test runs. If unspecified tests are only run once --debug Enable debug logging
Custom Server With Tests
Testbot is using http://www.getfirebug.com/ by default so, make sure this is what you want, otherwise specify your own server:
runFBTests --serverpath http://www.myserver.org/
Serverpath also specifies where
test-bot.config (the name of the config file is hardcoded) is located. The whole path is computed as follows:
serverpath + "releases/firebug/" + "test-bot.config"
So, make sure the config file is available on your custom server.
Custom Couch DB
If you want you can set up your own CouchDB and point testbot at that one.
runFBTests --couch http://mycouchdb.example.org/couchdb --database myfirebugdb
Custom Test List
You can also specify a test list on the command line:
$ runFBTests -c http://firebug.couchone.com/ -d firebug -s http://legoas/firebug -t http://legoas/firebug/firebug.html
- runFBTests depends on an external python package for scraping the latest firefox builds. There is a bug in this package where it won't return the download url for Firefox 3.6 on Mac OS X 64-bit systems.
- This package also depends on lxml which in turn depends on the libxml2 and libxslt packages. You may also need the python-dev package. In linux and mac this hasn't been a huge problem (just apt-get or port install the packages if it isn't done automatically) but building lxml has given me some headaches in windows (although maybe your development environment is set up better than mine). If setuptools can't compile lxml automatically, I'd recommend installing the pre-compiled binary of lxml found here: http://pypi.python.org/pypi/lxml/2.3beta1 (assuming you are using 32 bit Python) before running setuptools.
Test Results DB
Test results produced by running Firebug test suite are stored into a CouchDB (document-oriented database). The database is available online and all test results (stored as documents) can be accessed using XHR.
Testbot results are stored as documents and there are currently following types of docs.
- header this document represents a test-bot run. You can see list of these documents here.
- result represents a test results. The relation between header and result documents is one-to-many. There is many results from on test-bot run. The relation is made through a headerid filed that is part of each result doc. List of all results is here
The database contains also test-results reported by users who run automated tests manually using Firebug Test Console (FBTest). These results use following doc types.
- user-header represents a user test launch.
- user-results represents a test result from manual running of automated test suite.
Database schema (design documents) are described here.
Test Results UI
Custom CouchDB can be passed through URL as follows:
Automated running of tests is configurable using a config file. Testbot currently expects this file to be available here:
The file follows syntax of INI Files. Here is an example:
[Firebug1.10] FIREBUG_XPI=https://getfirebug.com/releases/firebug/1.10/firebug-1.10.6.xpi FBTEST_XPI=https://getfirebug.com/releases/fbtest/1.10/fbTest-1.10.0.xpi GIT_TAG=HEAD GIT_BRANCH=firebug1.10 GECKO_VERSION=aurora,beta [Firebug1.11] GIT_TAG=HEAD GIT_BRANCH=master GECKO_VERSION=aurora,beta,central DB_URL=http://firebug.couchone.com/ DB_NAME=firebug2 [Firebug1.11-bug800799] GIT_TAG=HEAD GIT_BRANCH=master GECKO_VERSION=aurora,beta,central TEST_LIST=http://getfirebug.com/tests/head/bug800799.html
Each section (there are three in this example) specifies one configuration. Name of the section should be descriptive, but isn't currently used by testbot.
- FIREBUG_XPI - URL of Firebug XPI file used in this configuration.
- FBTEST_XPI - URL of FBTest XPI (Firebug test harness) file used in this configuration.
- GIT_BRANCH - GIT branch in Firebug GIT repo that is used by testbot to pull all test files. Testbot puts these files on its own web server.
- GIT_TAG - GIT revisiion
- GECKO_VERSION - List of Firefox versions used for testing in the configuration
- TEST_LIST - Test suite used by test bot in this configuration.
- DB_URL - URL of an online CouchDB where all test-results should be uploaded
- DB_NAME=firebug2 - Name of the online database where to upload data.
Browser Window Size
Some tests depends on the browser window (with Firebug) size. For example, there can be a test that right clicks on an element in Firebug HTML Panel and opens a context menu on it - in order to execute an action. In case where the HTML Panel is too small the element doesn't have to be visible and the click wouldn't open the menu. The test would fail, even if the feature actually works.
For this reason, FBTest harness sets the browser window size to 1024x768 (as the minimum required size) and moves the window to [0, 0] position on the screen. This is done only in cases where the -runFBTests flag is specified for firefox.exe. So, users running tests as part of the daily development process are not bothered by the browser resizing.
The actual size and location can be specified in preferences:
extensions.fbtest.defaultOuterWidth [default 1024] extensions.fbtest.defaultOuterHeight [default 768] extensions.fbtest.defaultScreenX [default 0] extensions.fbtest.defaultScreenY [default 0]
- As soon as you have Firebug-Test-Runner installed try to execute runFBTests --help on your command line. If you see a list of possible arguments it works well and you can proceed to the next step.
- Now you need to setup your web server. Let's call it legoas.
- The server path is:
- So, the config file URL is:
- The config file can look like as follows:
[MyFirebugConfiguration] FIREBUG_XPI=https://getfirebug.com/releases/firebug/1.11/firebug-1.11.0b2.xpi FBTEST_XPI=https://getfirebug.com/releases/fbtest/1.11/fbTest-1.11b4.xpi GECKO_VERSION=beta TEST_LIST=http://getfirebug.com/tests/head/firebug.html DB_URL=http://firebug.couchone.com/ DB_NAME=firebug
- And finally, you just need to launch the test-bot and specify serverpath.
runFBTests --serverpath http://legoas/
- The test bot automatically runs all tests and uploads all results the specified database.
- Open test-result viewer and specify your database:
- CouchDB: http://firebug.couchone.com/_utils/index.html
- Result Viewer: https://getfirebug.com/testresults
- TestBot Source: https://github.com/mozilla/Firebug-Test-Runner
- XPI Archive: https://getfirebug.com/releases/
- Default Config: https://getfirebug.com/releases/firebug/test-bot.config
- Default Test List: http://getfirebug.com/tests/head/firebug.html