Firebug Test Bot

From FirebugWiki
Revision as of 09:40, 2 February 2011 by Honza (Talk | contribs)

Jump to: navigation, search

Firebug test-bot support for scheduled running of Firebug test suite.


Installing TestBot

See how you can install testbot on your machine. I have been installing it on Windows, but it's also possible to install on other OSes.

  • You need Python virtual environment
  • In case it doesn't come within the Python package you also need Python setuptools
  • In case you don't wont to install from the source, you can just (assuming easy_install is in your path):
easy_install runFBTests
  • Download Testbot source from GIT repository:
git clone
  • Install from the source:
cd Firebug-Test-Runner
python install
  • Using testbot:
runFBTests --help
  • Update tests from the GIT:
git pull origin master
python install

Custom Server With Tests

Testbot is using tests from by default so, make sure this what you want, otherwise specify your own server:

runFBTests --serverpath

Custom Couch DB

If you want you can set up your own CouchDB and point testbot at that one.

runFBTests --couch --database myfirebugdb

Known Issues

  • runFBTests depends on an external python package for scraping the latest firefox builds. There is a bug in this package where it won't return the download url for Firefox 3.6 on Mac OS X 64-bit systems.
  • This package also depends on lxml which in turn depends on the libxml2 and libxslt packages. You may also need the python-dev package. In linux and mac this hasn't been a huge problem (just apt-get or port install the packages if it isn't done automatically) but building lxml has given me some headaches in windows (although maybe your development environment is set up better than mine). If setuptools can't compile lxml automatically, I'd recommend installing the pre-compiled binary of lxml found here: (assuming you are using 32 bit Python) before running setuptools. I was trying to make the process on Windows as painless as possible, but I had to settle for doing it that way.
  • Mozilla is currently restructuring their build paths so this package will break when that happens (probably post ffx 4). I'll try and get things fixed asap when it does.

Test Results DB

Test results produced by running Firebug test suite are stored into a CouchDB (document-oriented database). The database is available online and all test results (stored as documents) can be accessed using XHR.

Testbot results are stored as documents and there are currently following types of docs.

  • header this document represents a test-bot run. You can see list of these documents here.
  • result represents a test results. The relation between header and result documents is one-to-many. There is many results from on test-bot run. The relation is made through a headerid filed that is part of each result doc. List of all results is here

The database contains also test-results reported by users who run automated tests manually using Firebug Test Console (FBTest). These results use following doc types.

  • user-header represents a user test launch.
  • user-results represents a test result from manual running of automated test suite.

Test Results UI

Summary of testbot results is available online. This page displays both (a) testbot and (b) user results as two separate lists. Source code of this application is part of Firebug SVN tree.

Testbot Config

Automated running of tests is configurable using a config file. Testbot currently expects this file to be available here:

The file follows syntax of INI Files. Here is an example:



Each section (there are two in this example) specifies one configuration. Name of the section should be descriptive, but isn't currently used by testbot.

  • FIREBUG_XPI - URL of Firebug XPI file used in this configuration.
  • FBTEST_XPI - URL of FBTest XPI (Firebug test harness) file used in this configuration.
  • SVN_REVISION - Revision number in Firebug SVN that is used by testbot to pull all test files. Testbot puts these files on its own web server.
  • FIREFOX_VERSION - Firefox version used in this configuration
  • TEST_LIST - Test suite used by test bot in this configuration.