Firebug Test Bot

From FirebugWiki

(Difference between revisions)
Jump to: navigation, search
(Custom Couch DB)
(Windows)
 
(27 intermediate revisions not shown)
Line 6: Line 6:
* [http://brasstacks.mozilla.com/couchdb/_utils/database.html?firebug/_all_docs Database] with reported test results.
* [http://brasstacks.mozilla.com/couchdb/_utils/database.html?firebug/_all_docs Database] with reported test results.
* Initial reported [https://bugzilla.mozilla.org/show_bug.cgi?id=559466 bug]
* Initial reported [https://bugzilla.mozilla.org/show_bug.cgi?id=559466 bug]
 +
 +
== Resources ==
 +
* CouchDB: http://firebug.couchone.com/_utils/index.html
 +
* Result Viewer: https://getfirebug.com/testresults
 +
* TestBot Source: https://github.com/mozilla/Firebug-Test-Runner
 +
* XPI Archive: https://getfirebug.com/releases/
 +
* Default Config: https://getfirebug.com/releases/firebug/test-bot.config
 +
* Default Test List: http://getfirebug.com/tests/head/firebug.html
 +
* Create CouchDB for TestBot: https://getfirebug.com/wiki/index.php/Firebug_Test_Bot_Database
== Installing TestBot ==
== Installing TestBot ==
-
See how you can install testbot on your machine. I have been installing it on Windows, but it's also possible to install on other OSes.
 
 +
=== Windows ===
* You need [http://www.python.org/download/ Python] virtual environment
* You need [http://www.python.org/download/ Python] virtual environment
* In case it doesn't come within the Python package you also need Python [http://pypi.python.org/pypi/setuptools setuptools]
* In case it doesn't come within the Python package you also need Python [http://pypi.python.org/pypi/setuptools setuptools]
Line 18: Line 27:
easy_install runFBTests
easy_install runFBTests
</source>
</source>
 +
''However, it is strongly recommended to install from source''
* Download Testbot source from GIT repository:
* Download Testbot source from GIT repository:
<source lang="bash">
<source lang="bash">
Line 30: Line 40:
<source lang="bash">
<source lang="bash">
runFBTests --help
runFBTests --help
 +
</source>
 +
* Run testbot:
 +
<source lang="bash">
 +
runFBTests --serverpath http://legoas/firebug
</source>
</source>
* Update tests from the GIT:
* Update tests from the GIT:
Line 36: Line 50:
python setup.py install
python setup.py install
</source>
</source>
 +
 +
=== Ubuntu ===
 +
Python Virtual Environment:
 +
* http://pypi.python.org/pypi/virtualenv
 +
* http://virtualenvwrapper.readthedocs.org/en/latest/install.html
 +
 +
Install pip and virtualenv for Ubuntu 10.10 Maverick and newer:
 +
* http://www.saltycrane.com/blog/2010/02/how-install-pip-ubuntu/
 +
<pre>
 +
$ sudo apt-get install python-pip python-dev build-essential
 +
$ sudo pip install --upgrade pip
 +
$ sudo pip install --upgrade virtualenv
 +
</pre>
 +
 +
Install virtualenvwrapper
 +
<pre>sudo pip install virtualenvwrapper</pre>
 +
 +
Update config file for virtual environments:
 +
<pre>gedit .bashrc</pre>
 +
 +
Put at the end of the file:
 +
<pre>
 +
export WORKON_HOME=$HOME/.virtualenvs
 +
export PROJECT_HOME=$HOME/Devel
 +
source /usr/local/bin/virtualenvwrapper.sh
 +
</pre>
 +
 +
Reload the config file:
 +
<pre>source ~/.bashrc</pre>
 +
 +
Create new virtual environment called ‘Firebug’:
 +
<pre>mkvirtualenv firebug</pre>
 +
 +
Install Firebug testRunner:
 +
 +
<pre>sudo apt-get install python-setuptools
 +
sudo apt-get install libxml2-dev libxslt-dev
 +
sudo apt-get install python-lxml
 +
</pre>
 +
 +
Install git:
 +
<pre>sudo apt-get install git</pre>
 +
 +
Download Firebug Test Runner source:
 +
<pre>git clone http://github.com/mozilla/Firebug-Test-Runner.git</pre>
 +
 +
Install Firebug Test Runner:
 +
<pre>
 +
cd Firebug-Test-Runner
 +
python setup.py install
 +
</pre>
 +
 +
Execute Test Runner:
 +
<pre>/usr/bin$ python runFBTests --serverpath http://10.0.3.102/firebug</pre>
 +
 +
Do not forget to activate your python virtual environment the next time you start up your computer before using the test runner:
 +
 +
<pre>
 +
workon firebug
 +
runFBTests --help
 +
</pre>
 +
 +
=== Mac OS ===
 +
'''This section needs to be yet revisited'''
 +
 +
Install pip, virtualenv and virtualenvwrapper:
 +
* http://darklaunch.com/2011/11/24/osx-install-pip-virtualenv-virtualenvwrapper-on-mac
 +
 +
Install GIT
 +
* http://code.google.com/p/git-osx-installer/downloads/detail?name=git-1.8.0-intel-universal-snow-leopard.dmg&can=3&q=&sort=-uploaded
 +
 +
Install GCC
 +
* https://github.com/kennethreitz/osx-gcc-installer
 +
 +
Install lxml
 +
* http://stackoverflow.com/questions/1277124/how-do-you-install-lxml-on-os-x-leopard-without-using-macports-or-fink
 +
* <pre>sudo env ARCHFLAGS="-arch i386 -arch x86_64" easy_install lxml</pre>
 +
 +
Run this to make the 'mkvirtualenv' command working:
 +
* <pre>source /usr/local/bin/virtualenvwrapper.sh</pre>
 +
 +
Run test-bot:
 +
* <pre>sudo runFBTests --debug --serverpath http://legoas/firebug</pre>
 +
 +
== Command Line Arguments ==
 +
 +
<pre>
 +
  -h, --help            show this help message and exit
 +
  -p PROFILE, --profile=PROFILE
 +
                        The profile to use when running Firefox
 +
  -b BINARY, --binary=BINARY
 +
                        The binary path to use. If unspecified appropriate
 +
                        binaries will be downloaded automatically
 +
  -s SERVERPATH, --serverpath=SERVERPATH
 +
                        The http server containing the firebug tests
 +
  -c COUCHURI, --couch=COUCHURI
 +
                        URI to couchdb server for log information
 +
  -d DATABASENAME, --database=DATABASENAME
 +
                        Database name to keep log information
 +
  -t TESTLIST, --testlist=TESTLIST
 +
                        Url to the testlist to use
 +
  --interval=WAITTIME  Number of hours to wait between test runs. If
 +
                        unspecified tests are only run once
 +
  --debug              Enable debug logging
 +
</pre>
== Custom Server With Tests ==
== Custom Server With Tests ==
Line 44: Line 163:
</source>
</source>
-
Serverpath also specifies where <code>test-bot.config</code> (the name of the config file is hardcoded) is located. The whole path is computed as follows:
+
Serverpath also specifies where <code>test-bot.config</code> (the name of the config file is [https://github.com/mozilla/Firebug-Test-Runner/blob/master/runFBTests/fb_run.py#L51 hardcoded]) is located. The whole path is computed as follows:
-
<pre>serverpath + releases/firebug/ + test-bot.config</pre>
+
<pre>serverpath + "releases/firebug/" + "test-bot.config"</pre>
So, make sure the config file is available on your custom server.
So, make sure the config file is available on your custom server.
Line 57: Line 176:
</source>
</source>
-
== Custom Test List ==
+
You can also specify the database in test-bot.config
-
You can also specify a test list on the command line:
+
-
<source lang="bash">
+
<pre>
-
$ runFBTests -c http://firebug.couchone.com/ -d firebug -s http://legoas/firebug -t http://legoas/src/github.com/firebug/firebug/tests/content/firebug.html
+
DB_URL=http://firebug.couchone.com/
-
</source>
+
DB_NAME=firebug3
 +
</pre>
-
== Known Issues ==
+
In order to preview your custom database, you can configure test result viewer through URL:
-
* runFBTests depends on an external python package for scraping the latest firefox builds. There is a bug in this package where it won't return the download url for Firefox 3.6 on Mac OS X 64-bit systems.
+
-
* This package also depends on lxml which in turn depends on the libxml2 and libxslt packages. You may also need the python-dev package.  In linux and mac this hasn't been a huge problem (just apt-get or port install the packages if it isn't done automatically) but building lxml has given me some headaches in windows (although maybe your development environment is set up better than mine). If setuptools can't compile lxml automatically, I'd recommend installing the pre-compiled binary of lxml found here: http://pypi.python.org/pypi/lxml/2.3beta1 (assuming you are using 32 bit Python) before running setuptools.
+
<pre>
 +
https://getfirebug.com/testresults?dburi=http://firebug.couchone.com/&dbname=firebug3
 +
</pre>
-
== Test Results DB ==
+
See how to create a new [[Firebug Test Bot Database]]
-
Test results produced by running Firebug test suite are stored into a [http://couchdb.apache.org/ CouchDB] (document-oriented database). The database is available [http://brasstacks.mozilla.com/couchdb/_utils/database.html?firebug/_all_docs online] and all test results (stored as documents) can be accessed using XHR.
+
-
Testbot results are stored as documents and there are currently following types of docs.
+
== Custom Test List ==
 +
You can also specify a test list on the command line:
-
* ''header'' this document represents a test-bot run. You can see list of these documents [http://brasstacks.mozilla.com/couchdb/_utils/database.html?firebug/_design/resultviews/_view/allheaders here].
+
<source lang="bash">
-
* ''result'' represents a test results. The relation between ''header'' and ''result'' documents is one-to-many. There is many results from on test-bot run. The relation is made through a ''headerid'' filed that is part of each ''result'' doc. List of all results is [http://brasstacks.mozilla.com/couchdb/_utils/database.html?firebug/_design/resultviews/_view/allresults here]
+
$ runFBTests -c http://firebug.couchone.com/ -d firebug -s http://legoas/firebug -t http://legoas/firebug/firebug.html
 +
</source>
-
The database contains also test-results reported by users who run automated tests manually using [http://getfirebug.com/wiki/index.php/Running_Automated_Test_Suite Firebug Test Console] (FBTest). These results use following doc types.
+
Again, you can specify custom test-list in test-bot.config file
-
* ''user-header'' represents a user test launch.
+
<pre>
-
* ''user-results'' represents a test result from manual running of automated test suite.
+
TEST_LIST=http://legoas/src/github.com/firebug/firebug/tests/content/firebug.html
-
 
+
</pre>
-
== Test Results UI ==
+
-
Summary of testbot results is available [https://getfirebug.com/testresults online]. This page displays both (a) testbot and (b) user results as two separate lists. Source code of this application is [https://github.com/firebug/test-result-viewer available].
+
-
 
+
-
Custom CouchDB can be passed through URL as follows:
+
-
<pre>https://getfirebug.com/testresults?dburi=http://firebug.couchone.com/&dbname=firebug2</pre>
+
== Testbot Config ==
== Testbot Config ==
Line 107: Line 223:
GIT_BRANCH=master
GIT_BRANCH=master
GECKO_VERSION=aurora,beta,central
GECKO_VERSION=aurora,beta,central
 +
DB_URL=http://firebug.couchone.com/
 +
DB_NAME=firebug2
[Firebug1.11-bug800799]
[Firebug1.11-bug800799]
Line 123: Line 241:
* GECKO_VERSION - List of Firefox versions used for testing in the configuration
* GECKO_VERSION - List of Firefox versions used for testing in the configuration
* TEST_LIST - [http://getfirebug.com/wiki/index.php/Running_Automated_Test_Suite#Test_List_File_Structure Test suite] used by test bot in this configuration.
* TEST_LIST - [http://getfirebug.com/wiki/index.php/Running_Automated_Test_Suite#Test_List_File_Structure Test suite] used by test bot in this configuration.
 +
* DB_URL - URL of an online CouchDB where all test-results should be uploaded
 +
* DB_NAME=firebug2 - Name of the online database where to upload data.
== Browser Window Size ==
== Browser Window Size ==
Line 137: Line 257:
extensions.fbtest.defaultScreenY [default 0]
extensions.fbtest.defaultScreenY [default 0]
</pre>
</pre>
 +
 +
== Example Scenario ==
 +
 +
* As soon as you have ''Firebug-Test-Runner'' installed try to execute ''runFBTests --help'' on your command line. If you see a list of possible arguments it works well and you can proceed to the next step.
 +
 +
* Now you need to setup your web server. Let's call it ''legoas''.
 +
 +
* The server path is: <code>http://legoas/</code>
 +
 +
* So, the config file URL is: <code>http://legoas/releases/firebug/test-bot.config</code>
 +
 +
* The config file can look like as follows:
 +
 +
<pre>
 +
[MyFirebugConfiguration]
 +
FIREBUG_XPI=https://getfirebug.com/releases/firebug/1.11/firebug-1.11.0b2.xpi
 +
FBTEST_XPI=https://getfirebug.com/releases/fbtest/1.11/fbTest-1.11b4.xpi
 +
GECKO_VERSION=aurora,beta,central
 +
TEST_LIST=http://getfirebug.com/tests/head/firebug.html
 +
DB_URL=http://firebug.couchone.com/
 +
DB_NAME=firebug
 +
</pre>
 +
 +
* And finally, you just need to launch the test-bot and specify ''serverpath''.
 +
 +
<pre>runFBTests --serverpath http://legoas/</pre>
 +
 +
* The test bot automatically runs all tests and uploads all results the specified database.
 +
 +
* Open test-result viewer and specify your database:
 +
 +
<pre>https://getfirebug.com/testresults?dburi=http://firebug.couchone.com/&dbname=firebug</pre>
 +
 +
== Known Issues ==
 +
* runFBTests depends on an external python package for scraping the latest firefox builds. There is a bug in this package where it won't return the download url for Firefox 3.6 on Mac OS X 64-bit systems.
 +
 +
* This package also depends on lxml which in turn depends on the libxml2 and libxslt packages. You may also need the python-dev package.  In linux and mac this hasn't been a huge problem (just apt-get or port install the packages if it isn't done automatically) but building lxml has given me some headaches in windows (although maybe your development environment is set up better than mine). If setuptools can't compile lxml automatically, I'd recommend installing the pre-compiled binary of lxml found here: http://pypi.python.org/pypi/lxml/2.3beta1 (assuming you are using 32 bit Python) before running setuptools.
 +
 +
* Do not use GIT_TAG and GIT_BRANCH fields in your custom config (needs to be fixed).

Latest revision as of 12:54, 5 December 2012

Firebug test-bot support for scheduled running of Firebug test suite.

Contents

[edit] Links

[edit] Resources

[edit] Installing TestBot

[edit] Windows

  • You need Python virtual environment
  • In case it doesn't come within the Python package you also need Python setuptools
  • You might also need to install lxml (tested with lxml-2.3.6.win32-py2.7.‌exe). See also Known Issues on this page.
  • You might want to add following to your PATH: C:/Python27/;C:/Python27/Scripts/ (related to Python 2.7)
  • In case you don't wont to install from the source, you can just (assuming easy_install is in your path):
easy_install runFBTests

However, it is strongly recommended to install from source

  • Download Testbot source from GIT repository:
git clone http://github.com/mozilla/Firebug-Test-Runner.git
  • Install from the source:
cd Firebug-Test-Runner
python setup.py install
  • Using testbot:
runFBTests --help
  • Run testbot:
runFBTests --serverpath http://legoas/firebug
  • Update tests from the GIT:
git pull origin master
python setup.py install

[edit] Ubuntu

Python Virtual Environment:

Install pip and virtualenv for Ubuntu 10.10 Maverick and newer:

$ sudo apt-get install python-pip python-dev build-essential 
$ sudo pip install --upgrade pip 
$ sudo pip install --upgrade virtualenv 

Install virtualenvwrapper

sudo pip install virtualenvwrapper

Update config file for virtual environments:

gedit .bashrc

Put at the end of the file:

export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/Devel
source /usr/local/bin/virtualenvwrapper.sh

Reload the config file:

source ~/.bashrc

Create new virtual environment called ‘Firebug’:

mkvirtualenv firebug

Install Firebug testRunner:

sudo apt-get install python-setuptools
sudo apt-get install libxml2-dev libxslt-dev 
sudo apt-get install python-lxml

Install git:

sudo apt-get install git

Download Firebug Test Runner source:

git clone http://github.com/mozilla/Firebug-Test-Runner.git

Install Firebug Test Runner:

cd Firebug-Test-Runner
python setup.py install

Execute Test Runner:

/usr/bin$ python runFBTests --serverpath http://10.0.3.102/firebug

Do not forget to activate your python virtual environment the next time you start up your computer before using the test runner:

workon firebug
runFBTests --help

[edit] Mac OS

This section needs to be yet revisited

Install pip, virtualenv and virtualenvwrapper:

Install GIT

Install GCC

Install lxml

Run this to make the 'mkvirtualenv' command working:

  • source /usr/local/bin/virtualenvwrapper.sh

Run test-bot:

  • sudo runFBTests --debug --serverpath http://legoas/firebug

[edit] Command Line Arguments

  -h, --help            show this help message and exit
  -p PROFILE, --profile=PROFILE
                        The profile to use when running Firefox
  -b BINARY, --binary=BINARY
                        The binary path to use. If unspecified appropriate
                        binaries will be downloaded automatically
  -s SERVERPATH, --serverpath=SERVERPATH
                        The http server containing the firebug tests
  -c COUCHURI, --couch=COUCHURI
                        URI to couchdb server for log information
  -d DATABASENAME, --database=DATABASENAME
                        Database name to keep log information
  -t TESTLIST, --testlist=TESTLIST
                        Url to the testlist to use
  --interval=WAITTIME   Number of hours to wait between test runs. If
                        unspecified tests are only run once
  --debug               Enable debug logging

[edit] Custom Server With Tests

Testbot is using http://www.getfirebug.com/ by default so, make sure this is what you want, otherwise specify your own server:

runFBTests --serverpath http://www.myserver.org/

Serverpath also specifies where test-bot.config (the name of the config file is hardcoded) is located. The whole path is computed as follows:

serverpath + "releases/firebug/" + "test-bot.config"

So, make sure the config file is available on your custom server.

[edit] Custom Couch DB

If you want you can set up your own CouchDB and point testbot at that one.

runFBTests --couch http://mycouchdb.example.org/couchdb --database myfirebugdb

You can also specify the database in test-bot.config

DB_URL=http://firebug.couchone.com/
DB_NAME=firebug3

In order to preview your custom database, you can configure test result viewer through URL:

https://getfirebug.com/testresults?dburi=http://firebug.couchone.com/&dbname=firebug3

See how to create a new Firebug Test Bot Database

[edit] Custom Test List

You can also specify a test list on the command line:

$ runFBTests -c http://firebug.couchone.com/ -d firebug -s http://legoas/firebug -t http://legoas/firebug/firebug.html

Again, you can specify custom test-list in test-bot.config file

TEST_LIST=http://legoas/src/github.com/firebug/firebug/tests/content/firebug.html

[edit] Testbot Config

Automated running of tests is configurable using a config file. Testbot currently expects this file to be available here:

http://getfirebug.com/releases/firebug/test-bot.config

The file follows syntax of INI Files. Here is an example:

[Firebug1.10]
FIREBUG_XPI=https://getfirebug.com/releases/firebug/1.10/firebug-1.10.6.xpi
FBTEST_XPI=https://getfirebug.com/releases/fbtest/1.10/fbTest-1.10.0.xpi
GIT_TAG=HEAD
GIT_BRANCH=firebug1.10
GECKO_VERSION=aurora,beta

[Firebug1.11]
GIT_TAG=HEAD
GIT_BRANCH=master
GECKO_VERSION=aurora,beta,central
DB_URL=http://firebug.couchone.com/
DB_NAME=firebug2

[Firebug1.11-bug800799]
GIT_TAG=HEAD
GIT_BRANCH=master
GECKO_VERSION=aurora,beta,central
TEST_LIST=http://getfirebug.com/tests/head/bug800799.html

Each section (there are three in this example) specifies one configuration. Name of the section should be descriptive, but isn't currently used by testbot.

  • FIREBUG_XPI - URL of Firebug XPI file used in this configuration.
  • FBTEST_XPI - URL of FBTest XPI (Firebug test harness) file used in this configuration.
  • GIT_BRANCH - GIT branch in Firebug GIT repo that is used by testbot to pull all test files. Testbot puts these files on its own web server.
  • GIT_TAG - GIT revisiion
  • GECKO_VERSION - List of Firefox versions used for testing in the configuration
  • TEST_LIST - Test suite used by test bot in this configuration.
  • DB_URL - URL of an online CouchDB where all test-results should be uploaded
  • DB_NAME=firebug2 - Name of the online database where to upload data.

[edit] Browser Window Size

Some tests depends on the browser window (with Firebug) size. For example, there can be a test that right clicks on an element in Firebug HTML Panel and opens a context menu on it - in order to execute an action. In case where the HTML Panel is too small the element doesn't have to be visible and the click wouldn't open the menu. The test would fail, even if the feature actually works.

For this reason, FBTest harness sets the browser window size to 1024x768 (as the minimum required size) and moves the window to [0, 0] position on the screen. This is done only in cases where the -runFBTests flag is specified for firefox.exe. So, users running tests as part of the daily development process are not bothered by the browser resizing.

The actual size and location can be specified in preferences:

extensions.fbtest.defaultOuterWidth [default 1024]
extensions.fbtest.defaultOuterHeight [default 768]
extensions.fbtest.defaultScreenX [default 0]
extensions.fbtest.defaultScreenY [default 0]

[edit] Example Scenario

  • As soon as you have Firebug-Test-Runner installed try to execute runFBTests --help on your command line. If you see a list of possible arguments it works well and you can proceed to the next step.
  • Now you need to setup your web server. Let's call it legoas.
  • The config file can look like as follows:
[MyFirebugConfiguration]
FIREBUG_XPI=https://getfirebug.com/releases/firebug/1.11/firebug-1.11.0b2.xpi
FBTEST_XPI=https://getfirebug.com/releases/fbtest/1.11/fbTest-1.11b4.xpi
GECKO_VERSION=aurora,beta,central
TEST_LIST=http://getfirebug.com/tests/head/firebug.html
DB_URL=http://firebug.couchone.com/
DB_NAME=firebug
  • And finally, you just need to launch the test-bot and specify serverpath.
runFBTests --serverpath http://legoas/
  • The test bot automatically runs all tests and uploads all results the specified database.
  • Open test-result viewer and specify your database:
https://getfirebug.com/testresults?dburi=http://firebug.couchone.com/&dbname=firebug

[edit] Known Issues

  • runFBTests depends on an external python package for scraping the latest firefox builds. There is a bug in this package where it won't return the download url for Firefox 3.6 on Mac OS X 64-bit systems.
  • This package also depends on lxml which in turn depends on the libxml2 and libxslt packages. You may also need the python-dev package. In linux and mac this hasn't been a huge problem (just apt-get or port install the packages if it isn't done automatically) but building lxml has given me some headaches in windows (although maybe your development environment is set up better than mine). If setuptools can't compile lxml automatically, I'd recommend installing the pre-compiled binary of lxml found here: http://pypi.python.org/pypi/lxml/2.3beta1 (assuming you are using 32 bit Python) before running setuptools.
  • Do not use GIT_TAG and GIT_BRANCH fields in your custom config (needs to be fixed).
Personal tools