Firebug Test Bot Database

From FirebugWiki

Jump to: navigation, search

Firebug Test Bot is using CouchDB for storing all test results.

Contents

[edit] Resources

[edit] Test Results DB

Test results produced by running Firebug test suite are stored into a CouchDB (document-oriented database). The database is available online and all test results (stored as documents) can be accessed using XHR.

Testbot results are stored as documents and there are currently following types of docs.

  • header this document represents a single test-bot run.
  • result represents a test result (one Firebug test executed). The relation between header and result documents is one-to-many. There are many results for one test-bot run (the number of results corresponds to number of tests in the used test list). The relation is made through a headerid filed that is part of each result doc.


The database contains also test-results reported by users who run automated tests manually using Firebug Test Console (FBTest). These results use following doc types.

  • user-header represents a user test launch.
  • user-results represents a user test result.

[edit] Test Results UI

Summary of testbot results is available online. This page displays both (a) testbot and (b) user results as two separate lists. Source code of this application is available.

Custom CouchDB can be passed through URL as follows:

https://getfirebug.com/testresults?dburi=http://firebug.couchone.com/&dbname=firebug2

[edit] Setup New DB

As soon as CouchDB is installed you can create databases. Firebug test-results database uses specific schema that the defined in design-documents. These documents are JSON based, just like any other doc in the database.

Database schema (design documents) are described here.

[edit] Instructions

Following instructions are using http://firebug.couchone.com/ as an existing CouchDB installation.

[edit] Create New Database

Open http://firebug.couchone.com/_utils/index.html and click Create Database and provide a name. This scenario is using Template, but you need to provide your own name of course.

[edit] Open The Database

The new database should be available here: http://firebug.couchone.com/_utils/database.html?template

[edit] Create New Temporary View

Pick Temporary view... option from the View Field.

New-test-results-db1.png

[edit] Create New Design Document

Click Save As... button at the bottom and provide allfailures for the Design Document and test for the view. The View name is not important now, it'll be changed later as we copy the template JSON document.

New-test-results-db2.png

[edit] Open Design Document

Click the list of Views again (you should see a new entry: allfailures). Pick the Design Documents option to open list of all design documents (we have one at this moment).

New-test-results-db3.png

[edit] Edit Design Document

Click on the only displayed entry (there is one design document) and consequently double-click on the source-content (JSON) to start editing.

New-test-results-db4.png

[edit] Define 'allfailures' Views

The selected part on the previous screenshot needs to be replaced by the following definition:

"views": {
       "allfailures": {
           "map": "function(doc)\n{\n if (doc.type == \"result\" && doc.result == \"TEST-UNEXPECTED-FAIL\")\n emit(doc.file, doc);\n}",
           "reduce": "function(keys, values, rereduce) {\n var output = {\n count: 0,\n file: null,\n };\n\n if (rereduce)\n {\n for (idx in values)\n {\n var doc = values[idx];\n if (doc.count !== undefined)\n output.count += doc.count;\n }\n }\n else\n {\n for (idx in values)\n {\n var doc = values[idx];\n output.count += 1;\n output.file = doc.file;\n }\n }\n return output;\n}"
       }
   }

Click Save Document at the top to save the changes.

[edit] Define 'resultviews' Views

Now go to the list of Design Documents by clicking on the Template (name of your database) at the top. Create a new Temporary View and save it.

  • Design Document: resultviews
  • View Name: test (again, will be redefined)

Open _design/resultviews Design Document and replace the Views part by:

   "views": {
       "allresults": {
           "map": "function(doc)\n{\n  if (doc.type == 'result')\n     emit([doc.headerid, doc._id], doc);\n}"
       },
       "allheaders": {
           "map": "function(doc)\n{\n  if (doc.type == 'header')\n     emit(doc._id, doc);\n}"
       },
       "results_by_header": {
           "map": "function(doc)\n{\n  if ((doc.type == 'result' || doc.type == 'user-result') &&\n    doc.result == 'TEST-UNEXPECTED-FAIL')\n  {\n      emit(doc.headerid, doc);\n  }\n}"
       },
       "headers": {
           "map": "function(doc) {\n  var date = (new Date(doc['Export Date'])).getTime();\n  if (doc.type == 'result') {\n    emit(date, doc);\n  } else if (doc.type == 'header') {\n    emit(date, doc);\n  }\n}",
           "reduce": "function(keys, values, rereduce) {\n  var output = {\n    failures: 0,\n    knownFailures: 0,\n    passes: 0,\n    testCount: 0,\n    doc: 0\n  };\n  if (rereduce) {\n    for (idx in values) {\n      var doc = values[idx];\n      if (doc.failures !== undefined)\n        output.failures += doc.failures;\n      if (doc.knownFailures !== undefined)\n        output.knownFailures += doc.knownFailures;\n      if (doc.passes !== undefined)\n        output.passes += doc.passes;\n      if (doc.testCount !== undefined)\n        output.testCount += doc.testCount;\n\n      if (typeof(doc.doc) != \"undefined\")\n        output.doc = doc.doc;\n    }\n  } else {\n    for (idx in values) {\n      var doc = values[idx];\n      if (doc.type == 'header') {\n        output.doc = {\n            '_id': doc._id,\n            'Export Date': doc['Export Date'],\n            'Total Tests': doc['Total Tests'],\n            'Firebug': doc.Firebug,\n            'App Name': doc['App Name'],\n            'Section Title': doc['Section Title'],\n            'App Version': doc['App Version'],\n            'OS Name': doc['OS Name'] + \" \" + doc['OS Detailed Name'],\n        };\n      }\n      else if (doc.type == 'result') {\n        output.testCount += 1;\n\n\tif (doc.result == 'TEST-UNEXPECTED-FAIL')\n          output.failures += 1;\n        else if (doc.result == 'TEST-KNOWN-FAIL')\n          output.knownFailures += 1;\n        else if (doc.result == 'TEST-PASS')\n          output.passes += 1;\n      }\n    }\n  }\n  return output;\n}"
       },
       "header": {
           "map": "function(doc) {\n  if (doc.type == 'header')\n    emit(doc._id, doc);\n  else if (doc.type == 'result')\n    emit(doc.headerid, doc);  \n}\n"
       },
       "user-header": {
           "map": "function(doc) {\n  if (doc.type == 'user-header')\n    emit(doc._id, doc);\n  else if (doc.type == 'user-result')\n    emit(doc.headerid, doc);  \n}\n"
       },
       "user-headers": {
           "map": "function(doc) {\n  var date = (new Date(doc['Export Date'])).getTime();\n  if (doc.type == 'user-result') {\n    if (doc.result == 'TEST-UNEXPECTED-FAIL')\n      emit(date, doc);\n  } else if (doc.type == 'user-header') {\n    emit(date, doc);\n  }\n}",
           "reduce": "function(keys, values, rereduce) {\n  var output = {failures: 0, doc: 0};\n  if (rereduce) {\n    for (idx in values) {\n      if (values[idx].failures !== undefined)\n        output.failures += values[idx].failures;\n      if (values[idx].doc !== undefined)\n        output.doc = values[idx].doc;\n    }\n  } else {\n    for (idx in values) {\n      if (values[idx].type == 'user-header') {\n        var doc = values[idx];\n        output.doc = {\n            '_id': doc._id,\n            'Export Date': doc['Export Date'],\n            'Total Tests': doc['Total Tests'],\n            'Firebug': doc.Firebug,\n            'App Name': doc['App Name'],\n            'App Version': doc['App Version'],\n            'OS Name': doc['OS Name'] + \" \" + doc['OS Detailed Name'],\n        };\n      }\n      else if (values[idx].type == 'user-result') {\n        output.failures += 1;\n      }\n    }\n  }\n  return output;\n}"
       },
       "user-results_by_header": {
           "map": "function(doc) {\n  if (doc.type == 'user-result' &&\n      doc.result == 'TEST-UNEXPECTED-FAIL')\n    emit(doc.headerid, doc);\n}"
       },
       "allcrashes": {
           "map": "function(doc) {\n  if (doc.type == 'crash')\n     emit(doc._id, doc);\n}"
       },
       "allfailures": {
           "map": "function(doc)\n{\n  var date = (new Date(doc['Export Date'])).getTime();\n\n  if ((doc.type == \"result\" || doc.type == \"user-result\") &&\n    doc.result == \"TEST-UNEXPECTED-FAIL\")\n  {\n    emit(date, doc.file);\n  }\n}"
       }
   }

Now replace the lists part by:

   "lists": {
       "json": "function(head, req) { start({'headers': {'Access-Control-Allow-Origin': '*', 'Content-Type' : 'application/json'}}); send('['); var row; var firstRow = true; while(row = getRow()) { if (!firstRow) send(','); send(JSON.stringify(row)); firstRow = false; } send(']'); }"
   },

[edit] Referencing Custom DB

The database is ready to use now and you can setup the test-bot to send all the test-results data to it:

Example of test-bot.config

DB_URL=http://firebug.couchone.com/
DB_NAME=template

Example of command line arguments:

runFBTests --couch http://mycouchdb.example.org/couchdb --database template

Example of the Viewer URL:

https://getfirebug.com/testresults?dburi=http://firebug.couchone.com/&dbname=template

dburi=http://firebug.couchone.com/ is used by default


See the template database online here: http://firebug.couchone.com/_utils/database.html?template/_design_docs

Personal tools