Firebug User Survey

From FirebugWiki

Revision as of 13:48, 28 January 2010 by Honza (Talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Contents

Motivation

Having stronger sense of which features Firebug users actually use, which could help Firebug developers and contributors to better direct the development and design efforts.


Goal

The main thing of a user study is to be clear about what specific question should be actually answered. The critical thing and whole point of the study is to get meaningful data that can be analyzed and results helps to improve Firebug itself. The recommendation from Test Pilot team is as follow:

The main thing is that we strongly recommend you come up with specific questions you want to answer. You can collect a huge amount of metrics, but that may not be as helpful as a more targeted approach. Think of the scientific method: you have a testable hypothesis in mind, "We think that more people use feature A than feature B" and then you design an experiment that attempts to disprove your hypothesis.

An example question in case of a Firefox study is: "How many tabs Firefox users actually open?"

Another reason why to start with simple question is to perform the first study quickly, analyze data and start to seeing some results. This helps to make sure it's effecting and working way how to improve Firebug, before investing a lot of development time.


Risks

The main risk here is obvious. In case of vague collected data, or missing contexts, there can be less chance to get an effective and helpful results. To minimalize this risk, it should be discussed what is expected from a study.


Test Pilot

There is already an existing platform aimed at collecting structured user feedback for various Mozilla labs experiments. So, the natural step is to consider how this existing service could be utilized in case of Firebug user studies and how Test Pilot tool can help.

From a preceding discussion, following seem to be a promising approach.

  • Firebug implements its own data collection mechanism. This way, the study is not limited to users who have the Test Pilot extension installed.
  • Firebug utilizes an existing Test Pilot infrastructure. In particular, servers for data collection and perhaps also help with data analysis.


Technical Details

Test Pilot uploads gathered data from the client to the data collection server by making an HTTP POST via an XMLHttpRequest. For this to work, the study needs to have a unique test ID number that is consequently used as a parameter of the request to identify particular study.

To summarize, two parameter are required for each such request.

  • testid Unique ID of a test study
  • data Arbitrary text string

Test pilot tools are optimized for processing CVS files and so, it could be effective for Firebug to use the same format. However, e.g. JSON format can be also considered.


Here is an example of an upload of collected data to the server:

// Upload URL and testID is defined here
const dataUploadURL = ...
const testID = ...

// Get collected data (CSV, JSON, ...)
var dataString = getCollectedData();
var params = "testid=" + testID + "&data=" + encodeURI(dataString);

// Construct and setup and send XHR
var request = Cc["@mozilla.org/xmlextras/xmlhttprequest;1"].
    createInstance(Ci.nsIXMLHttpRequest);
request.open("POST", dataUploadURL, true);
req.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
req.setRequestHeader("Content-length", params.length);
req.setRequestHeader("Connection", "close");
req.onreadystatechange = function(event)
{
     if (request.readyState == 4)
     {
         if (req.status == 200)
            // Success: Update the experiment state, notify the UI
            // stop collecting data, and delete local copy of data
         else
            // Errors: set up a timer here that will retry the submit
            // one hour later if it fails this time.
     }
};
request.send(params);


Instrumenting Firebug

The most promising way how to gather a statistical data about real feature usage within Firebug, is to implement a Firebug/Firefox extension. Requirements for such extension are following:

  • Minimal UI overhead, the existing Firebug UX should not be affected.
  • Automated upload of data with first time user verification.
  • It should be possible to preview uploaded data by the user (Perhaps by logging them into the Console panel on demand).


Related Sources

Personal tools