Firebug User Survey

From FirebugWiki

Revision as of 16:30, 29 January 2010 by Honza (Talk | contribs)
Jump to: navigation, search

Contents

Motivation

Having stronger sense of which features Firebug users actually use, which could help Firebug developers and contributors to better direct the development and design efforts.


Goal

The main thing of a (automated) user study is to be clear about what specific question should be actually answered. The critical thing and whole point of the study is to get meaningful data that can be analyzed and results helps to improve Firebug itself. The recommendation from Test Pilot team is as follow:

The main thing is that we strongly recommend you come up with specific questions you want to answer. You can collect a huge amount of metrics, but that may not be as helpful as a more targeted approach. Think of the scientific method: you have a testable hypothesis in mind, "We think that more people use feature A than feature B" and then you design an experiment that attempts to disprove your hypothesis.

An example question in case of a Firefox study can be: How many tabs have Firefox users usually opened at the same time?

Another reason why to start with simple question is to perform the first study quickly, analyze data and start to seeing some results. This helps to make sure it's effecting and working way how to improve Firebug, before investing a lot of development time.


Possible questions we can try to answer for Firebug could be as follows (subject for discussion)

  • What options are users actually changing?
  • What command line APIs users use the most?
  • What Firebug panels users use the most?
  • How many people use Firebug detached?


Of course, the other option is to perform a poll (manual questionnaire).


Analysis

As soon as a collected data give us a clear answer for a specific study, the other question is how this helps to improve Firebug? This is the hardest and final part of a study.


Here are some example analysis and possible impacts on Firebug UX (using the same studies/questions from the previous paragraph).

  • What options are users actually changing?
What if the result says that less than 1% of users never change the 'Show Preview Tooltips' option. Couldn't this be a hidden option accessible through about:config only then? The less options in the UI the better.
  • What command line APIs users use the most?
What if group of users using heavily the command line, almost never use the 'debug()' function? Does this mean that it's not documented properly and they don't know it exists or how to use it?
  • What Firebug panels users use the most?
What if less than 3% users use the CSS panel? Couldn't we hide it by default?
  • How many people use Firebug detached?
If more than 40% of users use Firebug UI detached, doesn't this mean that we need to put more efforts to testing this use case?


Risks

The main risk here is obvious. In case of vague collected data, or missing contexts, there can be less chance to get an effective and helpful results. To minimize this risk, it should be discussed what is expected from a study and what impact it could have on Firebug.


Test Pilot

There is already an existing platform aimed at collecting structured user feedback for various Mozilla labs experiments. So, the natural step is to consider how this existing service could be utilized in case of Firebug user studies and how Test Pilot tool can help.


From a preceding discussion, following seem to be the best approach.

  • Firebug implements its own data collection mechanism. This way, the study is not limited to users who have the Test Pilot extension installed.
  • Firebug utilizes an existing Test Pilot infrastructure. In particular, servers for data collection and perhaps also help with data analysis.


Technical Details

Test Pilot uploads gathered data from the client to the data collection server by making an HTTP POST via an XMLHttpRequest. For this to work, the study needs to have a unique test ID number that is consequently used as a parameter of the request to identify particular study.

To summarize, two parameter are required for each such request.

  • testid Unique ID of a test study
  • data Arbitrary text string


Test pilot tools are optimized for processing CVS files and so, it could be effective for Firebug to use the same format. However, e.g. JSON format can be also considered.


Here is an example of data upload to the server:

// Upload URL and testID is defined here
const dataUploadURL = ...
const testID = ...

// Get collected data (CSV, JSON, ...)
var dataString = getCollectedData();
var params = "testid=" + testID + "&data=" + encodeURI(dataString);

// Construct and setup and send XHR
var request = Cc["@mozilla.org/xmlextras/xmlhttprequest;1"].
    createInstance(Ci.nsIXMLHttpRequest);
request.open("POST", dataUploadURL, true);
req.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
req.setRequestHeader("Content-length", params.length);
req.setRequestHeader("Connection", "close");
req.onreadystatechange = function(event)
{
     if (request.readyState == 4)
     {
         if (req.status == 200)
            // Success: Update the experiment state, notify the UI
            // stop collecting data, and delete local copy of data
         else
            // Errors: set up a timer here that will retry the submit
            // one hour later if it fails this time.
     }
};
request.send(params);


Instrumenting Firebug

The most promising way how to gather a statistical data about real feature usage within Firebug, is to implement a Firebug/Firefox extension. Requirements for such extension are following:

  • Minimal UI overhead, the existing Firebug UX should not be affected.
  • Automated upload of data with first time user verification.
  • It should be possible to preview uploaded data by the user (Perhaps by logging them into the Console panel on demand).


Related Sources

Personal tools