Letter to my successors (or what do I know about the Xorg bug triage)
#####################################################################

:date: 2011-11-22T15:28:36
:category: computer
:tags: bugTriage, firefox, jetpack

For various reasons we have agreed with my supervisor, that my working
on the Xorg bug triage as a full time job will be terminated (feels like
a relief) and I will switch to working as a programmer for the Red Hat
internal Desktop QA team (which is a good thing!). As the last act in my
old position I was asked to write down brain dump of my experience
working on the Xorg bug triage for my potential successors. I am trying
also to pay some debts I feel I have in documenting some parts of my bug
triage workflow and tools.

First of all, it is not like there isn't any information available
already. We as a BugZappers team tried to collect as much information as
possible so anybody who would like to join our ranks would have
something to chew upon without us repeating it again and again. So,
there is `How to Triage`_ page and other pages coming from `this
navigation page`_ (e.g., you are expected to know `bug workflow`_ by
heart if drawn in the midnight from your bad).

Queries
-------

Probably the most important thing which can help you to survive a terror
of the never-ending inflow of new bugs are well prepared queries. Red
Hat Bugzilla (or any other bugzilla for that matter, as far as I know),
doesn't have a really good way to create coarser (or finer) organization
of bugs than on the level of components and products. The situation is
especially bad, because products are occupied by, well, products (like
Fedora, RHEL, etc.), so in fact the only classification of bugs we have
is on the basis of components. Which is obviously crazy huge group of
bugs for some components (kernel) and too little for many others. Other
options (using keywords in whiteboards, for example) requires much
manual additional work (although we use it as well, see below for Xorg
bugs categories).

The only way I found how to make dissecting bugs in useful ways possible
is by using queries. It is normal state of things for me to have more
than 50 defined queries in the bottom of my bugzilla pages. As an
example (and possibly foundation of your further development) I could
present you `this search`_. Notice that it selects just some components
of bugs (using a regular expression), eliminates bugs with needinfo flag
set (which is thus a very important variable to care for), and
eliminates already Triaged bugs. Of course, this is just a basic search
which can be further narrowed by selecting only bugs in some states
(although bug triagers deal mostly with NEW bugs), bugs changed in the
recent time (when the time is short, it is probably better to deal with
the active bugs, not stale ones), bugs for some versions of products,
etc.

Firefox addon
-------------

Although there has been concerned spent lately on making bugzilla work
better for developers, it is still true that its user interface is (if
for anybody) optimized for users and so there is little help for
avoiding repetitive work. In order to make my work with bugzilla more
palatable, I have created a Firefox addon which provides tools for
mechanizing a lot of stereotypic work. For example, reporters in the Red
Hat bugzilla have no meaningful guidance when filing Xorg bugs, so many
filed bugs are lacking logs and other information which is required for
developers to effectively diagnose the issue. Actually, making sure that
all this information is provided is the main task of Xorg bug triage
(where normal other tasks of bug triage, e.g., deduplication and testing
reproducers is hardly possible or completely impossible given the limits
on available hardware and knowledge of the code itself).

Back to the firefox addon. The main functionality has been described on
this blog, so just in the way of summary. The main point which you need
to understand is that almost all functionality can be controlled and
substantially changed by editing a JSON file which is loaded on the
startup of the addon. Per default it is `this JSON`_, but clicking on
“Triage configuration” link you can enter URL of any JSON file anywhere
on the Internet. All possible options are switched on in `my JSON`_,
which can be used as an example for developing of your own file. Let me
go through the main options available in such file.

Most famous and probably still most used are buttons filling
standardized comments. That's the biggest parts of JSON file in the
object (using Javascript terminology) commentPackages. The name evokes
that there are multiple packages which can be switched on for particular
bugzillas (one bug triager can certainly work on multiple bugzilla
instances ... see configData.enabledPackages for examples how to switch
on some packages for particular bugzillas). Each packages contains
multiple objects one for each button which looks like this::

   "addnoresponse": {
              "name": "NoRespns",
              "position": "topRow",
              "commentIdx": "noResponseString",
              "markTriaged": true,
              "status": "CLOSED",
              "resolution": "INSUFFICIENT_DATA"
         },

Each object has its own one-word name, which serves as an index and id
for the button itself. Each object is best understood as a package of
commands which will be executed when the button is pressed. Some of
these “commands” actually don’t are not commands but metadata ("name"
contains the label on the button, "position" shows to the addon where
the button should be located), but most of them actually do something.
So in this example, “commentIdx” points to the comment string in
“commentString” section of the JSON file. For shorter comment texts,
button package can contain directly “comment” property with the text
itself. Either of these texts can be included in the comment.
“markTriaged” will obviously make all necessary steps to make bug
triaged, and the remaining two commands will close the bug. “position”
deserves a little bit special treatment. It can be either “topRow” or
“bottomRow” which means that the button will be inserted into the row of
buttons above or below the comment box respectively. “position” property
can also contain the ID of the element after or before which the button
should be included (such string starts with “+” or “-” before the ID
string). So for example, “addexternalbugid” package contains
``"position": "+external_bugs_headline",`` which means that the button
should follow HTML element with “external\_bugs\_headline” ID.

After “commentPackages” and “commentStrings” sections, the next one is
“configData” which is a collection of various configuration options. I
have in my JSON file almost all of possible ones on. I will return to
this configuration section in a minute, but now let me just the mention
the last section “constantData” which contains various constants for
Xorg bug triagers, and which you should probably just take for your
JSON.

Back to the “configData” section. The biggest object (at least in my
file) is “killNodes” which is an object which for each bugzilla instance
where the triager is supposed to work lists IDs of all elements you
would like to remove from the page (some people especially from Mozilla
complained that there is just too much stuff on every page, so that for
some people the bugzilla is almost useless, but I had to admit I used it
well even with bugzilla.redhat.com). “downloadJSON” contains URLs for
additional data files which can be loaded for data which are just too
big to be included into “constantData” section (take a look at `that
file`_). “suspiciousComponents” is for situations when reporters
continuously file bugs into components which are there just for legacy
or organizational reasons (``xorg-x11`` and ``xorg-x11-drivers``). The
last is set of couple of Booleans, which directly allow some features of
the addon, which are usually not useful for “normal” bug triagers, but
were useful for me:

parseAbrtBacktraces
    This is mostly legacy option, for large piece of functionality,
    which pulled just a important part of backtrace from the attached
    abrt-produced log.

submitsLogging
    This is a switch for a functionality which is completely useless for
    anybody who is not hired for being a bug triager. This addons could
    ask you for every click on the Submit button to provide short
    message, which could be then collated into something which goes to
    my weekly status report. Obviously it should be set to ``false`` for
    almost everybody.

verboseInlineHistory
    This is a switch for unfinished attempt to merge in functionality
    from `Bugzilla Tweaks`_ `functionality of which`_ was `later
    included into the upstream bugzilla itself`_.

XorgLogAnalysis
    This is a tool which could be very useful for any Xorg bug triager.
    However, it is kind of dangerous, because if misused it could lead
    to quite sloppy work. What it does is that it adds to every
    Xorg.0.log (and dmesg output) a button which makes the addon to run
    the log through regular expressions which selects lines which can be
    interesting.  Obviously it is useful for fast overview of what's
    going on, but certainly detailed inspection of the log is required
    for any decision.

So, this is finally written down documentation for the mythical JSON
file, which I promised for so long.

A bit on process of treating a bug
----------------------------------

I will skip technical issues (they are described reasonably well on the
above mentioned BugZappers pages), but let me add here couple of notes
on the background of the process. First of all, although bug triage is
kind of lonely job, we stand on shoulders of some giants (or at least we
could if we knew about them). Luis Villa (former bugmaster of Ximian and
Gnome project) has `a nice collection of his presentations and talks`_.
Highly recommended. Also a lot of discussion about bugs management
(obviously) happens in the Bugzilla community, for which `Bugzilla
Planet`_ is a great entry point (or you can go all the way to `Mozilla
Planet`_)

(... to be continued ...)

.. _`How to Triage`: https://fedoraproject.org/wiki/BugZappers/How_to_Triage
.. _`this navigation page`: https://fedoraproject.org/wiki/BugZappers
.. _`bug workflow`: https://fedoraproject.org/wiki/BugZappers/BugStatusWorkFlow
.. _`this search`: https://bugzilla.redhat.com/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=all%20NEW&sharer_id=74116
.. _`this JSON`: https://fedorahosted.org/released/bugzilla-triage-scripts/Config_data.json
.. _`my JSON`: http://matej.ceplovi.cz/progs/data/RH_Data-packages.json
.. _`that file`: http://matej.ceplovi.cz/progs/data/chipNames.json
.. _`Bugzilla Tweaks`: http://ehsanakhgari.org/blog/2010-05-31/my-experience-jetpack-sdk
.. _`functionality of which`: http://ehsanakhgari.org/blog/2010-01-07/bugzilla-tweaks-enhanced
.. _`later included into the upstream bugzilla itself`: http://globau.wordpress.com/2011/08/23/bugzilla-inline-history-extension/
.. _`a nice collection of his presentations and talks`: http://tieguy.org/talks/
.. _`Bugzilla Planet`: http://planet.bugzilla.org/
.. _`Mozilla Planet`: http://planet.mozilla.org