Merged be.restructure, major internal reorganization.

Added a bunch of classes to make the commands, user interfaces, and
storage backends more abstract and distinct.  This should make it much
easier to extend and maintain BE.

Features:
 * Directory restructured:
   becommands/ -> libbe/commands
   submods sorted by functionality.
 * Lots of new classes:
   Option, Argument, Command
   InputOutput, StorageCallbacks, UserInterface
   Storage
 * Consolidated ID handling in libbe.util.id
 * Transitioned VCS backends for Python-based VCSs from subprocess
   calss to internal python calls.

Plus the user-visible changes:
 * New bugdir/bug/comment ID format replaces old bug:comment format.
 * Deprecated support for `be diff` on Arch and Darcs <= 2.3.1.  A new
   backend abstraction (Storage) makes the former implementation
   ungainly.
 * Improved command completion.
 * Removed commands close, open, email_bugs, 
 * Flipped some arguments
   `be assign BUG-ID [ASSIGNEE]` -> `be status ASSIGNED BUG-ID ...`
   `be severity BUG-ID SEVERITY` -> `be severity SEVERITY BUG-ID ...`
   `be status BUG-ID STATUS` -> `be status STATUS BUG-ID ...`

In the merge:
  * Added 'commit' to list of pagerless commands.
  * Updated doc/README.dev

See
#bea86499-824e-4e77-b085-2d581fa9ccab/1100c966-9671-4bc6-8b68-6d408a910da1#
for a discussion of why the changes were made and some of the
difficulties en-route.

3 files changed, 421 insertions, 0 deletions

diff --git a/misc/xml/be-mbox-to-xml b/misc/xml/be-mbox-to-xml
new file mode 100755
index 0000000..eda6d6e
--- /dev/null
+++ b/misc/xml/be-mbox-to-xml
@@ -0,0 +1,154 @@
+#!/usr/bin/env python
+# Copyright (C) 2009 W. Trevor King <wking@drexel.edu>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+"""
+Convert an mbox into xml suitable for input into be.
+  $ be-mbox-to-xml file.mbox | be import-xml -c <ID> -
+mbox is a flat-file format, consisting of a series of messages.
+Messages begin with a a From_ line, followed by RFC 822 email,
+followed by a blank line.
+"""
+
+import base64
+import email.utils
+from libbe.encoding import get_encoding, set_IO_stream_encodings
+from libbe.utility import time_to_str
+from mailbox import mbox, Message  # the mailbox people really want an on-disk copy
+from time import asctime, gmtime, mktime
+import types
+from xml.sax.saxutils import escape
+
+DEFAULT_ENCODING = get_encoding()
+set_IO_stream_encodings(DEFAULT_ENCODING)
+
+KNOWN_IDS = []
+
+def normalize_email_address(address):
+    """
+    Standardize whitespace, etc.
+    """
+    addr = email.utils.formataddr(email.utils.parseaddr(address))
+    if len(addr) == 0:
+        return None
+    return addr
+
+def normalize_RFC_2822_date(date):
+    """
+    Some email clients write non-RFC 2822-compliant date tags like:
+      Fri, 18 Sep 2009 08:49:02 -0400 (EDT)
+    with the non-standard (EDT) timezone name.  This funtion attempts
+    to deal with such inconsistencies.
+    """
+    time_tuple = email.utils.parsedate(date)
+    assert time_tuple != None, \
+        'unparsable date: "%s"' % date
+    return time_to_str(mktime(time_tuple))
+
+def comment_message_to_xml(message, fields=None):
+    if fields == None:
+        fields = {}
+    new_fields = {}
+    new_fields[u'alt-id'] = message[u'message-id']
+    new_fields[u'in-reply-to'] = message[u'in-reply-to']
+    new_fields[u'author'] = normalize_email_address(message[u'from'])
+    new_fields[u'date'] = message[u'date']
+    if new_fields[u'date'] != None:
+        new_fields[u'date'] = normalize_RFC_2822_date(new_fields[u'date'])
+    new_fields[u'content-type'] = message.get_content_type()
+    for k,v in new_fields.items():
+        if v != None and type(v) != types.UnicodeType:
+            fields[k] = unicode(v, encoding=DEFAULT_ENCODING)
+        elif v == None and k in fields:
+            new_fields[k] = fields[k]
+    for k,v in fields.items():
+        if k not in new_fields:
+            new_fields.k = fields[k]
+    fields = new_fields
+
+    if fields[u'in-reply-to'] == None:
+        if message[u'references'] != None:
+            refs = message[u'references'].split()
+            for ref in refs: # search for a known reference id.
+                if ref in KNOWN_IDS:
+                    fields[u'in-reply-to'] = ref
+                    break
+            if fields[u'in-reply-to'] == None and len(refs) > 0:
+                fields[u'in-reply-to'] = refs[0] # default to the first
+    else: # check for mutliple in-reply-to references.
+        refs = fields[u'in-reply-to'].split()
+        found_ref = False
+        for ref in refs: # search for a known reference id.
+            if ref in KNOWN_IDS:
+                fields[u'in-reply-to'] = ref
+                found_ref = True
+                break
+        if found_ref == False and len(refs) > 0:
+            fields[u'in-reply-to'] = refs[0] # default to the first
+
+    if fields[u'alt-id'] != None:
+        KNOWN_IDS.append(fields[u'alt-id'])
+
+    if message.is_multipart():
+        ret = []
+        alt_id = fields[u'alt-id']
+        from_str = fields[u'author']
+        date = fields[u'date']
+        for m in message.walk():
+            if m == message:
+                continue
+            fields[u'author'] = from_str
+            fields[u'date'] = date
+            if len(ret) > 0: # we've added one part already
+                fields.pop(u'alt-id') # don't pass alt-id to other parts
+                fields[u'in-reply-to'] = alt_id # others respond to first
+            ret.append(comment_message_to_xml(m, fields))
+            return u'\n'.join(ret)
+
+    charset = message.get_content_charset(DEFAULT_ENCODING).lower()
+    #assert charset == DEFAULT_ENCODING.lower(), \
+    #    u"Unknown charset: %s" % charset
+
+    if message[u'content-transfer-encoding'] == None:
+        encoding = DEFAULT_ENCODING
+    else:
+        encoding = message[u'content-transfer-encoding'].lower()
+    body = message.get_payload(decode=True) # attempt to decode
+    assert body != None, "Unable to decode?"
+    if fields[u'content-type'].startswith(u"text/"):
+        body = unicode(body, encoding=charset).rstrip(u'\n')
+    else:
+        body = base64.encode(body)
+    fields[u'body'] = body
+    lines = [u"<comment>"]
+    for tag,body in fields.items():
+        if body != None:
+            ebody = escape(body)
+            lines.append(u"  <%s>%s</%s>" % (tag, ebody, tag))
+    lines.append(u"</comment>")
+    return u'\n'.join(lines)
+
+def main(mbox_filename):
+    mb = mbox(mbox_filename)
+    print u'<?xml version="1.0" encoding="%s" ?>' % DEFAULT_ENCODING
+    print u"<be-xml>"
+    for message in mb:
+        print comment_message_to_xml(message)
+    print u"</be-xml>"
+
+
+if __name__ == "__main__":
+    import sys
+    main(sys.argv[1])
diff --git a/misc/xml/be-xml-to-mbox b/misc/xml/be-xml-to-mbox
new file mode 100755
index 0000000..ef7b714
--- /dev/null
+++ b/misc/xml/be-xml-to-mbox
@@ -0,0 +1,208 @@
+#!/usr/bin/env python
+# Copyright (C) 2009 W. Trevor King <wking@drexel.edu>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+"""
+Convert xml output of `be list --xml` into mbox format for browsing
+with a mail reader.  For example
+  $ be list --xml --status=all | be-xml-to-mbox | catmutt
+
+mbox is a flat-file format, consisting of a series of messages.
+Messages begin with a a From_ line, followed by RFC 822 email,
+followed by a blank line.
+"""
+
+#from mailbox import mbox, Message  # the mailbox people really want an on-disk copy
+import codecs
+import email.utils
+from libbe.encoding import get_encoding, set_IO_stream_encodings
+from libbe.utility import str_to_time as rfc2822_to_gmtime_integer
+from time import asctime, gmtime
+import types
+try: # import core module, Python >= 2.5
+    from xml.etree import ElementTree
+except ImportError: # look for non-core module
+    from elementtree import ElementTree
+from xml.sax.saxutils import unescape
+
+
+DEFAULT_DOMAIN = "invalid.com"
+DEFAULT_EMAIL = "dummy@" + DEFAULT_DOMAIN
+DEFAULT_ENCODING = get_encoding()
+set_IO_stream_encodings(DEFAULT_ENCODING)
+
+def rfc2822_to_asctime(rfc2822_string):
+    """Convert an RFC 2822-fomatted string into a asctime string.
+    >>> rfc2822_to_asctime("Thu, 01 Jan 1970 00:00:00 +0000")
+    "Thu Jan 01 00:00:00 1970"
+    """
+    if rfc2822_string == "":
+        return asctime(gmtime(0))
+    return asctime(gmtime(rfc2822_to_gmtime_integer(rfc2822_string)))
+
+class LimitedAttrDict (dict):
+    """
+    Dict with error checking, to avoid invalid bug/comment fields.
+    """
+    _attrs = [] # override with list of valid attribute names
+    def __init__(self, **kwargs):
+        dict.__init__(self)
+        for key,value in kwargs.items():
+            self[key] = value
+    def __setitem__(self, key, item):
+        self._validate_key(key)
+        dict.__setitem__(self, key, item)
+    def _validate_key(self, key):
+        if key in self._attrs:
+            return
+        elif type(key) not in types.StringTypes:
+            raise TypeError, "Invalid attribute type %s for '%s'" % (type(key), key)
+        else:
+            raise ValueError, "Invalid attribute name '%s'" % key
+
+class Bug (LimitedAttrDict):
+    _attrs = [u"uuid",
+              u"short-name",
+              u"severity",
+              u"status",
+              u"assigned",
+              u"reporter",
+              u"creator",
+              u"created",
+              u"summary",
+              u"comments",
+              u"extra-strings"]
+    def print_to_mbox(self):
+        if "creator" in self:
+            # otherwise, probably a `be show` uuid-only bug to avoid
+            # root comments.
+            name,addr = email.utils.parseaddr(self["creator"])
+            print "From %s %s" % (addr, rfc2822_to_asctime(self["created"]))
+            print "Message-id: <%s@%s>" % (self["uuid"], DEFAULT_DOMAIN)
+            print "Date: %s" % self["created"]
+            print "From: %s" % self["creator"]
+            print "Content-Type: %s; charset=%s" % ("text/plain", DEFAULT_ENCODING)
+            print "Content-Transfer-Encoding: 8bit"
+            print "Subject: %s: %s" % (self["short-name"], self["summary"])
+            if "extra-strings" in self:
+                for estr in self["extra_strings"]:
+                    print "X-Extra-String: %s" % estr
+            print ""
+            print self["summary"]
+            print ""
+        if "comments" in self:
+            for comment in self["comments"]:
+                comment.print_to_mbox(self)            
+    def init_from_etree(self, element):
+        assert element.tag == "bug", element.tag
+        for field in element.getchildren():
+            text = unescape(unicode(field.text).decode("unicode_escape").strip())
+            if field.tag == "comment":
+                comm = Comment()
+                comm.init_from_etree(field)
+                if "comments" in self:
+                    self["comments"].append(comm)
+                else:
+                    self["comments"] = [comm]
+            elif field.tag == "extra-string":
+                if "extra-strings" in self:
+                    self["extra-strings"].append(text)
+                else:
+                    self["extra-strings"] = [text]
+            else:
+                self[field.tag] = text
+
+class Comment (LimitedAttrDict):
+    _attrs = [u"uuid",
+              u"alt-id",
+              u"short-name",
+              u"in-reply-to",
+              u"author",
+              u"date",
+              u"content-type",
+              u"body",
+              u"extra-strings"]
+    def print_to_mbox(self, bug=None):
+        if bug == None:
+            bug = Bug()
+            bug[u"uuid"] = u"no-uuid"
+        name,addr = email.utils.parseaddr(self["author"])
+        print "From %s %s" % (addr, rfc2822_to_asctime(self["date"]))
+        if "uuid" in self:     id = self["uuid"]
+        elif "alt-id" in self: id = self["alt-id"]
+        else:                  id = None
+        if id != None:
+            print "Message-id: <%s@%s>" % (id, DEFAULT_DOMAIN)
+        print "Date: %s" % self["date"]
+        print "From: %s" % self["author"]
+        subject = ""
+        if "short-name" in self:
+            subject += self["short-name"]+u": "
+        if "summary" in bug:
+            subject += bug["summary"]
+        else:
+            subject += u"no-subject"
+        print "Subject: %s" % subject
+        if "in-reply-to" not in self.keys():
+            self["in-reply-to"] = bug["uuid"]
+        print "In-Reply-To: <%s@%s>" % (self["in-reply-to"], DEFAULT_DOMAIN)
+        if "extra-strings" in self:
+            for estr in self["extra_strings"]:
+                print "X-Extra-String: %s" % estr
+        if self["content-type"].startswith("text/"):
+            print "Content-Transfer-Encoding: 8bit"
+            print "Content-Type: %s; charset=%s" % (self["content-type"], DEFAULT_ENCODING)
+            print ""
+            print self["body"]
+        else: # content type and transfer encoding already in XML MIME output
+            print self["body"]
+        print ""
+    def init_from_etree(self, element):
+        assert element.tag == "comment", element.tag
+        for field in element.getchildren():
+            text = unescape(unicode(field.text).decode("unicode_escape").strip())
+            if field.tag == "extra-string":
+                if "extra-strings" in self:
+                    self["extra-strings"].append(text)
+                else:
+                    self["extra-strings"] = [text]
+            else:
+                if field.tag == "body":
+                    text+="\n"
+                self[field.tag] = text
+
+def print_to_mbox(element):
+    if element.tag == "bug":
+        b = Bug()
+        b.init_from_etree(element)
+        b.print_to_mbox()
+    elif element.tag == "comment":
+        c = Comment()
+        c.init_from_etree(element)
+        c.print_to_mbox()
+    elif element.tag in ["be-xml"]:
+        for elt in element.getchildren():
+            print_to_mbox(elt)
+
+if __name__ == "__main__":
+    import sys
+
+    if len(sys.argv) == 1: # no filename given, use stdin
+        xml_unicode = sys.stdin.read()
+    else:
+        xml_unicode = codecs.open(sys.argv[1], "r", DEFAULT_ENCODING).read()
+    xml_str = xml_unicode.encode("unicode_escape").replace(r"\n", "\n")
+    elist = ElementTree.XML(xml_str)
+    print_to_mbox(elist)
diff --git a/misc/xml/catmutt b/misc/xml/catmutt
new file mode 100755
index 0000000..601f14f
--- /dev/null
+++ b/misc/xml/catmutt
@@ -0,0 +1,59 @@
+#!/bin/sh
+
+# catmutt - wrap mutt allowing mboxes read from stdin.
+#
+# Copyright (C) 1998-1999 Moritz Barsnick <barsnick (at) gmx (dot) net>,
+#               2009 William Trevor King <wking (at) drexel (dot) edu>
+#
+#    This program is free software; you can redistribute it and/or
+#    modify it under the terms of the GNU General Public License
+#    version 2 as published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+#
+# developed from grepm-0.6
+#   http://www.barsnick.net/sw/grepm.html
+
+PROGNAME=`basename "$0"`
+export TMPDIR="${TMPDIR-/tmp}" # used by mktemp
+umask 077
+
+if [ $# -gt 0 ] && [ "$1" = "--help" ]; then
+  echo 1>&2 "Usage: ${PROGNAME} [--help] mutt-arguments"
+  echo 1>&2 ""
+  echo 1>&2 "Read a mailbox file from stdin and opens it with mutt."
+  echo 1>&2 "For example: cat somefile.mbox | ${PROGNAME}"
+  exit 0
+fi
+
+# Note: the -t/-p options to mktemp are deprecated for mktemp (GNU
+# coreutils) 7.1 in favor of --tmpdir but the --tmpdir option does not
+# exist yet for my 6.10-3ubuntu2 coreutils
+TMPFILE=`mktemp -t catmutt.XXXXXX` || exit 1
+
+trap "rm -f ${TMPFILE}; exit 1" 1 2 3 13 15
+
+cat > "${TMPFILE}" || exit 1
+
+# Now that we've read in the mailbox file, reopen stdin for mutt/user
+# interaction.  When in a pipe we're not technically in a tty, so use
+# a little hack from "greno" at
+# http://www.linuxforums.org/forum/linux-programming-scripting/98607-bash-stdin-problem.html
+tty="/dev/`ps -p$$ --no-heading | awk '{print $2}'`"
+exec < ${tty}
+
+if [ `wc -c "${TMPFILE}" | awk '{print $1}'` -gt 0 ]; then
+  echo 1>&2 "Calling mutt on temporary mailbox file (${TMPFILE})."
+  mutt -R -f "${TMPFILE}" "$@"
+else
+  echo 1>&2 "Empty mailbox input."
+fi
+
+rm -f "${TMPFILE}" && echo 1>&2 "Deleted temporary mailbox file (${TMPFILE})."