LyX, OpenOffice.org, structured authoring, and me
#################################################

:date: 2005-05-02T18:48:00
:category: computer

(This record is a slightly edited version of my `message to lyx-users
list`_. I would like to thank once more to the developers and all other
participants on the list for their very civilized tone of the
discussion.)

Sometimes around 1998+ I was working in law firm and was persuaded that
Word is certainly not ideal, but probably the best what we can get and
only occasionally I was dreaming about something better, which would be
more helpful than just Word with its stupid styles (if you think, that
Word style are best thing since sliced bread, or even better than that,
then please define a style, which would have a first line intended
except when it is a first paragraph after a heading style or list). Then
I have learned more about HTML, structured authoring (separation of
style from presentation etc.), XML, etc. and found out that (especially
because I was moving out of legal field, so I did not need compatibility
with the rest of the M$ world so much) what I really want is some tool
which would enable me to focus on what I write and leave the
presentation for sometimes later or to somebody else. After playing with
some other stuff (namely lout_ and Amaya_), I finally settled with
LaTeX, mainly because of LyX_ (LyX supports partially Docbook/SGML,
but many parts — including bibliography support — are missing), which
allowed me to work on my stuff without being bothered about the
underlying stuff and because truly good XML tools (Epic_ from
ArborText) are terribly expensive and absolute overshoot for my needs (I
do not want to maintain my thesis and personal letters in Oracle SQL
database and paying $10k for that; thank you). Moreover, they usually do
not run on Linux that well (or at all).

While working with LyX (currently I am working on my PhD thesis in
sociology/criminology/law-and-society) I’ve got really addicted to
BibTeX and nice support I get from LyX. Moreover, I love the simplicity
of the environment. You know, writing scientific papers is actually all
about sufficient number of characters put down on the paper — therefore,
I am really not that interested in OLE, WordArt, included live
spreadsheets, etc., but I need something where I could write a lot of
text and my attention (not that strong in the first place) is not
distracted by everything else (and the program is sufficiently fast in
the first place). Moreover, it is possible to change a lot of the
underlying programs to do more what could I do (so for example I am able
to include export of Docbook/XML into LyX menus — actually it is just a
small script which gets Docbook/SGML on input and then with sgml2xml
utility creates Docbook/XML. Unfortunately, there is no way how to
include support for some BibTeX-alike bibliographic system, so I am
stuck.

While working with LyX I have unfortunately met some of its limitations
and I am getting more and more frustrated, because I feel that there is
actually so little what I really want (see below).

* Although there is a pretty good crowd of LyX users, there is too small
  pool of actual developers (BTW, I am not the one of them either,
  because I just cannot spend a time to learn C++ and code in it) — just
  some six of them in fact.

* So although LyX probably cannot be considered dead yet, its
  development is even slower than global warming and there are many
  issues from small to big which are just not addressed. Moreover, the
  program is in the state of perpetual rewrite and for couple of years
  users have been calmed down by saying, that when this cycle of
  complete rewrite will be finished programmers of the code will address
  our wishes and concerns. I guess most of users gave up on hoping that
  their concerns will be ever addressed, as I did. I am not saying that
  these rewrites are not necessary, I have really no clue about
  underlying programming, but I just do not feel that my concerns are
  addressed at all and LyX is IMHO becoming to be “programmers’ project”
  rather than “users’ project”, if you know what I mean.

* Moreover, one of the projects which was silently dropped and will be
  addressed probably sometimes around the time orange farms move from
  Florida to Boston (or from Spain to my native Czechia) is support for
  scripting of LyX. So users like myself cannot address themsevels some
  small issues in the program itself — for example, LyX doesn’t have
  Transpose Characters function (Ctrl-T in Emacs) and there is no way
  how to emulate that. Kind of WYHIWYH (What You Have Is What You Have
  — and that’s it).

* I am getting more and more tired with LaTeX itself — it is not real
  document format, but really just a heap of hacks how to make life at
  least bearable and normal LaTeX document is usually just a mess of
  code (environment and macro definitions), presentation stuff and some
  actual text. LyX document is usually much better because most of the
  non-document stuff can be hidden in LyX stylesheets (.layout files),
  but then you get to utter dependency on LyX itself, and as I said
  above I am getting more and more worried than LyX will soon die out
  completely.

* Even more so, because of OpenOffice, we cannot hope anymore for that
  moment of conversion, when people would finally figure out that
  structured authoring (and LyX) is better than WYSIWYG. If they do,
  they will probably end up with OOo.

* I have checked out what is actually available now in OOo and I was
  quite shocked, that the situation is actually much better than what
  I expected. Actually, I think, that in rudimentary (and not always
  useable) form most of the stuff I am interested is there. And what is
  more important, there is at least hope that most of my concerns will
  be addressed — either through huge crowd of programmers (I know that
  it is not ideal, but still there are more than six real programmers
  working on OOo, aren’t they?) or (and that is for me the value of my
  exercise with bibliography with Docbook) I can do actually a lot
  myself — either in StarBasic or with XSLT in export filters.

So what are my requirements?

* easy-to-use WYSIWYM (What You See Is What You Mean — LyX’s community
  term for something very similar to “structured authoring”) tool. No,
  I am not willing to go to PSGML_/nXML_ or whatever mode of
  Emacs_, that’s not enough.

* support for bibliography (actually, the thing I like about BibTeX is
  that I can happily maintain the databse just with my text editor
  — I have never really falled in love with the idea of dedicated
  bibliographic managers; which makes my excitment about `MODS`_ etc.
  slighlty smaller — I really will not want to maintain XML document
  with vim anymore). Current support of bibliography sucks royally, but
  I think it could be fixed to at least somehow workable state (limited
  to just numerical citations for example) quite easily — citations
  compression and inclusion of the page numbers into refernces would
  make OOo working for me.

* international support (I would love UTF-8, but I still survive with
  inputenc, fontenc, & babel and 8bit encodings). And yes, I hate that
  I still do not know how to persuade OOo that although my locale is
  cs\_CZ I DO WANT to create mostly en\_US documents and so I want its
  fields to generate things like “Bibliography” and not “Seznam použité
  literatury”.

* Stable document format — I have six years of work in LyX and LaTeX and
  I do not want to use for my long-term storage any format which will
  change in next five years six times. Which is the reason why I am
  interested in Docbook_ and slightly cautious about new `OOo ISO
  document format`_ — the format which is actually not shipped yet in
  any real application is not what I am hoping for. However, export to
  Docbook would have to be make covering all my document needs.

Actually the last point illustrates some of my other frustration — it is
painful to see how many projects from many different sides are for so
long time really close my ideals and none of them really hits the bull’s
eye. I am mostly working in sociology, so I really do not need that much
advanced equations, complicated graphs, etc. and other complicated stuff
which is usually associated with `“document format for scholarly
authoring”`_ Probably the only four things I would need above HTML
(which I consider to be minimal format for this purpose) is:

1. footnotes  

2. bibliographical citations  

3. table of contents  

4. generated cross references

Some of these could be addressed with some simple namespaces over XHTML,
but there is just no-one who would provide complete set of tools and GUI
WYSIWYM environment for it. And it is frustrating to see how there are
many programs which are darn close — Amaya is the most obvious example
for many many years (`ThotEditor`_ seems to be failed attempt to do
something similar for Docbook). .

.. _`message to lyx-users list`:
    http://thread.gmane.org/gmane.editors.lyx.general/19370
.. _lout:
    http://snark.niif.spb.su/%7Euwe/lout/lout.html
.. _Amaya:
    http://www.w3.org/Amaya
.. _LyX:
    http://www.lyx.org
.. _Epic:
    http://www.arbortext.com/products/epic_editor_datasheet.htm
.. _PSGML:
    http://psgml.sourceforge.net
.. _nXML:
    http://www.thaiopensource.com/nxml-mode/
.. _Emacs:
    http://www.gnu.org/software/emacs/
.. _MODS:
    http://www.loc.gov/standards/mods/
.. _Docbook:
    http://www.docbook.org
.. _`OOo ISO document format`:
    http://xml.coverpages.org/starOfficeXML.html
.. _`“document format for scholarly authoring”`:
    https://web.archive.org/web/20060328034957/http://www.xmlshoestring.com/xml499/authoringrequirements
.. _ThotEditor:
    http://nongnu.org/thoteditor