Represent mdoc(7) .Pp (and .sp, and some SYNOPSIS and .Rs features)

by the <p> HTML element and use the html_fillmode() mechanism
for .Bd -unfilled, just like it was done for man(7) earlier, finally
getting rid both of the horrible <div class="Pp"></div> hack and
of the worst HTML syntax violations caused by nested displays.

Care is needed because in some situations, paragraphs have to remain
open across several subsequent macros, whereas in other situations,
they must get closed together with a block containing them.

Some implementation details include:
* Always close paragraphs before emitting HTML flow content.
* Let html_close_paragraph() also close <pre> for extra safety.
* Drop the old, now unused function print_paragraph().
* Minor adjustments in the top-level man(7) node formatter for symmetry.
* Bugfix: .Ss heads suspend no-fill mode, even though .Ss doesn't end it.
* Bugfix: give up on .Op semantic markup for now, see the comment.

29 files changed, 429 insertions, 23 deletions

diff --git a/regress/mdoc/Bd/Makefile b/regress/mdoc/Bd/Makefile
index 0d0b2fb9..f05f54b0 100644
--- a/regress/mdoc/Bd/Makefile
+++ b/regress/mdoc/Bd/Makefile
@@ -1,10 +1,11 @@
-# $OpenBSD: Makefile,v 1.21 2019/01/04 03:54:50 schwarze Exp $
+# $OpenBSD: Makefile,v 1.22 2019/01/07 06:51:37 schwarze Exp $
 
-REGRESS_TARGETS	 = beforeNAME blank centered nested nf spacing
+REGRESS_TARGETS	 = beforeNAME blank centered nested nf paragraph spacing
 REGRESS_TARGETS	+= badargs empty offset-empty offset-neg
 REGRESS_TARGETS	+= break broken unclosed
 
 LINT_TARGETS	 = beforeNAME blank nested badargs break broken unclosed
+HTML_TARGETS	 = nf paragraph
 
 SKIP_TMAN	 = beforeNAME centered
 
diff --git a/regress/mdoc/Bd/nf.in b/regress/mdoc/Bd/nf.in
index 94dc2438..a6adae29 100644
--- a/regress/mdoc/Bd/nf.in
+++ b/regress/mdoc/Bd/nf.in
@@ -1,4 +1,4 @@
-.\" $OpenBSD: nf.in,v 1.1 2019/01/04 03:54:50 schwarze Exp $
+.\" $OpenBSD: nf.in,v 1.2 2019/01/07 06:51:37 schwarze Exp $
 .Dd $Mdocdate$
 .Dt BD-NF 1
 .Os
@@ -6,6 +6,7 @@
 .Nm Bd-nf
 .Nd interaction of no-fill blocks and requests
 .Sh DESCRIPTION
+BEGINTEST
 initial
 text
 .nf
@@ -32,3 +33,7 @@ in filled block
 .Ed
 after end
 of filled block
+.br
+ENDTEST
+.br
+end of file
diff --git a/regress/mdoc/Bd/nf.out_ascii b/regress/mdoc/Bd/nf.out_ascii
index 4a42fe82..d8ceb0c9 100644
--- a/regress/mdoc/Bd/nf.out_ascii
+++ b/regress/mdoc/Bd/nf.out_ascii
@@ -4,7 +4,7 @@ NNAAMMEE
      BBdd--nnff - interaction of no-fill blocks and requests
 
 DDEESSCCRRIIPPTTIIOONN
-     initial text
+     BEGINTEST initial text
      after .nf
      request
      after .fi request
@@ -18,5 +18,7 @@ DDEESSCCRRIIPPTTIIOONN
      after .nf request
      in filled block
      after end of filled block
+     ENDTEST
+     end of file
 
-OpenBSD                         January 4, 2019                        OpenBSD
+OpenBSD                         January 7, 2019                        OpenBSD
diff --git a/regress/mdoc/Bd/nf.out_html b/regress/mdoc/Bd/nf.out_html
new file mode 100644
index 00000000..31642e07
--- /dev/null
+++ b/regress/mdoc/Bd/nf.out_html
@@ -0,0 +1,22 @@
+BEGINTEST initial text
+<pre>
+after .nf
+request
+</pre>
+after .fi request
+<div class="Bd Pp">
+<pre>
+in unfilled
+block
+</pre>
+after .fi request in unfilled block</div>
+after end of unfilled block
+<div class="Bd Pp">in filled block
+<pre>
+after .nf request
+in filled block
+</pre>
+</div>
+after end of filled block
+<br/>
+ENDTEST
diff --git a/regress/mdoc/Bd/nf.out_markdown b/regress/mdoc/Bd/nf.out_markdown
index 382bb786..46ece0a2 100644
--- a/regress/mdoc/Bd/nf.out_markdown
+++ b/regress/mdoc/Bd/nf.out_markdown
@@ -6,6 +6,7 @@ BD-NF(1) - General Commands Manual
 
 # DESCRIPTION
 
+BEGINTEST
 initial
 text
 after .nf
@@ -27,6 +28,8 @@ of unfilled block
 > in filled block
 
 after end
-of filled block
+of filled block  
+ENDTEST  
+end of file
 
-OpenBSD - January 4, 2019
+OpenBSD - January 7, 2019
diff --git a/regress/mdoc/Bd/paragraph.in b/regress/mdoc/Bd/paragraph.in
new file mode 100644
index 00000000..09629f56
--- /dev/null
+++ b/regress/mdoc/Bd/paragraph.in
@@ -0,0 +1,39 @@
+.\" $OpenBSD: paragraph.in,v 1.1 2019/01/07 06:51:37 schwarze Exp $
+.Dd $Mdocdate$
+.Dt BD-PARAGRAPH 1
+.Os
+.Sh NAME
+.Nm Bd-paragraph
+.Nd interaction of paragraphs and displays
+.Sh DESCRIPTION
+BEGINTEST
+initial
+text
+.Pp
+normal
+paragraph
+.Bd -filled
+filled
+display
+.Pp
+paragraph
+in display
+.Ed
+back
+to normal
+.Pp
+another
+paragraph
+.Bd -unfilled
+unfilled
+display
+.Pp
+unfilled
+paragraph
+.Ed
+again back
+to normal
+.br
+ENDTEST
+.br
+end of file
diff --git a/regress/mdoc/Bd/paragraph.out_ascii b/regress/mdoc/Bd/paragraph.out_ascii
new file mode 100644
index 00000000..216f4530
--- /dev/null
+++ b/regress/mdoc/Bd/paragraph.out_ascii
@@ -0,0 +1,27 @@
+BD-PARAGRAPH(1)             General Commands Manual            BD-PARAGRAPH(1)
+
+NNAAMMEE
+     BBdd--ppaarraaggrraapphh - interaction of paragraphs and displays
+
+DDEESSCCRRIIPPTTIIOONN
+     BEGINTEST initial text
+
+     normal paragraph
+
+     filled display
+
+     paragraph in display
+     back to normal
+
+     another paragraph
+
+     unfilled
+     display
+
+     unfilled
+     paragraph
+     again back to normal
+     ENDTEST
+     end of file
+
+OpenBSD                         January 7, 2019                        OpenBSD
diff --git a/regress/mdoc/Bd/paragraph.out_html b/regress/mdoc/Bd/paragraph.out_html
new file mode 100644
index 00000000..c2c6993e
--- /dev/null
+++ b/regress/mdoc/Bd/paragraph.out_html
@@ -0,0 +1,19 @@
+BEGINTEST initial text
+<p class="Pp">normal paragraph</p>
+<div class="Bd Pp">filled display
+<p class="Pp">paragraph in display</p>
+</div>
+back to normal
+<p class="Pp">another paragraph</p>
+<div class="Bd Pp">
+<pre>
+unfilled
+display
+
+unfilled
+paragraph
+</pre>
+</div>
+again back to normal
+<br/>
+ENDTEST
diff --git a/regress/mdoc/Bd/paragraph.out_markdown b/regress/mdoc/Bd/paragraph.out_markdown
new file mode 100644
index 00000000..13e43f11
--- /dev/null
+++ b/regress/mdoc/Bd/paragraph.out_markdown
@@ -0,0 +1,39 @@
+BD-PARAGRAPH(1) - General Commands Manual
+
+# NAME
+
+**Bd-paragraph** - interaction of paragraphs and displays
+
+# DESCRIPTION
+
+BEGINTEST
+initial
+text
+
+normal
+paragraph
+
+> filled
+> display
+
+> paragraph
+> in display
+
+back
+to normal
+
+another
+paragraph
+
+	unfilled
+	display
+
+	unfilled
+	paragraph
+
+again back
+to normal  
+ENDTEST  
+end of file
+
+OpenBSD - January 7, 2019
diff --git a/regress/mdoc/Bf/Makefile b/regress/mdoc/Bf/Makefile
index eb845e95..da5b2d91 100644
--- a/regress/mdoc/Bf/Makefile
+++ b/regress/mdoc/Bf/Makefile
@@ -1,7 +1,8 @@
-# $OpenBSD: Makefile,v 1.4 2014/07/04 16:11:42 schwarze Exp $
+# $OpenBSD: Makefile,v 1.6 2019/01/07 06:51:37 schwarze Exp $
 
-REGRESS_TARGETS	= nest multiargs badargs break broken
-LINT_TARGETS	= multiargs badargs break broken
+REGRESS_TARGETS	 = badargs break broken multiargs nest paragraph
+LINT_TARGETS	 = badargs break broken multiargs
+HTML_TARGETS	 = paragraph
 
 # mandoc -T markdown ignores .Bf, at least for now
 
diff --git a/regress/mdoc/Bf/paragraph.in b/regress/mdoc/Bf/paragraph.in
new file mode 100644
index 00000000..dc2836a0
--- /dev/null
+++ b/regress/mdoc/Bf/paragraph.in
@@ -0,0 +1,22 @@
+.\" $OpenBSD: paragraph.in,v 1.1 2019/01/07 06:51:37 schwarze Exp $
+.Dd $Mdocdate$
+.Dt BF-PARAGRAPH 1
+.Os
+.Sh NAME
+.Nm Bf-paragraph
+.Nd interaction of paragraphs and font blocks
+.Sh DESCRIPTION
+BEGINTEST
+.Pp
+normal
+text
+.Bf -literal
+literal
+text
+.Pp
+literal
+paragraph
+.Ef
+ENDTEST
+.br
+end of file
diff --git a/regress/mdoc/Bf/paragraph.out_ascii b/regress/mdoc/Bf/paragraph.out_ascii
new file mode 100644
index 00000000..25b2fd89
--- /dev/null
+++ b/regress/mdoc/Bf/paragraph.out_ascii
@@ -0,0 +1,14 @@
+BF-PARAGRAPH(1)             General Commands Manual            BF-PARAGRAPH(1)
+
+NNAAMMEE
+     BBff--ppaarraaggrraapphh - interaction of paragraphs and font blocks
+
+DDEESSCCRRIIPPTTIIOONN
+     BEGINTEST
+
+     normal text literal text
+
+     literal paragraph ENDTEST
+     end of file
+
+OpenBSD                         January 7, 2019                        OpenBSD
diff --git a/regress/mdoc/Bf/paragraph.out_html b/regress/mdoc/Bf/paragraph.out_html
new file mode 100644
index 00000000..e86b979a
--- /dev/null
+++ b/regress/mdoc/Bf/paragraph.out_html
@@ -0,0 +1,6 @@
+BEGINTEST
+<p class="Pp">normal text</p>
+<div class="Bf Li">literal text
+<p class="Pp">literal paragraph</p>
+</div>
+ENDTEST
diff --git a/regress/mdoc/D1/Makefile b/regress/mdoc/D1/Makefile
index bed27d1b..b94d5727 100644
--- a/regress/mdoc/D1/Makefile
+++ b/regress/mdoc/D1/Makefile
@@ -1,6 +1,7 @@
-# $OpenBSD: Makefile,v 1.2 2015/02/06 02:04:35 schwarze Exp $
+# $OpenBSD: Makefile,v 1.5 2019/01/07 06:51:37 schwarze Exp $
 
 REGRESS_TARGETS	 = spacing
 LINT_TARGETS	 = spacing
+HTML_TARGETS	 = spacing
 
 .include <bsd.regress.mk>
diff --git a/regress/mdoc/D1/spacing.in b/regress/mdoc/D1/spacing.in
index 0f6f83fa..16661257 100644
--- a/regress/mdoc/D1/spacing.in
+++ b/regress/mdoc/D1/spacing.in
@@ -1,4 +1,4 @@
-.\" $OpenBSD: spacing.in,v 1.3 2017/07/04 14:53:24 schwarze Exp $
+.\" $OpenBSD: spacing.in,v 1.4 2019/01/07 06:51:37 schwarze Exp $
 .Dd $Mdocdate$
 .Dt D1-SPACING 1
 .Os
@@ -6,8 +6,15 @@
 .Nm D1-spacing
 .Nd spacing in and around one-line displays
 .Sh DESCRIPTION
-preceding text
+BEGINTEST
+.Pp
+preceding
+paragraph
 .D1 spacing  in  and around one-line displays
 empty display:
 .D1
 following text
+.br
+ENDTEST
+.br
+end of file
diff --git a/regress/mdoc/D1/spacing.out_ascii b/regress/mdoc/D1/spacing.out_ascii
index 40945936..c0d1556d 100644
--- a/regress/mdoc/D1/spacing.out_ascii
+++ b/regress/mdoc/D1/spacing.out_ascii
@@ -4,9 +4,13 @@ NNAAMMEE
      DD11--ssppaacciinngg - spacing in and around one-line displays
 
 DDEESSCCRRIIPPTTIIOONN
-     preceding text
+     BEGINTEST
+
+     preceding paragraph
            spacing in and around one-line displays
      empty display:
      following text
+     ENDTEST
+     end of file
 
-OpenBSD                          July 4, 2017                          OpenBSD
+OpenBSD                         January 7, 2019                        OpenBSD
diff --git a/regress/mdoc/D1/spacing.out_html b/regress/mdoc/D1/spacing.out_html
new file mode 100644
index 00000000..b1a1d4da
--- /dev/null
+++ b/regress/mdoc/D1/spacing.out_html
@@ -0,0 +1,8 @@
+BEGINTEST
+<p class="Pp">preceding paragraph</p>
+<div class="Bd Bd-indent">spacing in and around one-line displays</div>
+empty display:
+<div class="Bd Bd-indent"></div>
+following text
+<br/>
+ENDTEST
diff --git a/regress/mdoc/D1/spacing.out_lint b/regress/mdoc/D1/spacing.out_lint
index 5ac4d0c0..0a2790bb 100644
--- a/regress/mdoc/D1/spacing.out_lint
+++ b/regress/mdoc/D1/spacing.out_lint
@@ -1 +1 @@
-mandoc: spacing.in:12:2: WARNING: empty block: D1
+mandoc: spacing.in:15:2: WARNING: empty block: D1
diff --git a/regress/mdoc/D1/spacing.out_markdown b/regress/mdoc/D1/spacing.out_markdown
index eac16dfe..d1f12f63 100644
--- a/regress/mdoc/D1/spacing.out_markdown
+++ b/regress/mdoc/D1/spacing.out_markdown
@@ -6,12 +6,17 @@ D1-SPACING(1) - General Commands Manual
 
 # DESCRIPTION
 
-preceding text
+BEGINTEST
+
+preceding
+paragraph
 
 > spacing in and around one-line displays
 
 empty display:
 
-following text
+following text  
+ENDTEST  
+end of file
 
-OpenBSD - July 4, 2017
+OpenBSD - January 7, 2019
diff --git a/regress/mdoc/Rs/Makefile b/regress/mdoc/Rs/Makefile
index 3732dc0a..2881b874 100644
--- a/regress/mdoc/Rs/Makefile
+++ b/regress/mdoc/Rs/Makefile
@@ -1,8 +1,9 @@
-# $OpenBSD: Makefile,v 1.7 2015/02/04 18:03:28 schwarze Exp $
+# $OpenBSD: Makefile,v 1.11 2019/01/07 06:51:37 schwarze Exp $
 
-REGRESS_TARGETS	 = allch args break empty three_authors
+REGRESS_TARGETS	 = allch args break empty paragraph three_authors
 UTF8_TARGETS	 = allch break empty three_authors
 LINT_TARGETS	 = allch args empty
+HTML_TARGETS	 = paragraph
 
 # groff-1.22.3 defect:
 # - arguments after .Rs cause the macro to be ignored
diff --git a/regress/mdoc/Rs/paragraph.in b/regress/mdoc/Rs/paragraph.in
new file mode 100644
index 00000000..9a679eb2
--- /dev/null
+++ b/regress/mdoc/Rs/paragraph.in
@@ -0,0 +1,37 @@
+.\" $OpenBSD: paragraph.in,v 1.1 2019/01/07 06:51:37 schwarze Exp $
+.Dd $Mdocdate$
+.Dt RS-PARAGRAPH 1
+.Os
+.Sh NAME
+.Nm Rs-paragraph
+.Nd interaction of paragraphs and reference blocks
+.Sh DESCRIPTION
+BEGINTEST
+.br
+initial reference:
+.Rs
+.%A author name
+.%B book title
+.Re
+.Pp
+in a paragraph:
+.Rs
+.%A another author
+.%B another book
+.Re
+.Sh SEE ALSO
+initial reference:
+.Rs
+.%A author name
+.%B book title
+.Re
+.Pp
+in a paragraph:
+.Rs
+.%A another author
+.%B another book
+.Re
+.nf
+ENDTEST
+.br
+end of file
diff --git a/regress/mdoc/Rs/paragraph.out_ascii b/regress/mdoc/Rs/paragraph.out_ascii
new file mode 100644
index 00000000..821e721d
--- /dev/null
+++ b/regress/mdoc/Rs/paragraph.out_ascii
@@ -0,0 +1,23 @@
+RS-PARAGRAPH(1)             General Commands Manual            RS-PARAGRAPH(1)
+
+NNAAMMEE
+     RRss--ppaarraaggrraapphh - interaction of paragraphs and reference blocks
+
+DDEESSCCRRIIPPTTIIOONN
+     BEGINTEST
+     initial reference: author name, _b_o_o_k _t_i_t_l_e.
+
+     in a paragraph: another author, _a_n_o_t_h_e_r _b_o_o_k.
+
+SSEEEE AALLSSOO
+     initial reference:
+
+     author name, _b_o_o_k _t_i_t_l_e.
+
+     in a paragraph:
+
+     another author, _a_n_o_t_h_e_r _b_o_o_k.
+     ENDTEST
+     end of file
+
+OpenBSD                         January 7, 2019                        OpenBSD
diff --git a/regress/mdoc/Rs/paragraph.out_html b/regress/mdoc/Rs/paragraph.out_html
new file mode 100644
index 00000000..19e9362c
--- /dev/null
+++ b/regress/mdoc/Rs/paragraph.out_html
@@ -0,0 +1,17 @@
+BEGINTEST
+<br/>
+initial reference: <cite class="Rs" title="Rs"><span class="RsA">author
+  name</span>, <i class="RsB">book title</i>.</cite>
+<p class="Pp">in a paragraph:
+    <cite class="Rs" title="Rs"><span class="RsA">another author</span>,
+    <i class="RsB">another book</i>.</cite></p>
+<h1 class="Sh" title="Sh" id="SEE_ALSO"><a class="permalink" href="#SEE_ALSO">SEE
+  ALSO</a></h1>
+initial reference:
+<p class="Pp"><cite class="Rs" title="Rs"><span class="RsA">author name</span>,
+    <i class="RsB">book title</i>.</cite></p>
+<p class="Pp">in a paragraph:</p>
+<p class="Pp"><cite class="Rs" title="Rs"><span class="RsA">another
+    author</span>, <i class="RsB">another book</i>.</cite></p>
+<pre>
+ENDTEST
diff --git a/regress/mdoc/Rs/paragraph.out_markdown b/regress/mdoc/Rs/paragraph.out_markdown
new file mode 100644
index 00000000..c8bb4429
--- /dev/null
+++ b/regress/mdoc/Rs/paragraph.out_markdown
@@ -0,0 +1,32 @@
+RS-PARAGRAPH(1) - General Commands Manual
+
+# NAME
+
+**Rs-paragraph** - interaction of paragraphs and reference blocks
+
+# DESCRIPTION
+
+BEGINTEST  
+initial reference:
+author name,
+*book title*.
+
+in a paragraph:
+another author,
+*another book*.
+
+# SEE ALSO
+
+initial reference:
+
+author name,
+*book title*.
+
+in a paragraph:
+
+another author,
+*another book*.
+ENDTEST  
+end of file
+
+OpenBSD - January 7, 2019
diff --git a/regress/mdoc/Sh/Makefile b/regress/mdoc/Sh/Makefile
index 9610ec4c..1c20f13f 100644
--- a/regress/mdoc/Sh/Makefile
+++ b/regress/mdoc/Sh/Makefile
@@ -1,9 +1,10 @@
-# $OpenBSD: Makefile,v 1.7 2017/01/08 00:10:22 schwarze Exp $
+# $OpenBSD: Makefile,v 1.11 2019/01/07 06:51:37 schwarze Exp $
 
 REGRESS_TARGETS	 = badNAME before empty emptyNAME first nohead order
-REGRESS_TARGETS	+= orderNAME parbefore parborder punctNAME subbefore
+REGRESS_TARGETS	+= orderNAME paragraph parbefore parborder punctNAME subbefore
 LINT_TARGETS	 = badNAME before empty emptyNAME first nohead order
 LINT_TARGETS	+= orderNAME parbefore parborder punctNAME subbefore
+HTML_TARGETS	 = paragraph
 
 # groff-1.22.3 defects:
 # - .Pp before .Sh NAME causes a blank line before the header line
diff --git a/regress/mdoc/Sh/paragraph.in b/regress/mdoc/Sh/paragraph.in
new file mode 100644
index 00000000..908ab49a
--- /dev/null
+++ b/regress/mdoc/Sh/paragraph.in
@@ -0,0 +1,19 @@
+.\" $OpenBSD: paragraph.in,v 1.1 2019/01/07 06:51:37 schwarze Exp $
+.Dd $Mdocdate$
+.Dt SH-PARAGRAPH 1
+.Os
+.Sh NAME
+.Nm Sh-paragraph
+.Nd interaction of paragraph and section macros
+.Sh DESCRIPTION
+BEGINTEST
+.Pp
+descriptive text
+.Ss Subsection
+initial subsection text
+.Pp
+subsection paragraph
+.Sh EXAMPLES
+ENDTEST
+.br
+end of file
diff --git a/regress/mdoc/Sh/paragraph.out_ascii b/regress/mdoc/Sh/paragraph.out_ascii
new file mode 100644
index 00000000..0ef1787f
--- /dev/null
+++ b/regress/mdoc/Sh/paragraph.out_ascii
@@ -0,0 +1,20 @@
+SH-PARAGRAPH(1)             General Commands Manual            SH-PARAGRAPH(1)
+
+NNAAMMEE
+     SShh--ppaarraaggrraapphh - interaction of paragraph and section macros
+
+DDEESSCCRRIIPPTTIIOONN
+     BEGINTEST
+
+     descriptive text
+
+   SSuubbsseeccttiioonn
+     initial subsection text
+
+     subsection paragraph
+
+EEXXAAMMPPLLEESS
+     ENDTEST
+     end of file
+
+OpenBSD                         January 7, 2019                        OpenBSD
diff --git a/regress/mdoc/Sh/paragraph.out_html b/regress/mdoc/Sh/paragraph.out_html
new file mode 100644
index 00000000..3eeb9fe8
--- /dev/null
+++ b/regress/mdoc/Sh/paragraph.out_html
@@ -0,0 +1,7 @@
+BEGINTEST
+<p class="Pp">descriptive text</p>
+<h2 class="Ss" title="Ss" id="Subsection"><a class="permalink" href="#Subsection">Subsection</a></h2>
+initial subsection text
+<p class="Pp">subsection paragraph</p>
+<h1 class="Sh" title="Sh" id="EXAMPLES"><a class="permalink" href="#EXAMPLES">EXAMPLES</a></h1>
+ENDTEST
diff --git a/regress/mdoc/Sh/paragraph.out_markdown b/regress/mdoc/Sh/paragraph.out_markdown
new file mode 100644
index 00000000..4f0affe4
--- /dev/null
+++ b/regress/mdoc/Sh/paragraph.out_markdown
@@ -0,0 +1,24 @@
+SH-PARAGRAPH(1) - General Commands Manual
+
+# NAME
+
+**Sh-paragraph** - interaction of paragraph and section macros
+
+# DESCRIPTION
+
+BEGINTEST
+
+descriptive text
+
+## Subsection
+
+initial subsection text
+
+subsection paragraph
+
+# EXAMPLES
+
+ENDTEST  
+end of file
+
+OpenBSD - January 7, 2019