pod2man generates invalid roff #11977

p5pRT · 2012-02-29T21:04:43Z

Migrated from rt.perl.org#111448 (status was 'open')

Searchable as RT111448$

p5pRT · 2012-02-29T21:04:43Z

From [email protected]

On the systems I tested it on, the manpages whose pods contain non-ASCII
are botched, some to the point of complete illegibility.

This is wrong:

$ man perlhist| grep var
Steve Hay, Matt S Trout, David Golden, Florian Ragwitz, Tatsuhiko Miyagawa, Chris "BinGOs" Williams, Zefram and var Arnfjoer` Bjarmason.
Chip Salzenberg gets credit for that, with a nod to his cow orker, David Croy. We had passed around various names (baton, token, hot potato) but none caught on.
var 5.13.10 2011-Feb-20

And this is intolerable:

$ man perlcn
PERLCN(1) Perl Programmers Reference Guide PERLCN(1)

NAME
perlcn - XXXX Perl XX

DESCRIPTION
XXXX Perl XXX!

X 5.8.0 XXX, Perl XXXXXX Unicode (XXX) XX, XXXXXXXXXXXXXXXXXXX; CJK (XXX) XXXXXXXX. Unicode XXXXXXX, XXXXXXXXXXXX: XXXX, XXXX,
XXXXXXXX (XXX, XXXX, XXXX, XXXX, XXX, XXXX, XX). XXXXXXXXXXXXXX (X PC XXXX).

Perl XXX Unicode XXXX. XXX Perl XXXXXXXXXX Unicode XX; Perl XXXXXX (XXXXXXXXX) XXX Unicode XXXX. XXXXXXX, XXXXX Unicode
XXXXXXXXXXXX, Perl XXX Encode XXXX, XXXXXXXXXXXXXXXXXXX.

The bug is that while you are allowed to and encouraged to declare
your podfiles' encoding, this is severely disrespected.

The following podpages show the problem:

perlebcdic.pod
perlgit.pod
perlhist.pod
perlpodspec.pod
perlthrtut.pod

perl588delta.pod
perl5100delta.pod
perl5120delta.pod
perl5121delta.pod
perl5122delta.pod
perl5123delta.pod
perl5124delta.pod
perl5140delta.pod
perl5141delta.pod
perl5142delta.pod
perl5150delta.pod
perl5151delta.pod
perl5152delta.pod
perl5153delta.pod
perl5154delta.pod
perl5156delta.pod
perl5157delta.pod
perl5158delta.pod

perlcn.pod
perljp.pod
perlko.pod
perltw.pod

--tom

Summary of my perl5 (revision 5 version 14 subversion 0) configuration:

Platform:
osname=openbsd, osvers=4.4, archname=OpenBSD.i386-openbsd
uname='openbsd chthon 4.4 generic#0 i386 '
config_args='-des'
hint=recommended, useposix=true, d_sigaction=define
useithreads=undef, usemultiplicity=undef
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=undef, use64bitall=undef, uselongdouble=undef
usemymalloc=y, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include',
optimize='-O2',
cppflags='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
ccversion='', gccversion='3.3.5 (propolice)', gccosandvers='openbsd4.4'
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags ='-Wl,-E -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib
libs=-lgdbm -lm -lutil -lc
perllibs=-lm -lutil -lc
libc=/usr/lib/libc.so.48.0, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
cccdlflags='-DPIC -fPIC ', lddlflags='-shared -fPIC -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl):
Compile-time options: MYMALLOC PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP
PERL_PRESERVE_IVUV USE_LARGE_FILES USE_PERLIO
USE_PERL_ATOF
Built under openbsd
Compiled at Jun 11 2011 11:48:28
%ENV:
PERL_UNICODE="SA"
@INC:
/usr/local/lib/perl5/site_perl/5.14.0/OpenBSD.i386-openbsd
/usr/local/lib/perl5/site_perl/5.14.0
/usr/local/lib/perl5/5.14.0/OpenBSD.i386-openbsd
/usr/local/lib/perl5/5.14.0
/usr/local/lib/perl5/site_perl/5.12.3
/usr/local/lib/perl5/site_perl/5.11.3
/usr/local/lib/perl5/site_perl/5.10.1
/usr/local/lib/perl5/site_perl/5.10.0
/usr/local/lib/perl5/site_perl/5.8.7
/usr/local/lib/perl5/site_perl/5.8.0
/usr/local/lib/perl5/site_perl/5.6.0
/usr/local/lib/perl5/site_perl/5.005
/usr/local/lib/perl5/site_perl
.

p5pRT · 2012-09-27T02:07:49Z

From @jkeenan

On Wed Feb 29 13:04:43 2012, tom christiansen wrote:

On the systems I tested it on, the manpages whose pods contain non-
ASCII
are botched, some to the point of complete illegibility.

This is wrong:
$ man perlhist| grep var
   Steve Hay\, Matt S Trout\, David Golden\, Florian Ragwitz\,
Tatsuhiko Miyagawa, Chris "BinGOs" Williams, Zefram and var Arnfjoer`
Bjarmason.
Chip Salzenberg gets credit for that, with a nod to his cow
orker, David Croy. We had passed around various names (baton, token,
hot potato) but none caught on.
var 5.13.10 2011-Feb-20

And this is intolerable:
$ man perlcn
PERLCN$1$                                            Perl
Programmers Reference Guide
PERLCN(1)
NAME
   perlcn \- XXXX Perl XX

DESCRIPTION
   XXXX Perl XXX\!

   X 5\.8\.0 XXX\, Perl XXXXXX Unicode $XXX$ XX\, XXXXXXXXXXXXXXXXXXX;
CJK (XXX) XXXXXXXX. Unicode XXXXXXX, XXXXXXXXXXXX: XXXX, XXXX,
XXXXXXXX (XXX, XXXX, XXXX, XXXX, XXX, XXXX, XX). XXXXXXXXXXXXXX (X
PC XXXX).
   Perl XXX Unicode XXXX\. XXX Perl XXXXXXXXXX Unicode XX; Perl XXXXXX
(XXXXXXXXX) XXX Unicode XXXX. XXXXXXX, XXXXX Unicode
XXXXXXXXXXXX, Perl XXX Encode XXXX, XXXXXXXXXXXXXXXXXXX.

The bug is that while you are allowed to and encouraged to declare
your podfiles' encoding, this is severely disrespected.

The following podpages show the problem:
perlebcdic\.pod
perlgit\.pod
perlhist\.pod
perlpodspec\.pod
perlthrtut\.pod

perl588delta\.pod
perl5100delta\.pod
perl5120delta\.pod
perl5121delta\.pod
perl5122delta\.pod
perl5123delta\.pod
perl5124delta\.pod
perl5140delta\.pod
perl5141delta\.pod
perl5142delta\.pod
perl5150delta\.pod
perl5151delta\.pod
perl5152delta\.pod
perl5153delta\.pod
perl5154delta\.pod
perl5156delta\.pod
perl5157delta\.pod
perl5158delta\.pod

perlcn\.pod
perljp\.pod
perlko\.pod
perltw\.pod
--tom

Is this a problem that can be fixed in the 'pod2man' utility?

Or does it require corrections of individual .pod files?

Or both?

Or neither?

Thank you very much.
Jim Keenan

p5pRT · 2012-09-27T02:07:49Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2012-09-27T04:35:40Z

From @mrallen1

On Wed Sep 26 19:07:49 2012, jkeenan wrote:

On Wed Feb 29 13:04:43 2012, tom christiansen wrote:

On the systems I tested it on, the manpages whose pods contain non-
ASCII
are botched, some to the point of complete illegibility.

Is this a problem that can be fixed in the 'pod2man' utility?

Or does it require corrections of individual .pod files?

Or both?

Or neither?

Hi. I'm the current maintainer of Pod::Perldoc. In 90% of the cases where the output from
pod2man is displayed as "X" instead of an proper glyph, it is typically groff itself which botches
the handling non-ASCII characters.

Getting clean non-ASCII characters typically requires a groff >= 1.20.1 and within the context
of perldoc (at least) a recent version (>=3.18). As far as pod2man is concerned, I believe it
handles non-ASCII characters as well as it can given the various points where encoding matters
to display which are (among others), the shell itself, perl, groff, the pager program, and
whatever terminal one is using.

So I would say this report is inconclusive. It may well be that pod2man is generating proper roff
output, and groff is mangling it when it tries to format and display it.

Mark

p5pRT · 2012-09-27T04:35:41Z

From @mrallen1

On Wed Sep 26 19:07:49 2012, jkeenan wrote:

On Wed Feb 29 13:04:43 2012, tom christiansen wrote:

On the systems I tested it on, the manpages whose pods contain non-
ASCII
are botched, some to the point of complete illegibility.

Is this a problem that can be fixed in the 'pod2man' utility?

Or does it require corrections of individual .pod files?

Or both?

Or neither?

Hi. I'm the current maintainer of Pod::Perldoc. In 90% of the cases where the output from
pod2man is displayed as "X" instead of an proper glyph, it is typically groff itself which botches
the handling non-ASCII characters.

Getting clean non-ASCII characters typically requires a groff >= 1.20.1 and within the context
of perldoc (at least) a recent version (>=3.18). As far as pod2man is concerned, I believe it
handles non-ASCII characters as well as it can given the various points where encoding matters
to display which are (among others), the shell itself, perl, groff, the pager program, and
whatever terminal one is using.

So I would say this report is inconclusive. It may well be that pod2man is generating proper roff
output, and groff is mangling it when it tries to format and display it.

Mark

khwilliamson · 2025-05-10T02:24:10Z

I just ran the two test cases on Ubuntu 24.04, and they both worked fine.

I think that in the interval since this was filed, the underlying tools were fixed.

I think this is closable; I'm self-assigning this, and if no one complains by June 9, 2025, I will consider this closable, and when I remember to, will close it. If I forget, anyone else feel free to close it then.

jkeenan · 2025-05-15T22:03:58Z

I just ran the two test cases on Ubuntu 24.04, and they both worked fine.

I think that in the interval since this was filed, the underlying tools were fixed.

I think this is closable; I'm self-assigning this, and if no one complains by June 9, 2025, I will consider this closable, and when I remember to, will close it. If I forget, anyone else feel free to close it then.

I believe this ticket is almost closable ... but not quite.

If you're working on Ubuntu 24.04 (as I am) or, more precisely, with any perl 5.38 or newer, then you should get good results. That's because in the 5.37 development cycle we upgraded the podlators distribution, which includes Pod::Man, to CPAN version 5.00.

commit 20616d517ad4726a85ee27750db6e24443343601
Author:     Russ Allbery <[email protected]>
AuthorDate: Tue Dec 6 22:40:57 2022 +0000
Commit:     James E Keenan <[email protected]>
CommitDate: Fri Dec 9 10:43:28 2022 -0500

    Update podlators to CPAN version 5.00

Investigation:

You referred to the interval between the filing of this ticket and the present, which made me wonder when the problem was fixed, i.e., a bisection problem. Since I have a number of perls installed via perlbrew on my laptop, I constructed commands to run the problematic cases described by Tom back in 2012. Here are excerpts from my results.

Ubuntu Linux 24.04 LTS

$ for f in `find /home/jkeenan/perl5/perlbrew/perls/ -type f -name '*perlhist.1' | xargs ls`; do echo $f;man $f | grep var; done
...
/home/jkeenan/perl5/perlbrew/perls/perl-5.36.3/man/man1/perlhist.1
       Golden, Florian Ragwitz, Tatsuhiko Miyagawa, Chris "BinGOs" Williams, Zefram, AEvar
       passed around various names (baton, token, hot potato) but none caught on.  Then, Chip
        AEvar     5.13.10       2011-Feb-20     
/home/jkeenan/perl5/perlbrew/perls/perl-5.38.0/man/man1/perlhist.1
       Golden, Florian Ragwitz, Tatsuhiko Miyagawa, Chris "BinGOs" Williams, Zefram, Ævar
       passed around various names (baton, token, hot potato) but none caught on.  Then, Chip
        Ævar     5.13.10       2011-Feb-20
...


$ for f in `find /home/jkeenan/perl5/perlbrew/perls/ -type f -name '*perlcn.1' | xargs ls`; do echo $f; head -n5 $f; done
...
/home/jkeenan/perl5/perlbrew/perls/perl-5.36.0/man/man1/perlcn.1
.\" Automatically generated by Pod::Man 4.14 (Pod::Simple 3.43)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
/home/jkeenan/perl5/perlbrew/perls/perl-5.36.3/man/man1/perlcn.1
.\" Automatically generated by Pod::Man 4.14 (Pod::Simple 3.43)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
/home/jkeenan/perl5/perlbrew/perls/perl-5.38.0/man/man1/perlcn.1
.\" -*- mode: troff; coding: utf-8 -*-
.\" Automatically generated by Pod::Man 5.01 (Pod::Simple 3.43)
.\"
.\" Standard preamble:
.\" ========================================================================

This suggested that the problem was corrected between 5.36 and 5.38. I tested the commit cited above and verified it was the corrective commit.

$ man /usr/home/jkeenan/testing/20616d517ad4726a85ee27750db6e24443343601^/man/man1/perlcn.1 | head -n 5
PERLCN(1)              Perl Programmers Reference Guide              PERLCN(1)


NAME
       perlcn - XXXX Perl XX

$ man /usr/home/jkeenan/testing/20616d517ad4726a85ee27750db6e24443343601/man/man1/perlcn.1 | head -n 5
PERLCN(1)              Perl Programmers Reference Guide              PERLCN(1)


NAME
       perlcn - 简体中文 Perl 指南

FreeBSD-14 (/usr/local/bin/perl: perl-5.36.3)

$ man perlhist | grep var
       Zefram, AEvar Arnfjoer` Bjarmason, Stevan Little, Dave Rolsky, Max
       David Croy.  We had passed around various names (baton, token, hot
        AEvar     5.13.10       2011-Feb-20

$ man perlcn | head -n 5
PERLCN(1)              Perl Programmers Reference Guide              PERLCN(1)

XXXXXXXXXXXXXXXXXX, XXXXXXXXXXXX.  XXXXXX POD (XXXXXX) XX; XXXXXXXXXXXXXX,
XXXXXX. XXXXXXXXXXX, XXX perlpod XXXX.

perl-5.38.1 built with man pages

$ man /usr/home/jkeenan/testing/v5.38.1-with_man/man/man1/perlhist.1 | grep var
       Zefram, Ævar Arnfjörð Bjarmason, Stevan Little, Dave Rolsky, Max
       David Croy.  We had passed around various names (baton, token, hot
        Ævar     5.13.10       2011-Feb-20

$ man /usr/home/jkeenan/testing/v5.38.1-with_man/man/man1/perlcn.1 | head -n 5
PERLCN(1)              Perl Programmers Reference Guide              PERLCN(1)


NAME
       perlcn - 简体中文 Perl 指南

So provided we're using perl-5.38 or newer on FreeBSD, this DWIMs.

OpenBSD-6.9 (/usr/bin/perl: perl-5.32.1)

$ man perlhist | grep var
       Zefram, AEvar Arnfjordh Bjarmason, Stevan Little, Dave Rolsky, Max
       David Croy.  We had passed around various names (baton, token, hot
        AEvar     5.13.10       2011-Feb-20

$ man perlcn | head -n 5
man: No entry for perlcn in the manual.

So I don't have all the data I want for OpenBSD. Furthermore, I was unable to build and install a post-5.38 perl with man pages on OpenBSD, which led to #23292. If I could the results I want on OpenBSD, then I would feel comfortable closing this 13-year-old ticket.

jkeenan · 2025-05-17T16:02:22Z

So I don't have all the data I want for OpenBSD. Furthermore, I was unable to build and install a post-5.38 perl with man pages on OpenBSD, which led to #23292. If I could the results I want on OpenBSD, then I would feel comfortable closing this 13-year-old ticket.

So the upshot of the discussion in #23292 (at least as I read it), is that I'm not going to be able to get that data from OpenBSD. We've identified what Perl has done (Russ Allbery's work on Pod::Man) to address the OP's complaint, and it appears that we've reached the limits of what Perl can do, I'm closing this ticket now.

p5pRT added Severity Low distro-openbsd labels Oct 19, 2019

xenu removed the Severity Low label Dec 29, 2021

jkeenan removed the distro-openbsd label May 11, 2024

khwilliamson self-assigned this May 10, 2025

khwilliamson added the Closable? We might be able to close this ticket, but we need to check with the reporter label May 10, 2025

jkeenan closed this as completed May 17, 2025

jkeenan mentioned this issue May 17, 2025

OpenBSD: why can't I install man pages locally? #23292

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pod2man generates invalid roff #11977

pod2man generates invalid roff #11977

p5pRT commented Feb 29, 2012

p5pRT commented Feb 29, 2012

p5pRT commented Sep 27, 2012

p5pRT commented Sep 27, 2012

p5pRT commented Sep 27, 2012

p5pRT commented Sep 27, 2012

khwilliamson commented May 10, 2025

jkeenan commented May 15, 2025

jkeenan commented May 17, 2025

pod2man generates invalid roff #11977

pod2man generates invalid roff #11977

Comments

p5pRT commented Feb 29, 2012

p5pRT commented Feb 29, 2012

From [email protected]

p5pRT commented Sep 27, 2012

From @jkeenan

p5pRT commented Sep 27, 2012

p5pRT commented Sep 27, 2012

From @mrallen1

p5pRT commented Sep 27, 2012

From @mrallen1

khwilliamson commented May 10, 2025

jkeenan commented May 15, 2025

jkeenan commented May 17, 2025