pod2man generates invalid roff

[p5pRT]

Migrated from rt.perl.org#111448 (status was 'open')

Searchable as RT111448$

[p5pRT]

From tchrist@perl.com

On the systems I tested it on, the manpages whose pods contain non-ASCII
are botched, some to the point of complete illegibility.

This is wrong​:

  $ man perlhist| grep var
  Steve Hay, Matt S Trout, David Golden, Florian Ragwitz, Tatsuhiko Miyagawa, Chris "BinGOs" Williams, Zefram and var Arnfjoer` Bjarmason.
  Chip Salzenberg gets credit for that, with a nod to his cow orker, David Croy. We had passed around various names (baton, token, hot potato) but none caught on.
  var 5.13.10 2011-Feb-20

And this is intolerable​:

  $ man perlcn
  PERLCN(1) Perl Programmers Reference Guide PERLCN(1)

  NAME
  perlcn - XXXX Perl XX

  DESCRIPTION
  XXXX Perl XXX!

  X 5.8.0 XXX, Perl XXXXXX Unicode (XXX) XX, XXXXXXXXXXXXXXXXXXX; CJK (XXX) XXXXXXXX. Unicode XXXXXXX, XXXXXXXXXXXX​: XXXX, XXXX,
  XXXXXXXX (XXX, XXXX, XXXX, XXXX, XXX, XXXX, XX). XXXXXXXXXXXXXX (X PC XXXX).

  Perl XXX Unicode XXXX. XXX Perl XXXXXXXXXX Unicode XX; Perl XXXXXX (XXXXXXXXX) XXX Unicode XXXX. XXXXXXX, XXXXX Unicode
  XXXXXXXXXXXX, Perl XXX Encode XXXX, XXXXXXXXXXXXXXXXXXX.

The bug is that while you are allowed to and encouraged to declare
your podfiles' encoding, this is severely disrespected.

The following podpages show the problem​:

  perlebcdic.pod
  perlgit.pod
  perlhist.pod
  perlpodspec.pod
  perlthrtut.pod

  perl588delta.pod
  perl5100delta.pod
  perl5120delta.pod
  perl5121delta.pod
  perl5122delta.pod
  perl5123delta.pod
  perl5124delta.pod
  perl5140delta.pod
  perl5141delta.pod
  perl5142delta.pod
  perl5150delta.pod
  perl5151delta.pod
  perl5152delta.pod
  perl5153delta.pod
  perl5154delta.pod
  perl5156delta.pod
  perl5157delta.pod
  perl5158delta.pod

  perlcn.pod
  perljp.pod
  perlko.pod
  perltw.pod

--tom

Summary of my perl5 (revision 5 version 14 subversion 0) configuration​:
 
  Platform​:
  osname=openbsd, osvers=4.4, archname=OpenBSD.i386-openbsd
  uname='openbsd chthon 4.4 generic#0 i386 '
  config_args='-des'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=undef, use64bitall=undef, uselongdouble=undef
  usemymalloc=y, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include',
  optimize='-O2',
  cppflags='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
  ccversion='', gccversion='3.3.5 (propolice)', gccosandvers='openbsd4.4'
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=4, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags ='-Wl,-E -fstack-protector -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib
  libs=-lgdbm -lm -lutil -lc
  perllibs=-lm -lutil -lc
  libc=/usr/lib/libc.so.48.0, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
  cccdlflags='-DPIC -fPIC ', lddlflags='-shared -fPIC -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl)​:
  Compile-time options​: MYMALLOC PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP
  PERL_PRESERVE_IVUV USE_LARGE_FILES USE_PERLIO
  USE_PERL_ATOF
  Built under openbsd
  Compiled at Jun 11 2011 11​:48​:28
  %ENV​:
  PERL_UNICODE="SA"
  @​INC​:
  /usr/local/lib/perl5/site_perl/5.14.0/OpenBSD.i386-openbsd
  /usr/local/lib/perl5/site_perl/5.14.0
  /usr/local/lib/perl5/5.14.0/OpenBSD.i386-openbsd
  /usr/local/lib/perl5/5.14.0
  /usr/local/lib/perl5/site_perl/5.12.3
  /usr/local/lib/perl5/site_perl/5.11.3
  /usr/local/lib/perl5/site_perl/5.10.1
  /usr/local/lib/perl5/site_perl/5.10.0
  /usr/local/lib/perl5/site_perl/5.8.7
  /usr/local/lib/perl5/site_perl/5.8.0
  /usr/local/lib/perl5/site_perl/5.6.0
  /usr/local/lib/perl5/site_perl/5.005
  /usr/local/lib/perl5/site_perl
  .

[p5pRT]

From @jkeenan

On Wed Feb 29 13​:04​:43 2012, tom christiansen wrote​:

On the systems I tested it on, the manpages whose pods contain non-
ASCII
are botched, some to the point of complete illegibility.

This is wrong​:

$ man perlhist| grep var
   Steve Hay\, Matt S Trout\, David Golden\, Florian Ragwitz\,

Tatsuhiko Miyagawa, Chris "BinGOs" Williams, Zefram and var Arnfjoer`
Bjarmason.
Chip Salzenberg gets credit for that, with a nod to his cow
orker, David Croy. We had passed around various names (baton, token,
hot potato) but none caught on.
var 5.13.10 2011-Feb-20

And this is intolerable​:

$ man perlcn
PERLCN\(1\)                                            Perl

Programmers Reference Guide
PERLCN(1)

NAME
   perlcn \- XXXX Perl XX

DESCRIPTION
   XXXX Perl XXX\!

   X 5\.8\.0 XXX\, Perl XXXXXX Unicode \(XXX\) XX\, XXXXXXXXXXXXXXXXXXX;

CJK (XXX) XXXXXXXX. Unicode XXXXXXX, XXXXXXXXXXXX​: XXXX, XXXX,
XXXXXXXX (XXX, XXXX, XXXX, XXXX, XXX, XXXX, XX). XXXXXXXXXXXXXX (X
PC XXXX).

   Perl XXX Unicode XXXX\. XXX Perl XXXXXXXXXX Unicode XX; Perl XXXXXX

(XXXXXXXXX) XXX Unicode XXXX. XXXXXXX, XXXXX Unicode
XXXXXXXXXXXX, Perl XXX Encode XXXX, XXXXXXXXXXXXXXXXXXX.

The bug is that while you are allowed to and encouraged to declare
your podfiles' encoding, this is severely disrespected.

The following podpages show the problem​:

perlebcdic\.pod
perlgit\.pod
perlhist\.pod
perlpodspec\.pod
perlthrtut\.pod

perl588delta\.pod
perl5100delta\.pod
perl5120delta\.pod
perl5121delta\.pod
perl5122delta\.pod
perl5123delta\.pod
perl5124delta\.pod
perl5140delta\.pod
perl5141delta\.pod
perl5142delta\.pod
perl5150delta\.pod
perl5151delta\.pod
perl5152delta\.pod
perl5153delta\.pod
perl5154delta\.pod
perl5156delta\.pod
perl5157delta\.pod
perl5158delta\.pod

perlcn\.pod
perljp\.pod
perlko\.pod
perltw\.pod

--tom

Is this a problem that can be fixed in the 'pod2man' utility?

Or does it require corrections of individual .pod files?

Or both?

Or neither?

Thank you very much.
Jim Keenan

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @mrallen1

On Wed Sep 26 19​:07​:49 2012, jkeenan wrote​:

On Wed Feb 29 13​:04​:43 2012, tom christiansen wrote​:

On the systems I tested it on, the manpages whose pods contain non-
ASCII
are botched, some to the point of complete illegibility.

Is this a problem that can be fixed in the 'pod2man' utility?

Or does it require corrections of individual .pod files?

Or both?

Or neither?

Hi. I'm the current maintainer of Pod​::Perldoc. In 90% of the cases where the output from
pod2man is displayed as "X" instead of an proper glyph, it is typically groff itself which botches
the handling non-ASCII characters.

Getting clean non-ASCII characters typically requires a groff >= 1.20.1 and within the context
of perldoc (at least) a recent version (>=3.18). As far as pod2man is concerned, I believe it
handles non-ASCII characters as well as it can given the various points where encoding matters
to display which are (among others), the shell itself, perl, groff, the pager program, and
whatever terminal one is using.

So I would say this report is inconclusive. It may well be that pod2man is generating proper roff
output, and groff is mangling it when it tries to format and display it.

Mark

[p5pRT]

From @mrallen1

On Wed Sep 26 19​:07​:49 2012, jkeenan wrote​:

On Wed Feb 29 13​:04​:43 2012, tom christiansen wrote​:

On the systems I tested it on, the manpages whose pods contain non-
ASCII
are botched, some to the point of complete illegibility.

Is this a problem that can be fixed in the 'pod2man' utility?

Or does it require corrections of individual .pod files?

Or both?

Or neither?

Hi. I'm the current maintainer of Pod​::Perldoc. In 90% of the cases where the output from
pod2man is displayed as "X" instead of an proper glyph, it is typically groff itself which botches
the handling non-ASCII characters.

Getting clean non-ASCII characters typically requires a groff >= 1.20.1 and within the context
of perldoc (at least) a recent version (>=3.18). As far as pod2man is concerned, I believe it
handles non-ASCII characters as well as it can given the various points where encoding matters
to display which are (among others), the shell itself, perl, groff, the pager program, and
whatever terminal one is using.

So I would say this report is inconclusive. It may well be that pod2man is generating proper roff
output, and groff is mangling it when it tries to format and display it.

Mark

[khwilliamson]

I just ran the two test cases on Ubuntu 24.04, and they both worked fine.

I think that in the interval since this was filed, the underlying tools were fixed.

I think this is closable; I'm self-assigning this, and if no one complains by June 9, 2025, I will consider this closable, and when I remember to, will close it. If I forget, anyone else feel free to close it then.

[jkeenan]

I just ran the two test cases on Ubuntu 24.04, and they both worked fine.

I think that in the interval since this was filed, the underlying tools were fixed.

I think this is closable; I'm self-assigning this, and if no one complains by June 9, 2025, I will consider this closable, and when I remember to, will close it. If I forget, anyone else feel free to close it then.

I believe this ticket is almost closable ... but not quite.

If you're working on Ubuntu 24.04 (as I am) or, more precisely, with any perl 5.38 or newer, then you should get good results. That's because in the 5.37 development cycle we upgraded the podlators distribution, which includes Pod::Man, to CPAN version 5.00.

commit 20616d517ad4726a85ee27750db6e24443343601
Author:     Russ Allbery <eagle@eyrie.org>
AuthorDate: Tue Dec 6 22:40:57 2022 +0000
Commit:     James E Keenan <jkeenan@cpan.org>
CommitDate: Fri Dec 9 10:43:28 2022 -0500

    Update podlators to CPAN version 5.00

Investigation:

You referred to the interval between the filing of this ticket and the present, which made me wonder when the problem was fixed, i.e., a bisection problem. Since I have a number of perls installed via perlbrew on my laptop, I constructed commands to run the problematic cases described by Tom back in 2012. Here are excerpts from my results.

Ubuntu Linux 24.04 LTS

$ for f in `find /home/jkeenan/perl5/perlbrew/perls/ -type f -name '*perlhist.1' | xargs ls`; do echo $f;man $f | grep var; done
...
/home/jkeenan/perl5/perlbrew/perls/perl-5.36.3/man/man1/perlhist.1
       Golden, Florian Ragwitz, Tatsuhiko Miyagawa, Chris "BinGOs" Williams, Zefram, AEvar
       passed around various names (baton, token, hot potato) but none caught on.  Then, Chip
        AEvar     5.13.10       2011-Feb-20     
/home/jkeenan/perl5/perlbrew/perls/perl-5.38.0/man/man1/perlhist.1
       Golden, Florian Ragwitz, Tatsuhiko Miyagawa, Chris "BinGOs" Williams, Zefram, Ævar
       passed around various names (baton, token, hot potato) but none caught on.  Then, Chip
        Ævar     5.13.10       2011-Feb-20
...


$ for f in `find /home/jkeenan/perl5/perlbrew/perls/ -type f -name '*perlcn.1' | xargs ls`; do echo $f; head -n5 $f; done
...
/home/jkeenan/perl5/perlbrew/perls/perl-5.36.0/man/man1/perlcn.1
.\" Automatically generated by Pod::Man 4.14 (Pod::Simple 3.43)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
/home/jkeenan/perl5/perlbrew/perls/perl-5.36.3/man/man1/perlcn.1
.\" Automatically generated by Pod::Man 4.14 (Pod::Simple 3.43)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
/home/jkeenan/perl5/perlbrew/perls/perl-5.38.0/man/man1/perlcn.1
.\" -*- mode: troff; coding: utf-8 -*-
.\" Automatically generated by Pod::Man 5.01 (Pod::Simple 3.43)
.\"
.\" Standard preamble:
.\" ========================================================================

This suggested that the problem was corrected between 5.36 and 5.38. I tested the commit cited above and verified it was the corrective commit.

$ man /usr/home/jkeenan/testing/20616d517ad4726a85ee27750db6e24443343601^/man/man1/perlcn.1 | head -n 5
PERLCN(1)              Perl Programmers Reference Guide              PERLCN(1)


NAME
       perlcn - XXXX Perl XX

$ man /usr/home/jkeenan/testing/20616d517ad4726a85ee27750db6e24443343601/man/man1/perlcn.1 | head -n 5
PERLCN(1)              Perl Programmers Reference Guide              PERLCN(1)


NAME
       perlcn - 简体中文 Perl 指南

FreeBSD-14 (/usr/local/bin/perl: perl-5.36.3)

$ man perlhist | grep var
       Zefram, AEvar Arnfjoer` Bjarmason, Stevan Little, Dave Rolsky, Max
       David Croy.  We had passed around various names (baton, token, hot
        AEvar     5.13.10       2011-Feb-20

$ man perlcn | head -n 5
PERLCN(1)              Perl Programmers Reference Guide              PERLCN(1)

XXXXXXXXXXXXXXXXXX, XXXXXXXXXXXX.  XXXXXX POD (XXXXXX) XX; XXXXXXXXXXXXXX,
XXXXXX. XXXXXXXXXXX, XXX perlpod XXXX.

perl-5.38.1 built with man pages

$ man /usr/home/jkeenan/testing/v5.38.1-with_man/man/man1/perlhist.1 | grep var
       Zefram, Ævar Arnfjörð Bjarmason, Stevan Little, Dave Rolsky, Max
       David Croy.  We had passed around various names (baton, token, hot
        Ævar     5.13.10       2011-Feb-20

$ man /usr/home/jkeenan/testing/v5.38.1-with_man/man/man1/perlcn.1 | head -n 5
PERLCN(1)              Perl Programmers Reference Guide              PERLCN(1)


NAME
       perlcn - 简体中文 Perl 指南

So provided we're using perl-5.38 or newer on FreeBSD, this DWIMs.

OpenBSD-6.9 (/usr/bin/perl: perl-5.32.1)

$ man perlhist | grep var
       Zefram, AEvar Arnfjordh Bjarmason, Stevan Little, Dave Rolsky, Max
       David Croy.  We had passed around various names (baton, token, hot
        AEvar     5.13.10       2011-Feb-20

$ man perlcn | head -n 5
man: No entry for perlcn in the manual.

So I don't have all the data I want for OpenBSD. Furthermore, I was unable to build and install a post-5.38 perl with man pages on OpenBSD, which led to #23292. If I could the results I want on OpenBSD, then I would feel comfortable closing this 13-year-old ticket.

[jkeenan]

So I don't have all the data I want for OpenBSD. Furthermore, I was unable to build and install a post-5.38 perl with man pages on OpenBSD, which led to #23292. If I could the results I want on OpenBSD, then I would feel comfortable closing this 13-year-old ticket.

So the upshot of the discussion in #23292 (at least as I read it), is that I'm not going to be able to get that data from OpenBSD. We've identified what Perl has done (Russ Allbery's work on Pod::Man) to address the OP's complaint, and it appears that we've reached the limits of what Perl can do, I'm closing this ticket now.