sv_parser: Critical lexer/parser compilation failures

[InterLinked1]

This library no longer compiles with (recent?) versions of flex/bison:

Some missing declarations:

make  all-am
make[1]: Entering directory '/usr/src/libsieve'
  CC       src/sv_test/example.o
  CC       src/sv_interface/callbacks2.lo
  CC       src/sv_interface/context2.lo
  CC       src/sv_interface/message2.lo
  CC       src/sv_interface/message.lo
  CC       src/sv_interface/script2.lo
  CC       src/sv_interface/script.lo
  CC       src/sv_interface/tree.lo
  CC       src/sv_parser/addr.lo
In file included from src/sv_parser/addr.y:43:
src/sv_parser/addr.h:85:32: warning: ‘struct sieve2_context’ declared inside parameter list will not be visible outside of this definition or declaration
   85 | int libsieve_addrparse (struct sieve2_context *context, void *addr_scan);
      |                                ^~~~~~~~~~~~~~
In file included from src/sv_parser/addr.y:44:
src/sv_parser/addrinc.h:8:5: error: conflicting types for ‘libsieve_addrparse’
    8 | int libsieve_addrparse(struct sieve2_context *context, void *yyscanner);
      |     ^~~~~~~~~~~~~~~~~~
In file included from src/sv_parser/addr.y:43:
src/sv_parser/addr.h:85:5: note: previous declaration of ‘libsieve_addrparse’ was here
   85 | int libsieve_addrparse (struct sieve2_context *context, void *addr_scan);
      |     ^~~~~~~~~~~~~~~~~~
src/sv_parser/addr.c:68:25: error: conflicting types for ‘libsieve_addrparse’
   68 | #define yyparse         libsieve_addrparse
      |                         ^~~~~~~~~~~~~~~~~~
src/sv_parser/addr.c:990:1: note: in expansion of macro ‘yyparse’
  990 | yyparse (struct sieve2_context *context, void *addr_scan)
      | ^~~~~~~
In file included from src/sv_parser/addr.y:43:
src/sv_parser/addr.h:85:5: note: previous declaration of ‘libsieve_addrparse’ was here
   85 | int libsieve_addrparse (struct sieve2_context *context, void *addr_scan);
      |     ^~~~~~~~~~~~~~~~~~
src/sv_parser/addr.y: In function ‘libsieve_addr_parse_buffer’:
src/sv_parser/addr.y:186:5: error: unknown type name ‘YY_BUFFER_STATE’
  186 |     YY_BUFFER_STATE buf = libsieve_addr_scan_string((char*)*ptr, addr_scan);
      |     ^~~~~~~~~~~~~~~
src/sv_parser/addr.y:186:27: warning: implicit declaration of function ‘libsieve_addr_scan_string’ [-Wimplicit-function-declaration]
  186 |     YY_BUFFER_STATE buf = libsieve_addr_scan_string((char*)*ptr, addr_scan);
      |                           ^~~~~~~~~~~~~~~~~~~~~~~~~
src/sv_parser/addr.y:197:9: warning: implicit declaration of function ‘libsieve_addr_delete_buffer’; did you mean ‘libsieve_addr_parse_buffer’? [-Wimplicit-function-declaration]
  197 |         libsieve_addr_delete_buffer(buf, addr_scan);
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      |         libsieve_addr_parse_buffer
make[1]: *** [Makefile:687: src/sv_parser/addr.lo] Error 1
make[1]: Leaving directory '/usr/src/libsieve'
make: *** [Makefile:445: all] Error 2

The first issue is easily fixed by adding a forward declaration for struct sieve2_context, near the bottom of addr.h:

#include "src/sv_interface/context2.h"

I'm not sure why YY_BUFFER_STATE isn't defined here, other than I'd think that would need to be in a .l file, not a .y file. But nonetheless, this fixes that:

diff --git a/src/sv_parser/addr.y b/src/sv_parser/addr.y
index 87b306f..e38b3e4 100644
--- a/src/sv_parser/addr.y
+++ b/src/sv_parser/addr.y
@@ -173,6 +173,8 @@ void libsieve_addrerror(struct sieve2_context *context, void *yyscanner, const c
     libsieve_do_error_address(context, msg);
 }

+#define YY_BUFFER_STATE void*
+
 /* Wrapper for addrparse() which sets up the
  * required environment and allocates variables
  */

Then, for these warnings:

src/sv_parser/addr.y: In function ‘libsieve_addr_parse_buffer’:
src/sv_parser/addr.y:188:27: warning: implicit declaration of function ‘libsieve_addr_scan_string’ [-Wimplicit-function-declaration]
  188 |     YY_BUFFER_STATE buf = libsieve_addr_scan_string((char*)*ptr, addr_scan);
      |                           ^~~~~~~~~~~~~~~~~~~~~~~~~
src/sv_parser/addr.y:188:27: warning: initialization of ‘void *’ from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
src/sv_parser/addr.y:199:9: warning: implicit declaration of function ‘libsieve_addr_delete_buffer’; did you mean ‘libsieve_addr_parse_buffer’? [-Wimplicit-function-declaration]
  199 |         libsieve_addr_delete_buffer(buf, addr_scan);
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      |         libsieve_addr_parse_buffer

I'm not sure why context2.h can't be included from addr.y but when I try that it doesn't work. Only in addr.h, but every time addr.y is modified than that gets regenerated so I had to keep readding it.

With those changes, I'm able to get as far as:

make[1]: Entering directory '/usr/src/libsieve'
  CC       src/sv_parser/addr.lo
  CC       src/sv_parser/addr-lex.lo
  CC       src/sv_parser/header.lo
src/sv_parser/header.y:28:10: fatal error: header-lex.h: No such file or directory
   28 | #include "header-lex.h"
      |          ^~~~~~~~~~~~~~
compilation terminated.

I'm assuming header-lex.h should get generated from header-lex.l, but, well, it's not. Haven't been able to get that generated. I can force building header-lex.lo first:

echo "  CC      " src/sv_parser/header-lex.lo;depbase=`echo src/sv_parser/header-lex.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;/bin/bash ./libtool --silent --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I.    -Wall -I. -I./src/sv_include -I. -fvisibility=hidden -g -O2 -MT src/sv_parser/header-lex.lo -MD -MP -MF $depbase.Tpo -c -o src/sv_parser/header-lex.lo src/sv_parser/header-lex.c &&mv -f $depbase.Tpo $depbase.Plo

But that doesn't produce a header file as a byproduct.

In short, compilation of the lexer/parser stuff has really fallen apart and is a total mess right now. This doesn't compile on any system anymore. Any ideas for a fix?

[InterLinked1]

On, in case this is helpful, version info:

root@debian11:/usr/src/libsieve# flex -V
flex 2.6.4
root@debian11:/usr/src/libsieve# bison -V
bison (GNU Bison) 3.7.5
Written by Robert Corbett and Richard Stallman.

Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[sodabrew]

I haven't maintained this repo in quite some time, unfortunately. Can you tell me the use case you're working on?

[InterLinked1]

I haven't maintained this repo in quite some time, unfortunately. Can you tell me the use case you're working on?

Hi Aaron, thanks for the response! I was wondering if that was the case, I was kind of afraid of it.

The use case is fairly standard, I'm using this for implementing a Sieve backend in my mail server software[1]. The existing packages work okay, but there are some limitations for what I'd like to do, they are outdated so still contain fixed segfaults and possibly fixed memory leaks, and I'd most of all like to extend it by implementing more Sieve extensions, so that's why I'm trying to compile from source.

I'd be happy to do some work on extending this library if you no longer have time or interest in it, but I've never worked with flex or bison before and of course that's the part I haven't been able to get to compile.

[1] https://github.com/InterLinked1/lbbs/blob/master/modules/mod_sieve.c

[coeurtech]

I'm trying to get this working in the current Alpine Linux and believe I'm seeing the same or similar issues.

flex 2.6.4
bison (GNU Bison) 3.8.2

make[1]: Entering directory '/opt/src/libsieve'
CC src/sv_test/example.o
CC src/sv_interface/callbacks2.lo
CC src/sv_interface/context2.lo
CC src/sv_interface/message2.lo
CC src/sv_interface/message.lo
CC src/sv_interface/script2.lo
CC src/sv_interface/script.lo
CC src/sv_interface/tree.lo
CC src/sv_parser/addr.lo
In file included from src/sv_parser/addr.y:43:
src/sv_parser/addr.h:86:32: warning: 'struct sieve2_context' declared inside parameter list will not be visible outside of this definition or declaration
86 | int libsieve_addrparse (struct sieve2_context *context, void *addr_scan);
| ^~~~~~~~~~~~~~
In file included from src/sv_parser/addr.y:44:
src/sv_parser/addrinc.h:8:5: error: conflicting types for 'libsieve_addrparse'; have 'int(struct sieve2_context *, void *)'
8 | int libsieve_addrparse(struct sieve2_context *context, void *yyscanner);
| ^~~~~~~~~~~~~~~~~~
src/sv_parser/addr.h:86:5: note: previous declaration of 'libsieve_addrparse' with type 'int(struct sieve2_context *, void *)'
86 | int libsieve_addrparse (struct sieve2_context *context, void *addr_scan);
| ^~~~~~~~~~~~~~~~~~
src/sv_parser/addr.y:45:10: fatal error: addr-lex.h: No such file or directory
45 | #include "addr-lex.h"
| ^~~~~~~~~~~~
compilation terminated.

[coeurtech]

I manually generated the addr-lex.h, header-lex.h, and sieve-lex.h files by executing:

cd src/sv_parser

for i in addr header sieve
do
flex --header-file=${i}-lex.h ${i}-lex.l
done

This should be done as part of dist-hook but it looks like that's executed much later. Also flex has changed and the --header-file parameter needs to be used.

Also, for the bison files to update grammer

for i in addr.yheader.y sieve.y
do
bison--update ${i}
done

A note about bison...

* Noteworthy changes in release 3.8.1 (2021-09-11) [stable]

  The generation of prototypes for yylex and yyerror in Yacc mode is
  breaking existing grammar files.  To avoid breaking too many grammars, the
  prototypes are now generated when `-y/--yacc` is used *and* the
  `POSIXLY_CORRECT` environment variable is defined.

  Avoid using `-y`/`--yacc` simply to comply with Yacc's file name
  conventions, rather, use `-o y.tab.c`.  Autoconf's AC_PROG_YACC macro uses
  `-y`.  Avoid it if possible, for instance by using gnulib's gl_PROG_BISON.

which got me here:

localhost:/opt/src/libsieve# make
  YACC     src/sv_parser/addr.c
addr.tab.c is unchanged
addr.tab.h is unchanged
  YACC     src/sv_parser/header.c
header.tab.c is unchanged
header.tab.h is unchanged
  YACC     src/sv_parser/sieve.c
sieve.tab.c is unchanged
sieve.tab.h is unchanged
make  all-am
make[1]: Entering directory '/opt/src/libsieve'
  CC       src/sv_test/example.o
  CC       src/sv_interface/callbacks2.lo
  CC       src/sv_interface/context2.lo
  CC       src/sv_interface/message2.lo
  CC       src/sv_interface/message.lo
  CC       src/sv_interface/script2.lo
  CC       src/sv_interface/script.lo
  CC       src/sv_interface/tree.lo
  YACC     src/sv_parser/addr.c
addr.tab.c is unchanged
addr.tab.h is unchanged
  CC       src/sv_parser/addr.lo
src/sv_parser/addr.y:37: warning: "YYSTYPE" redefined
   37 | #define YYSTYPE char *
      |
src/sv_parser/addr.c:67: note: this is the location of the previous definition
   67 | #define YYSTYPE         LIBSIEVE_ADDRSTYPE
      |
In file included from src/sv_parser/addr.y:43:
src/sv_parser/addr.h:94:32: warning: 'struct sieve2_context' declared inside parameter list will not be visible outside of this definition or declaration
   94 | int libsieve_addrparse (struct sieve2_context *context, void *addr_scan);
      |                                ^~~~~~~~~~~~~~
In file included from src/sv_parser/addr.y:44:
src/sv_parser/addrinc.h:8:5: error: conflicting types for 'libsieve_addrparse'; have 'int(struct sieve2_context *, void *)'
    8 | int libsieve_addrparse(struct sieve2_context *context, void *yyscanner);
      |     ^~~~~~~~~~~~~~~~~~
src/sv_parser/addr.h:94:5: note: previous declaration of 'libsieve_addrparse' with type 'int(struct sieve2_context *, void *)'
   94 | int libsieve_addrparse (struct sieve2_context *context, void *addr_scan);
      |     ^~~~~~~~~~~~~~~~~~
src/sv_parser/addr.c:69:25: error: conflicting types for 'libsieve_addrparse'; have 'int(struct sieve2_context *, void *)'
   69 | #define yyparse         libsieve_addrparse
      |                         ^~~~~~~~~~~~~~~~~~
src/sv_parser/addr.c:991:1: note: in expansion of macro 'yyparse'
  991 | yyparse (struct sieve2_context *context, void *addr_scan)
      | ^~~~~~~
src/sv_parser/addr.h:94:5: note: previous declaration of 'libsieve_addrparse' with type 'int(struct sieve2_context *, void *)'
   94 | int libsieve_addrparse (struct sieve2_context *context, void *addr_scan);
      |     ^~~~~~~~~~~~~~~~~~
make[1]: *** [Makefile:685: src/sv_parser/addr.lo] Error 1
make[1]: Leaving directory '/opt/src/libsieve'
make: *** [Makefile:443: all] Error 2

[InterLinked1]

bison--update ${i}

I'm assuming this should be bison --update ${i}?

[InterLinked1]

One thing I did before that moves this along further is adding this before line 86 of src/sv_parser/addr.h:

struct sieve2_context;

There's probably some other header file that should be included somewhere but this shouldn't do any harm.
With that, I get to this point, where I'm a bit more lost with the exact usage of these yacc typedefs:

make  all-am
make[1]: Entering directory '/usr/local/src/libsieve'
  CC       src/sv_parser/addr.lo
  CC       src/sv_parser/addr-lex.lo
  CC       src/sv_parser/comparator.lo
  CC       src/sv_parser/header.lo
  CC       src/sv_parser/header-lex.lo
  CC       src/sv_parser/sieve.lo
  CC       src/sv_parser/sieve-lex.lo
In file included from src/sv_parser/sieve-lex.l:40:
src/sv_parser/sieveinc.h:9:40: error: unknown type name 'YYSTYPE'; did you mean 'YYSTATE'?
    9 | #define YY_DECL int libsieve_sievelex (YYSTYPE *yylval_param, struct sieve2_context *context, void *yyscanner)
      |                                        ^~~~~~~
src/sv_parser/sieve-lex.l:43:8: note: in expansion of macro 'YY_DECL'
   43 | extern YY_DECL;
      |        ^~~~~~~
src/sv_parser/sieve-lex.c:2606:5: error: unknown type name 'YYSTYPE'
 2606 |     YYSTYPE * yylval_r;
      |     ^~~~~~~
src/sv_parser/sieve-lex.c:2635:19: error: unknown type name 'YYSTYPE'; did you mean 'YYSTATE'?
 2635 | void yyset_lval ( YYSTYPE * yylval_param , yyscan_t yyscanner );
      |                   ^~~~~~~
      |                   YYSTATE
src/sv_parser/sieveinc.h:9:40: error: unknown type name 'YYSTYPE'; did you mean 'YYSTATE'?
    9 | #define YY_DECL int libsieve_sievelex (YYSTYPE *yylval_param, struct sieve2_context *context, void *yyscanner)
      |                                        ^~~~~~~
src/sv_parser/sieve-lex.c:2761:1: note: in expansion of macro 'YY_DECL'
 2761 | YY_DECL
      | ^~~~~~~
src/sv_parser/sieve-lex.c:3569:26: warning: 'yy_try_NUL_trans' defined but not used [-Wunused-function]
 3569 |     static yy_state_type yy_try_NUL_trans  (yy_state_type yy_current_state , yyscan_t yyscanner)
      |                          ^~~~~~~~~~~~~~~~
src/sv_parser/sieve-lex.c:3543:26: warning: 'yy_get_previous_state' defined but not used [-Wunused-function]
 3543 |     static yy_state_type yy_get_previous_state (yyscan_t yyscanner)
      |                          ^~~~~~~~~~~~~~~~~~~~~
src/sv_parser/sieve-lex.c:3404:12: warning: 'yy_get_next_buffer' defined but not used [-Wunused-function]
 3404 | static int yy_get_next_buffer (yyscan_t yyscanner)
      |            ^~~~~~~~~~~~~~~~~~
src/sv_parser/sieve-lex.c:2495:27: warning: 'yy_rule_can_match_eol' defined but not used [-Wunused-const-variable=]
 2495 | static const flex_int32_t yy_rule_can_match_eol[76] =
      |                           ^~~~~~~~~~~~~~~~~~~~~
make[1]: *** [Makefile:687: src/sv_parser/sieve-lex.lo] Error 1
make[1]: Leaving directory '/usr/local/src/libsieve'
make: *** [Makefile:445: all] Error 2

If I add a missing definition to this header file:

src/sv_parser/sieveinc.h:

+ #ifndef YYSTYPE
+ #define YYSTYPE char *
+ #endif

#undef YY_DECL

then I get down to this:

make  all-am
make[1]: Entering directory '/usr/local/src/libsieve'
  CC       src/sv_parser/sieve-lex.lo
src/sv_parser/sieve-lex.l: In function 'libsieve_sievelex':
src/sv_parser/sieve-lex.l:67:36: error: request for member 'sval' in something not a structure or union
   67 |                           (*yylval).sval = libsieve_catbuf_free(yyextra);
      |                                    ^
src/sv_parser/sieve-lex.l:81:12: error: request for member 'sval' in something not a structure or union
   81 | <QSTRING>([^\"]|\\\"|{CRLF})+ { (*yylval).sval = libsieve_strndup(yytext, yyleng);
      |            ^
src/sv_parser/sieve-lex.l:90:12: error: request for member 'sval' in something not a structure or union
   90 | <INITIAL>\"\"           { (*yylval).sval = libsieve_strdup("");
      |            ^
src/sv_parser/sieve-lex.l:93:12: error: request for member 'nval' in something not a structure or union
   93 | <INITIAL>[0-9]+[KMG]?   { (*yylval).nval = libsieve_strtonum(yytext);
      |            ^
make[1]: *** [Makefile:687: src/sv_parser/sieve-lex.lo] Error 1
make[1]: Leaving directory '/usr/local/src/libsieve'
make: *** [Makefile:445: all] Error 2

It's possible I might have caused this with some change, though I'm not entirely sure how to work around this yet.

@sodabrew Any thoughts on any of this, even if you're not actively maintaining the library? We would really appreciate your thoughts. It will be a lot easier to get this library working again then start writing a whole new library from scratch.

[InterLinked1]

@sodabrew Just wanted to reach out again to see if you had any thoughts. Once the build issues are resolved, we can make changes in a separate fork, but these issues have been kind of elusive thus far.