Handling end of file

Question

I am trying to write a reentrant C++ parser with bison, I followed the Complete C++ Example and I am using bison 3.5.1 and flex 2.6.4 (the versions available via apt-get in ubuntu buster).

After completing I had an issue in the lexer ‘make_YYEOF’ is not a member of ‘yy::parser’

My solution to that was to declare %token YYEOF

Then it compiles but the parser gives the syntax error, unexpected YYEOF, expecting $end, for an apparently valid input.

Then I made the simplified rule unit: IDENTIFIER YYEOF but again, the parser reports the same error.

/*
The grammar file parser.yy starts by asking for the C++ deterministic
parser skeleton, the creation of the parser header file.
Because the C++ skeleton changed several times, it is safer to require 
the version you designed the grammar for.
*/
// %skeleton "lalr1.cc" // -*- C++ -*-
%require "3.0"
%language "c++"
// %header

/* Because our scanner returns only genuine tokens and never simple characters
(e.g., it returns ‘PLUS’, not ‘'+'’), we can avoid conversions. */
%define api.token.raw

/* This example uses genuine C++ objects as semantic values, therefore,
we require the variant-based storage of semantic values. To make sure 
we properly use it, we enable assertions. To fully benefit from type-safety
and more natural definition of “symbol”, we enable api.token.constructor. */

%define api.token.constructor
%define api.value.type variant
%define parse.assert

/* Then come the declarations/inclusions needed by the semantic values.
Because the parser uses the parsing driver and reciprocally, both would
like to include the header of the other, which is, of course, insane.
This mutual dependency will be broken using forward declarations.
Because the driver’s header needs detailed knowledge about the parser class
(in particular its inner types), it is the parser’s header which will use 
a forward declaration of the driver. 
See https://www.gnu.org/software/bison/manual/html_node/_0025code-Summary.html. 

Code qualifies for C/C++: requires, provides, top.
Unqualified code will be equivalent to %{ %} for most purposes */

%code requires {
  # include <string>
  class driver;
}

/* The driver is passed by reference to the parser and to the scanner.
This provides a simple but effective pure interface, not relying on global variables.
*/
%param { driver& drv }

%locations

/* Use the following two directives to enable parser tracing and detailed error messages.
However, detailed error messages can contain incorrect information if lookahead correction 
is not enabled (see LAC https://www.gnu.org/software/bison/manual/html_node/LAC.html). */

%define parse.trace
%define parse.error verbose 
%define parse.lac full // none or full

/** The code between ‘%code {’ and ‘}’ is output in the *.cc file;
it needs detailed knowledge about the driver. */
%code {
# include "driver.hh"
}

/* User friendly names are provided for each symbol. 
To avoid name clashes in the generated files (see Calc++ Scanner),
prefix tokens with TOK_ */

%define api.token.prefix {TOK_}
%token YYEOF;

/* Since we use variant-based semantic values, %union is not used, 
and %token, %nterm and %type expect genuine types, not type tags. */
%token <std::string> IDENTIFIER "identifier"

/* No %destructor is needed to enable memory deallocation during error recovery; 
the memory, for strings for instance, will be reclaimed by the regular destructors.
All the values are printed using their operator<< (see Printing Semantic Values
https://www.gnu.org/software/bison/manual/html_node/Printer-Decl.html).
*/
%printer { yyo << $$; } <*>;

/* The grammar itself is straightforward (see Location Tracking Calculator: ltcalc
https://www.gnu.org/software/bison/manual/html_node/Location-Tracking-Calc.html). */


%%
%start unit;
unit: IDENTIFIER YYEOF // assignments exp { drv.result = $2; };

%%

/* Finally the error member function reports the errors. */
void
yy::parser::error (const location_type& l, const std::string& m)
{
  std::cerr << l << ": " << m << '\n';
}

driver.cc

#include "driver.hh"
#include "parser.hh"

driver::driver ()
  : trace_parsing (false), trace_scanning (false){}

int driver::parse (const std::string &f)
{
  file = f;
  location.initialize (&file);
  scan_begin ();
  yy::parser parse (*this);
  parse.set_debug_level (trace_parsing);
  int res = parse ();
  scan_end ();
  return res;
}

driver.hh

/*
To support a pure interface with the parser (and the scanner) the technique of the
"parsing context" is convenient: a structure containing all the data to exchange.
Since, in addition to simply launch the parsing, there are several auxiliary tasks
to execute (open the file for scanning, instantiate the parser etc.), we recommend
transforming the simple parsing context structure into a fully blown parsing driver class.

The declaration of this driver class, in driver.hh, is as follows. The first part includes
the CPP guard and imports the required standard library components, 
and the declaration of the parser class. 
*/
#ifndef DRIVER_HH
# define DRIVER_HH
# include <string>
# include <map>
# include "parser.hh"

/*
Then comes the declaration of the scanning function. Flex expects the signature
of yylex to be defined in the macro YY_DECL, and the C++ parser expects it to be
declared. We can factor both as follows.
*/

// Give Flex the prototype of yylex we want ...
# define YY_DECL \
  yy::parser::symbol_type yylex (driver& drv)
// ... and declare it for the parser's sake.
YY_DECL;
/* The driver class is then declared with its most obvious members. */
// Conducting the whole scanning and parsing of Calc++.
class driver
{
public:
  driver ();

  std::map<std::string, int> variables;

  int result;

  // Run the parser on file F.  Return 0 on success.
  int parse (const std::string& f);
  // The name of the file being parsed.
  std::string file;
  // Whether to generate parser debug traces.
  bool trace_parsing;
  // Handling the scanner.
  void scan_begin ();
  void scan_end ();
  // Whether to generate scanner debug traces.
  bool trace_scanning;
  // The token's location used by the scanner.
  yy::location location;
};

#endif // ! DRIVER_HH

id.cc

#include <iostream>
#include "driver.hh"

int main (int argc, char *argv[])
{
  int res = 0;
  driver drv;
  for (int i = 1; i < argc; ++i)
    if (argv[i] == std::string ("-p"))
      drv.trace_parsing = true;
    else if (argv[i] == std::string ("-s"))
      drv.trace_scanning = true;
    else if (!drv.parse (argv[i]))
      std::cout << drv.result << '\n';
    else
      res = 1;
  return res;
}

bison -d parser.yy -o parser.cc
flex -o lex.yy.cc lexer.l
g++ -std=c++11 *.cc -o parser
echo b | ./parser -p -s  -

That gives me

Starting parse
Entering state 0
Reading a token: --(end of buffer or a NUL)
--accepting rule at line 66 ("b")
Next token is token "identifier" (-:1.1: b)
Shifting token "identifier" (-:1.1: b)
Entering state 1
Reading a token: --(end of buffer or a NUL)
--accepting rule at line 65 ("
")
--(end of buffer or a NUL)
--EOF (start condition 0)
Next token is token YYEOF (-:2.1: )
Shifting token YYEOF (-:2.1: )
Entering state 3
Reducing stack by rule 1 (line 86):
   $1 = token "identifier" (-:1.1: b)
   $2 = token YYEOF (-:2.1: )
-> $$ = nterm unit (-:1.1-2.0: )
Stack now 0
Entering state 2
Reading a token: --(end of buffer or a NUL)
--EOF (start condition 0)
Next token is token YYEOF (-:2.1: )
LAC: initial context established for YYEOF
LAC: checking lookahead YYEOF: Err
LAC: checking lookahead $end: S4
LAC: checking lookahead YYEOF: Err
LAC: checking lookahead "identifier": Err
-:2.1: syntax error, unexpected YYEOF, expecting $end
Error: popping nterm unit (-:1.1-2.0: )
Stack now 0
Cleanup: discarding lookahead token YYEOF (-:2.1: )

What do I have to change to handle the end of input gracefully?

Thank you

Answer 1

If you use bison-3.5.1, you need to also use the documentation for bison-3.5.1. You're trying to apply the documentation for the most recent bison version (now 3.8.1), and the C++ interface has changed in some details, one of which is how you signal an end-of-input token.

In bison-3.5.1, it was necessary to give the end-of-input token a name, using an explicit declaration of token number 0, like:

%token YYEOF 0

(In the actual calc++ example from the bison 3.5.1, they used the name END, not YYEOF. And in other examples, other names.)

Once you declared the token name, Bison would generate an appropriate make_ member function.

Sometime in early 2020, YYEOF was introduced as a built-in name for the end-of-input token. Since then it has not been necessary to provide an explicit %token declaration, and make_YYEOF is automatically generated. So the example program in the current version doesn't include that declaration.

You might want to consider updating your bison installation. If not, you should install the package bison-doc which will include some examples (in /usr/share/doc/bison-doc/examples) and the various documentation formats (pdf, html, and info) for the same version of bison as you have installed. Once you've done that, you should be able to look at the documentation locally by using the command info bison or by pointing a browser at file:///usr/share/doc/bison-doc/html/index.html.