is there a way to perform a look-ahead with fslex/Lexing.LexBuffer

Question

I'm using fslex and in one of my tokenizer rules I need to look for the next char so I can decide whether to continue consuming tokens or to stop, unwind and let the previous rule consume it.

I've spent some checking its source code, and looks like the only way would be to compile a version that exposes some internal methods. Maybe someone here has a better solution?

Answer 1

Maybe looking at the source code of LexBuffer can help. Although there are properties (BufferScanStart and BufferScanLength) to set the position of the lexer in the input stream, those are internal. I'm not sure what would be required to expose those in a safe way.

There is also a question on HubFS which deals with putting back. The conclusion seems to be the same: fslex does not currently support it.