Is there a way to mix a LISP's macro &optional and &key params?

Question

I want to define a LISP macro like dolist that lets me define an optional output argument. In the following case study, this macro will be called doread. It will read lines from a file and return the number of lines found that way.

(let ((lines 0))
  (doread (line file lines)
     ;; do something with line
     (incf lines)))

The problem is that getting that lines to work in the above macro

I can do what I want with &key , but not with &optional &key (and the &key is needed since I want to control how a file is read; e.g with read or read-line or whatever).

Now the following works BUT to works the wrong way. Here the out argument has to be a &key and not a &optional:

;; this way works... 

(defmacro doread ((it f  &key out (take #'read)) &body body)
  "Iterator for running over files or strings."
  (let ((str (gensym)))
    `(with-open-file (,str f)
       (loop for ,it = (funcall ,take ,str nil)
             while ,it do
             (progn ,@body))
       ,out)))

;; lets me define something that reads first line of a file
(defun para1 (f)
  "Read everything up to first blank line."
  (with-output-to-string (s)
    (doread (x f :take #'read-line)
      (if (equalp "" (string-trim '(#\Space #\Tab) x))
        (return)
        (format s "~a~%" x)))))

(print (para1 sometime)) ; ==> shows all up to first blank line

What I'd like to do is this is the following (note that out has now moved into &optional:

(defmacro doread ((it f &optional out &key   (take #'read)) &body body)
  "Iterator for running over files or strings."
  (let ((str (gensym)))
    `(with-open-file (,str f)
       (loop for ,it = (funcall ,take ,str nil)
             while ,it do
             (progn ,@body))
       ,out)))

and if that worked, I could do something like.

(defun para1 (f)
  "Print  everything up to first blank line. 
   Return the lines found in that way"
  (let ((lines 0))
      (doread (x f lines :take #'read-line)
        (if (equalp "" (string-trim '(#\Space #\Tab) x))
            (return)
            (and (incf lines) (format t "~a~%" x)))))

but it I use &optional out I get

 loading /Users/timm/gits/timm/lisp/src/lib/macros.lisp
*** - GETF: the property list (#'READ-LINE) has an odd length

Answer 1

You cannot mix &optional and &key and expect to be able to pass only the keyword arguments. You can however define a syntax that allow for an optional list of arguments associated with the source.

For example:

(defpackage :so (:use :cl :alexandria))
(in-package :so)

(defmacro do-read ((entry source &optional result) &body body)
  (destructuring-bind (source &key (take '#'read)) (ensure-list source)
    (once-only (take)
      `(loop
          :with ,entry
          :do (setf ,entry (handler-case (funcall ,take ,source)
                             (end-of-file () (loop-finish))))
            (progn ,@body)
          :finally (return ,result)))))

The syntax for DO-READ could be written as:

(DO-READ (ENTRY [SOURCE|(SOURCE &KEY TAKE)] &OPTIONAL RESULT) . BODY)

This is not an unusual syntax w.r.t. standard Lisp forms (see LET, keyword synax in lambda-lists, defstruct, etc.). You could add more keyword parameters along with TAKE.

Remarks

• In macros, I prefer to emit LOOP keywords as keywords, not symbols in the macro's definition package; otherwise, when macroexpanding the code, you are likely to get the symbols prefixed by the macro's package (i.e. SO::WITH instead of :WITH), which becomes quickly unreadable.

• Returning NIL from READ-LINE is fine, but not from READ, as NIL could be a successfully read value. In general, since TAKE is provided by the user, you don't know if NIL is an acceptable result or not. That's why I catch END-OF-FILE instead. In case you want to read from other sources you may also check a secondary return value, or document that they signal a condition too.

• The ENTRY variable's scope is extended so that RESULT can be ENTRY itself; in your case, OUT could not be equal to IT, because once you exit the loop, you don't have access to it anymore. This is a minor point, but that can be useful.

• I did not include WITH-OPEN-FILE, in case you want to read from something else than files (streams).

• #'READ is quoted, this is not important here but a good habit to have in macros, so that you actually evalute things at evaluation time, not at macroexpansion time.

Examples

(with-input-from-string (in "abcdef")
  (do-read (char (in :take #'read-char) char)
    (print char)))

Print all characters and return #\f.

(with-input-from-string (in (format nil "~{~a~%~}" *features*))
  (let ((lines 0))
    (do-read (line in lines)
      (incf lines))))

Print the number of lines in a string.