                 The Foreign Function Call Facility
                 ==================================

A foreign function description is written as a Lisp file,
and when compiled it produces a .c file which is then compiled
by the C compiler and may be linked together with lisp.a.

All symbols relating to the foreign function interface are exported from
the package FFI. To use them, (USE-PACKAGE "FFI").

Special FFI forms may appear anywhere in the Lisp file.

                                Overview
                                --------

These are the special FFI forms. We have taken a pragmatic approach:
the only foreign languages we support for now are C and ANSI C.

(DEF-C-TYPE name <c-type>)

(DEF-C-VAR name {option}*)
  option ::=
      (:name <c-name>)
    | (:type <c-type>)
    | (:read-only <boolean>)
    | (:alloc <allocation>)

(DEF-CALL-OUT name {option}*)
  option ::=
      (:name <c-name>)
    | (:arguments {(arg-name <c-type> [<param-mode> [<allocation>]])}*)
    | (:return-type <c-type> [<allocation>])
    | (:language <language>)

(DEF-CALL-IN name {option}*)
  option ::=
      (:name <c-name>)
    | (:arguments {(arg-name <c-type> [<param-mode> [<allocation>]])}*)
    | (:return-type <c-type> [<allocation>])
    | (:language <language>)

(DEF-C-CALL-OUT name {option}*)
  option ::=
      (:name <c-name>)
    | (:arguments {(arg-name <c-type> [<param-mode> [<allocation>]])}*)
    | (:return-type <c-type> [<allocation>])

(DEF-C-CALL-IN name {option}*)
  option ::=
      (:name <c-name>)
    | (:arguments {(arg-name <c-type> [<param-mode> [<allocation>]])}*)
    | (:return-type <c-type> [<allocation>])

(DEF-C-STRUCT name (<ident> <c-type>)*)

(DEF-C-ENUM name {<ident> | (<ident> [<value>])}*)

(C-LINES format-string {argument}*)

(ELEMENT c-place {index}*)
(DEREF c-place)
(SLOT c-place slot-name)
(CAST c-place <c-type>)

(TYPEOF c-place)
(SIZEOF c-place), (SIZEOF <c-type>)
(BITSIZEOF c-place), (BITSIZEOF <c-type>)

(VALIDP foreign-entity)

name is any Lisp symbol.

<c-name> is a string.

                       (Foreign) C types
                       -----------------

Foreign C types are used in the FFI. They are *not* regular Common Lisp
types or CLOS classes.

A <c-type> is either a predefined C type or the name of a type defined by
DEF-C-TYPE.

The simple C types are these:

 Lisp name     Lisp equiv           C equiv        ILU equiv
  nil           NIL                  void                             (o)
  boolean       (MEMBER NIL T)       int            BOOLEAN
  character     STRING-CHAR          char           SHORT CHARACTER
  char          INTEGER              signed char
  uchar         INTEGER              unsigned char
  short         INTEGER              short
  ushort        INTEGER              unsigned short
  int           INTEGER              int
  uint          INTEGER              unsigned int
  long          INTEGER              long
  ulong         INTEGER              unsigned long
  uint8         (UNSIGNED-BYTE 8)    uint8          BYTE
  sint8         (SIGNED-BYTE 8)      sint8
  uint16        (UNSIGNED-BYTE 16)   uint16         SHORT CARDINAL
  sint16        (SIGNED-BYTE 16)     sint16         SHORT INTEGER
  uint32        (UNSIGNED-BYTE 32)   uint32         CARDINAL
  sint32        (SIGNED-BYTE 32)     sint32         INTEGER
  uint64        (UNSIGNED-BYTE 64)   uint64         LONG CARDINAL     (*)
  sint64        (SIGNED-BYTE 64)     sint64         LONG INTEGER      (*)
  single-float  SINGLE-FLOAT         float
  double-float  DOUBLE-FLOAT         double
(o) as a result type only.
(*) does not work on all platforms.

The predefined C types are:

  c-type ::=
      <simple-c-type>
    | C-POINTER
    | C-STRING
    | (C-STRUCT <class> (<ident> <c-type>)*)
    | (C-UNION (<ident> <c-type>)*)
    | (C-ARRAY <c-type> dimensions)
        dimensions ::= number | ({number}*)
    | (C-ARRAY-MAX <c-type> maxdimension)
        maxdimension ::= number
    | (C-FUNCTION {option}*)
        option ::=
            (:arguments {(arg-name <c-type> [<param-mode> [<allocation>]])}*)
          | (:return-type <c-type> [<allocation>])
          | (:language <language>)
    | (C-PTR <c-type>)
    | (C-PTR-NULL <c-type>)
    | (C-ARRAY-PTR <c-type>)

(DEF-C-TYPE name <c-type>)
makes name a shortcut for <c-type>. Note that <c-type> may already refer
to name. Forward declarations of types are not possible, however.

The type C-POINTER corresponds to what C calls "void*", an opaque pointer.

The type C-STRING corresponds to what C calls "char*", a zero-terminated
string. Its Lisp equivalent is a string, without the trailing zero character.

The type (C-STRUCT class (ident1 type1) ... (ident2 type2)) is equivalent to
what C calls "struct { type1 ident1; ...; type2 ident2; }". Its Lisp
equivalent is: if class is VECTOR, a simple-vector; if class is LIST, a list;
if class is a symbol naming a structure or CLOS class: an instance of this
class, with slots of names ident1,...,ident2.

The type (C-UNION (ident1 type1) ... (ident2 type2)) is equivalent to what C
calls "union { type1 ident1; ...; type2 ident2; }". Conversion to and from
Lisp assumes that a value is to be viewed as being of type1.

The type (C-ARRAY type dim1 ... dim2) is equivalent to what C calls
"type [dim1]...[dim2]". Note that when an array is passed as an argument to
a function in C, it is actually passed as a pointer; you therefore have to
write (C-PTR (C-ARRAY ...)) for this argument's type.

The type (C-ARRAY-MAX type maxdim) is equivalent to what C calls
"type [maxdim]", an array containing up to maxdim elements. The array is
zero-terminated if it contains less than maxdim elements. Conversion from Lisp
of an array with more than maxdim elements silently ignores the superfluous
elements.

The type (C-PTR type) is equivalent to what C calls "type *": a pointer to
a single item of the given type.

The type (C-PTR-NULL type) is also equivalent to what C calls "type *": a
pointer to a single item of the given type, with the exception that C NULL
corresponds to Lisp NIL.

The type (C-ARRAY-PTR type) is equivalent to what C calls "type (*)[]":
a pointer to a zero-terminated array of items of the given type.

The type (C-FUNCTION (:return-type rtype) (:arguments (arg1 type1 ...) ...))
designates a C function that can be called according to the given prototype
(rtype (*) (type1, ...)).
The <language> is either :C (denotes K&R C) or :STDC (denotes ANSI C) or
:STDC-STDCALL (denotes ANSI C with `stdcall' calling convention). It
specifies whether the C function has been compiled by a K&R C compiler or by
an ANSI C compiler, and possibly the calling convention.
Conversion between C functions and Lisp functions is transparent.

(DEF-C-STRUCT <name> (<ident> <c-type>)*) defines <name> to be both a
DEFSTRUCT structure type and a foreign C type with the given slots.

(DEF-C-ENUM <name> {<ident> | (<ident> [<value>])}*) defines <ident>s as
constants, similarly to the C declaration  enum { <ident> [= <value>], ... };

(C-LINES format-string {argument}*)
outputs the string (FORMAT nil format-string {argument}*) to the C output
file. This is a rarely needed low-level facility.

The form (SIZEOF <c-type>) returns the size and alignment of a C type,
measured in bytes.

The form (BITSIZEOF <c-type>) returns the size and alignment of a C type,
measured in bits.

The predicate (VALIDP foreign-entity) returns NIL if the foreign-entity
(e.g. the Lisp equivalent of a C-POINTER) refers to a pointer which is
invalid because it comes from a previous Lisp session. It returns T if
foreign-entity can be used within the current Lisp process.

                       Foreign variables
                       -----------------

Foreign variables are variables whose storage is allocated in the foreign
language module. They can nevertheless be evaluated and modified through SETQ,
just as normal variables can, except that the range of allowed values is
limited according to the variable's foreign type. Note that for a foreign
variable X the form (EQL X X) is not necessarily true, since every time X is
evaluated its foreign value is converted to a freshly created Lisp value.

(DEF-C-VAR name {option}*)
  option ::=
      (:name <c-name>)
    | (:type <c-type>)
    | (:read-only <boolean>)
    | (:alloc <allocation>)

defines a foreign variable. `name' is the Lisp name, a regular Lisp symbol.

The :name option specifies the name, as seen from C, as a string. If not
specified, it is derived from the print name of the Lisp name.

The :type option specifies the variable's foreign type.

If the :read-only option is specified and non-NIL, it will be impossible
to change the variable's value from within Lisp (using SETQ or similar).

The :alloc option can be either :NONE or :MALLOC-FREE and defaults to :NONE.
If it is :MALLOC-FREE, any values of type C-STRING, (C-PTR ...),
(C-PTR-NULL ...), (C-ARRAY-PTR ...) within the foreign value are assumed to
be pointers to malloc()-allocated storage, and when SETQ replaces an old
value by a new one, the old storage is freed using free() and the new storage
allocated using malloc(). If it is :NONE, SETQ assumes that the pointers
point to good storage (not NULL!) and overwrites the old values by the new
ones. This is dangerous (just think of overwriting a string with a longer one
or storing some data in a NULL pointer...) and deprecated.

                   Operations on foreign places
                   ----------------------------

A foreign variable `name' defined by DEF-C-VAR defines a "place", i.e.
a form which can also be used as argument to SETF. (An "lvalue" in C
terminology.) The following operations are available on foreign places:

(ELEMENT place index1 ... indexn)
Array element: If place is of foreign type (C-ARRAY <c-type> dim1 ... dimn)
and 0 <= index1 < dim1, ..., 0 <= indexn < dimn, this will be the place
corresponding to (aref place index1 ... indexn) or place[index1]...[indexn].
It is a place of type <c-type>.
If place is of foreign type (C-ARRAY-MAX <c-type> dim) and 0 <= index < dim,
this will be the place corresponding to (aref place index) or place[index].
It is a place of type <c-type>.

(DEREF place)
Dereference pointer: If place is of foreign type (C-PTR <c-type>) or
(C-PTR-NULL <c-type>), this will be the place the pointer points to. It is a
place of type <c-type>. For (C-PTR-NULL <c-type>), the place may not be NULL.

(SLOT place slot-name)
Struct or union component: If place is of foreign type
(C-STRUCT <class> ... (slot-name <c-type>) ...) or of type
(C-UNION ... (slot-name <c-type>) ...), this will be of type <c-type>.

(CAST place <c-type>)
Type change: A place denoting the same memory locations as the original place,
but of type <c-type>.

(TYPEOF place)
returns the <c-type> corresponding to the place.

(SIZEOF place) returns the size and alignment of the C type of place,
measured in bytes.

(BITSIZEOF place) returns the size and alignment of the C type of place,
measured in bits.

                       Foreign functions
                       -----------------

Foreign functions are functions which are defined in the foreign language.
There are named foreign functions (imported via DEF-CALL-OUT or created via
DEF-CALL-IN) and anonymous foreign functions; they arise through conversion
of function pointers.

A "call-out" function is a foreign function called from Lisp: control flow
temporarily leaves Lisp.
A "call-in" function is a Lisp function called from the foreign language:
control flow temporary enters Lisp.

(DEF-CALL-OUT name {option}*)
  option ::=
      (:name <c-name>)
    | (:arguments {(arg-name <c-type> [<param-mode> [<allocation>]])}*)
    | (:return-type <c-type> [<allocation>])
    | (:language <language>)

defines a named call-out function. Any Lisp function call to #'name is
redirected to call the C function <c-name>.

DEF-C-CALL-OUT is equivalent to DEF-CALL-OUT with :LANGUAGE :C.

(DEF-CALL-IN name {option}*)
  option ::=
      (:name <c-name>)
    | (:arguments {(arg-name <c-type> [<param-mode> [<allocation>]])}*)
    | (:return-type <c-type> [<allocation>])
    | (:language <language>)

defines a named call-in function. Any C function call to the C function
<c-name> is redirected to call the Lisp function #'name.

DEF-C-CALL-IN is equivalent to DEF-CALL-IN with :LANGUAGE :C.

              Argument and result passing conventions
              ---------------------------------------

When passed to and from functions, allocation of arguments and results is
handled as follows:

Values of <simple-c-type>, C-POINTER are passed on the stack, with dynamic
extent. The <allocation> is effectively ignored.

Values of type C-STRING, (C-PTR ...), (C-PTR-NULL ...), (C-ARRAY-PTR ...) need
storage. The <allocation> specifies the allocation policy:
  <allocation> is :NONE          means that no storage is allocated.
  <allocation> is :ALLOCA        means allocation of storage on the stack,
                                       which has dynamic extent.
  <allocation> is :MALLOC-FREE   means that storage will be allocated
                                       via malloc() and freed via free().
If no <allocation> is specified, the default <allocation> is :NONE for most
types, but :ALLOCA for C-STRING and (C-PTR ...) and (C-PTR-NULL ...) and
(C-ARRAY-PTR ...) and for :OUT arguments. [Subject to change!]
The :MALLOC-FREE policy provides the ability to pass arbitrarily nested
structs containing pointers pointing to structs ... within a single conversion.

For call-out functions:
  For arguments passed from Lisp to C:
    If <allocation> is :MALLOC-FREE,
       Lisp allocates the storage using malloc() and never deallocates it.
       The C function is supposed to call free() when done with it.
    If <allocation> is :ALLOCA,
       Lisp allocates the storage on the stack, with dynamic extent. It is
       freed when the C function returns.
    If <allocation> is :NONE,
       Lisp assumes that the pointer already points to a valid area of the
       proper size and puts the result value there. This is dangerous! and
       deprecated.
  For results passed from C to Lisp:
    If <allocation> is :MALLOC-FREE,
       Lisp calls free() on it when done.
    If <allocation> is :NONE,
       Lisp does nothing.
For call-in functions:
  For arguments passed from C to Lisp:
    If <allocation> is :MALLOC-FREE,
       Lisp calls free() on it when done.
    If <allocation> is :ALLOCA or :NONE,
       Lisp does nothing.
  For results passed from Lisp to C:
    If <allocation> is :MALLOC-FREE,
       Lisp allocates the storage using malloc() and never deallocates it.
       The C function is supposed to call free() when done with it.
    If <allocation> is :NONE,
       Lisp assumes that the pointer already points to a valid area of the
       proper size and puts the result value there. This is dangerous! and
       deprecated.

A function parameter's <param-mode> may be
either :IN (means: read-only):
       The caller passes information to the callee.
or     :OUT (means: write-only):
       The callee passes information back to the caller on return.
       When viewed as a Lisp function, there is no Lisp argument corresponding
       to this, instead it means an additional return value.
or     :IN-OUT (means: read-write):
       Information is passed from the caller to the callee and then back to
       the caller. When viewed as a Lisp function, the ":OUT" value is
       returned as an additional multiple value.
The default is :IN.

[Currently, only :IN is fully implemented. :OUT works only with
<allocation> = :ALLOCA.]

On AmigaOS, <allocation> may not be :MALLOC-FREE because there is no commonly
used malloc()/free() library function.

On AmigaOS, the <allocation> may be followed by a register specification,
any of the symbols :D0, :D1, :D2, :D3, :D4, :D5, :D6, :D7, :A0, :A1, :A2,
:A3, :A4, :A5, :A6, each representing one 680x0 register. This works only
for integral types: integers, pointers, C-STRING, C-FUNCTION.

Passing C-STRUCT, C-UNION, C-ARRAY, C-ARRAY-MAX values as arguments (not via
pointers) is only possible to the extent the C compiler supports it. Most C
compilers do it right, but some C compilers (such as gcc on hppa) have
problems with this.

                           Examples
                           --------

Ex. 1: The C declaration

       struct foo {
           int a;
           struct foo * b[100];
       };

corresponds to

       (def-c-struct foo
         (a int)
         (b (c-array (c-ptr foo) 100))
       )

The element access

       struct foo f;
       f.b[7].a

corresponds to

       (declare (type foo f))
       (foo-a (aref (foo-b f) 7)) or (slot-value (aref (slot-value f 'b) 7) 'a)

Ex. 2: Here is an example of an external C variable and some accesses:

       struct bar {
           short x, y;
           char a, b;
           int z;
           struct bar * n;
       };

       extern struct bar * my_struct;

       my_struct->x++;
       my_struct->a = 5;
       my_struct = my_struct->n;

corresponds to

       (def-c-struct bar
         (x short)
         (y short)
         (a char)
         (b char) ; or (b character) if it represents a character, not a number
         (z int)
         (n (c-ptr bar))
       )

       (def-c-var my_struct (:type (c-ptr bar)))

       (setq my_struct (let ((s my_struct)) (incf (slot-value s 'x)) s))
       or (incf (slot my_struct 'x))
       (setq my_struct (let ((s my_struct)) (setf (slot-value s 'a) 5) s))
       or (setf (slot my_struct 'a) 5)
       (setq my_struct (slot-value my_struct 'n))
       or (setq my_struct (deref (slot my_struct 'n)))

Ex. 3: An example for calling an external function:
On ANSI C systems, <stdlib.h> contains the declarations

       typedef struct {
         int quot;   /* Quotient */
         int rem;    /* Remainder */
       } div_t;
       extern div_t div (int numer, int denom);

This translates to

       (def-c-struct div_t
         (quot int)
         (rem int)
       )
       (def-c-call-out div (:arguments (numer int) (denom int))
                           (:return-type div_t)
       )

Sample call from within Lisp:

       > (div 20 3)
       #S(DIV :QUOT 6 :REM 2)

Ex. 4: Another example for calling an external function:

Suppose the following is defined in a file "cfun.c":

       struct cfunr { int x; char *s; };
       struct cfunr * cfun (i,s,r,a)
           int i;
           char *s;
           struct cfunr * r;
           int a[10];
       {
           int j;
           struct cfunr * r2;
           printf("i = %d\n", i);
           printf("s = %s\n", s);
           printf("r->x = %d\n", r->x);
           printf("r->s = %s\n", r->s);
           for (j = 0; j < 10; j++) printf("a[%d] = %d.\n", j, a[j]);
           r2 = (struct cfunr *) malloc (sizeof (struct cfunr));
           r2->x = i+5;
           r2->s = "A C string";
           return r2;
       }

It is possible to call this function from Lisp using the file "callcfun.lsp"
(don't call it "cfun.lsp" - COMPILE-FILE would overwrite "cfun.c") whose
contents is:

       (in-package "TEST-C-CALL" :use '("LISP" "FFI"))
       (def-c-struct cfunr (x int) (s c-string))
       (def-c-call-out cfun (:arguments (i int)
                                        (s c-string)
                                        (r (c-ptr cfunr) :in :alloca)
                                        (a (c-ptr (c-array int 10)) :in :alloca)
                            )
                            (:return-type (c-ptr cfunr))
       )
       (defun call-cfun ()
         (cfun 5 "A Lisp string" (make-cfunr :x 10 :s "Another Lisp string")
               '#(0 1 2 3 4 5 6 7 8 9)
       ) )

Use the module facility:

       $ clisp-link create-module-set cfun callcfun.c
       $ cc -O -c cfun.c
       $ cd cfun
       $ ln -s ../cfun.o cfun.o
       Add cfun.o to NEW_LIBS and NEW_FILES in link.sh.
       $ cd ..
       $ base/lisp.run -M base/lispinit.mem -c callcfun.lsp
       $ clisp-link add-module-set cfun base base+cfun
       $ base+cfun/lisp.run -M base+cfun/lispinit.mem -i callcfun
       > (test-c-call::call-cfun)
       i = 5
       s = A Lisp string
       r->x = 10
       r->s = Another Lisp string
       a[0] = 0.
       a[1] = 1.
       a[2] = 2.
       a[3] = 3.
       a[4] = 4.
       a[5] = 5.
       a[6] = 6.
       a[7] = 7.
       a[8] = 8.
       a[9] = 9.
       #S(TEST-C-CALL::CFUNR :X 10 :S "A C string")
       > 
       $ rm -r base+cfun

Note that there is a memory leak here: The return value r2 of cfun() is
malloc()ed but never free()d. Specifying
       (:return-type (c-ptr cfunr) :malloc-free)
is not an alternative because this would also free(r2->x) but r2->x is a
pointer to static data.

Ex. 5: To sort an array of double-floats using the Lisp function SORT
instead of the C library function qsort(), one can use the following
interface code "sort1.c". The main problem is to pass a variable-sized array.

       extern void lispsort_begin (int);
       void* lispsort_function;
       void lispsort_double (int n, double * array)
       {
           double * sorted_array;
           int i;
           lispsort_begin(n); /* store #'sort2 in lispsort_function */
           sorted_array = ((double * (*) (double *)) lispsort_function) (array);
           for (i = 0; i < n; i++) array[i] = sorted_array[i];
           free(sorted_array);
       }

This is accompanied by "sort2.lsp":

       (use-package "FFI")
       (def-call-in lispsort_begin (:arguments (n int))
                                   (:return-type nil)
                                   (:language :stdc)
       )
       (def-c-var lispsort_function (:type c-pointer))
       (defun lispsort_begin (n)
         (setf (cast lispsort_function
                     `(c-function
                        (:arguments (v (c-ptr (c-array double-float ,n))))
                        (:return-type (c-ptr (c-array double-float ,n))
                                      :malloc-free
                      ) )
               )
               #'sort2
       ) )
       (defun sort2 (v)
         (declare (type vector v))
         (sort v #'<)
       )

To test this, use the following test file "sorttest.lsp":

       (def-call-out sort10
                     (:name "lispsort_double")
                     (:language :stdc)
                     (:arguments (n int)
                                 (array (c-ptr (c-array double-float 10))
                                        :in-out
       )             )           )

Now try

       $ clisp-link create-module-set sort sort2.c sorttest.c
       $ cc -O -c sort1.c
       $ cd sort
       $ ln -s ../sort1.o sort1.o
       Add sort1.o to NEW_LIBS and NEW_FILES in link.sh.
       $ cd ..
       $ base/lisp.run -M base/lispinit.mem -c sort2.lsp sorttest.lsp
       $ clisp-link add-module-set sort base base+sort
       $ base+sort/lisp.run -M base+sort/lispinit.mem -i sort2 sorttest
       > (sort10 10 '#(0.501d0 0.528d0 0.615d0 0.550d0 0.711d0
                       0.523d0 0.585d0 0.670d0 0.271d0 0.063d0))
       #(0.063d0 0.271d0 0.501d0 0.523d0 0.528d0 0.55d0 0.585d0 0.615d0 0.67d0 0.711d0)
       $ rm -r base+sort

