cffi-objectFast and convenient foreign object interoperation via CFFI.

1Introduction

When developing projects that heavily use CFFI, interfacing with foreign libraries and managing memory are unavoidable issues.The following are three commonly used approaches:

1. Expose raw pointers to the high level code. \\This approach is very lightweight and efficient, but it requires programmers to manually manage memory.Even if macros that expand to unwind-protect can be used to manage resources with dynamic extent,it can sometimes make the code style unnatural under some scenarios that don't require deterministic time for resource acquisition and release.
2. Provide high-level wrapper classes or structs (all the fields are of Lisp's native types) with cffi:translate-from/to-foreign or cffi:expand-from/to-foreign defined. \\This hands over the memory management to Lisp's GC and makes it more natural when operating data received from or passed to the foreign.However, for foreign functions that directly accept structs by value, it requires cffi-libffi, which has significant overhead under frequent invocations.CFFI does not automatically call the translation mechanism mentioned above for foreign functions that accept struct pointers as parameters,so users or library developers need to first allocate memory on the stack using with-foreign-object(s) (if the Lisp implementation does not support it, it may be allocated on the heap),and CFFI will perform the conversion with cffi:translate-from/to-foreign at runtime or cffi:expand-from/to-foreign at compile-time.The overhead involved in this process is not negligible for large structs, especially for real-time media processing, gaming, and other CPU intensive applications.
3. Define structs for each CFFI type, wrap a pointer inside, and selectively use trivial-garbage to manage the memory. \\This approach seems to combine the advantages of the above two methods. In most cases, programmers don't need to concern themselves with memory.Except for making the timing of resource release uncertain and putting some potential pressure on the GC,it has good performance because many implementations (such as SBCL and ECL) operate foreign memory efficiently.Additionally, this approach does not have the overhead brought by cffi:translate-from/to-foreign or cffi:expand-from/to-foreign,making it ideal for applications that require frequent calls to foreign functions,such as calling foreign functions for SIMD-accelerated matrix calculations or outputting audio buffers to audio devices.

cffi-object adopts the third approach above and provides a uniform way to directly convert existing CFFI type definitions (which can be generated by autowrapping tools like claw) into Lisp's struct and function definitions, allowing you to operate on foreign data types as if they were native types in Lisp, without having to write glue code by hand.

cffi-object should run on any implementation that supports CFFI and trivial-garbage. To test the system, simply eval (asdf:test-system :cffi-object) in the REPL.

2Features

• Generate CLOS classes for foreign types and use them as if they are native Lisp types \\You can generate the structure definition for a existing CFFI type:

      (cffi:defcstruct vector2
        (x :float)
        (y :float))
  
      (cobj:define-cobject-class (vector2 (:struct vector2)))
  

  Or you can generate structure definitions for all the CFFI types declared in a package. This can be useful if you have an existing library that already defined those CFFI types:

      (cl:defpackage #:mylib
        (:use #:cl))
  
      (cl:in-package #:mylib)
  
      (cffi:defcstruct vector2
        (x :float)
        (y :float))
  
      (cffi:defcstruct camera-2d
        (offset (:struct vector2))
        (target (:struct vector2))
        (rotation :float)
        (zoom :float))
  
      (cobj:define-cobject-class #:mylib)
  

  Then you can create or modify objects of these types just like using structs defined with defstruct:

      MYLIB> (make-vector2)
      #<VECTOR2 :X -1.5046615e-36 :Y 4.5643094e-41 @0x00007F3C84001210>
      MYLIB> ; The memory is unintialized by default
      ; No values
      MYLIB> (make-vector2 :x 1.0 :y 2.0)
      #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C84001190>
      MYLIB> (make-camera-2d :offset * :target * :rotation 0.0 :zoom 1.0)
      #<CAMERA-2D 
        :OFFSET #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C840011B0> 
        :TARGET #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C840011B8>
        :ROTATION 0.0 
        :ZOOM 1.0
       @0x00007F3C840011B0>
      MYLIB> (camera-2d-offset *)
      #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C840011B0>
      MYLIB> (copy-vector2 *)
      #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C840011D0>
      MYLIB> (setf (vector2-x *) 2.0)
      2.0
      MYLIB> (copy-vector2 ** ***) ; In-place copy
      #<VECTOR2 :X 2.0 :Y 2.0 @0x00007F3C840011B0>
      MYLIB> (vector2-equal * ***)
      T
  

  You can also define generic methods specialized for these foreign types:

      MYLIB> (defmethod position2 ((camera camera-2d)) (camera-2d-offset camera))
      #<STANDARD-METHOD MYLIB::POSITION2 (CAMERA-2D) {100467F273}>
      MYLIB> (defmethod position2 ((vector vector2)) vector)
      #<STANDARD-METHOD MYLIB::POSITION2 (VECTOR2) {10046EE753}>
      MYLIB> (position2 (make-camera-2d))
      #<VECTOR2 :X -1.5046902e-36 :Y 4.5643094e-41 @0x00007F3C840012E0>
      MYLIB> (position2 (make-vector2))
      #<VECTOR2 :X -1.5046586e-36 :Y 4.5643094e-41 @0x00007F3C84001300>
  

• Low overhead when interfacing with foreign functions \\All the objects created with cffi-object are fixed in memory and have the same memory representation as C,which means that structures can be passed directly to C functions or objects can be created directlyby returning a pointer to a structure from a C function without conversion needed.

      (cl:in-package #:mylib)
  
      (declaim (inline vector2-add))
      (cffi:defcfun ("__claw_Vector2Add" vector2-add) (:pointer (:struct vector2))
        (%%claw-result- (:pointer (:struct vector2)))
        (v1 (:pointer (:struct vector2)))
        (v2 (:pointer (:struct vector2))))
  
      (let ((v1 (make-vector2 :x 1.0 :y 2.0))
            (v2 (make-vector2 :x 3.0 :y 4.0)))
        (vector2-add (cobj:cobject-pointer v1)
                     (cobj:cobject-pointer v1)
                     (cobj:cobject-pointer v2))
        v1) ; => #<VECTOR2 :X 4.0 :Y 6.0 @0x00007F3C7C000EF0>
  

• Automatic and safe memory management \\All objects created by Lisp are automatically managed by the GC (Garbage Collector),and any reference to an object or its fields will prevent the memory of that object from being released:

      (let* ((cam (make-camera-2d))
             (vec (camera-2d-offset cam)))
        ;; VEC is a reference to the OFFSET field of CAMERA-2D, 
        ;; which will share memory in a certain region.
        vec) ; => #<VECTOR2 :X -3.1651653e31 :Y 9.809089e-45 @0x00007F3C7C001170>
      ;; This is safe because VEC holds a reference to CAM,
      ;; which will prevent both GC from collecting CAM and
      ;; releasing the corresponding memory.
  

  Exchanging object ownership with C functions is convenient:

      (cl:in-package #:mylib)
  
      (declaim (inline malloc))
      (cffi:defcfun malloc :pointer           ; cffi:foreign-alloc
        (size :size))
  
      (declaim (inline free))
      (cffi:defcfun free :void                ; cffi:foreign-free
        (size :pointer))
  
      (let* ((vec1 (cobj:manage-cobject       ; Take ownership of the object from foreign and responsible for freeing the memory.
                    (cobj:pointer-cobject   
                     (malloc (cffi:foreign-type-size
                              '(:struct vector2)))
                     'vector2)))
             (vec2 (cobj:pointer-cobject      ; Share the memory of this object with foreign and not responsible for freeing the memory.
                    (cobj:cobject-pointer vec1)
                    'vector2)))
        (assert (vector2-equal vec1 vec2))
        (free (cobj:unmanage-cobject vec1)))  ; Transfer ownership of the object to foreign and no longer responsible for freeing its memory.
  

  But when you transfer the deallocation of memory to foreign code, you should be aware that the memory of this object may become invalid at any time if it is deallocated by the foreign.

• Bring unboxed struct/array and by-value assignment to Common Lisp \\cffi-object is capable of creating unboxed structs or arrays, which are fully compatible with C,so pointers can be directly passed to foreign:

      (cl:in-package #:mylib)
  
      (cffi:defcstruct named-vector2-buffer
        (name :string)
        (buffer (:array (:struct vector2) 64))
        (size :size))
  
      (cobj:define-cobject-class (:struct named-vector2-buffer))
  

      MYLIB> (cffi:foreign-type-size '(:struct named-vector2-buffer))
      528
      MYLIB> (make-named-vector2-buffer :name "DEFAULT" :size 0)
      #<NAMED-VECTOR2-BUFFER
        :NAME "DEFAULT"
        :BUFFER #<#<VECTOR2 :X -1.5046586e-36 :Y 4.5643094e-41 @0x00007F3C8400FCC8>
                  #<VECTOR2 :X 0.0            :Y 0.0           @0x00007F3C8400FCD0>
                  #<VECTOR2 :X 0.0            :Y 0.0           @0x00007F3C8400FCD8>
                  #<VECTOR2 :X 1.1382681e27   :Y 2.1868875e-10 @0x00007F3C8400FCE0>
                  #<VECTOR2 :X 7.3027877e31   :Y 7.1538162e22  @0x00007F3C8400FCE8>
                  #<VECTOR2 :X 2.7199348e23   :Y 6.4820554e-10 @0x00007F3C8400FCF0>
                  #<VECTOR2 :X 1.0256189e-8   :Y 8.1793216e23  @0x00007F3C8400FCF8>
                  #<VECTOR2 :X 1.3900956e31   :Y 5.1765536e22  @0x00007F3C8400FD00>
                  #<VECTOR2 :X 7.673137e34    :Y 3.0880886e29  @0x00007F3C8400FD08>
                  #<VECTOR2 :X 8.435921e26    :Y 1.0326977e-38 @0x00007F3C8400FD10> ... [54 elements elided]>
        :SIZE 0
       @0x00007F3C8400FCC0>
      MYLIB> (cobj:cfill (named-vector2-buffer-buffer *) (make-vector2 :x 1.0 :y 2.0))
      #<#<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C8400FCC8>
        #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C8400FCD0>
        #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C8400FCD8>
        #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C8400FCE0>
        #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C8400FCE8>
        #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C8400FCF0>
        #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C8400FCF8>
        #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C8400FD00>
        #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C8400FD08>
        #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C8400FD10> ... [54 elements elided]>
      MYLIB> (cobj:make-carray 5 :element-type 'vector2
                                 :initial-contents (loop :for i :below 5
                                                         :collect (make-vector2 :x (coerce i 'single-float)
                                                                                :y (coerce i 'single-float))))
      #<#<VECTOR2 :X 0.0 :Y 0.0 @0x00007F3C8401BED0>
        #<VECTOR2 :X 1.0 :Y 1.0 @0x00007F3C8401BED8>
        #<VECTOR2 :X 2.0 :Y 2.0 @0x00007F3C8401BEE0>
        #<VECTOR2 :X 3.0 :Y 3.0 @0x00007F3C8401BEE8>
        #<VECTOR2 :X 4.0 :Y 4.0 @0x00007F3C8401BEF0>>
      MYLIB> (cobj:creplace ** *)
      #<#<VECTOR2 :X 0.0 :Y 0.0 @0x00007F3C8400FCC8>
        #<VECTOR2 :X 1.0 :Y 1.0 @0x00007F3C8400FCD0>
        #<VECTOR2 :X 2.0 :Y 2.0 @0x00007F3C8400FCD8>
        #<VECTOR2 :X 3.0 :Y 3.0 @0x00007F3C8400FCE0>
        #<VECTOR2 :X 4.0 :Y 4.0 @0x00007F3C8400FCE8>
        #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C8400FCF0>
        #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C8400FCF8>
        #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C8400FD00>
        #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C8400FD08>
        #<VECTOR2 :X 1.0 :Y 2.0 @0x00007F3C8400FD10> ... [54 elements elided]>
  

3Related Projects

• unboxables \\unboxables can provide unboxed struct/array features for Common Lisp too,and it uses a more compact memory layout, which can potentially have lower memory consumption,while cffi-object , by default, uses the C memory representation which may have padding between fields,allowing you to pass pointers to foreign functions directly.Currently, cffi-cobject may not have the high-performance array operations that unboxables provides.It is more focused on interoperation with foreign anyway.
• cffi-ops \\cffi-ops provides some macros expanded at compile-time, so it doesn't cons and can be used in performance-sensitive functions,which allows you to implement GC-free and high performance algorithms.System cffi-object.ops provides cffi-object the integration with cffi-ops, which can be enabled by (cobj.ops:enable-cobject-ops) at compile-time:

      (cl:in-package #:mylib)
  
      (eval-when (:compile-toplevel :load-toplevel :execute)
        (cobj.ops:enable-cobject-ops))
  
      (let ((vec1 (make-vector2 :x 1.0 :y 2.0))
            (vec2 (make-vector2 :x 3.0 :y 4.0)))
        (clocally (declare (ctype (:object (:struct vector2)) vec1 vec2))
          (vector2-add (& vec1) (& vec1) (& vec2))
          (assert (= (-> vec1 x) 4.0))
          (assert (= (-> (& vec1) y) 6.0))))