diff options
Diffstat (limited to 'doc/misc/grant-cffi-1.0.rst')
-rw-r--r-- | doc/misc/grant-cffi-1.0.rst | 124 |
1 files changed, 124 insertions, 0 deletions
diff --git a/doc/misc/grant-cffi-1.0.rst b/doc/misc/grant-cffi-1.0.rst new file mode 100644 index 0000000..b026209 --- /dev/null +++ b/doc/misc/grant-cffi-1.0.rst @@ -0,0 +1,124 @@ + +=========================== +Grant Proposal for CFFI 1.0 +=========================== + +*Accepted by the PSF board on April 4, 2015* + +This Grant Proposal is to give a boost towards "CFFI 1.0". Two main +issues with the current CFFI need to be solved: the difficulties of +installation, and the potentially large time taken at import. + +1. The difficulties of installation can be seen from outside by looking +at various workarounds and 3rd-party documentation that have grown into +existence. For example, the `setup.py` of projects like cryptography, +PyNaCl and bcrypt deploys workarounds that are explicitly documented in +https://caremad.io/2014/11/distributing-a-cffi-project/. + +2. The time taken at import is excessive in some cases. For example, +importing `pygame-cffi` on a Raspberry Pi ARM board takes on the order +of 10 to 20 seconds (and this is the "fast" case where the compiler +doesn't need to be invoked any more). + + +Technical Overview +------------------ + +"CFFI" is an existing Python project which complements the ctypes, +SWIG and Cython approaches to ease writing C Extension Modules for +Python. It has several advantages over the previous approaches, which +are presented at the start of the documentation at +http://cffi.readthedocs.org/en/latest/ . It has been very successful +so far: http://pypi-ranking.info/alltime records almost 7 million +downloads (for comparison, the #1 of all packages has almost 36 +million downloads). CFFI works on any Python >= 2.6, including 3.x, +as well as on PyPy. + +One problem is that while getting started with CFFI is very easy, the +installation process of a package that uses CFFI has got its rough +edges. CFFI (at least in its "verify()" mode) is based on calling the +C compiler to get information about the exact C types, structures, +argument types to functions, and so on. The C compiler is invoked +transparently at run-time, and the results cached. A +correctly-installed package using CFFI should cache the results at +installation time, but it can be difficult to ensure that no more +run-time compiler invocation is needed; doing so requires following +some extra guidelines or understanding some internal details. (The +problem is particularly acute on Windows where a typical user might +not have a proper C compiler installed.) + +To fix this, we have in mind adding a different CFFI mode (replacing +"verify()"), while keeping the access to the underlying C library +unmodified. In this mode, the code containing the cdef() and verify() +invocations would be moved to a separate Python source file. Running +that Python file would produce a dynamically-linked library. There +would be no caching logic involved; you would need to run it +explicitly during development whenever you made changes to it, to +re-generate and re-compile the dynamically-linked library. + +When distributed, the same file would be run (once) during +installation. This can be fully automated in setuptools-based +setup.py files; alternatively, it can be done in distutils-based +setup.py files by requiring prior manual installation of CFFI itself. + +A major difference with the existing verify() approach would be that +the ``.so/.dll/.dylib`` file would not be immediately loaded into the +process; you would load it only from the installed program at +run-time, and get the ``ffi`` and ``lib`` objects in this way (these +are the two objects that you use so far to access a C library with +verify()). + +Additionally, this would solve another issue: every import of a large +CFFI-using package takes a while so far. This is caused by CFFI +needing to parse again the C source code given in the cdef() (adding a +run-time dependency to the ``pycparser`` and ``ply`` packages). CFFI +also computes a CRC to know if it can reuse its cache. In the +proposed change, all the cdef() code would be pre-parsed and stored in +the dynamically-linked library, and no CRC would be needed. This +would massively reduce the import times. + + +Grant objective +--------------- + +The objective is to give a boost towards "CFFI 1.0", which needs to have +the functionalities described above in order to solve the two main +issues with the current CFFI: the difficulties of installation, and the +time taken at import. + +Included in the objective: the internal refactorings of CFFI that are +needed to get it done cleanly. The goal is to avoid simply adding +another layer on top of the old unchanged CFFI. + +This work may happen eventually in any case, but support from the PSF +would help make it happen sooner rather than later. + + +Grant size +---------- + +2'500 US$ for supporting the development time. This would cover 2.5 +weeks of full-time work at the part-time cost of 25 US$ per hour. + +The estimated work time until the CFFI 1.0 release is a bit larger +than that (I estimate it at roughly 4 weeks), but 2.5 weeks should +cover all the basics. An extended grant size of 4'000 US$ would be +appreciated but not required ``:-)`` + + +Grant beneficiaries +------------------- + +Armin Rigo, main author of CFFI, committing 2.5 weeks of full-time +work. + + +Grant follow-up +--------------- + +I will report on the success of the grant on the CFFI mailing list and +on the blog I usually post to (the PyPy blog) and mention the PSF as +providing the grant. The PSF will receive an email pointing to these +postings once they are out. Moreover a full CFFI 1.0 release should +follow (likely starting with beta versions); the PSF will receive +another email pointing to it. |