CFFI

What is PyPy?

Other Highlights (since 4.0.1 released in November 2015)

• New features:
  • Support embedding PyPy in a C-program via cffi and static callbacks in cffi.
    This deprecates the old method of embedding PyPy
  • Refactor vmprof to work cross-operating-system, deprecate using buggy
    libunwind on Linux platforms. Vmprof even works on Windows now.
  • Support more of the C-API type slots, like tp_getattro, and fix C-API
    macros, functions, and structs such as _PyLong_FromByteArray(),
    PyString_GET_SIZE, f_locals in PyFrameObject, Py_NAN, co_filename in
    PyCodeObject
  • Use a more stable approach for allocating PyObjects in cpyext. (see
    blog post). Once the PyObject corresponding to a PyPy object is created,
    it stays around at the same location until the death of the PyPy object.
    Done with a little bit of custom GC support. It allows us to kill the
    notion of “borrowing” inside cpyext, reduces 4 dictionaries down to 1, and
    significantly simplifies the whole approach (which is why it is a new
    feature while technically a refactoring) and allows PyPy to support the
    populart lxml module (as of the next release) with no PyPy specific
    patches needed
  • Make the default filesystem encoding ASCII, like CPython
  • Use hypothesis in test creation, which is great for randomizing tests
   
• Bug Fixes
  • Backport always using os.urandom for uuid4 from cpython and fix the JIT as well
    (issue #2202)
  • More completely support datetime, optimize timedelta creation
  • Fix for issue #2185 which caused an inconsistent list of operations to be
    generated by the unroller, appeared in a complicated DJango app
  • Fix an elusive issue with stacklets on shadowstack which showed up when
    forgetting stacklets without resuming them
  • Fix entrypoint() which now acquires the GIL
  • Fix direct_ffi_call() so failure does not bail out before setting CALL_MAY_FORCE
  • Fix (de)pickling long values by simplifying the implementation
  • Fix RPython rthread so that objects stored as threadlocal do not force minor
    GC collection and are kept alive automatically. This improves perfomance of
    short-running Python callbacks and prevents resetting such object between
    calls
  • Support floats as parameters to itertools.isslice()
  • Check for the existence of CODESET, ignoring it should have prevented PyPy
    from working on FreeBSD
  • Fix for corner case (likely shown by Krakatau) for consecutive guards with
    interdependencies
  • Fix applevel bare class method comparisons which should fix pretty printing
    in IPython
  • Issues reported with our previous release were resolved after reports from users on our issue tracker at https://bitbucket.org/pypy/pypy/issues or on IRC at #pypy
   
• Numpy:
  • Updates to numpy 1.10.2 (incompatibilities and not-implemented features
    still exist)
  • Support dtype=((‘O’, spec)) union while disallowing record arrays with
    mixed object, non-object values
  • Remove all traces of micronumpy from cpyext if –withoutmod-micronumpy option used
  • Support indexing filtering with a boolean ndarray
  • Support partition() as an app-level function, together with a cffi wrapper
    in pypy/numpy, this now provides partial support for partition()
   
• Performance improvements:
  • Optimize global lookups
  • Improve the memory signature of numbering instances in the JIT. This should
    massively decrease the amount of memory consumed by the JIT, which is
    significant for most programs. Also compress the numberings using variable-
    size encoding
  • Optimize string concatenation
  • Use INT_LSHIFT instead of INT_MUL when possible
  • Improve struct.unpack by casting directly from the underlying buffer.
    Unpacking floats and doubles is about 15 times faster, and integer types
    about 50% faster (on 64 bit integers). This was then subsequently
    improved further in optimizeopt.py.
  • Optimize two-tuple lookups in mapdict, which improves warmup of instance
    variable access somewhat
  • Reduce all guards from int_floordiv_ovf if one of the arguments is constant
  • Identify permutations of attributes at instance creation, reducing the
    number of bridges created
  • Greatly improve re.sub() performance
   
• Internal refactorings:
  • Refactor and improve exception analysis in the annotator
  • Remove unnecessary special handling of space.wrap().
  • Support list-resizing setslice operations in RPython
  • Tweak the trace-too-long heuristic for multiple jit drivers
  • Refactor bookkeeping (such a cool word - three double letters) in the
    annotater
  • Refactor wrappers for OS functions from rtyper to rlib and simplify them
  • Simplify backend loading instructions to only use four variants
  • Simplify GIL handling in non-jitted code
  • Refactor naming in optimizeopt
  • Change GraphAnalyzer to use a more precise way to recognize external
    functions and fix null pointer handling, generally clean up external
    function handling
  • Remove pure variants of getfield_gc_* operations from the JIT by
    determining purity while tracing
  • Refactor databasing
  • Simplify bootstrapping in cpyext
  • Refactor rtyper debug code into python.rtyper.debug
  • Seperate structmember.h from Python.h Also enhance creating api functions
    to specify which header file they appear in (previously only pypy_decl.h)
  • Fix tokenizer to enforce universal newlines, needed for Python 3 support