Teach you to use Cython to make your own wheels

Teach you to use Cython to make your own wheels

"Gotham" by James Gilleard

Author : Nugine

Column address : zhuanlan.zhihu.com/c_168195059

In this article, I want to show you the techniques of using Cython to extend Python.

If you have both C/C++ and Python coding skills, I believe you will like this.

The wheel we want to build is the simplest implementation of the stack. Writing in C/C++ can reduce unnecessary overhead and bring significant speedup.

step

  1. Create a catalog
  2. Write C++ files
  3. Write pyx file
  4. Direct compilation
  5. test

1. Create a catalog

1. create our working directory.

mkdir pystack
cd pystack

The 32-bit version and the 64-bit version bring different problems. My C library is 32-bit, so the python library must also be 32-bit.

Use pipenv to specify the python version and install Cython.

pipenv --python P:\Py3.6.5\python.exe
pipenv install Cython

2. Write C++ files

According to the official Python documentation, C++ must be compiled in C, so extern "C" needs to be added.

"c_stack.h"

#include "python.h"

extern "C"{
    class C_Stack {
        private:
        struct Node {
            PyObject* val;
            Node* prev;
        };
        Node* tail;

        public:
        C_Stack();

        ~C_Stack();

        PyObject* peek();

        void push(PyObject* val);

        PyObject* pop();
    };
}

"c_stack.cpp"

extern "C"{
    #include "c_stack.h"
}

C_Stack::C_Stack() {
    tail = new Node;
    tail->prev = NULL;
    tail->val = NULL;
};

C_Stack::~C_Stack() {
    Node *t;
    while(tail!=NULL){
        t=tail;
        tail=tail->prev;
        delete t;
    }
};

PyObject* C_Stack::peek() {
    return tail->val;
}

void C_Stack::push(PyObject* val) {
    Node* nt = new Node;
    nt->prev = tail;
    nt->val = val;
    tail = nt;
}

PyObject* C_Stack::pop() {
    Node* ot = tail;
    PyObject* val = tail->val;
    if (tail->prev != NULL) {
        tail = tail->prev;
        delete ot;
    }
    return val;
}

The simplest stack implementation, only three interfaces push, peek, pop, is enough as an example.

3. Write pyx file

Cython uses a mixed C and Python syntax to simplify the steps of extending Python.

It is very simple to write, provided that you know its syntax in advance.

"pystack.pyx"

# distutils: language=c++
# distutils: sources = c_stack.cpp

from cpython.ref cimport PyObject,Py_INCREF,Py_DECREF

cdef extern from'c_stack.h':
    cdef cppclass C_Stack:
        PyObject* peek();

        void push(PyObject* val);

        PyObject* pop();

class StackEmpty(Exception):
    pass

cdef class Stack:
    cdef C_Stack _c_stack

    cpdef object peek(self):
        cdef PyObject* val
        val=self._c_stack.peek()
        if val==NULL:
            raise StackEmpty
        return <object>val

    cpdef object push(self,object val):
        Py_INCREF(val);
        self._c_stack.push(<PyObject*>val);
        return None

    cpdef object pop(self):
        cdef PyObject* val
        val=self._c_stack.pop()
        if val==NULL:
            raise StackEmpty
        cdef object rv=<object>val;
        Py_DECREF(rv)
        return rv

Divided into four parts:

  1. The comment specifies the corresponding cpp file.
  2. Import C symbols from CPython: PyObject, PyINCREF, PyDECREF.
  3. Import C symbols from "cstack.h": CStack, and its interface.
  4. Wrap it as a Python object.

be careful:

  1. In the C implementation, when the stack is empty, a null pointer is returned. The Python implementation checks for null pointers and throws exception StackEmpty.
  2. PyObject* and object are not the same, type conversion is required.
  3. When pushing and poping, the reference counting must be operated correctly, otherwise the Python interpreter will crash directly. I didn't know this at the beginning, and I was confused for a long time, but occasionally saw that the error was related to gc, and I thought about the issue of reference counting.

4. Direct compilation

pipenv run cythonize -a -i pystack.cpp

3.files are generated: pystack.cpp, pystack.html, pystack.cp36-win32.pyd

pyx is compiled to cpp, and then compiled to pyd by the C compiler.

html is a cython prompt, indicating the degree of interaction with python in the pyx code.

pyd is the ultimate Python library.

5. Test it

"test.py"

from pystack import *
st=Stack()
print(dir(st))
try:
    st.pop()
except StackEmpty as exc:
    print(repr(exc))

print(type(st.pop))
for i in ['1',1,[1.0],1,dict(a=1)]:
    st.push(i)
while True:
    print(st.pop())


pipenv run python test.py

['__class__','__delattr__','__dir__','__doc__','__eq__','__format__','__ge__','__getattribute__','__gt__','__hash__','__init__','__init_subclass__', ' __le__','__lt__',
'__ne__','__new__','__pyx_vtable__','__reduce__','__reduce_ex__','__repr__','__setattr__','__setstate__','__sizeof__','__str__','__subeekclasshook__','peek','pop ','push']

<class'list'>
{'a': 1}
1
[1.0]
1
1
Traceback (most recent call last):
File "test.py", line 13, in <module>
    print(st.pop())
File "pystack.pyx", line 32, in pystack.Stack.pop
    cpdef object pop(self):
File "pystack.pyx", line 36, in pystack.Stack.pop
    raise StackEmpty
pystack.StackEmpty

It behaves the same as a normal Python object, perfect!

6. Application

pipenv run python test_polish_notation.py

from operator import add, sub, mul, truediv
from fractions import Fraction
from pystack import Stack

def main():
    exp = input('exp:')
    val = eval_exp(exp)
    print(f'val: {val}')


op_map = {
    '+': add,
    '-': sub,
    '*': mul,
    '/': truediv
}


def convert(exp):
    for it in reversed(exp.split('')):
        if it in op_map:
            yield True, op_map[it]
        else:
            yield False, Fraction(it)


def eval_exp(exp):
    stack = Stack()

    for is_op, it in convert(exp):
        if is_op:
            left = stack.pop()
            right = stack.pop()
            stack.push(it(left, right))
        else:
            stack.push(it)
    return stack.pop()


if __name__ =='__main__':
    main()
    # exp: + 5-2 * 3/4 ​​7
    # val: 37/7

This article shows the simplest Cython wheel making skills, hoping to provide a stepping stone for students who are about to enter the pit and those who have already entered the pit.

Reference: https://cloud.tencent.com/developer/article/1167153 teach you to use Cython to make your own wheels-Cloud + Community-Tencent Cloud