如何从boost线程调用Python？

Question

我有一个Python应用程序调用C++ boost python库，它都可以工作。不过，我有一个回调C++的Python场景，其中来自boost线程的C++调用python，并在C++端获得访问冲突。如果我使用python线程完成相同的回调，它可以完美地工作。因此，我怀疑我不能使用boost线程从C++中简单地回调Python，但需要额外的工作才能使用它。

Answer 1

最有可能的罪魁祸首是Global Interpreter Lock（GIL）在调用Python代码时未被线程占用，导致未定义的行为。验证所有直接或间接Python调用的路径，在调用Python代码之前获取GIL。

---

GIL是围绕CPython解释器的互斥锁。这个互斥体阻止了在Python对象上执行并行操作。因此，在任何时候，允许最多一个线程（获得GIL的线程）对Python对象执行操作。当存在多个线程时，调用Python代码而不保留GIL会导致未定义的行为。

C或C++线程有时在Python文档中被称为外来线程。 Python解释器无法控制外来线程。因此，外来线程负责管理GIL以允许与Python线程并行或并行执行。必须仔细考虑：

• 堆栈展开，因为Boost.Python可能会引发异常。
• 间接调用到Python，如拷贝构造函数和析构函数

一种解决方案是包装的Python回调与知道GIL管理的自定义类型。

---

使用RAII-style类来管理GIL提供了一个优雅的异常安全解决方案。例如，使用以下with_gil类，创建对象with_gil时，调用线程将获取GIL。当with_gil对象被破坏时，它恢复GIL状态。

/// @brief Guard that will acquire the GIL upon construction, and 
///  restore its state upon destruction. 
class with_gil 
{ 
public: 
    with_gil() { state_ = PyGILState_Ensure(); } 
    ~with_gil() { PyGILState_Release(state_); } 

    with_gil(const with_gil&)   = delete; 
    with_gil& operator=(const with_gil&) = delete; 
private: 
    PyGILState_STATE state_; 
}; 

及其用法：

{ 
    with_gil gil;      // Acquire GIL. 
    // perform Python calls, may throw 
}         // Restore GIL. 

与正在能够通​​过with_gil管理GIL，下一步是创建正确管理GIL函子。下面py_callable类将包装一个boost::python::object并获取GIL对于其中Python代码被调用的所有路径：

/// @brief Helper type that will manage the GIL for a python callback. 
/// 
/// @detail GIL management: 
///   * Acquire the GIL when copying the `boost::python` object 
///   * The newly constructed `python::object` will be managed 
///    by a `shared_ptr`. Thus, it may be copied without owning 
///    the GIL. However, a custom deleter will acquire the 
///    GIL during deletion 
///   * When `py_callable` is invoked (operator()), it will acquire 
///    the GIL then delegate to the managed `python::object` 
class py_callable 
{ 
public: 

    /// @brief Constructor that assumes the caller has the GIL locked. 
    py_callable(const boost::python::object& object) 
    { 
    with_gil gil; 
    object_.reset(
     // GIL locked, so it is safe to copy. 
     new boost::python::object{object}, 
     // Use a custom deleter to hold GIL when the object is deleted. 
     [](boost::python::object* object) 
     { 
     with_gil gil; 
     delete object; 
     }); 
    } 

    // Use default copy-constructor and assignment-operator. 
    py_callable(const py_callable&) = default; 
    py_callable& operator=(const py_callable&) = default; 

    template <typename ...Args> 
    void operator()(Args... args) 
    { 
    // Lock the GIL as the python object is going to be invoked. 
    with_gil gil; 
    (*object_)(std::forward<Args>(args)...); 
    } 

private: 
    std::shared_ptr<boost::python::object> object_; 
}; 

通过在自由空间管理boost::python::object，可以自由复制该shared_ptr而不必保持GIL。这允许我们安全地使用默认生成的拷贝构造函数，赋值运算符，析构函数等。

一个将使用py_callable如下：

// thread 1 
boost::python::object object = ...; // GIL must be held. 
py_callable callback(object);  // GIL no longer required. 
work_queue.post(callback); 

// thread 2 
auto callback = work_queue.pop(); // GIL not required. 
// Invoke the callback. If callback is `py_callable`, then it will 
// acquire the GIL, invoke the wrapped `object`, then release the GIL. 
callback(...); 

---

下面是一个完整的例子具有Python扩展调用Python对象从一个C++线程回调demonstrating：

#include <memory> // std::shared_ptr 
#include <thread> // std::this_thread, std::thread 
#include <utility> // std::forward 
#include <boost/python.hpp> 

/// @brief Guard that will acquire the GIL upon construction, and 
///  restore its state upon destruction. 
class with_gil 
{ 
public: 
    with_gil() { state_ = PyGILState_Ensure(); } 
    ~with_gil() { PyGILState_Release(state_); } 

    with_gil(const with_gil&)   = delete; 
    with_gil& operator=(const with_gil&) = delete; 
private: 
    PyGILState_STATE state_; 
}; 

/// @brief Helper type that will manage the GIL for a python callback. 
/// 
/// @detail GIL management: 
///   * Acquire the GIL when copying the `boost::python` object 
///   * The newly constructed `python::object` will be managed 
///    by a `shared_ptr`. Thus, it may be copied without owning 
///    the GIL. However, a custom deleter will acquire the 
///    GIL during deletion 
///   * When `py_callable` is invoked (operator()), it will acquire 
///    the GIL then delegate to the managed `python::object` 
class py_callable 
{ 
public: 

    /// @brief Constructor that assumes the caller has the GIL locked. 
    py_callable(const boost::python::object& object) 
    { 
    with_gil gil; 
    object_.reset(
     // GIL locked, so it is safe to copy. 
     new boost::python::object{object}, 
     // Use a custom deleter to hold GIL when the object is deleted. 
     [](boost::python::object* object) 
     { 
     with_gil gil; 
     delete object; 
     }); 
    } 

    // Use default copy-constructor and assignment-operator. 
    py_callable(const py_callable&) = default; 
    py_callable& operator=(const py_callable&) = default; 

    template <typename ...Args> 
    void operator()(Args... args) 
    { 
    // Lock the GIL as the python object is going to be invoked. 
    with_gil gil; 
    (*object_)(std::forward<Args>(args)...); 
    } 

private: 
    std::shared_ptr<boost::python::object> object_; 
}; 

BOOST_PYTHON_MODULE(example) 
{ 
    // Force the GIL to be created and initialized. The current caller will 
    // own the GIL. 
    PyEval_InitThreads(); 

    namespace python = boost::python; 
    python::def("call_later", 
    +[](int delay, python::object object) { 
     // Create a thread that will invoke the callback. 
     std::thread thread(+[](int delay, py_callable callback) { 
     std::this_thread::sleep_for(std::chrono::seconds(delay)); 
     callback("spam"); 
     }, delay, py_callable{object}); 
     // Detach from the thread, allowing caller to return. 
     thread.detach(); 
    }); 
} 

互动用法：

>>> import time 
>>> import example 
>>> def shout(message): 
...  print message.upper() 
... 
>>> example.call_later(1, shout) 
>>> print "sleeping"; time.sleep(3); print "done sleeping" 
sleeping 
SPAM 
done sleeping