Lorenzo Rovigatti

Building a Python Package with CMake, pybind11, and scikit-build-core

Lorenzo Rovigatti — Wed, 05 Nov 2025 15:00:48 GMT

Building a Python Package with CMake, pybind11, and scikit-build-core

I’m the main developer and maintainer of oxDNA, a simulation engine implementing oxDNA and oxRNA, two of the most widely-used coarse-grained models used in nucleic-acid nanotechnology. The software has been used in hundreds of publications, and it is part of a larger ecosystem (see e.g. the oxDNA.org webserver, or oxview, a browser-based editing/visualising tool).

A few years ago I started working making some of oxDNA’s features available to Python through pybind11. As a result, recent oxDNA versions can optionally compile and install an oxpy module (which, for instance, can be used to streamline analysis of simulation results, or run metadynamics simulations). For practical reasons, it is possible to install oxpy in two ways:

By using pip:

pip install . # from the root folder

By using the usual CMake pipeline

mkdir build
cd build
cmake .. -DPython=On # you can add more options, such as -DCUDA=On
make install

The requirement of having two ways of compiling the code forced me to conjure up, mostly by trial and error, a rather obscure compilation process. Unfortunately (but perhaps not unsurprisingly), it turns out that it procedure was not only obscure, but also fragile, as upgrading pip broke it. I then spent a weekend trying to fix things up. This post is the result of this effort, and I decided to write mostly to help my future self. Throughout the post I’ll be using PIP and CMAKE to refer to the two installation methods introduced before, respectively.

Here I will explain how to create a Python package that includes a C++ extension built with CMake and pybind11, and how to package it cleanly using scikit-build-core so that it fulfills my (oxDNA’s?) needs.

A tar file containing a full example (e.g. all the files described in this post) can be downloaded here. The example combines:

CMake for building the C++ extension
pybind11 for Python bindings
scikit-build-core for packaging
setuptools_scm for automatic versioning from git tags/commits

Project layout

oxpy-mwe/
├─ CMakeLists.txt
├─ example.py
├─ pyproject.toml
├─ README.md
└─ src/
   └─ oxpy_mwe/
      ├─ __init__.py
      ├─ core.cpp
      └─ pyproject_cmake.toml

Module source files

The src/oxpy_mwe/core.cpp file uses pybind11 to create one module that exposes a simple function:

#include <pybind11/pybind11.h>

namespace py = pybind11;

int add(int a, int b) { return a + b; }

PYBIND11_MODULE(core, m) {
    m.doc() = "oxpy_mwe core module";
    m.def("add", &add, "Add two integers");
}

Then we create an __init__.py file that imports the add function from the core module and handles the module version by importing a _version.py that will be generated at compile time:

from .core import add

try:
    from ._version import version as __version__
except Exception:
    __version__ = "0.0.0"

CMakeLists.txt

Here I split the file in a few sections, so that they can be commented separately.

cmake_minimum_required(VERSION 3.18)
project(oxpy_mwe LANGUAGES CXX)

find_package(Python COMPONENTS Interpreter Development.Module REQUIRED)
find_package(pybind11 CONFIG QUIET)
if(NOT pybind11_FOUND)
  include(FetchContent)
  FetchContent_Declare(
    pybind11
    GIT_REPOSITORY https://github.com/pybind/pybind11.git
    GIT_TAG v2.12.0
  )
  FetchContent_MakeAvailable(pybind11)
endif()

We look for python and pybind11 (which is fetched from the internet if not found on the system).

# Create a staging directory for the Python package
set(OXPY_BASE_DIR "${CMAKE_BINARY_DIR}/python")
set(OXPY_PACKAGE_DIR "${OXPY_BASE_DIR}/oxpy_mwe")
# Make sure the directory exists
file(MAKE_DIRECTORY ${OXPY_PACKAGE_DIR})

We create a python/oxpy_mwe folder relative to the building directory where all the package files will be stored.

pybind11_add_module(core MODULE src/oxpy_mwe/core.cpp)

# Place the compiled extension (.so) in that folder
set_target_properties(core
  PROPERTIES
    LIBRARY_OUTPUT_DIRECTORY "${OXPY_PACKAGE_DIR}"
    SUFFIX ".so"
)

We tell CMake to compile the module, make it a shared .so object and put it in the output folder (i.e. python/oxpy_mwe).

configure_file(src/oxpy_mwe/__init__.py ${OXPY_PACKAGE_DIR}/__init__.py COPYONLY)

We copy __init__.py to the output folder.

if(SKBUILD)
  # copy the _version.py file generated by setuptools_scm
  configure_file(${CMAKE_SOURCE_DIR}/src/oxpy_mwe/_version.py ${OXPY_PACKAGE_DIR}/_version.py COPYONLY)
  # install the package to a location where scikit-build can find it
  install(DIRECTORY ${OXPY_PACKAGE_DIR}/ DESTINATION oxpy_mwe)
else()
  configure_file(${CMAKE_SOURCE_DIR}/README.md ${OXPY_PACKAGE_DIR}/README.md COPYONLY)
  configure_file(${CMAKE_SOURCE_DIR}/src/oxpy_mwe/pyproject_cmake.toml ${OXPY_BASE_DIR}/pyproject.toml)
  # Install the package using pip
  option(OXPY_SYSTEM_INSTALL "Install system-wide via pip" OFF)
  if(OXPY_SYSTEM_INSTALL)
    install(CODE "execute_process(COMMAND ${Python_EXECUTABLE} -m pip install ${OXPY_BASE_DIR})")
  else()
    install(CODE "execute_process(COMMAND ${Python_EXECUTABLE} -m pip install --user ${OXPY_BASE_DIR})")
  endif()
endif()

Here the compilation proceeds differently for PIP and CMAKE: in the former case pip uses scikit-build-core, which defines a CMake variable SKBUILD that is used to determine what to do:

In the case of PIP, the _version.py file, generated by setuptools_scm, see below, is copied from the /src/oxpy_mwe/ folder to the output folder. Then, when the install step is invoked, the the output folder is copied as-is to the target folder, which in this case is again a local folder (oxpy_mwe). There, if the correct path is given in the wheel.packages option of pyproject.toml, scikit-build-core will be able to find the package and use it to copy it to the correct Python’s site-packages folder.
For the CMAKE method, we copy the README and the correct pyproject.toml (the one stored in the src/oxpy_mwe folder) to the target folder, and then install the package directly with pip.

The key point here is that we use slightly different pyproject.toml files for the PIP and CMAKE methods, as described below.

pyproject.toml

The pyproject.toml file for the PIP method is as follows:

[build-system]
requires = ["scikit-build-core>=0.9", "setuptools_scm>=8", "pybind11>=2.10"]
build-backend = "scikit_build_core.build"

[project]
name = "oxpy_mwe"
dynamic = ["version"]

[...]

[tool.scikit-build]
metadata.version.provider = "scikit_build_core.metadata.setuptools_scm"
wheel.packages = ["oxpy_mwe"]

[tool.setuptools_scm]
version_file = "src/oxpy_mwe/_version.py"
fallback_version = "unknown"
local_scheme = "no-local-version"

Here I’m omitting standard project-related fields (such as classifiers, authors, etc). Here we use scikit-build-core as the build backend, and tell it to look for the package in the oxpy_mwe folder. If this is not specified, omitted or if the path is wrong, scikit-build-core will be happy to consider src/oxpy_mwe as the package folder, and silently copy that to the site-packages folder. I believe that this happens because there is a src/oxpy_mwe/__init__.py file, and that, according to scikit-build-core, makes the folder a Python package.

The last lines tell setuptools_scm to write a version file that will be copied to the output folder by CMake. The local_scheme = "no-local-version" makes sure that no dates or times are appended to the version string, which would, very confusingly, generate mismatches when installing the package with pip.

For CMAKE, the pyproject.toml file is copied to the right folder at runtime by CMake starting from the following /src/oxpy_mwe/pyproject_cmake.toml file:

[build-system]
requires = ["scikit-build-core>=0.9", "setuptools_scm>=8"]
build-backend = "setuptools.build_meta"

[...]

[tool.setuptools.package-data]
oxpy_mwe = ["*.so"]

[tool.setuptools_scm]
version_file = "oxpy_mwe/_version.py"
fallback_version = "0.0.0"
local_scheme = "no-local-version"
root = "${CMAKE_SOURCE_DIR}"

Again, I omit the project metadata in order to focus on the important content. Here we use setuptools.build_meta as the build backend, and tell it that the compiled extension (*.so) is part of the package. Then, we again use setuptools_scm to write a version file, but in this case we have to point it to the right root folder, where it can find the SCM (i.e. git) metadata. We do this with the option root, which is populated by CMake when it copies the file to the output folder. Note that in this case the version file is put directly into the output folder (since the pip install . command is invoked from the /build/python, where CMake has put the pyproject.toml file).

Continuous integration

As a last note, when using GitHub Actions, make sure the workflow checks out the full repository with all tags so that setuptools_scm can detect the correct version:

- uses: actions/checkout@v4
  with:
    fetch-depth: 0
    fetch-tags: true

reveal.js presentations

Lorenzo Rovigatti — Tue, 08 Aug 2023 12:12:36 GMT

The problem

A few years ago I started using reveal.js to prepare my scientific talks and I never looked back since then. Now I use it also for the programming class I teach to first-year students!

The only thing I hate about reveal.js is its PDF-exporting features: perhaps it is my fault (although I have rtfm’d), but I never managed to generate a PDF that faithfully reproduced my presentation. And of course PDFs don’t support movies (in a portable and reliable manner).

When I decided to use Nikola to build my website I thought that I could maybe find a way of making my presentations available online as HTML pages. I didn’t find any resources on the topic online, but I used what I learned about Nikola in those last weeks and I hacked up my own solution, which I’m going to describe here.

The strategy

The basic idea is to have a custom template page reveal.tmpl that

loads the necessary reveal (and possibly custom) javascript and css files required to display the presentation
sets up the HTML bits that will contain the markdown/HTML code.

The (markdown/HTML) content of each presentation is then copied in an HTML file containing the special .. template: reveal.tmpl metadata overriding the default template file for pages.

The template file

If you are curious you can look at (and download) the custom template I have written here, which depends on the theme I’m using (a modified version of bnw) to output the first part of the HTML page. Here I will describe the most important parts of the template.

I decided to use reveal.js version 4.1.2 and load the relevant file through a CDN, but the same procedure should work with any version and/or files stored elsewhere.

We start with loading the CSS files in the <head> element:

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/reveal.js/4.1.2/reset.min.css">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/reveal.js/4.1.2/reveal.min.css">
<link rel="stylesheet" href="/assets/css/slides.css">

Here slides.css is the file containing my custom CSS styles.

Within the <body> element there is the reveal.js HTML boilerplate, where ${post.text()} will be replaced by each presentation’s content:

<div class="reveal">
    <div class="slides">
            % if post.meta('is_html') and post.meta('is_html') == 'true':
            ${post.text()}
            % else:
            <section data-markdown>
                <textarea data-template>
                    ${post.text()}
                </textarea>
            </section>
            % endif
    </div>
</div>

I tend to write my teaching slides with markdown and my talks directly in HTML. With the structure above, an HTML presentation should contain a .. is_html: true metadata to be correctly displayed.

Finally, just before </body> there is the import and initialisation of reveal.js’s javascript machinery:

<script src="https://cdnjs.cloudflare.com/ajax/libs/reveal.js/4.1.2/reveal.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/reveal.js/4.1.2/plugin/notes/notes.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/reveal.js/4.1.2/plugin/markdown/markdown.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/reveal.js/4.1.2/plugin/highlight/highlight.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/reveal.js/4.1.2/plugin/math/math.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/reveal.js/4.1.2/plugin/zoom/zoom.min.js"></script>
<script>
Reveal.initialize({
    hash: true,
    
    math: {
        mathjax: 'https://cdn.jsdelivr.net/gh/mathjax/mathjax@2.7.8/MathJax.js',
        config: 'TeX-AMS_HTML-full',
    },
    
    markdown: {
        smartypants: true
    },
    
    slideNumber: 'c/t',
    showSlideNumber: 'all',

    plugins: [ RevealMarkdown, RevealHighlight, RevealNotes, RevealMath, RevealZoom ]
});
</script>

Tip

I have added the following link to help the user go back to the “normal” part of the website

<a class="back_button" href="#" onclick="history.back()"></a>

The link is fixed in the topmost left corner and is styled as an arrow with the following CSS code:

a.back_button {
    position: fixed;
    top: 20px;
    left: 20px;
    z-index: 999 !important;
}

a.back_button:before {
    content: "\f060";
    font-family: 'fontawesome';
    font-weight: normal;
    font-style: normal;
    font-size: 110%;
}

The presentation page file

Since we use HTML files to store presentations styled with reveal.js, we first need to make sure that HTML pages are enabled in the conf.py file. The relevant part of my configuration is as follows:

PAGES = (
    ("pages/*.md", "", "page.tmpl"),
    ("pages/*.rst", "", "page.tmpl"),
    ("pages/*.html", "", "page.tmpl"),
)

We then create a new HTML page (with nikola new_page -f html for instance) and update its metadata to something like this:

<!--
.. title: My fancy seminar
.. slug: fancy_seminar
.. date: 2023-08-08 13:12:44 UTC+02:00
.. tags: 
.. category: seminars
.. link: 
.. description: 
.. type: text
.. template: reveal.tmpl
.. has_math: true
-->

where the most important bit is .. template: reveal.tmpl, where we override the default template.

The rest of the file should contain the presentation text. Here is an example:

# This is a fancy seminar

## by Lorenzo Rovigatti

---

# Introduction

<img src="fancy_image.png">

Nikola tips

Lorenzo Rovigatti — Sun, 02 Jul 2023 19:41:48 GMT

Templates

The theme you are using may lack a template file for a specific page. In my case, listing.tmpl was missing. Therefore I copied the NIKOLA/data/themes/base/templates/listing.tmpl file over to /themes/my_theme/templates and edited it to make its structure similar to my theme’s other tmpl files.

Math support

With Nikola you can enable math support for a specific page by adding has_math: true to its metadata. However, if you use long equations and want to support devices with small screens, remember to add "HTML-CSS": { linebreaks: { automatic: true } }, to MathJax’s config in conf.py.

Images

By default Nikola creates thumbnails for each image you put into the images folder (or subfolders). If you don’t need thumbnails and want to save compiling time and disk space you can put everything in files/images instead. This has the benefit of keeping all the paths valid, since in either case images will be placed in /images/path/to/image.jpg.

Relative links

I wrote a javascript function that takes paths to data files stored in a given folder as arguments. As always, paths can be relative to the current page or absolute. If one chooses the former approach, if the page is moved somewhere else the links may break, which is why I tend to prefer absolute links. Often the best of way of specifying absolute links on a website is to have something like /path/to/dest.

As far as I understand, nikola translates all absolute links specified in this way to relative links, so that these links work regardless of the root URL (i.e. if the website is browsed locally, if it’s deployed on a first-level domain or as a subfolder). Unfortunately, nikola doesn’t do the same to paths specified not in link (e.g. in javascript code, as in my case). As a result, I needed a way of translating these absolute paths to relative paths. In order to do so I tried to use template-based shortcodes and put the following code in the /shortcodes/rel_url.tmpl path:

${rel_link(post.permalink(), dst)}

This shortcode calls nikola’s rel_link method, passing it the current post’s permalink and the destination path specified by the user, and can be used like this in a page or post:

let my_path = '../../my_file/file.dat';

Unfortunately, the snippet above will result in something like this:

let my_path = '../../my_file/file.dat
';

since, as far as I understand, nikola adds a trailing new line to the output of template-based shortcodes. A workaround would be to surround the shortcode’s code with single (or double) quotes, making the plugin less general and more prone to subtle bugs.

The solution I found was to add a “proper” plugin, since nikola apparently uses its output verbatim, without adding newlines or other whitespace. In order to do so I simply created a /plugins/rel_url folder and added these two files in it:

rel_url.plugin

[Core]
Name = rel_url
Module = rel_url

[Nikola]
PluginCategory = Shortcode

[Documentation]
Author = Lorenzo Rovigatti
Version = 0.1
Website = https://www.roma1.infn.it/~rovigatl/
Description = Generate a relative URL

rel_url.py

from nikola.plugin_categories import ShortcodePlugin

class RelUrlShortcodePlugin(ShortcodePlugin):
    """Return a relative URL."""
    
    name = "rel_url"

    def handler(self, dst, site=None, data=None, lang=None, post=None):
        link = site.rel_link(post.permalink(), dst)
        
        return link, []