Rust in Python part 3: publish a Python package

2024-11-03T00:00:00+00:00

In the previous entries in this series, the concept of moving functionality from Python to Rust was illustrated using examples out of real-life application. On this page, I want to talk about my experience of doing a similar addition to my intervalues</code> package, which can be found here</a>. I will not fully discuss the Rust implementation details, but just the overall learnings and suggestions.

What is intervalues?</a> </h2> intervalues</code> is a Python package aimed at processing valued intervals efficiently. With valued intervals, I mean something like "All values between 2 and 4, with a value of 3". Say you note this down as [2-4; 3]</code>, and you have any valued interval [1-3; 2]</code>. Then you might want to add these together, which would produce the three valued intervals [1-2; 2]</code>, [2-3; 5]</code> and [3-4; 3]</code> that would summarize for each input the total value. Tasks like this have popped up in some previous Advent of Code challenges, and I can imagine practical applications as well, with the value part often being interpreted as "counts". For example, how many planes are there on an airfield at any given time for a recurring daily or weekly schedule? In that case, each input interval would be a 1-count interval with bounds set to the landing and departure time of the plane. For planes that land at the end of the schedule and depart at the beginning, there would be two intervals for both parts. Then, all these intervals could be combined to find the overlaps and counts across the schedule in an exact and efficient way, which can be used to identify potential bottlenecks or expansion opportunities. Although other interval-themed Python packages exist, the 'count' or 'value' perspective of it was not represented. Instead, these other packages are often limited to looking at it from a Set perspective, so finding either the union or the intersection of the input intervals. Translated to the airfield example above: "At what times is there at least one plane on the airfield?" and "At what times are all scheduled planes on the airfield simultaneously?". These questions can be useful as well ("When should the airfield be open?" for the first case for example) but could be answered by intervalues</code> as well by looking at all intervals with a count >= 1 for the first case, and all intervals with the count equal to the number of planes for the second case. The package provides functionality for all kinds of data management-like tasks you might want to do with one or more intervals: adding together, shifting, finding the most-common interval in an interval collection, finding the value of a particular input, etc. At the core of this, there is an algorithm for combining the intervals together, appropriately called combine_intervals</code>. This algorithm is the key thing that could slow the package down in case of a huge dataset, and because of that it is the key candidate to consider for porting over to Rust for a potential speed boost. The Rust implementation</a> </h2> Implementing combine_intervals</code> in Rust was a task that I tackled from the core up. An interesting challenge in this was that the types of the numbers have to be defined beforehand as Rust is a typed language. First, I avoided that by implementing the function just for isize</code> input for both the bounds and the value, and then I made variations of it where either could be f32</code> as well, as proof of concepts. It was good to start with this, because a potential blocker quickly became apparent: the combine_intervals</code> algorithm requires the bounds to be used as index in a HashMap</code> (the Rust version of a dict</code> in Python) but the standard floats in Rust are not hashable. So first, this challenge had to be solved, for which I ended up with two solutions: using Decimals</code> from rust_decimals</code>, which is an alternative numeric implementation for (among other) floats</li> creating my own intfloat</code> package</a> and Struct, which is faster than Decimals but also less reliable for accuracy the issue with the latter is in how numbers are stored: as 2 integers, a base integer and a power-of-10 integer. In the conversion of, say, 2 x 10^-4 back to a float there is a risk of a small error to be introduced, for example to get 0.00020000001 instead of 0.0002. Since the '4' above is an input on the Python side, you can simply round it again in Python to the same number of digits - but this might be a problem in a more generic use of intfloat</code> so in that case using Decimals</code> might be smart.</li> </ul> </li> </ul> Then, as next steps I slowly generalized the functions by making use of the Numeric trait (if you don't know about Rust traits yet: don't worry about it): first combining all integer inputs into one function call, and all float inputs into another; and then generalizing that to end up with a single function that is reused across all potential input types. It supports the above mentioned Decimal</code> and IntFloat</code> together with all default integer types. At first, the combined result was just a Vec</code> containing the relevant numeric values for each aggregated valued subinterval. It simply made sense to also collect this in a well-defined Struct, and so also the Rust version of having an IntervalCollection. Because of this, the Rust code is also available and usable by itself without Python on crates.io</a>, although the main goal of the project remains the Python code. Speed comparison</a> </h2> The main motivation of both the Rust version of intervalues</code> and of the intfloat</code> package is speed, so we need to actually verify that both provide a speed boost compared to the Python version of intervalues</code> and the Decimals</code> struct respectively. Intfloat vs Decimal</a> </h3> For this comparison, I use the demo implementation of the intervalues</code> Rust package in its main.rs</code>. This runs 4 setups in which one million intervals are generated and combined, with its inputs being: i32</code> for both integer bounds and value -> runs in 38.1 ms</li> i32</code> for integer bounds and Decimal</code> for float value -> runs in 60.6 ms</li> Decimal</code> for both float bounds and value -> runs in 143.6 ms</li> IntFloat</code> for both float bounds and value -> runs in 68.4 ms</li> </ol> This shows that using IntFloat</code> is roughly twice as slow as using i32</code> but twice as fast as using Decimal</code>, at least for this setup. On the one hand, this shows the added value of the existence of IntFloat</code>. On the other hand, in the rare situation you would need to combine one million intervals, having to wait 0.1s is not too bad, so Decimal</code> is still a good option even if it's the slowest. combine_intervals: Python vs Rust</a> </h3> For the secondary speed check, I had to make my Rust package available for Python again using maturin, and then make a wrapper to use either the Python or the Rust implementation depending on some input flag. To test for speed, I added an example script that runs both of them, for both integer and float inputs. The results are (for one million intervals) as follows: integers: 1774 ms (Python) vs 900 ms (Rust)</li> floats: 1976 ms (Python) vs 857 ms (Rust)</li> </ul> This pattern holds for smaller number of intervals as well: Rust is about as fast for integers and floats whereas Python is about twice as slow for integers, and a bit extra slow for floats. Also note that this call of Rust via Python is significantly slower than the 38.1ms or 68.4ms for the same setup in pure Rust above. Again the same two factors appear like in the previous entry in this blog series: there is the extra overhead of code that is needed to get this translation to work, and next to that there are a million intervals (so 3 million values) that needs to be transferred from Python to Rust. Together, this shows that for the Python package, the Rust implementation of the core functionality can result in a good speedup, but that it is also worthwhile to maintain the pure Rust version of the package for cases where extra speed is absolutely critical. Having it confirmed that all these efforts are worth it, now we move on to the next step: including the Rust code in my published PyPI package as well. Building a Python package with Rust code</a> </h2> In order to build intervalues</code> with the Rust code in it as well in the way I did it, there were a few things I had to do: Update my pyproject.toml</code> to include setuptools-rust</code></li> Add a rust folder to my Python repo with a Rust-to-Python wrapper in it</li> Add a MANIFEST.in to make the build process aware of the above files</li> </ul> Let's go over them one by one! Updating pyproject.toml</a> </h3> To adjust the build process to make use of the Rust packages as well, we need to update the file that is the input to the build process: pyproject.toml</code> (of an alternative to that if you use it). In this toml, we need to update the build-system</code> section: [build-system] requires = ["setuptools>=61.0", "setuptools-rust"] build-backend = "setuptools.build_meta" </code></pre> Also, we need to add a new section specifically for setuptools-rust</code>: [[tool.setuptools-rust.ext-modules]] # Private Rust extension module to be nested into the Python package target = "intervalues_pyrust" # Name of the package/module as defined in Cargo.toml path = "rust/Cargo.toml" # The location of the Cargo.toml binding = "PyO3" # Default value, can be omitted </code></pre> You can see here already a peak of the name and location of the rust files in the next subsection. Next to this, you can specify which binding to use, which is here defined as the default PyO3</code> (which is that same technology as what maturin uses). Finally, an issue I encountered was that I specified my version in a version.py</code>. For my setup, this will not work, since the python modules can't be imported before the rust-wrapper has been made available to Python. If you are in the same situation, you have two basic choices: overhaul your code to delay the import of the rust wrapper, or instead put your version information in a non-Python file. I have chosen the latter using the good old VERSION</code> setup: [tool.setuptools.dynamic] version = {file = "VERSION"} </code></pre> Add a rust folder with the wrapper functionality</a> </h3> I created a very simple rust</code> folder in the root of my Python repository containing (in essence) not much more than a Cargo.toml</code> and a src/lib.rs</code> file with all its functionality. In the lib.rs</code> file the two combine_intervals</code> functions for int and float input are defined, which both call the same combine_intervals</code> from the actual Rust intervalues</code> package but with different before steps (the ints are isize</code>, whereas the floats are converted to IntFloat</code>). This is confirmed by the associated Cargo.toml</code> as well, which is setup using the same changes as needed for a Maturin build: the dependencies.pyo3</code> section with the specified ABI, the crate_type</code> being set to cdylib</code>, and the opt-level</code> set to 3. Next to that, it has dependencies set to the main intervalues</code> and intfloat</code> since it uses them both (and also itertools</code>). The name is specified as intervalues_pyrust</code> in both the package and lib section of the toml. Note that these two files are all that is technically needed (plus the automatically generated Cargo.lock</code>), but I also included a LICENSE and README.md file for publishing this intervalues_pyrust</code> to crates.io as well, and a separate pyproject.toml</code> with all the Maturin-specific definitions in case I want to try out something quickly without fully building the Python wheel. Why all of this work for this separate wrapper with a new name? A couple of reasons: I wanted to split the Rust and Python work as much as possible in separate repositories.</li> I also wanted to split the functionality of the pure Rust version and the wrapper, such that I can individually maintain and update those</li> With the default intervalues</code> name for the rust package, there was some potential for conflict with the folder with my Python code having the same name (and during the Maturin test phase, this really was an issue). Of course, I can also just rename that folder without impacting the name of the package, but I prefer not to overhaul the Python part due to some non-Python reason.</li> </ul> I didn't have to publish this intervalues_pyrust</code> on crates.io as well, but I felt like doing that anyway, at the very least to claim that name. Add a MANIFEST.in file</a> </h3> Finally, the shortest addition is a MANIFEST.in file at the project root. This file will make sure the build process can find and use the files in the rust folder. It is in my case defined as: include rust/Cargo.toml recursive-include rust/ *.rs </code></pre> The result of all this is a buildable Python wheel that contains the Rust code. Before uploading to PyPI I did the basic checks: can I install it locally on my system? Can I install it in a Docker image? In both cases, can I also use it to run the Rust specific functions? All systems were go, so let's publish to PyPI using twine like I normally do. And then it happens: the error that I can't upload my wheel because it is compiled specifically for my system. (Like the issue described here</a> on StackOverflow). Publishing a Python package with Rust code</a> </h2> So, the main lesson from the problem above: by compiling my Rust code, I can't share it in the same way I can share my Python packages. Since "normal" intervalues</code> is written in Pure Python, it can be made available in the broadest way possible, with the only restriction due to me using some syntax introduced in Python 3.10. But now, in the way I have build my Python&Rust combined intervalues</code>, it can only be used on a Linux system setup similar to mine, and that is not acceptable for PyPI. So, how to solve this? Like mentioned on the StackOverflow link above, for this we can use the manylinux</code> project and the auditwheel</code> tool. This project has as goal to make tools available that will make it you can build a Python package with compiled code and still make it available across "many linux setups" (hence the name). They do this by offering Docker images that have been configured with compiler toolchains from some time ago, making sure that all recent Linux distributions support the output of it. They offer multiple versions depending on how far back you want to go. One of the other software in these Docker images is the auditwheel</code> tool to be used to make sure the wheels that are built indeed conform to all requirements needed for uploading to PyPI. In order to use this, download the build-wheels.sh</code> file on this link</a> and adjust it to your project needs. For intervalues</code> you can see this in the repository</a>. My adjustments are to target Python 3.10 and up, together with Pypy 3.10. Next to that, of course, I had to adjust the example project name to intervalues</code>. Then, to use this with the Docker image, pull the one you want to use (e.g. docker pull quay.io/pypa/manylinux2014_x86_64</code>) and from the project root run the script above using this Docker image with the following command: docker run --rm -v `pwd`:/io quay.io/pypa/manylinux2014_x86_64 bash /io/build-wheels.sh </code></pre> Note that this will take more time than a normal Python package build, because it will do the build and checks for all specified versions of CPython and PyPy. The result is also a set of wheels for each of these, instead of a single one for Python in general. These need to all be uploaded to PyPI to make available for installation. But then after all this.. it works! I can pip install</code> the new version of intervalues</code> and make use of the Rust functionality in both Python 3.10 and PyPy 3.10. The only downside is that it is now (afaik) Linux specific: this newest version (as of writing) can't be installed on Windows whereas the version before can. I can't verify this since I don't have a Windows box available right now, but this is what I have been able to gather from what I read about it. To be honest, I personally don't really care about that. But if there is an intervalues</code> user who wants me to find out if/how it can still be used on Windows even with this Rust functionality, let me know (or, of course, submit a PR yourself). Alternatively, the old version still works on Windows and there are no real changes from that version to this one outside of the Rust wrapper. Alternative setups</a> </h2> There are several alternatives to consider to some of the steps that I have taken, which I might experiment with in the future. Experiences from readers are also welcome, especially if these alternatives work really good or really bad. Maturin allows you to directly publish the Python package with Rust functionality using maturin publish</code> instead of maturin develop</code>. I have not tried this, so I don't know whether the result is OS-agnostic or if it uses some kind of manylinux</code> restriction as well, or something else. Also, this feels like it would make most sense if the wrapper was included in the base Rust intervalues</code> package, since it will introduce another package itself (a Python package that the main Python intervalues</code> package can include as a dependency).</li> I want to look into how to make the Rust component optional, and if that makes the core Python functionality still usable on Windows. I could (either by using Maturin as above, or manually) publish the Python side of the Python/Rust layer separately from the rest of the Python code, and make it an optional dependency of the main intervalues</code>.</li> Perhaps there also other ways of doing this</li> </ul> </li> Ideally, I want to have the Rust setup also usable on Windows. Like discussed in the previous section, I will likely not look into this myself until a Windows user requests this, especially if I can figure out how to create non-Rust future versions as well that are more universally usable. Probably the first candidate to explore for this functionality is to use the cibuildwheel</code> as mentioned in the setuptools-rust</a> docs as well.</li> </ul> </li> Let me know if you have other suggestions as well for how to best set this up!</li> </ul> Depending on these investigations and/or suggestions, there might be a future entry in this series to discuss them. But for now, the next (and potentially final) entry in this series will be on my overall learnings and experience</a> of using Rust in Python. Alternatively, have a look at some of the links down below for further reading. Links</a> </h2> Some of the other Python interval packages: portion</li> pyinterval</li> </ul> </li> My repo's and packages related to intervals: Python intervalues: github</a> and PyPI</a></li> Rust intervalues: github</a> and crates.io</a></li> intervalues_pyrust: crates.io</a></li> intfloat: github</a> and crates.io</a></li> </ul> </li> More resources on setuptools-rust: Overall doc on setuptools-rust</a></li> If you want to use setup.py instead of pyproject.toml</a></li> Building wheels</a></li> </ul> </li> More resources on manylinux and auditwheel: The manylinux repository</a> with details on what the different manylinux</code> versions imply for Python and distribution support.</li> The auditwheel tool</a> showing examples of other commands that can be supplied to it, for example for only checking (not repairing) existing wheels for PyPI compatibility.</li> </ul> </li> </ul>

Rust in Python part 2: more detailed example

2024-10-30T00:00:00+00:00

After the instructions on setting up Maturin for the "simple" Fibonacci example, you might have questions how well this generalizes to bigger actual problems. Of course, the answer will be "it depends", and to get some more intuition on what it might depend on, let's first look into a bigger example that is still not an actual problem: day 5 of Advent of Code 2018.

Rust in Python part 1: getting started

2024-10-23T00:00:00+00:00

On this page, I will talk you through how to get started with using Rust in Python. For this, I will assume you know some basic components of Python (defining a function, importing a package, etc). To add on that, this page will present:

Rust, & how to get started with that up to a sufficient level for the goal of a single Python function call</li>

An example Python project</li>

The same functionality written in Rust</li>

How to compile and expose Rust code for use in Python</li>

How to adjust Python code to use this Rust code</li>

Comparison of run time</li>

More links and references</li> </ul>
The goal of this is to present a single solution that you can just copy to get started. Later on I will introduce some choices and alternatives that you might consider, but I want you to get to the power Rust from Python as fast as possible. So let's dive in!
Getting started with Rust </a> </h2>
here</a>.
To install Rust, follow instructions on section 1.1</a> of the book. In case you are using Linux like I do, that amounts to running:
curl --proto '=https' --tlsv1.2 https://sh.rustup.rs -sSf | sh </code></pre> This downloads a shell script that installs the rustup</code> tool which is used to maintain versions of Rust. More details can be found on the linked page. The next step is to "just learn Rust". But don't feel that this is too big of a hurdle! As mentioned on the introduction page, you don't need to know all concepts of the Rust language to get to a stage where you can already replace a specific Python function with Rust calculations - often the ones for which this matter the most can be written using the most basic components of the language. More complicated stuff will only be relevant when more of your codebase will be written in Rust, which will come later together with learning those details later. So to get that basic level of Rust as needed for this series, continue reading "The Book" up to (and including) chapter 4. This means you will learn about how to compile and run a piece of Rust code, how to work with variables, functions, if-statements, for-loops and comments, which are all things that should be familiar to you from the Python world, although some things in Rust are defined stricter. Next to that, you will be introduced to a thing that almost uniquely defines Rust: its ownership rules and borrowing system.Then, you can come back here. Or, you can continue on reading about Rust. If you do, you will learn more about how Rust works which allows you to write more advanced stand-alone or library code. I can suggest combining that with doing the Rustlings</a> exercises, which mostly follow along with The Book. In any case, at some point I hope you will be back here, either with some basic or more advanced knowledge of Rust. (If you decide to just read along without knowing Rust, I think you will be able to understand most concepts as well, although some terms might not make sense to you). Example project</a> </h2> On this page, we will show an example project in Python, that we will adjust to make use of calculations in Rust instead. In this, for instructive purposes, suppose we want to calculate the nth number in the Fibonacci</a> sequence, defined by the sum of the previous two values with 0 and 1 (or 1 and 1) as first two values. We will do this in a stupid way in both Python and Rust, which will help to demonstrate the speed gain without being smart in the first way</a>. Initialize the project</a> </h3> To start out, first go into the folder in which you put your projects and run cargo new rust_in_python</code>. This will create a folder of that name with in it a few Rust related files that we will adjust along the way. But first, let's make sure our Python setup that we will aim to replace is there. Note: in this setup we will make use of Maturin for building our Python packages with Rust compute. Make sure in your Python setup you have maturin</code> and patchelf</code> available, either by running pip install maturin[patchelf]</code> or separately installing both packages. I recommend installing these not for your system python interpreter, but using a virtual environment or something similar (which is outside the scope of this series, but if you are new to it, you can read up on it here</a>). Python implementation</a> </h3> The function that we will use to calculate the nth entry in the Fibonacci sequence in an inefficient ("stupid") way is: def fibo(n): if n == 1: return 1 if n == 2: return 1 return fibo(n-2) + fibo(n-1) </code></pre> Now I can talk about what the stupid approach is: this approach will recalculate the same fibonacci number multiple times. The number at n-1</code> is calculated 1, but the number at n-2</code> is calculated twice: once for calculating n-1</code>, and once in the final sum for n</code>. It only gets worse from here: n-3</code> is calculated three times (two times for n-2</code>, and then once more for n-1</code>), n-4</code> five times, n-5</code> eight times, et cetera. (Do you see the pattern in here?) A more efficient calculation would store the already-calculated values in some way or avoid having to do the calculation multiple times in another way. But remember: we are not trying to be type-1 smart here. So, we use the calculation above to calculate, say, the 40th entry in the sequence: import time before = time.time() result = fibo(40) after = time.time() print(f"Result in Python after {after-before:.2f} seconds is: {result}.") </code></pre> which prints on my PC Result in Python after 19.07 seconds is: 102334155. </code></pre> Let's see what this looks like in Rust! Rust implementation</a> </h3> The analogue function definition in Rust could be written like: pub fn fibo(n: usize) -> usize { if n == 1 || n == 2 { 1 } else { fibo(n - 2) + fibo(n - 1) } } </code></pre> Note that if you got further than chapter 4 in the Book of Rust, you will have been introduced to the match</code> syntax which is an alternative and possibly more "rustonic"; but this version works as well and contains no Rust syntax that is not introduced in the first 4 chapters. To run the above as pure Rust code, you could put the above directly in main.rs</code> in your src/</code> folder. The result could be like what follows, which also includes Instant</code> from the standard Rust library to create a simple timer: use std::time::Instant; pub fn fibo(n: usize) -> usize { if n == 1 || n == 2 { 1 } else { fibo(n - 2) + fibo(n - 1) } } fn main() { let before = Instant::now(); let res = fibo(40); let after = Instant::now(); println!("Result in Rust after {:?} is: {}.", after - before, res) } </code></pre> On my PC executing this with cargo run</code> results in the output: Result in Rust after 650.475884ms is: 102334155. </code></pre> Two important observations: Happy to see the answer is the same! Otherwise we'd have a problem..</li> Also, happy to see that Rust is much faster! Otherwise, we'd be doing this without gain.</li> </ol> Rust adjustments for use in Python</a> </h3> To convert the above to something that can be used in Python, some things have to be changed: Move the fibo</code> function above to lib.rs</code> in the src/</code> folder</li> In main.rs</code>, add use fibo_rust::fibo;</code> to the top</li> In lib.rs</code>, adjust fibo</code> to be a pyfunction (see below)</li> Also add fiborust</code> as a pymodule (also see below)</li> Update the Cargo.toml</code> to make use of pyo3 (see even further below)</li> </ul> The resulting lib.rs</code> becomes: use pyo3::prelude::*; #[pyfunction] pub fn fibo(n: usize) -> usize { if n == 1 || n == 2 { 1 } else { fibo(n - 2) + fibo(n - 1) } } #[pymodule] #[pyo3(name = "fibo_rust")] pub fn fiborust(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(fibo, m)?)?; Ok(()) } </code></pre> In this, the name fibo_rust</code> is used to make clear what you will import from in Python. What additionally is important to note: we are importing from pyo3</code> on top, which is the package Maturin uses behind the scenes for the Rust-Python integration</li> each function you want to use in Python needs to be flagged with #[pyfunction]</code>. This will (among other things) make sure the Rust compiler will check whether this Rust variable can be converted to a Python variable (which is fine for builtin variable types like usize</code>)</li> each function you want to use in Python also needs to be included in the #[pymodule]</code> section using the add_function</code> call. The other syntax used in the fiborust</code> definition is outside the scope of this blog.</li> </ul> The updated Cargo.toml</code> will have to contain the following lines: [lib] name = "fibo_rust" crate-type = ["cdylib", "lib"] [dependencies.pyo3] version = "0.22.5" features = ["abi3-py310"] </code></pre> This specifies: the name the library will have in Python (as a module) - I can recommend something different than the folder name</li> the library-type, which next to normal lib</code> now also needs to be cdylib</code> (you can also leave out lib</code> but then the result is no longer runnable in Rust by itself using cargo run</code>; in that case, you can remove main.rs</code>)</li> the version of pyo3 to use</li> which Python ABI to use as a lowerbound - I have selected Python 3.10 here but other values can be used as well (up from py38 for Python 3.8 as the lowest)</li> </ul> Compiling the Rust package for Python use</a> </h3> Next, we will finally be using Maturin for actually making the Rust function available to be called in Python. From the same directory as that contains your src</code> folder and your Cargo.toml</code>, run the command maturin develop</code> (with your virtual environment active, if you are choosing to make use of that setup). This will automatically compile the Rust code, wrap it with some Python-to-Rust and Rust-to-Python variable converters, and put it in your site-packages</code> in the currently used Python interpreter. That means it is importable from Python! Adjustments in Python code</a> </h3> To import and use the package in Python, adjust the above Python script to the following: from fibo_rust import fibo import time before = time.time() result = fibo(40) after = time.time() print(f"Result in Python with Rust after {after-before:.2f} seconds is: {result}.") </code></pre> Note that this is the same code as above, with just the fibo</code> function definition replaced with importing fibo</code> from the fibo_rust</code> module. As a Python user, nothing would make it seem that this module is not just a native Python function. Except for the second note, namely that your IDE might not (immediately) recognize this new module and flag it as an import that shouldn't work, even though it does work. Your mileage may vary. On my computer, running the above code prints: Result in Python with Rust after 0.67 seconds is: 102334155. </code></pre> It works! The runtime of 0.67 seconds is roughly comparable with the 650ms reported by Rust, and significantly lower than the initial 19.07 seconds from the first Python version. For this, we needed only very basic Rust knowledge, a very simple reimplementation of the calculation in Rust without changing the internal logic, and the setup with Maturin and the adjustments in Cargo.toml</code>. Not too much effort for around a 30x speed boost. Great! At least.. as long as this actually is a realistic representation of what you might achieve in practice. Is it? It can be, but by how much can differ. Let's look at a different example that illustrates a potential limitation in the next part</a>, or check out some additional background and reading below first. Links</a> </h2> For full code used in this example, have a look at the rust_in_python</code> folder in my websites_examples</a> repository. In case you'd like to view some other examples of using Maturin, have a look at these links: On the main website of maturin, there is a different tutorial</a> to get you started. This one uses a bit more advanced Rust syntax, but can be helpful if you are also a bit further in your Rust journey.</li> Some well-known Python packages use Rust and Maturin under the hood, like polars</a> and ruff</a>.</li> </ul>

Rust in Python part 0: why and how?

2024-10-18T00:00:00+00:00

When coding in Python, you will occasionally be in a situation where a single function or a few functions are responsible for the majority of your execution time. In most cases, that is fine: if for a particular task, 95% of your time is spent in a single function but the total run time is less than a second, and this particular task is not ran frequently, then there is no reason to worry about speed improvements.

But of course, the opposite will happen as well: this 1 second task needs to be ran 1000 or 1000000 times each time you do a particular broader task (say, a model run in your data science research; or a ingestion of data in your pipeline; or every time a page loads on your website). Bringing down this 1 second to a tenth of second will be worth it in aggregate. If for this or another reason you want to speed it up, there are roughly two categories of things you can do:

Links </a> </h3>

The Advent of Code</a> website</li>
My Advent of Code personal repository</a></li>
The section in the Rust book that discusses compiler flags in more detail is section 14.1</a></li> </ul>

Initialize the project
`here</a>).</p>`

Rust adjustments for use in Python

Why is Rust faster than Python?
Does this only hold for Rust?
What is coming up?
first page</a> of this series.</p>

Does this only hold for Rust?
What is coming up?
first page</a> of this series.</p>

What is coming up?
first page</a> of this series.</p>

My Blog

Advent of Code is coming. Why I participate again and again.

Rust in Python: summary of learnings so far

Rust in Python part 3: publish a Python package

Rust in Python part 2: more detailed example

Rust in Python part 1: getting started

Rust in Python part 0: why and how?