Embedding a Python Application in uWSGI

Jan 4, 2024Wyatt SheffieldJan 9, 2024

WyWeb (my software that is running this website right now) uses uWSGI to communicate with the server. Previously, WyWeb was a loose collection of python files that needed to be run in a virtualenv and started with a separate uWSGI command and a configuration file. This was fine when WyWeb was fairly bare bones and only used by me, but as I began to expand its functionality, I wanted to easily share it with others. Having people clone a git repo, manage dependencies, manually set up systemd services, and configure a server is too much to ask. Luckily, I found uWSGI’s official documentation on how to embed a python script in a standalone binary! Unluckily, the instructions are out-of-date at best, objectively wrong at worst, and written in a tone so intolerable that I thought I had been catapulted back to the heckin’ epicsause year of our Lord 2010. Luckily for you I am recording my steps for success so that hopefully you can be spared the torment I have endured.

The Official Documentation is bad

Before being helpful, allow me to complain about the official documentation of this topic for a moment. The very first command the official documentation says to run will fail because it cannot find a file called bootstrap.py. Frustrating, sure, but I found it in the examples directory quickly enough. Why is something required for building an embedded app in the examples directory? I have no answer.

Step 2 is not so bad, though they doforget to mention that having the line embed_plugins = python may be useful and could even be absolutely necessary depending on your use case. To even find the existence of that option, I had to dig through the source code. I wasn’t even looking for it at the time, but I am so glad I made a note of it.

After failing at steps 1 and 2, the title of step 3, “embedding flask itself” gave me hope - It looks like we have instructions on how to embed a real app along with its dependencies. My hope was immediately and violently exorcised from my body, rent from my psyche, and severed from my soul when my eyes had the misfortune of these words falling upon their retinas. (If you didn’t want to suffer my purple prose, you shouldn’t have been so cringe.)

Now, we are ready to kick asses with uWSGI ninja awesomeness.

Oh god no. Just tell me how to do it. Do not subject me to this. Ok. I can get through this.

I could not get through this. The guide is too short, does not provide resources for any configuration options, and does not describe why certain options are used. In the next section, I will report what finally worked for me, and as I discover more information, I will continue to update this page.

An Actually Helpful Guide

I will assume that you are in the directory for your application, that you have successfully used uWSGI in some way before (not strictly necessary but helpful), and that all the source files are located in a directory called src/. I will also assume that you are using Linux.

Set up Files and Folders

To begin, you will need the uWSGI source. I will tell you where to actually get it, and how. We don’t need the full commit history, so we will clone only the files in the commit for release 2.0.23 of uWSGI (the most recent release at the time of writing).

1	`git clone --depth 1 --branch 2.0.23 https://github.com/unbit/uwsgi.git`

Next, we move into the newly cloned repo with cd uwsgi. Now we need to make some configuration files and set up for building. First, let’s copy all our source files into here and grab bootstrap.py. Is there a better way than scattering everything into the root folder of the repo? Probably, but it’s not listed in the documentation anywhere.

1 2	`cp ../src/* . cp examples/bootstrap.py .`

Does your project have any dependencies? If so, create a directory called site-packages and copy your dependencies in there. WyWeb requires lxml, markdown, pygments, Pillow, and ruamel.yaml so I ran the following. You will need to change which directories you copy unless you happen to need exactly the same dependencies as me. Again, there is probably a better way. I simply do not care to find out.

1
2
3

python -m site      # list where python packages live
cd /path/to/system/site-packages
rsync -av --progress markdown pygments ruamel lxml PIL /path/to/your/project/uwsgi/site-packages/.

Configuration

Now, we need to make a configuration file in buildconf/. Go back to the uwsgi/ directory and using your favorite editor, create buildconf/myapp.ini with

[uwsgi]
inherit = base
main_plugin = python
embed_plugins = python  ; I had to dig through the source code to figure out that this was needed.
bin_name = myapp
embed_files = bootstrap.py,myapp.py,mymodule.py,site-packages/lxml,site-packages/markdown,site-packages/PIL,site-packages/pygments,site-packages/ruamel
embed_config = myapp-config.ini

Note: Make sure to include all your modules on that incredibly long embed_files option!

If your app does not already have a configuration file for uWSGI, go ahead and create myapp-config.ini. This is similar to what I am using for WyWeb.

[uwsgi]
module = myapp:app
master = true
processes = 1       ; or however many processes you want
need-app = true
enable-threads = true

socket = 127.0.0.1:3031     ; or unix socket or whatever
import = sym://bootstrap_py
plugins=python

disable-logging = false

Building

Finally, we can build our app. Navigate back to the uwsgi/ directory and run

1
2
3

python uwsgiconfig.py --plugin python --build myapp
# the flag --plugin python might not be necessary but it worked for me
# and I'm too much of a coward to go through with all of this again.

And now you should have an executable called myapp, and if it didn’t work you can curse my name ten thousand times as you venture forth on your own journey of ten thousand open tabs. Godspeed.

In my research, here are some links I found helpful: