Embedding a Python Application in uWSGI
WyWeb (my software that is running this website right now) uses uWSGI to communicate
with the server. Previously, WyWeb was a loose collection of python files that needed to be run in a virtualenv and
started with a separate uWSGI command and a configuration file. This was fine when WyWeb was fairly bare bones and only
used by me, but as I began to expand its functionality, I wanted to easily share it with others. Having people clone a
git repo, manage dependencies, manually set up systemd services, and configure a server is too much to ask. Luckily, I
found uWSGI’s official documentation on how to embed a python
script in a standalone binary! Unluckily, the instructions are out-of-date at best, objectively wrong at worst, and
written in a tone so intolerable that I thought I had been catapulted back to the heckin’ epicsause year of our Lord
2010. Luckily for you I am recording my steps for success so that hopefully you can be spared the torment I have
endured.
The Official Documentation is bad
Before being helpful, allow me to complain about the official documentation of this topic for a moment. The very first
command the official documentation says to run will fail because it cannot find a file called bootstrap.py.
Frustrating, sure, but I found it in the examples directory quickly enough. Why is something required for building an
embedded app in the examples directory? I have no answer.
Step 2 is not so bad, though they doforget to mention that having the line embed_plugins = python may be useful and
could even be absolutely necessary depending on your use case. To even find the existence of that option, I had to dig
through the source code. I wasn’t even looking for it at the time, but I am so glad I made a note of it.
After failing at steps 1 and 2, the title of step 3, “embedding flask itself” gave me hope - It looks like we have instructions on how to embed a real app along with its dependencies. My hope was immediately and violently exorcised from my body, rent from my psyche, and severed from my soul when my eyes had the misfortune of these words falling upon their retinas. (If you didn’t want to suffer my purple prose, you shouldn’t have been so cringe.)
Now, we are ready to kick asses with uWSGI ninja awesomeness.
Oh god no. Just tell me how to do it. Do not subject me to this. Ok. I can get through this.
I could not get through this. The guide is too short, does not provide resources for any configuration options, and does not describe why certain options are used. In the next section, I will report what finally worked for me, and as I discover more information, I will continue to update this page.
An Actually Helpful Guide
I will assume that you are in the directory for your application, that you have successfully used uWSGI in some way
before (not strictly necessary but helpful), and that all the source files are located in a directory called src/. I
will also assume that you are using Linux.
Set up Files and Folders
To begin, you will need the uWSGI source. I will tell you where to actually get it, and how. We don’t need the full commit history, so we will clone only the files in the commit for release 2.0.23 of uWSGI (the most recent release at the time of writing).
1git clone --depth 1 --branch 2.0.23 https://github.com/unbit/uwsgi.git
Next, we move into the newly cloned repo with cd uwsgi. Now we need to make some configuration files and set up for
building. First, let’s copy all our source files into here and grab bootstrap.py. Is there a better way than
scattering everything into the root folder of the repo? Probably, but it’s not listed in the documentation anywhere.
1cp ../src/* .
2cp examples/bootstrap.py .
Does your project have any dependencies? If so, create a directory called site-packages and copy your dependencies in
there. WyWeb requires lxml, markdown, pygments, Pillow, and ruamel.yaml so I ran the following. You will need to change
which directories you copy unless you happen to need exactly the same dependencies as me. Again, there is probably
a better way. I simply do not care to find out.
1python -m site # list where python packages live
2cd /path/to/system/site-packages
3rsync -av --progress markdown pygments ruamel lxml PIL /path/to/your/project/uwsgi/site-packages/.
Configuration
Now, we need to make a configuration file in buildconf/. Go back to the uwsgi/ directory and using your favorite
editor, create buildconf/myapp.ini with
1[uwsgi]
2inherit = base
3main_plugin = python
4embed_plugins = python ; I had to dig through the source code to figure out that this was needed.
5bin_name = myapp
6embed_files = bootstrap.py,myapp.py,mymodule.py,site-packages/lxml,site-packages/markdown,site-packages/PIL,site-packages/pygments,site-packages/ruamel
7embed_config = myapp-config.ini
Note: Make sure to include all your modules on that incredibly long embed_files option!
If your app does not already have a configuration file for uWSGI, go ahead and create myapp-config.ini. This is
similar to what I am using for WyWeb.
1[uwsgi]
2module = myapp:app
3master = true
4processes = 1 ; or however many processes you want
5need-app = true
6enable-threads = true
7
8socket = 127.0.0.1:3031 ; or unix socket or whatever
9import = sym://bootstrap_py
10plugins=python
11
12disable-logging = false
Building
Finally, we can build our app. Navigate back to the uwsgi/ directory and run
1python uwsgiconfig.py --plugin python --build myapp
2# the flag --plugin python might not be necessary but it worked for me
3# and I'm too much of a coward to go through with all of this again.
And now you should have an executable called myapp, and if it didn’t work you can curse my name ten thousand times as
you venture forth on your own journey of ten thousand open tabs. Godspeed.
In my research, here are some links I found helpful: