How to write systemd daemons using Python

Note: You can find all the code at every step of the process in my GitHub repository https://github.com/martinberoiz/daemon.

A daemon is a basically a background process; a program that runs continuously, most of the time waiting for external input to do specific work.

In this article I explain how to write and deploy a Linux systemd service using Python.

Linux adopted systemd as its “System and Service Manager”, replacing the previous init system.

Systemd is tasked with starting and managing several “services”, ie., background processes that provide system functionality on demand.

It is customary to add the letter d (for daemon) to programs that will run as services.

Run a Python script as a background process

So, starting from the basic, Python scripts are files that are meant to run from the command line.

For example

import time
DELAY = 10
while True:
    time.sleep(DELAY)
    print("Hello from a script!"
          " It's been {} seconds since last time.".format(DELAY))

If we run it,

$ python myscript.py

It will run forever and print a statement every 10 s.
To stop it, simply enter ctrl+c (^C) on the console and it will kill the running process.

To make the process run in the background, we can append & to the command

$ python myscript.py &

The use of the end ampersand will push myscript to the background and return the prompt for the next command. This way we have created a background process.

There are two problems with this approach.

The first one is that the process is terminated as soon as the user logs off.

The second is that there is no redirection of stdout and stderr (we could improve the example with pipes redirecting these two, but it’s beyond the scope of this article).

To remedy these, we can use the command nohup, which is a POSIX command that ignores terminal’s termination signal once we log out, so that the process can keep running.

$ nohup python myscript.py &

This last command would be enough to create a daemon, and we could finish our tutorial at this point.

Note: To terminate background processes initiated by nohup or appending ampersand, you can find the process PID with `ps -e` and terminate with `kill`

$ ps -e | grep python
...
17490 ttys000 0:00.03 python myscript.py
...
$ kill 17490

Systemd Services

We can also make use of systemd to launch the process at start-up and restart it whenever it crashes to make sure we can always have it running. Systemd also provides a unique interface to start, stop and restart services. This last one (restart) is useful if we changed any configuration file and we want to restart the service with the new one.

The process to create a systemd service is fairly simple. All we need to do, is create a .service file in /etc/systemd/system/ directory with the name of the service.

For our example we’ll use the file name mydaemond.service. Like we said before, it is customary to append a letter d after the service name.

Below is a minimal example of a service file.
We will copy the following to the file /etc/systemd/system/mydaemond.service
(Like we said before, it is customary to append a letter d after the service name.)

[Unit]
Description=My Awesome Service
After=network.target

[Service]
Type=simple
ExecStart=/home/myuser/myscript.py
Restart=on-failure

Once the file is created we can start our service invoking `systemctl`:

$ systemctl start mydaemond

And stop it:

$ systemctl stop mydaemond

Simplifying the installation

Installing the python module with pip

It may be convenient to provide our users with an easy way to install both our python script and to register our service.

The standard way to install packages in Python is using pip, and for that we need a setup.py file.

A minimal setup.py to install a Python module looks like this:

from setuptools import setup
setup(name='mymodule',
      version='0.1',
      description='My Awesome Python Module',
      py_modules=['myscript', ],
     )

The difference between a Python module and a Python script is that the former is intended to be imported inside other modules or scripts, while the latter is intended to be run completely from the command line.

Modules don’t usually have executable instructions in the global scope, only function definitions (and occasional variable declarations).

Scripts, on the other hand, have mostly executable instructions in the global scope.

We have written a script myscript.py so far (not a module), so this setup.py file would not help us.

We can reach a compromise between a module and a script, however, by encapsulating functionality in function definitions and only call them on the condition that it’s run from the command line.

Let’s modify our myscript.py like this:

import time
DELAY = 10
def main():
    while True:
    time.sleep(DELAY)
    print("Hello from a script!"
          " It's been {} seconds since last time.".format(DELAY))

If we run the script as it is, nothing will be done because the main loop is inside a function that is never called. On the upside, importing it will not do any harm, since the loop is not going to be executed.

Most importantly, our script can already be installed with pip:

$ pip install .
$ python
>>> import myscript
>>> myscript.main()
Hello from a script! It's been 10 seconds since last time.
Hello from a script! It's been 10 seconds since last time.
...

It will still do nothing if executed from the command line:

$ python myscript.py
$

Python modules that can be executed from command line

To fix our broken script, what we need is to add a condition at the end of the file that calls main() only if it’s being executed as a script.

How can we detect if our code has been called from the command line or imported from a module?

When we import our code, python fills the module .__name__ variable to the called module’s name

For example:

import numpy as np
print(np.__name__)

Will print the string numpy, which is the name of the imported module.

But when the module is not imported but rather called from the command line, Python will populate the __name__ variable with the string “__main__”.

So we can bifurcate the code in our script and execute main() only when it’s intended to be run as a script:

import time
DELAY = 10
def main():
    while True:
    time.sleep(DELAY)
    print("Hello from a script!"
          " It's been {} seconds since last time.".format(DELAY))

if __name__ == "__main__":
    main()

This time calling myscript.py from the command line will run forever as it is intended, but importing it as a module will not hang our computer in an infinite loop.

With this modification, our script can be installed as a module with pip as before but can be also run from the command line.

$ python myscript.py
Hello from a script! It's been 10 seconds since last time.
Hello from a script! It's been 10 seconds since last time.
...

Using pip to create an installable script

Using setuptools gives us an extra feature, which is, that now pip can create a small script from our module and place it in our $PATH. If it is installed in a virtual environment, it will be installed in the `bin/` directory, it it is installed globally, it will be installed in /usr/local/bin.

To create this script, we just have to modify our setup.py like this:

from setuptools import setup
setup(name='mymodule',
      version='0.1',
      description='My Awesome Python Module',
      py_modules=['myscript', ],
      entry_points={
          'console_scripts': [
              'mydaemon = myscript:main',
          ],
      },
     )

If we install this with pip, it will create two things.

First, it will install myscript.py as a module (so it can be imported as `import myscript`), this is of not much use for this example.

Secondly, it will create a mydaemon script, which basically calls main from myscript.

$ mydaemon
Hello from a script! It's been 10 seconds since last time.
Hello from a script! It's been 10 seconds since last time.
...

This mydaemon is the one we will use as the systemd service.

You can check that the previous functionality still works as expected.

Automating the installation of the systemd unit

We could ask our user to edit and copy himself the file mydaemond.service, but instead we will create a makefile, which is more familiar to most Linux users, and make the installation process easier.

all: mydaemon mydaemond.service
.PHONY: all mydaemon install uninstall clean

service_dir=/etc/systemd/system
awk_script='BEGIN {FS="="; OFS="="}{if ($$1=="ExecStart") {$$2=exec_path} if (substr($$1,1,1) != "\#") {print $$0}}'

mydaemon: myscript.py setup.py
    pip install .

mydaemond.service: myscript.py
# awk is needed to replace the absolute path of mydaemon executable in the .service file
    awk -v exec_path=$(shell which mydaemon) $(awk_script) mydaemond.service.template > mydaemond.service

install: $(service_dir) $(conf_dir) schedulerd.service scheduler.conf.yml
    cp mydaemond.service $(service_dir)

uninstall:
    -systemctl stop mydaemond
    -rm -r $(service_dir)/mydaemond.service

clean:
    -rm mydaemond.service

The makefile makes use of a .service template file, that we provide below.

# mydaemond.service.template
[Unit]
Description=My Awesome Service
After=network.target

[Service]
Type=simple
ExecStart=/home/myuser/myscript.py
Restart=on-failure

Now to install the service, we just need to make and make install.

$ make
$ sudo make install

The installation requires root permissions because it will copy files over /etc/ which is usually owned by root.

The makefile will basically replace the full path of the mydaemon executable into the ExecStart line using awk and then copy the file over to /etc/.

Makefile commands clean and uninstall are also provided for convenience.

Advertisements
How to write systemd daemons using Python

A web server on a Raspberry Pi

I recently bought a Raspberry Pi mini-computer.

Here I offer a small tutorial for those trying to use it as a simple web server.
This will assume you already built your raspberry pi box with Raspbian installed in the SD card.

I compiled a list of recipes from different websites and pages, in the hope that this will be useful to someone with a similar setup as mine (a macbook with OS X and a wifi router at home.)

You don’t need your pi to be connected to an external monitor and keyboard to make it work. If your pi came with the Raspbian OS then it’s prepared to be accessed through SSH out of the box.

Connect your pi to your router with an ethernet cable or set up your wifi connection (you need to enable wifi first on your pi.) Then you need to find out which internal IP was the raspberry assigned to by your router, and then ssh with user pi and password raspberry. To find the IP of your pi you can check on your router settings, the DHCP Client List. Your pi should be named raspberrypi or similarly.

In principle it’s not guaranteed that the IP will be the same every time the device connects, but in my experience with my router, the IP assigned to each device is always the same.

The first time you log in you will be greeted with a setup menu. I suggest you follow the instructions on this site to get it up and running. Most importantly, change the default password to log in!

Once you know your pi’s IP (in my case was 192.168.2.3, yours may differ) you can log in normally.

ssh pi@192.168.2.3
password: raspberry

Installing Apache 2

The following are steps to install an Apache server on your raspberry pi. Apache is a popular web server in the Linux side of computers. Another emerging contender on the open source side is Nginx (pronounced ‘Engine X’), but I have no experience with it. I should probably buy another pi and try this new server; after all, one of the advantages of the pi is the relatively inexpensive hardware!

Back to Apache, installing it is relatively easy, you just type

sudo apt-get install apache2 apache2-doc apache2-utils

on the terminal and that should be enough.

Test Apache with a simple html page

You can manage several sites on your pi with Apache, all available sites are stored in files under /etc/apache2/sites-available, of those you can enable or disable the ones you like with the commands a2ensite and a2dissite (a2 is for apache2 and the rest is enable/disable site.) These commands basically create and delete symlinks into /etc/apache2/sites-enabled.

Go to /etc/apache2/sites-available. We’ll make a test page to confirm apache is running well. First we copy the default site provided by apache to have a template to work with.

sudo cp default test

Edit the test file replacing instances of “/var/www” to “/home/pi/www”
and create a simple welcome index.html in /home/pi/www

Below is an example of a simple html. If emacs is your preferred command line editor, you will have to install it now, because Raspbian doesn’t include it by default.

sudo apt-get install emacs

then you can create index.html with emacs on /home/pi/www

cd
mkdir www
emacs www/index.html

and copy the following text in it.

<!DOCTYPE>
<html>
<body>
<h1>Welcome</h1>
<p>Website coming soon</p>
</body>
</html>

Then execute the following commands to enable the site (a2ensite), disable the old site (a2dissite) and reload the sites.

sudo a2ensite test
sudo a2dissite 000-default
sudo service apache2 reload

You should see the Welcome page you set up in /home/pi/www, when you visit the pi’s IP (type http://192.168.2.3 on your favorite web browser)

If you want to access your website from outside your internal network (given by your router) you need to direct the external HTTP requests on port 80 to your raspberry pi’s port 80.

If you’re connected through Wifi like I am, then you have a wireless router and you have to configure your router to do the redirection. Select incoming port 80 to map to your pi’s internal IP’s (192.168.2.3 in my case) port 80.

Once you do that, you can access your website through your IP (the one provided by your ISP) on the web browser. You may want to hire a Domain Name Service to assign your IP number to some memorable web address. Some websites offer this service for free, noip.com is one of them.

Please leave comments if you find typos or mistakes!

A web server on a Raspberry Pi

Craster’s Incestual Recursion

Black Holes and Revelations

He marries his daughters, and they give him more daughters, and on and on it goes.
Edison Tollett

During one of my usual afternoon discussions with fellow graduate students, we strayed on the precarious path of incest and argued as to where incest would fit on the nature vs nurture spectrum. I preferred to attribute the cultural embargo on incest to nurture, as incest had been widespread in antiquity, whereas my friend was of the opinion that the incest taboo is hard-wired in human beings, citing the near-universal forbiddance of sexual relations within close family.

This made me wonder about the sustainability of a population where the only means of population growth (for whatever reason) is incest. Those of you who watch/read Game of Thrones will be able to think of just such a population, the Craster’s keep. Craster (who the above quote refers to) is an old…

View original post 655 more words

Craster’s Incestual Recursion

Stupid const, stupid mutable, stupid C++

code monk

Every now and then someone drags me into a C++ discussion. I have friends who code in C++. Some of them do it for money. I’m sorry about that.

Recently I heard about “const methods”. “What’s that?” I ask; “Methods that are contractually obliged not to change their object.”. Oh, I think. That seems really useful, but a feature like that would never make into C++ (I reason, based on what I know of C). Still, not knowing much about C++ I shut up and go and learn some more C++ instead. I consult my friends who actually know more C++ than I do. The replies are not consistent, not confident, and sometimes go all slippery and vague when I press for details.

Then, it dawns on me. A “const method” is simply a method whose this pointer is const qualified. It’s an inevitable feature of the language…

View original post 1,266 more words

Stupid const, stupid mutable, stupid C++

A Camera Class

For those like me that are interested in Game Development as a hobby, one of the main challenges to build a 3D game is understanding the OpenGL framework.

The distinction between the different coordinates systems is mandatory if one doesn’t want to get ultimately lost in coordinates transformations. A world coordinate system is a system of coordinates with which the scene is best described. It is the system that arises naturally from the game constrains and scene. The eye coordinate system is the system OpenGL understands, with the z coordinate sticking out from the screen and x and y running along the screen edges.

Even further, we also need a camera in our scene, that will be the point of view from where we render the scene.

It is very useful to split an OpenGL program into things our virtual objects do in the scene and how the camera moves in the scene. To achieve that split of responsibilities, it is necessary to have a Camera class that encapsulates the state of the camera at all times. The camera should know how to return its view transformation matrix to modify the point of view from which the objects will be rendered. This camera or view transformation will be the matrix that transform the vertices from the world coordinates to the eye coordinates.

For the impatient: a direct link to the github repository containing the Camera Class implementation on Objective-C can be found on this link. It can be easily modified for C++.

This article will try to explain hot to set a view transformation for a Camera class.

One way to implement a camera is to set the position and orientation of the camera in world coordinates. With this one can work out the transformation matrix to be applied to the vertices on the scene. The following will assume that we have a vector that points where the camera is looking at (lookAt), an approximate vector for what is considered “up” in the camera (upVector), and the coordinates of the camera (pos) on the scene. All three vectors in world coordinates. The upVector doesn’t necessarily must be the true upwards vector. It only must have a non-zero component along the true “up” direction.

In what follows, {i, j, k} are the basis vectors of the world coordinate system and {i’, j’, k’} are the basis vectors of the eye coordinate system. In the eye coordinate system -the only one that OpenGL cares about- the eye is always pointing in the -z direction. From the pos and lookAt vectors, we can find what is the k’ direction of the camera in world coordinates.

k’ = normalize(lookAt – pos)

k’ = normalize(poslookAt)

“Up” is in the y direction in the eye coordinate system. The upVector would be j’ if it weren’t for the fact that it’s not necessarily completely upwards. Nevertheless, the cross product between upVector and k’ will be i’, as long as upVector has a component in the true j’ direction.

i’ = normalize(upVector x k’).

Finally the cross product of k’ and i’ will give us the true j’.

j’ = k’ x i’

Now that we have the relative orientation of the camera {i’, j’, k’} with respect to world coordinates {i, j, k} we can figure out the transformation (it’s a rotation) between the two frames. This can be done easily noting that i’ = (i’.i) i + (i’.j) j + (i’.k) k etc. which can be written formally (it’s not a true matrix-vector multiplication)

\left[ \begin{array}{c}    i' \\    j' \\    k'    \end{array} \right]    = \begin{bmatrix} i'.i & i'.j & i'.k\\    j'.i & j'.j & j'.k \\    k'.i & k'.j & k'.k    \end{bmatrix}    \left[ \begin{array}{c} i \\    j \\    k \end{array} \right] =    \left[ \begin{array}{c} i' \rightarrow \\    j' \rightarrow \\    k' \rightarrow \end{array} \right]    \left[ \begin{array}{c} i \\    j \\    k \end{array} \right] \equiv R \left[ \begin{array}{c} i \\    j \\    k \end{array} \right]

where i' \rightarrow means write the components of i’ in the {i,j,k} base in that row. For any other vector (x’,y’,z’) the transformation is the same. It can be seen by writing

\begin{array}{l} v = x' i' + y' j' + z'k' = (x',y',z') (i', j', k')^t = (x',y',z') R (i, j, k)^{t} = (x,y,z).(i, j, k)^t \\    \Rightarrow (x',y',z') R = (x,y,z) \\    \Rightarrow (x',y',z') = (x,y,z) R^t \\    \Rightarrow (x',y',z')^t = R.(x,y,z)^t    \end{array}

(R^t=R^{-1} since it is an orthonormal matrix).

R is the rotation that takes a vector from world coordinates to eye coordinates. Its transpose will take from eye to world coordinates. To put the camera in the world we did two operations. First we rotated the camera at the origin by matrix R^t and then we translated it by vector pos.

Mcam = T(pos).RT

The matrix we have to apply to vertices to simulate the camera motion is the inverse of this Mcam

Mv = R . T(-pos)

Implementation in OpenGL|ES:

OpenGL|ES stores the matrices in a column-major format. (We can equivalently think that it stores the transpose of the matrix in a row-major format.) To construct the matrix R, we have to fill each row with the primed direction versors in the array R[16].

\left[ \begin{array}{c}    i' \rightarrow 0 \\    j' \rightarrow 0 \\    k' \rightarrow 0 \\    0 \ldots 1    \end{array}    \right] =    \begin{bmatrix} R_0 & R_4 & R_8 & R_{12} \\    R_1 & R_5 & R_9 & R_{13}    \\ R_2 & R_6 & R_{10} & R_{14}    \\ R_3 & R_7 & R_{11} & R_{15}    \end{bmatrix}

 for (int i = 0; i &lt; 3; i++) {
   for (int j = 0; j &lt; 3; j++) {
     viewMatrix[i*4 + j] = prime[j][i];
   }
   viewMatrix[i*4 + 3] = 0.;
 }
 for (int i = 12; i &lt; 15; i++) viewMatrix[i] = 0.;
 viewMatrix[15] = 1.;

The translation as an array T[16] is the usual one and has the form:

\begin{bmatrix} 1 & 0 & 0 & t_x    \\ 0 & 1 & 0 & t_y    \\ 0 & 0 & 1 & t_z    \\ 0 & 0 & 0 & 1    \end{bmatrix} =    \begin{bmatrix} T_0 & T_4 & T_8 & T_{12} \\    T_1 & T_5 & T_9 & T_{13} \\    T_2 & T_6 & T_{10} & T_{14} \\    T_3 & T_7 & T_{11} & T_{15}    \end{bmatrix}

But since T is mostly empty we can make the multiplication R.T without storing the array T. The rotation block will remain untouched and R_{12}, R_{13}, R_{14} will be replaced by a linear combination of the rotation terms and the translation terms.

\begin{array}{l} R_{12} \leftarrow t_x R_0 + t_y R_4 + t_z R_8 \\    R_{13} \leftarrow t_x R_1 + t_y R_5 + t_z R_9 \\    R_{14} \leftarrow t_x R_2 + t_y R_6 + t_z R_{10}    \end{array}

In code:

 viewMatrix[12] = - position[0] * viewMatrix[0] \
                  - position[1] * viewMatrix[4] \
                  - position[2] * viewMatrix[8];

 viewMatrix[13] = - position[0] * viewMatrix[1] \
                  - position[1] * viewMatrix[5] \
                  - position[2] * viewMatrix[9];

 viewMatrix[14] = - position[0] * viewMatrix[2] \
                  - position[1] * viewMatrix[6] \
                  - position[2] * viewMatrix[10];

The whole project (WIP) can be found on this link: https://github.com/martinberoiz/CameraClass

Note: I will update this post with more details.

A Camera Class

Sorting Algorithms and the Strategy Design Pattern

There’s an excellent course on Algorithms in the Coursera website. You should go check it out. It is a series of videos and exercises on several well known basal algorithms in Computer Science. If you are not a CS major, but you program as part of your daily routine, I recommend you at least watch the videos, since they are very informative.

Even though the language they use to implement the algorithms is Java, I decided it would be a good exercise for me to try them in C++. I am not particularly fond of C++, but I thought it would be an opportunity to explore more the language.

The first set of algorithms deal with sorting arrays of various types. I think that’s an excellent use case for the Strategy Design Pattern, in which each different sorting method is encapsulated into a subclass of a general ‘sorting’ class. This allows each strategy (method) to share the same outlets and they can be interchanged even at runtime if necessary.

Each individual strategy and its superclass Strategy class are all templates, to allow for sorting of different types of elements (double’s, int’s).

Each strategy will have a name() method that returns a string with the name of the method, a sort() method that does the trick, and a few other class methods.

Helper functions for a particular sorting method, should be kept private so we don’t contaminate the public scope of the strategies and possibly confuse the user of the class.

With the Strategy Pattern we can iterate over sorting strategies (like Selection Sort, Quick Sort, Merge Sort, etc).

In the following example, I have a simple test function called sorttest() that will take a strategy and an array of numbers (called ‘array’ in the code) and will test if the given strategy correctly sorts the array.

The sorting test for a vector of strategies can be done in one loop as is shown below:

 for (typename vector<SortStrategy<T>*>::iterator aStrat = allStrats->begin();
      aStrat != allStrats->end();
      ++aStrat)
 {
   T* data_copy = (T*)malloc(len*sizeof(*array));
   memcpy(data_copy, array, len*sizeof(*array));
   bool pass = sorttest(data_copy, len, *aStrat);
   printf("Test%spassed.\n\n", pass ? " ": " not ");
 }

In summary, the Strategy Design Pattern is very useful in cases when one task can be accomplished in many different ways and one wants to abstract the method to do the task. Pluggable classes are neat!

A copy of my code using the Strategy Pattern and the Sorting classes can be found in this repository https://github.com/martinberoiz/algorithms.

Sorting Algorithms and the Strategy Design Pattern

The percolation problem

The percolation problem deals with connected paths between nodes in a 2D grid. The grid nodes could be in either of two states: “open” or “closed”; “conducting” or “insulating” or any other two opposite physical properties. When open, a node will connect with all the open nodes surrounding it. The system percolates where there is a path of connected nodes from any node on the top row to any node at the bottom row.

The system can be used to model diffusion of a liquid through a porous surface or the electric conductivity of a dielectric.

In the case of liquid diffusion, the grid is said to percolate when there is a connected path from any of the sites at the top row to any of the sites at the bottom row. On that moment, there is a free path for the liquid to move through the grid.

For a given grid size and a given fraction of open sites, there is a certain probability that the grid will percolate. It depends, of course, of the distribution of open sites on the grid.

It turns out that there is a phase transition on the probability of percolation of a two dimensional grid. Below a certain fraction of open sites, the probability of percolation is almost null. Above that fraction, the probability of percolation is almost certain. The percent value that divides both phases is very hard, if not impossible, to find theoretically. A good algorithm to simulate percolation for random grids is in need, to experimentally find this value.

The fast O(N lgN) algorithm to check for connected paths used in Sedgewick‘s Algorithm book is Quick Find or some optimized version of it.

I decided to implement myself these algorithms on C++, using the OO Strategy Pattern.

When I finished coding these command line tests, I thought it would be fun to see it in action for some concrete instance of a grid. That’s why I programmed a simple GUI for OS X.

The MVC (Model-View-Controller) was of great help here. Since I had my strategy classes for Quick Union, Quick Find and Weighted Quick Find as separate classes, I already had my model part done.

I then coded an Objective-C wrapper for the grid. This will be the class aware of the 2D nature of the grid. The model only knows about nodes, but it’s not spatially aware.

I only needed a general controller class, the application delegate provided by Xcode, and a view class to display the grid.

Once all the connections were done, I only added some bells and whistles to the app, and it was done.

An OS X GUI for a Percolation example
An OS X GUI for a Percolation example

I leave here a link to download the app, and I will post the source code in another post.

 

I hope you enjoy it!

The percolation problem