elektito programming & stuff

DPDK, tamed!

DPDK is a fantastic piece of software. I’ve used it both for work and in my hobby projects (yes, I crunch packets as a hobby, say what you will about me!) and it works great. My only grievance with it has always been the complicated build system it forces on you. It’s ugly and inflexible and it sometimes drives you crazy.

Yesterday, I saw that DPDK is in Ubuntu’s repositories. It took me some time to realize wha that means. I was looking for the value of the RTE_SDK variable when it clicked in my head; I could just use the damn thing like any normal library; with -l, -L, -I and all that. I just needed to add a -msse4.2 flag for the compilation to work properly (I’ve been adding that flag to the TOOLCHAIN_CFLAGS make variable on some of my machines to make DPDK compile anyway).

Does it have any drawbacks? I have no idea, as I never managed to penetrate the many layers of DPDK make files to see what they really do. It’s quite possible that this is not as efficient as compiling DPDK yourself so it can detect and use the full capabilities of your machine, but at the very least I learned a thing or two about building DPDK by looking at the the Git repository of the Ubuntu maintainers. Just do a git clone https://git.launchpad.net/~ubuntu-server/dpdk. Switch to the ubuntu-xenial branch to see the interesting bits.

This seems to be a Canonical endeavor, so thank you kind folks in Canonical!

Controlling rtorrent with XML-RPC

I’m a huge fan of rtorrent. It’s simple, lean and elegant. I have so far only used it as an interactive BitTorrent client inside screen/tmux or as a batch downloader with a watch directory.

There is more to rtorrent however. It supports XML-RPC which means you can control it programmatically. As I was in dire need of a client I can manipulate from a script, I spent some time today to setup rtorrent correctly and call it remotely.

First we’re going to need something like this line in our .rtorrent.rc file.

scgi_port = localhost:5005

You can also make rtorrent listen to a UNIX socket:

scgi_local = /tmp/rtorrent-listening-here.sock

After that, we need a web server to forward our requests to rtorrent, and responses back to us. The web server will be communicating with rtorrent through SCGI. I use nginx for this purpose. Here’s the relevant nginx config for this:

server {
    listen 8000;
    server_name localhost;
    root /rtorrent/;
    location ^~ /RPC2 {
        include scgi_params;
#        scgi_pass  unix:/tmp/rtorrent-listening-here.sock;
        scgi_pass   127.0.0.1:5005;
    }
}

After this, reload nginx config, make sure rtorrent is running, and you’re ready to go.

In order to test your setup, or contact rtorrent from a shell script, you can use the xmlrpc utility accompanied by the libxmlrpc library. You can get this in Ubuntu by running sudo apt-get install libxmlrpc-core-c3-dev. After that, run this:

xmlrpc localhost:8000 system.listMethods

If everything is done correctly, you will get a list of RPC methods supported by rtorrent.

Some useful commands:

  • Add a torrent file and start downloading it:

     xmlrpc localhost:8000 load_start ~/foo.torrent
    
  • See list of current downloads:

     xmlrpc localhost:8000 download_list
    
  • Start a torrent:

     xmlrpc localhost:8000 d.start 07BF391725A0891F3A1E84784B078D5EF337F82F
    
  • Stop a torrent:

     xmlrpc localhost:8000 d.stop 07BF391725A0891F3A1E84784B078D5EF337F82F
    
  • Get the status of a torrent:

     xmlrpc localhost:8000 d.state 07BF391725A0891F3A1E84784B078D5EF337F82F
    

There are a lot more commands. Try some of them for yourself. I admit that it’s difficult to find out how each command works this way, but in the absence of a comprehensive documentation, it seems to be the only way. There are a few examples here.

Complete Vagrant Setup

I have put the complete setup for use with vagrant in this github repository. Simply clone the repo, and run vagrant up in its directory. After the VM is up, you can access rtorrent on the forwarded port 8080:

xmlrpc localhost:8080 load_start /vagrant/foo.torrent

Python/asyncio based BitTorrent tracker

I just uploaded a Python/asyncio-based UDP BitTorrent tracker to my github account. It’s called pybtracker and you can find it here.

You can install pybtracker using pip by running pip3 install pybtracker. You’ll need Python 3.5. After installing pybtracker you can simply run it like pybtracker -b 127.0.0.1:8000 -O. An interactive client is also included which can be used by running pybtracker-client udp://127.0.0.1:8000 (update server address as you wish).

Letting AsyncIO choose a port for you

Say you are writing a test case for an AsyncIO-based network function. You want to write a test server and have the code being tested connect to it. You can choose a port number and hope it’s not taken when the test is run, or you can have a free port chosen for you each time the test is run. Simply pass 0 as the local port number:

    transport, protocol = await loop.create_datagram_endpoint(
        ServerProto, local_addr=('127.0.0.1', 0))
    

Afterwards, you can query the transport object for the port number chosen for you:

    server_addr, server_port = transport._sock.getsockname()
    

And that’s it; now you know the port number. I’m not sure if using a member variable with an underscore at the start of its name is the best or the official way to do this but it’s the best method I’ve found so far.

Python 3.5 in Emacs

April 2016 Update: Native support for Python 3.5 has finally landed in Emacs development branch. You don’t need this patch anymore.

Native co-routine support in Python 3.5 is really cool. So cool in fact that it finally made me switch from Python 2. I never was a big fan of Python 3, but that’s another story. The point is I’m finally ready to write some Python 3 code and I find out that my favorite editor (Emacs) does not support the new functionality. I don’t need anything fancy really. I just need proper syntax coloring, proper indentation and working navigation commands.

The indentation seems to be working fine in Emacs 24 already, but the other two are not. The best I could find was the patch accompanying this message on the Emacs mailing list. All I needed to do was to get Emacs source code (git clone git://git.savannah.gnu.org/emacs.git; there, I probably just spared you one Google search!), make some minor modifications (see below) and apply the patch (git apply py35.diff).

One small thing missing in the patch was proper coloring for the new keyword await. Another problem was that the file locations are not the same as in the git head. I’ve made the proper adjustments to make it possible to apply the patch to the current master branch. You can download the updated patch from the link below.

I should really thank Lele Gaifax from the Emacs mailing list. You made my day stranger!

Download: py35.diff