Networking without an operating system

Please consider subscribing to LWN Subscriptions are the lifeblood of LWN.net. If you appreciate this content and would like to see more of it, your subscription will help to ensure that LWN continues to thrive. Please visit this page to join up and keep LWN on the net.

At last year's PyCon in Montréal, Josh Triplett introduced the work he and others have done to port Python to run in the GRUB boot loader. At this year's PyCon in Portland, Oregon, he updated attendees on progress that has been made in the BIOS Implementation Test Suite (BITS) to add networking support. True to form, his presentation came with an eye-opening demonstration of the networking implemented in BITS.

BITS is an implementation of Python for GRUB and EFI without an underlying operating system. It is used in production at Intel to explore and test its hardware and firmware. He noted that in previous talks he has demonstrated doing graphics (rendering the Mandelbrot set at last year's PyCon) and an interactive video game at LinuxCon North America, both using BITS.

Most of the Python standard library is available in BITS, but there are parts that are not. Because there is no underlying operating system, functions like os.execve() and os.fork() are not supported. In consequence, there is no support for multi-processing using the subprocess or popen2 modules. There is also no support for the webbrowser module, which "sadly means that you cannot import antigravity ... sorry", he said.

Early on in the project, he and his colleagues looked at what pieces of the standard library made sense for BITS. Modules like socket and select, along with things built atop them, such as urllib2 and httplib, were considered, but he remembers clearly recognizing that those would never be ported to BITS.

Networking in BITS

But EFI supports networking and there are some useful things that could be done if BITS also supported it. It would allow sending scripts and test data to the hardware or reading test logs and other data from devices. That would eliminate the reliance on a writable filesystem or any local storage on the hardware.

As usual, Triplett was using BITS in QEMU on his laptop to display his slides [Speaker Deck] and to run some demos. His first was to run a SimpleHTTPServer on the laptop (outside of QEMU); it just serves up the directory from which it is run over HTTP and is "highly recommended", he said. Then, inside BITS, he did:

    >>> import sys
    >>> print sys.platform
    BITS-EFI
    >>> import urllib2
    >>> print urllib2.urlopen("http://[ip:port]/hello").read()

That took a second, while BITS set up the IP address in the virtual machine, then printed the canonical "Hello world!" string from the file. Doing so again was instantaneous since the networking was already set up. Going back to the HTTP server terminal showed two requests for the hello file as expected.

As might be guessed, it took some work to get there. One way BITS could have provided networking would be for it to call the EFI network protocols directly; BITS already has the capability to do that. But that wouldn't provide compatibility with existing Python network code and modules. "Who knows, someday maybe you will run Django inside your firmware—BIOS as a service", he said with a chuckle.

socket and select

With that compatibility in mind, then, the Python socket and select modules need to be available. In BITS, the socket module from the standard library is used directly. It imports the _socket module, which is written in C and implements the low-level socket handling code for CPython. _socket is where BITS does the work to translate between the sockets API and the EFI networking protocols.

He then gave a bit of an overview of sockets and how they are used. The basic idea is that sockets are the network endpoints that can be used either to connect to a remote endpoint (in a client role) or to listen for and accept incoming connections (in a server role). select() is typically used when you "have a giant pile of sockets" and want to know "which I can do something with". There are a set of standard C calls that are used (e.g. socket(), connect(), bind(), etc.) that are mostly faithfully reproduced in the Python API. Those wanting more details on the socket overview may want to consult the YouTube video of the talk at around the four-minute mark (though it should be noted that the first few minutes of the talk are missing).

In CPython, the _socket and _select modules are implemented in socketmodule.c and selectmodule.c. Both extensively use the POSIX sockets API for C. In order to port them to BITS, that API would need to be implemented. But that would mean calling the EFI protocols from C code, which is something that is not done anywhere else in BITS.

Instead, BITS implements all of it in Python, with a "tiny C helper" to safely handle asynchronous events. BITS already had a ctypes wrapper for EFI. That allows the socket and select modules to call the EFI protocols.

There are some quirks, however. EFI uses manual memory management, while Python uses garbage collection that doesn't know anything about references from EFI. That means BITS must keep Python objects alive as long as EFI is referencing them and explicitly free EFI resources when they are no longer referenced from Python.

EFI has several protocols that are used to do networking tasks. EFI_IP4_CONFIG2_PROTOCOL is used to get the current IP configuration or to start it (and use DHCP to get an IP address) if it isn't. The EFI_TCP4_SERVICE_BINDING_PROTOCOL creates an EFI_TCP4_PROTOCOL, which effectively acts like a socket. It has methods like Configure, Accept, Transmit, and so on. There are a number of quirks, bugs, and workarounds, as well as error handling and compatibility with earlier EFI versions that he said he was glossing over in his presentation, but the result of all of that is present in the BITS code.

There is an impedance mismatch between select() and the EFI protocols, though. EFI can only check for data by receiving it, while select() needs to signal that there is pending data without consuming it. The solution is similar to what operating system kernels do: receive the data (or accept the connection), but buffer it and have select() and other operations consult the buffers to determine if data or a pending connection is available.

The EFI socket is not updated directly from the low-level network interrupts and the data is processed infrequently. A user is expected to call the Poll operation periodically if it is waiting. If that is not done, network performance suffers. So the _select module makes Poll calls in all of its helper functions when "they have nothing better to do".

There is another mismatch in that all EFI calls are asynchronous; callers need to provide a callback function to signal that the call is complete. But there are times when the Python networking calls will need to block. The callbacks are C functions using the EFI API, which needs to be handled safely in Python. But the Python concurrency model makes that tricky.

In Python, all bytecode operations and C calls are expected to run to completion and the global interpreter lock (GIL) helps enforce that. If callbacks can occur at any time, the interpreter data may have inconsistent state at that time. So the callback cannot call any CPython functions. The same problem exists when you hit control-c while the interpreter is running: the signal handler cannot make any CPython calls.

The Py_AddPendingCall() CPython API function is an exception. It can be called from interrupts to register a callback that is to be run at the next safe point (between bytecodes or after a C function has returned). That callback can make other CPython calls. So the EFI callback simply registers a pending call to the Python callback that dispatches the event to an event-specific callback via a dictionary. That dictionary also has the advantage of keeping the Python objects alive while EFI is still referencing them.

So to implement select(), there is a busy loop over the list of sockets calling _read_ready() or _write_ready() as appropriate until the timeout expires. _read_ready() checks the queue, initiates a new Receive or Accept EFI call if needed, or calls Poll if those calls are already pending. _write_ready() simply calls Poll and returns true if the socket is connected since there is no write buffering.

With that all in place, the socket class can be written all in Python. The various socket methods get mapped to the appropriate EFI calls. With the socket and select modules in place, a wide array of higher-level Python network modules will then just work in BITS.

More demos

At that point, Triplett turned back to the demo console. He showed both client and server code within the BITS virtual machine communicating with their counterparts on the outside. He poked around inside the socket object to show how the receive queue worked. Beyond that, he did a POST of some data from outside to a server running in BITS—doing the server HTTP by hand, essentially, and using curl from his laptop. That could be used to implement a read-eval-print-loop (REPL) console in the browser for BITS hardware, which is something he is looking at doing to access devices that are easier to get to via the network, rather than via a console device.

His final demo streamed some data from the SimpleHTTPServer to do "something useful with it". In this case, the useful bit was to display the data frame by frame on the BITS console: a Pinky and the Brain video. "Live video streaming into a pre-OS environment playing video a frame at a time via EFI and Python without an OS", he said to a round of applause. There was no sound and he thought it unlikely that the audio interfaces would be added; on the other hand, if history is any guide, someone may well add that "and it might be us", he said with a grin.

An audience member asked about IPv6 support and Triplett said that it was definitely on the list. EFI supports it as do the various BIOS implementations. One of the other missing pieces, though, is DNS. He used IP addresses in all of the demos because DNS is missing. It is not necessarily all that important for a test lab, so it has not been added yet. EFI has support for DNS, but most of the BIOS vendors do not support it, so it will need to be written as a Python module making DNS queries directly.

The final question was about turning an inherently asynchronous network API into a synchronous one, but Triplett said that was mostly an artifact of the demos that he chose. BITS does implement socket options for non-blocking behavior and he has a multi-user chat server demo that uses that functionality. If asynchronous behavior is desired, it can be done in BITS.

[ I would like to thank LWN subscribers for supporting my travel to Portland for PyCon. ]

Index entries for this article
Conference	PyCon/2016

---

(Log in to post comments)