Cross-platform network programming in wasm/libc

I’m still working on my fledger project. It’s goal is to create a node for a decentralized system directly in the browser. For this I want the following:

  • Works in the browser or with a CLI: have a common codebase but use different network implementations
  • Direct browser to browser communication: use the WebRTC protocol for communication
  • Use rust: because the rust/wasm abstraction is very nice, produces short code, and I can learn more rust

Technical Requirements

After a year of development as a side-project, I discovered the webrtc crate for libc. So I changed the previous node-based CLI to work with this new crate. But I had the following constraints:

  • Libc must use threads, so structures need to implement the Send trait
  • Wasm structures for webrtc don’t support the Send trait
  • Writing networking code with callbacks leads to a lot of Arc<Mutex<>>
  • Dependencies between modules:
    • Unit-testing modules that call other modules is difficult
    • Locking may fail, but the call should not be dropped

So I did what every good programmer does: I created a new library. I called it Broker, because it transfers messages from one module to another. With it the programmer can define modules with input- and output-messages, and then link these modules together.

Modules

In fledger, independent modules hold the functionality of the node. Every module is implemented using the broker library. There are two modules with two separate implementations for libc and wasm: the WebRTC and  the Websocket module. All other modules have a single implementation and can work for both wasm and libc compilation.

Networking

To have a cross-platform code with a common core, two message enums are defined: one for the WebRTC and one for the Websocket broker. Like this, the network module can depend on a Broker<WebRTC> and a Broker<WebsocketMessage>. A rough overview of these messages is like this, very similar for both Websocket and WebRTC:

  • Input
    • (Dis)Connect requests a connection or to closes it
    • SendMessage over an established connection or putting it in a queue
    • RequestStatus of the current connection
  • Output
    • (Dis)Connected once the connection is up or down
    • MessageReceived
    • Status of the current connection, including connection-type

The network broker itself offers more complex messages like SendMessage(NodeID, Message). It takes care of setting up connections as needed. Finally, two crates exist: one for a libc-implementation of WebRTC and Websocket, and one for the wasm-implementation of these brokers.

Functionality

Different modules use the networking module. Each one has its own abstraction and uses its own input/output messages. Currently four modules are connected to the networking module:

  • random connections: setting up a randomly but connected network of nodes
  • gossip events: passing events to and from these nodes
  • ping: test liveness of connections and request a change of connection if there is no response
  • stat: holds the state of the connections as reported by the WebRTC module

Implementations

Wasm and Libc

The wasm modules are implemented using the web-sys crate. This crate offers great compatibility with most of the necessary functions to set up webrtc and websocket connections. It works great in both the browser and as a node module.

For the libc modules, the websocket implementation is straightforward. The WebRTC module is only possible since November 2021 and the arrival of the webrtc-crate. Before the CLI used node to run, which was really not nice and quite performance hungry.

The repository has an implementation of the Broker<WebsocketMessage> and Broker<WebRTCMessage> for both libc and wasm. The CLI depends on the libc implementation, creates a broker using it, and then passes it to the generic flnode structure. And the wasm depends on the corresponding implementation, and does the same. So the node only sees a Broker<NetworkMessage> and can ignore whether the messages are sent back and forth using libc or wasm.

As I wrote above, one thing that still needs to be treated separately is the async_trait and async_trait(?Send) (meaning not to use Send) for the trait-definitions. Keeping it all async_trait doesn’t work for wasm, because some of the fields in the structures do not implement Send.

Broker

The broker is a message passing system that works with async, in Send, and in Send? environments. Every module uses its own broker, but the different brokers can be linked together with translators, which pass messages between two brokers. Internally, each broker has two parts: one that is cloneable and has an Arc<Mutex> of a second part with which it communicates over channels.

To send messages to a broker, one can use an async method which will process the message directly. Another method is to enqueue a message, but then it won’t be processed before the process method is called. As the broker can easily be cloned, it is not a problem to send messages from multiple callbacks: connection completed, message received, channel state changed.

Processing messages from a broker can be done in many ways: through a channel, using a callback, or defining the module as a handler. In the fledger code I use the last one a lot: by implementing an async process_msg method, the module can be passed to the broker and then process the messages.

Finally two brokers can be linked by defining one or two methods to translate the messages between the two brokers. This allows to plug different modules together in a lego-like style. Whenever a broker emits a message, and a translation exists, the corresponding broker will be called to process the new messages.

One disadvantage is that the modules still need a definition of the broker-handlers that is different for the wasm and the libc implementation:

#[cfg_attr(feature = "wasm", async_trait(?Send))]
#[cfg_attr(not(feature = "wasm"), async_trait)]
impl SubsystemListener<GossipMessage> for Translate {
    async fn messages(&mut self, msgs: Vec<GossipMessage>) -> Vec<(Destination, GossipMessage)> {

Theoretically the feature= could’ve been replaced by an architecture=, but this somewhat didn’t work out with VisualCode which uses cargo check. So for now the modules need to be included using a feature-flag.

Interacting with Modules

The main Node structure instantiates all these modules. One of the disadvantages of the broker library is that the modules themselves are passed by value to the englobing broker object. So to interact with the modules, the Node structure can only use the send and receive messages. Each module regularly sends a copy of its states as a message. The Node structure uses this copy to update an internal view of the different modules.

— Linus