Welcome to aiosmf

Please see the links below or in the sidebar for instructions on installing aiosmf and getting started with an example project. The main project can be found at https://github.com/noahdesu/aiosmf where issues and pull requests can be submitted.

Installation

There are three main components needed to use aiosmf:

  1. The flatbuffers compiler
  2. The smf rpc service definition compiler
  3. The aiosmf package (you’re in the right place!)

The aiosmf library may be installed using pip:

pip install aiosmf

Most Linux distributions include the flatbuffers compiler. Please consult the flatbuffers website or your distribution package manaager for installation instructions.

Installation instructions for the smf compiler can be found at https://github.com/smfrpc/smf which must currently be installed from source. There you can also find links to instructions on using smf to build high-performance rpc servers with c++.

Getting started

The current verison of aiosmf only support Python-based smf clients. In order to have a working example we need a server, which can be built using C++, Go, or Java. We’ll be using the C++-based server described in this introductory smf post:

This is the rpc specification for the example server. The Put interface receives a PutRequest and returns a PutResponse. The first step in using aiosmf is compiling this specification.

namespace kvstore.fbs;

table PutRequest {
  key: string;
  value: string;
}

table PutResponse {
  text: string;
}

rpc_service MemoryNode {
  Put(PutRequest):PutResponse;
}

First use the flatbuffers compiler to generate a Python library that implements each of the PutRequest and PutResponse types.

[user@localhost testtest]$ flatc -o . --python kvstore.fbs
[user@localhost testtest]$ ls -l kvstore/fbs/
total 8
-rw-rw-r--. 1 user user    0 Mar 30 08:18 __init__.py
-rw-rw-r--. 1 user user 1277 Mar 30 08:18 PutRequest.py
-rw-rw-r--. 1 user user  939 Mar 30 08:18 PutResponse.py

Next use the smf compiler to generate a Python smf client specific to this service.

[user@localhost testtest]$ smfc --filename kvstore.fbs --language=python
[user@localhost testtest]$ ls -l kvstore/fbs/
total 12
-rw-rw-r--. 1 user user    0 Mar 30 08:18 __init__.py
-rw-rw-r--. 1 user user  420 Mar 30 08:19 kvstore_smf_client.py
-rw-rw-r--. 1 user user 1277 Mar 30 08:18 PutRequest.py
-rw-rw-r--. 1 user user  939 Mar 30 08:18 PutResponse.py

This is the contents of the generated client. The client accepts an smf connection (which we’ll discuss next), and a method for each of the service interfaces. In this case that is a single Put method.

# Generated by smfc.
# Any local changes WILL BE LOST.
# source: kvstore.fbs

import kvstore.fbs.PutResponse

class MemoryNodeClient:
    def __init__(self, conn):
        self._conn = conn

    async def Put(self, x):
        # request id = 504045560 ^ 3345117782
        buf, status = await self._conn.call(x, 3647565230)
        return kvstore.fbs.PutResponse.PutResponse.GetRootAsPutResponse(buf, 0), status

Finally we can begin building a sample client. The first thing that is done is to establish a connection to the smf server, and pass this connection to an instance of the generated client:

import asyncio
import flatbuffers
import kvstore
import aiosmf
from kvstore.fbs.kvstore_smf_client import MemoryNodeClient
import kvstore.fbs.PutRequest

async def main():
    conn = await aiosmf.create_connection("127.0.0.1:20776")
    client = MemoryNodeClient(conn)

Once the connection is established the rpc methods can be invoked. To invoke the Put method we must first create a PutRequest. This is done using the standard flatbuffers api which results in a buffer containing the serialized form of the request:

# build an rpc request buffer
builder = flatbuffers.Builder(32)
key = builder.CreateString("my.key")
value = builder.CreateString("my.value")
kvstore.fbs.PutRequest.PutRequestStart(builder)
kvstore.fbs.PutRequest.PutRequestAddKey(builder, key)
kvstore.fbs.PutRequest.PutRequestAddValue(builder, value)
put = kvstore.fbs.PutRequest.PutRequestEnd(builder)
builder.Finish(put)
buf = builder.Output()

And finally the remote method is invoked and we print out the results.

resp, status = await client.Put(buf)
print(resp.Text(), status)

The client and the connection should be shutdown to cleanup resources:

conn.close()
await conn.wait_closed()

Invoke this sample client using an asyncio loop:

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

If your server uses zstd compression add incoming and outgoing filters to the connection:

conn = await aiosmf.create_connection("127.0.0.1:20776",
    incoming_filters=(aiosmf.ZstdDecompressionFilter(),),
    outgoing_filters=(aiosmf.ZstdCompressionFilter(128),))

Development notes

When making changes to the smf flatbuffers idl make the following updates if still relevant.

diff --git a/aiosmf/smf/rpc/compression_flags.py b/aiosmf/smf/rpc/compression_flags.py
index b8bbfb6..2234e32 100644
--- a/aiosmf/smf/rpc/compression_flags.py
+++ b/aiosmf/smf/rpc/compression_flags.py
@@ -17,4 +17,4 @@ class compression_flags(object):
     zstd = 2
 # /// \brief lz4 compression
     lz4 = 3
-
+    max = lz4

Convenience copies of the compression flags should also be updated in constants.py

diff --git a/aiosmf/connection.py b/aiosmf/connection.py
index 69ba45c..81dc102 100644
--- a/aiosmf/connection.py
+++ b/aiosmf/connection.py
@@ -148,7 +148,13 @@ class SMFConnection:
         header = aiosmf.smf.rpc.header.Createheader(builder, ctx.compression, 0,
                 ctx.session_id, len(ctx.payload), checksum, ctx.meta)
         builder.Finish(header)
-        return builder.Output()[4:]
+        # XXX: the flatbuffers python code doesn't offer a sizeof option for
+        # structs and the serialization also adds a size header into the buffer.
+        # so when integrating an update the codegen make sure that the header is
+        # stripped off and the size is correct.
+        buf = builder.Output()[4:]
+        assert len(buf) == 16
+        return buf