Network features

Using pysig for intra-process communication it’s useful, especially for big applications or aplications divided in modules that want to comunicate events to each other without requiring to be aware of when the module is loaded by the main application. It simply favors a lite binding between two endpoints that want to signal specific events.

For small applications though, using a centralized event dispatching framework, sounds more like over-engineering than a good decision. How would be like to have a python application with a couple of functions that signals events to each other via a centralized event dispatching framework?!

But for those apps and not only, a very useful feature of pysig would be to communicate their events over the network. How would be like for your python application, even if it’s small and straight-forward, to communicate it’s results to another application, running in the same network (or even outside the local network) in real time?!

Important

When pysig is running over a network, it still supports the same number of features described before. That is, the following features are still available:

  • registration to specific sender events
  • connect/disconnect events
  • broadcast events
  • channels
  • requests

Use cases

We would like to list just a couple of possible applications for communicating events over a network, using a simple framework like pysig.

Distributed applications

You can create a listener that subscribes itself to several senders and several events, that simply logs the incoming data and processes it in a meaningful way. The senders can connect when they want and signal events to the router whenever their finish their jos and have new meaningful data (e.g. a python script that measures disk usage on each machine).

You can make several different python applications that run on a schedule, by unix-like cron (or Windows Scheduler) and signals their results to the router. Whenever the application that listens for this events it’s up, the data is stored or logged or processed.

This creates a very flexible setup, that can be adjusted on the run with no hassle.

Sensors

Almoast the same scenario as [1], just that in this case the sensor sends the data to our pysig router, based on a trigger that may happen spuriously (e.g. when the light sensor is detecting day or night) and not on a regular basis, as [1] implies.

Anyone interested on the information signaled by these sensors will register theirselfs to the router, for a specific event or for all events triggered by a sensor.

Both the sensors and the listeners can be installed on different machines communicating via the local network or the internet and connected to the centralized pysig router.

Push-like service

We can run a push-like service for your python applications and by using the dispatching mechanism implemented in pysig a listener that connects to the service, will register itself just for the events that presents interest.

The design in pysig is flexible, so you may implement your own message carrier, focusing only on how to transmit and receive data over the network and not the entire dispatching logic, that is already assured by pysig.

Therefore, you can make an UDP carrier, TCP carrier or even HTTP carrier that may run on plain or encrypted channels.

Inter-process communication

Of course, IPC is a good application example of pysig.

If your application requires dispatching events to another application, running on the same machine, pysig can do the job. You can run a server on the targeted machine that handles all the message dispatching, with different processes connect to it via sockets or pipes.

Custom transport

You can define custom carriers for pysig events that can transport them over different communication environments, like serial connections.

Once you define and implement your custom carrier, you can decide how the messages are packed, encrypted or compressed over this transport medium.

Design

In pysig there are three classes which allows senders, listeners and routers to be inter-connected via a common data transportation channel.

Server Router

The first one is the ServerRouter, which is reponsible with receiving commands and messages from its connected clients and dispatch them accordingly. The ServerRouter class is declaring several RPC methods (where RPC stands for Remote Procedure Call) in order to allow a remotely connected sender or listener to register and receive events.

All the senders registered to this endpoint, whether they where registered by the python application that created the object (using direct API calls like ServerRouter.addSender) or they are registered remotely (via a message), are visible to any listener connected to it.

The ServerRouter design is stateful meaning it’s aware of each currently connected listener and each connected sender. Whenever one of them disconnects, it is automatically removed from the dispatching framework. To exemplify this more clearly, if you register a sender on a machine that somehow looses network connection with the ServerRouter, the server will automatically remove sender (i.e. just like if ServerRouter.removeSender was called) and it will fire the sig.EVENT_DISCONNECT for all listeners registered to this event.

Client Router

As you may expect, there is a implementation for the client side too. This ClientRouter allows you to register remote listeners and remote senders to a ServerRouter via whatever transportation carrier you are using.

Using this router, you can register senders and their corresponding events to the centralized ServerRouter and trigger events almoast the same way you would have done using a simple, local Router implementation. Those events will be communicated over network by the ClientRouter.

Important

Please note that there is a slight distiction between the Router and the ClientRouter class. If you use the functions addListener/addSender respectively, you will register listeners/senders only locally visible.

If you intend to add listeners or senders connected to the ServerRouter, you must use the following corresponding set of functions:

  • addRemoteListener / removeRemoteListener
  • addRemoteSender / removeRemoteSender

This distinction is valid only for ClientRouter and not for the ServerRouter where all registered senders or listeners are visible to the connected clients.

For firing requests you must use remoteRequest instead of request function.

Carrier

The Carrier implementation is the main actor of this remote signaling feature of pysig. The class is expected to be inherited by the one that really implements the transportation layery.

The role of the Carrier is to provide an abstract API for the ServerRouter and ClientRouter for sending and receiving messages.

The Carrier class defines the following methos, that MAY or MUST be implemented.

Carrier.pack(self, message)

This methods packs the received message to a format that is acceptable by the carrier. It returns the object containing the packed data or None in case of an exception.

The default implementation uses json module and encodes the message in a json object, therefore the method MAY be overrided.

Carrier.unpack(self, data)

This method unpacks the received data and returns the python dictionary object containing the message. In case of an exception it returns None.

The default implementation uses json module to decode the data, therefore it MAY be overrided.

Carrier.handleRX(self, clientid, message)

This method MUST be invoked by the carrier implementation whenever a new message is received. The message passed to this function must be already unpacked and ready to be interpreted.

The main purpose of this method is to translate the message and run the corresponding RPC method. The RPC methods supported will be listed by the Carrier.methods dictionary, in the following format:

Carrier.methods = { “method_name” : method_callback, […] }

When this function is called, it will execute Carrier.handleRPC function to search for methods defined by Carrier.methods and execute them accordingly. The function will always return a reply message, that must be sent back to the client, even if the method is unsupported (is not present in Carrier.methods) or it fails during execution. The method will not raise an exception.

The clientid paremeter, uniquely identifies the client from which this mesage was received and it will be used mostly by the ServerRouter class to distinguish between multiple connected clients. It can be in any form (e.g. int or str), as long as it is uniquely identifying the client. For ClientRouter implementation it can be anything, it will be ignored.

For example, for a TCP Server, the clientid will be unique for each client connected to the listening socket. The identifier can be the id of the instance that is processing the communication with the client. When the Carrier receives this parameter on its Carrier.handleTX function, it will select the proper client to send its message to.

This method MAY NOT be overrided.

Message format

{
  "id" : "23",
  "method" : "add_sender",
  "params" :
      {
        "sender" : "lorem",
        "events" : ["ipsum"]
      }
}

Reply format

{
  "id" : "23",
  "status" : 0,
  "response" : None
}

As you can see in this format, the method name is stored in the method field while it’s parameter within the params field. Each message must contain these two fields, including the id field that will be described below.

Each reply however, returns back the same id field received within the message, a status field containing the error code that Carrier.handleRPC returned and the response field that stores that the RPC method responded with.

In case the RPC method raised an exception, the reply will also store a field called exception containing the string representation of the exception raised.

Carrier.handleTX(self, clientid, message)

This method MUST be implemented in order to allow sending data to a specific client. The message passed to this function must be packed before doing the actual send operation. The default implementation does nothing.

The method MUST return the reply received by the client specified by clientid, in a python dictionary object, threfore it must be unpacked. In pysig no message is sent without receiving a reply.

This method MUST add to the message an unique identifier called the id field. This is useful for avoiding the case where the immediate data received after sending the message is NOT the reply that actually corresponds to this message but some other sent before it. The Carrier.handleRX default implementation, will store the id field from the message and sent it back in the reply (see the example above).

Note:

There are several rules that you must respect, when implementing a carrier: * each message sent respects the format above (contains id, method, params as fields) * each reply respects the format above (contains id, status, response and may contain exception as fields) * each call to Carrier.handleTX will return the corresponding reply in an unpacked form * the method Carrier.handleRX must be called for each message (not reply) received in an unpacked form

Carrier.handle_client_connected(self, clientid)

This method MUST be invoked by the implementation whenever a new client is connected. It is useful for ServerRouter and not for a carrier used in the context of the ClientRouter. Upon calling this method ServerRouter will map the corresponding data to this client.

Returns nothing.

Carrier.handle_client_disconnected(self, clientid)

This method MUST be called by the implementation whenever an existing client is disconnected. It is useful only for ServerRouter. Upon calling this method the ServerRouter will detach all registered listeners or senders corresponding by this client.

Returns nothing.

Carrier.handle_all_clients_diconnected(self)

This method MAY be called by the implementation whenever the server looses connection with all of his clients. This method is useful for ServerRouter and it’s an optimized version for callind Carrier.handle_client_disconnected for each client in particular.

Returns nothing.

Built-in carriers

Currently pysig suppors several ready-to-use carriers, as follows:

  • TCP Server for using it in conjuction with ServerRouter
  • TCP Client for using it in conjuction with ClientRouter
  • Local carrier, for testing purposes only

TCP Server

pysig provides a ready-to-use TCP Server carrier for connecting it to the ServerRouter. The way you use it is pretty simple

import time
import sig
from sig.carrier.tcpserver import *

# create the tcp server
tcp_server = CarrierTCPServer()

# create the server router
router = sig.ServerRouter(tcp_server)

# add a sender
sender_timer = router.addSender("timer", ["tic"])
signal_tic = sender_timer.getSignal("tic")

# start server
tcpserver.start("localhost", 3000)

# loop
try:
  while True:
    # tic every ten seconds
    signal_tic.trigger(None)
    time.sleep(10)
except KeyboardInterrupt:
  print "Stopping server.."
  tcpserver.stop()
  print "Done."

Very well, we have a server router that sends a tic signal every ten seconds. This signal can be listened by anyone on the network that can connect to this machine to tcp port 3000.

TCP Client

Also, pysig has a ready-to-use TCP Client carrier for pairing it with ClientRouter. This carrier of course can communicate with the built-in TCP server presented above.

Let’s see how we can listen for the tic signal sent above:

import sig
import time
from sig.carrier.tcpclient import *

# create the tcp client
tcpclient = carrier.CarrierTCPClient()

# create the client router
router = sig.ClientRouter(tcpclient)

# connect client to the server
tcpclient.connect("localhost", 3000)

# register for the tic signal
def listen_for_tic(info, data):
  print "'%s' received from '%s' (data: %s)" % (info.get("event"), info.get("sender"), data)

router.addRemoteListener(listen_for_tic, "timer", "tic")
router.addRemoteListener(listen_for_tic, "another_timer", "tic")

# loop
try:
  while True: time.sleep(10)
except KeyboardInterrupt:
  print "Disconnecting client.."
  tcpclient.disconnect()
  print "Stop"

Great, we have our client connected to the server above. Notice how we used addRemoteListener and not addListener. The difference between these two is that the last one only registers a listener to local senders and not the senders registered to our ServerRouter. That is, you can still use addListener to connect to senders directly registered to ClientRouter and not the ServerRouter we’ve created in our first example.

Also, please notice our second listener registration, to a sender called another_timer. For now, this client will register itself to an unexisting sender, which is quite legit in pysig. Wouldn’t be nice to use a client for adding senders to the entire scheme?

Let’s see how we can do that in our next example.

import sig
import time
from sig.carrier.tcpclient import *

# create the tcp client
tcpclient = carrier.CarrierTCPClient()

# create the client router
router = sig.ClientRouter(tcpclient)

# connect client to the server
tcpclient.connect("localhost", 3000)

# add our remote sender
sender = router.addRemoteSender("another_timer", ["tic"])
signal_tic = sender.getSignal("tic")

# loop
try:
  while True:
     time.sleep(10)
     signal_tic.trigger(None)
except KeyboardInterrupt:
  print "Disconnecting client.."
  tcpclient.disconnect()
  print "Stop"

That’s it. If we run all three examples in the same time, we will have: * a server that triggers a tic event in the name of timer as sender * a client that triggers a tic event in the name of another_timer as sender * a client that registers for listening both tic events

Of course, as a consequence, the client that listens for the tic events will receive events from another_timer only when the client that registers the remote sender is running. But will always receive the tic events from the timer sender, that is directly registered to the ServerRouter.