This article describes the mapping of the active object client-server model to the OSCL C++ Inter Thread Communication library.

Introduction

To be useful, the active object Client-Server model notation needs to be mapped to the implemenation language (C++). This is done using the primitives defined by the OSCL Multithreading ITC (Inter Thread Communication) source code libraries.

Given the interface centric principles of the client-server model, it follows that most of the work involves the client-server interface.

The communication between the client and server is done using ITC messages. Fundamentally, the messages are blocks of memory that are put into the linked list (mailbox) belonging to the thread to receive the message.

If the receiving thread is blocked waiting for a message, it will be awakened and find the new message in its mailbox.

While this describes the fundamental how of the implementation, the ITC framework is designed to support even higher level notions including:

• The client is responsible for the communication memory.
• Servers are to be as independent as possible about their clients.

Since messages are central to the client-server interface the evolution of their implementation is presented next.

Messages Passing

Messages are objects used to convey information between concurrent processes and threads. Messaging services are as old as operating systems themselves and have taken many forms.

Traditional Message Queues

Most systems implement a form of messaging known as message queues, which take the form of an array that is treated as a circular fixed length buffer. Messages are queued in FIFO order to the buffer by one process/thread and removed by a receiving process/thread. UNIX pipes and fifos are similar structures.

There are several issues with these mechanisms.

• Queue full
• Copy in, copy out.

When the queue to becomes full, the client must take one of three actions:

• Try again later.
• Block until the queue becomes not full.
• Handle the exception in some application specific way.
• Ignore the exception.
• Log the exception and abort the program.

Which method is chosen is highly application dependent. Ideally, it would be better that queue full exceptions could be eliminated.

Message queues of this type also require that data be copied into the queue during the send operation and out of the queue during the receive operation. This is inefficient, but it does allow these types of mechanisms to be used across address spaces (e.g. TCP/IP sockets, UNIX fifos and pipes.)

For communication between threads in the same address space, the amount of copying can be reduced by only copying a pointer to the data into the message queue itself.

Linked List Message Queues

For the purposes of communications between threads in the same address space, message passing can be structured such that the act of sending a message never fails. This is done by using a linked list as the receiving mailbox. Using such a mechanism, once the sender has the memory required to send a message, it can link it into the receiver's mailbox without the need to handle exceptions.

This is as efficient as the placement of pointers in traditional message queues, without the need to wory about queue full conditions.

It is this linked-list method that is used for message passing in the OSCL ITC framework. The OSCL ITC framework is designed such that all messages contain a link field such they can be linked into the mailbox of the receiving thread. The OSCL provides oscl/queue/queue.h a type-safe singly linked list queue implementation, used to implement the ITC mailboxes implemented by oscl/mt/itc/mbox/mbox.h and oscl/mt/itc/mbox/mbox.cpp.

Any element linked into an Oscl::Queue must have a public link field member with the following signature:

void* __qitemlink;

This field is usually supplied by inheriting the Oscl::QueueItem type defined in oscl/queue/queueitem.h.

Thus, the most simple message is simply an instance of Oscl::QueueItem.

Of course such a message is not very useful, and indeed in reality cannot even be used with the Oscl::Mt::Itc::Mailbox since its linked list is not publicly accessible as it must be shared by multiple threads and access to the list must be protected by a mutual exclusion mechanism.

The Oscl::Mt::Itc::Mailbox accepts messages which are of the type Oscl::Mt::Itc::Msg.

This abstract class, defined in the file oscl/mt/itc/mbox/msg.h inherits the required queue link field from Oscl::QueueItem and defines a polymorphic operation (pure virtual) called process().

The receiver of a message must examine the message to determine its type, and ultimately what to do with the message. The type indicates to the receiver what the sender intends the receiver to do with the message, and the types of data that accompany the message.

Traditionally, this is done by defining a field common to all messages that contains an integer that is mapped to the message type. The receiver would examine the type using a switch statement, and take the appropriate action with the remainder of the message, which might include casting data to the appropriate type.

Seeing the switch-on-type construct, the object oriented practioner can clearly see an oportunity for improvement. Thus the purpose of the process() operation, to polymorphically dispatch the appropriate message handling based on its type.

Client-Server Message Types

Before we consider how to apply the polymorphic process() operation, lets first consider the different needs of clients and servers.

Generally, interfaces define a set of requests that a server is expected to satisfy. Requests are ITC messages in the active object ITC framework.

It is usually the case in a client-server relationship that the client needs to know when the server has completed the request, either to obtain the resulting data from the request or to simply know that the server is finished with the client resources used for the communication (e.g. the ITC message.)

There are many methods the server might use to indicate that it has completed the request. For example, the server might set a semaphore on which the client is blocked, or the server might send a reply message to the client.

It is desirable to leave this choice of acknowledgment mechanism to the client, and thus there is a need to hide the mechanism from the server.

This is done using another polymorphic operation called returnToSender() as illustrated by the Oscl::Mt::Itc::SrvMsg as defined in the file oscl/mt/itc/mbox/srvmsg.h.

The returnToSender() operation is invoked by the server when it completes completes the request.

For reasons of symmetry and type-safety the ITC framework defines Oscl::Mt::Itc::CliMsg another message subclass to differentiate messages received by clients (responses) from those received by servers (requests.)

Notice that clients have no need for a returnToSender() operation since they are the source of the original request.

The Return Handler

A return handler determines what happens when a server invokes the returnToSender operation on a server request message. There are two return handler implementations that correspond to the two primary acknowledgement mechanisms:

• Asynchronous Return Handler
• Synchronous Return Handler

Notice that these two types correspond to the mode in which clients may send request messages to a server.

It is possible to create other types of return handlers, for very specialized uses, but my experience to this point has proven that to be unnecessary. As a result, it is strongly recommended that any decision to create a new type of return handler be undertaken only after considering other alternatives.

An important property of the synchronous and asynchronous return handlers is that both methods notify the client when the server has completed processing the request. This promotes and simplifies the principle of the client being responsible for the memory used in the communications with other active objects.

The message memory is considered busy or in-use until the server invokes the request message return to sender operation. At that point, the memory used for the message may be reclaimed and reused by the client.

To avoid unnecessary coupling, however, we do not want the server to know its clients.

• Server Messages
• Client Messages

message code Client messages are received by clients, while server messages are received by servers.

The process operation is invoked by the receiving thread when the message is received from the thread's mailbox. Invoking the process() operation polymorphically invokes the code responsible for handling the received message. This mechanism replaces traditional systems that use switch constructs to dispatch received messages.

Notice that the SrvMsg (server message) has an additional operation called returnToSender() which is invoked by the server implementation when it has completed the action requested by the message. This will be covered in more detail below.

To simplify the usage of client and server messages, and to capture common implementation details, two template classes are used define request and response messages.

Request Messages

Server request messages (SrvRequest) are server messages (SrvMsg) defined by an interface instantiated by a client, and sent to a server. Server messages have a return to sender operation, distinguishing them from client messages, that is invoked by the server when it has completed the request.

Request messages are defined in terms of the Oscl::Mt::Itc::SrvRequest template as defined in the "oscl/mt/itc/mbox/srvreq.h" header file.

Notice in the diagram the types ReqApi and Payload are template parameters. The ReqApi type must have a member operation that matches the signature:

void request(SrvRequest& msg) throw();

where SrvRequest is the same type as this particular server request message.

The Payload type may be a class or struct, which may optionally have no members.

Response Messages

Client response messages (CliResponse) are client messages (CliMsg) that are defined in terms of a specific request message (SrvRequest.) In fact, the CliResponse template is defined such that it actually contains (is composed of) the corresponding SrvRequest message.

The Oscl::Mt::Itc::CliResponse template captures the common behavior of response messages . The template is defined in the "oscl/mt/itc/mbox/clirsp.h" header file.

A response message is purely a client concept used when a client sends a request asynchronously to a server. As such, the CliResponse template class contains an AsyncReturnHandler.

Notice that servers know nothing about the existence of a client response message.

Response messages are sent from the server to the client during the execution of the corresponding request message's return to sender operation, which is dispatched polymorphically.

Server requests which are sent synchronously do not have the need for the client response message. In fact, the server request message and payload are typically allocated on the thread's stack during a synchronous post, and the thread blocks on a semaphore thereafter until the semaphore is released by the server's invocation of the returnToSender() operation, the implementation of which simply signals the semaphore releasing the client from its blocked condition. The result is semantically indistinguishable from a simple function call from the client point-of-view.

Payloads

The client-server interface also defines a payload for each request message. The payload type actually distinguishes individual messages defined by the interface.

Payloads contain any data transferred between the client and server as the result of the request messasge and may be structures or classes. Payloads may also contain the result of a request. However, the payload may be an empty type (struct or class) when there is no data transfer.

The Server Role

The server receives request messages defined by the interface; performs the action required to satisfy the request; and subsequently invokes the return to sender operation of the request message in response to completing the request.

The Client Role

The client creates and sends requestmessages (as defined by the interface), and sends (posts) the messages to the server mailbox associated with the interface. The client has the option of posting the message synchronously or asynchronously.

When a client posts a request synchronously, the message is placed into the server's mailbox, and the client thread blocks until the server has fulfilled the request and invokes the return to sender operation of the request message.

When a client posts a request asynchronously, the message is placed into the server's mailbox, and the client thread continues to execute according to its priority. In this case, when the server finishes processing the client's request by invoking the return to sender operation on the request message, the response message is placed into the client's mailbox.

The Client Determines The Message Mode

Notice that in the descriptions of the roles that the server is unaware of whether the request message was sent synchronously or asynchronously. In both cases, the server simply services the request message and subsequently invokes the return to sender operation of the message when the request is complete.

To obtain this property, there is a difference between synchronous and asynchronous mesasges.

The difference is a parameter used when creating a request message known as the return handler.

The Return Handler

A return handler determines what happens when a server invokes the return to sender operation on a request message. There are two primary return handler implementations:

• Asynchronous Return Handler
• Synchronous Return Handler

Notice that these two types correspond to the mode in which clients may send request messages to a server.

It is possible to create other types of return handlers, for very specialized uses, but my experience to this point has proven that to be unnecessary. As a result, it is strongly recommended that any decision to create a new type of return handler be undertaken only after considering other alternatives.

An important property of the synchronous and asynchronous return handlers is that both methods notify the client when the server has completed processing the request. This promotes and simplifies the principle of the client being responsible for the memory used in the communications with other active objects.

The message memory is considered busy or in-use until the server invokes the request message return to sender operation. At that point, the memory used for the message may be reclaimed and reused by the client.