Distributed Information System (DIS)
  • Home
  • The blog
  • Contact

Bleams explained...

6/12/2007

0 Comments

 

IDR is the data encoding used with DITP. It is to DITP what XML is to SOAP. IDR uses bleams that combines the benefits of block and stream encoding:

   - no need to specify total size in front of it
   - no upper size limit
   - may be encapsulated without depth limit
   - no constrain on data and doesn't rely on markers or tags
   - no need to parse and search payload data to locate the end of bleam

A bleam is encoded as a sequence of byte blocks of at most 16KB. Each block is preceded by an unsigned short value whose 14 less significant bits encodes the number of bytes of data that follow. Its most significant bit is set to one if the block is not first, and its second most significant bit is set to one if the block is not last.

When encapsulating a bleam, its sequence of blocks is simply inserted in the sequence of blocks of the encapsulating bleam. The encapsulated bleam can be stored in the data of the the encapsulating bleam block if it fully fits in one of its block.

The maximum payload size is 16382 so that the biggest block will be 2^14 byte long. The invalid size value 16383 (0x3FFF) is then used as a signal. A signal block has no payload data.

The signal is used in IDR to inform the receiver that the expected sequence of data is interrupted because an exception or an error occured. If the signal block is flagged as end of bleam, the interruption is anonymous. Otherwise subsequent bleam data provides information on the reason of the interruption. In IDR it is the serialized exception object and the objects it may be referencing. Such interruption will eventually propagate to encloding bleams and be encoded as anonymous interruptions.

Encoding and decoding bleams require some care, but the effort is worth it because of its multiple benefits. With small messages DITP is as performant as common inter-object communication protocols. With bigger message, DITP benefits from the stream oriented encoding and reduced latency and memory usage requirements. DITP can thus be used to send huge files, streamed films, etc.

0 Comments

More on stream versus block oriented protocol...

6/12/2007

0 Comments

 

Most inter-object communication protocols encode message into blocks with its size encoded in front. This requires that the message is fully encoded to compute the size of blocks before it is sent.

A stream oriented protocol doesn't need this. The communication process can be pipelined. As shown in the figure below, this reduces the communication latency. Note that in a two way transaction the saving is doubled.

In absence of the block size, one needs a marker to signal the end of the block. For instance SOAP uses xml tags as markers. There are two drawbacks with such encoding: the marker can't show up in regular data and the marker must be searched to locate the end of a block. When the block size is available, locating the end of the block is trivial and very fast.

IDR and DITP combines the benefits of blocks and streams encoding by introducing a new encoding called BLEAM. Here is a short list of the their properties:

   - bleams have no size limits
   - bleams may be encapsulated with unlimited depth
   - bleams impose no constrain on contained data
   - no need to search and parse contained data to locate the end of bleam
 
The last property is what makes the difference with SOAP and its xml encoding.
See next blog note for a description of the bleam encoding.

0 Comments

Why is DITP a stream oriented protocol ?

5/28/2007

0 Comments

 

 ... because it reduces communication latency and memory requirement.

Communication latency:
In a message oriented protocol the sender has to fully encode the message before it can be sent, and the receiver must receive the whole message before he can start decoding and processing it. The receiver has thus to wait for the whole message to be generated, sent and received before doing anything.

With a stream oriented protocol the communication process is pipelined. The sender encodes and sends a first chunk of the message allowing the receiver to start decoding and processing it, while the sender encodes and sends the next chunk of the message. The sender and the receiver then work in parallel and not in sequence.

Manual data split is possible with message oriented protocol but at the price of a significant complexity increase of user code, and such splitting can become very tricky when dealing with object aggregates.

Memory requirement:
A message oriented protocol requires that the sender and the receiver hold an encoded copy of the full message. This puts the memory management system under stress because these memory blocks have a short life span, they are of varying size and they are sometime big.

With a stream oriented protocol, the memory requirement is limited to the storage of the encoded message chunk, and this even if the message size is huge. The buffer holding a message chunk can also be easily recycled because it is of a fixed size.


DITP has thus been designed to be a stream oriented protocol, while most inter-object communication protocols are classically message oriented (i.e. CORBA, RMI, ICE). SOAP is the only inter-object communication protocol I know that is stream oriented. It thus benefits from the reduced latency and small memory requirement; but these benefits are spoilt by the ASCII and xml encoding. 

0 Comments

DITP status

5/14/2007

0 Comments

 

DITP was designed early, together with IDR. Since IDR is now operational and ready for throughout testing, DITP could be implemented and tested.

DITP core is very trivial and easy to implement and a first prototype is already operational. This gave the opportunity to test readabilty and lightness of the IDR API which needed some refinements.

This also allowed me to check what is needed to build a server and client objects which of course have to be made as simple as possible.

A critical and time consuming task was adding garbage collector support into C++. I'll discuss the C++ and GC issue in another post. I used libgc v6.8 and managed yesterday to get a running client server prototype with full reliance on the GC and showing no memory leaks. Since the client and server may host live objects, they have to support multithreading.

So far, so good! ... as said a guy at each storey when falling from a building.

0 Comments
Forward>>

    Author

    Christophe Meessen is a  computer science engineer working in France.

    Any suggestions to make DIS more useful ? Tell me by using the contact page.

    Categories

    All
    Business Model
    Database
    Dis
    Ditp
    Dvcs
    Git
    Gob
    Idr
    Misc
    Murphys Law
    Programming Language
    Progress Status
    Startup
    Suggested Reading
    Web Site

    Archives

    December 2017
    November 2015
    September 2015
    February 2013
    December 2012
    November 2012
    May 2012
    February 2012
    March 2010
    October 2009
    September 2009
    July 2009
    June 2009
    May 2009
    February 2009
    January 2009
    November 2008
    September 2008
    August 2008
    July 2008
    May 2008
    April 2008
    March 2008
    February 2008
    January 2008
    December 2007
    October 2007
    August 2007
    July 2007
    June 2007
    May 2007

    RSS Feed

    Live traffic feed
    You have no departures or arrivals yet. Wait a few minutes and check again.
    Powered by FEEDJIT
Powered by Create your own unique website with customizable templates.