Distributed Information System (DIS)
  • Home
  • The blog
  • Contact

The 8 fallacies of distributed computing

10/17/2009

4 Comments

 
The following two paragraphs are the introductory paragraphs of the document Fallacies of distributed computing (pdf) by Arnon Rotem-Gal-Oz that presents the 8 fallacies of distributed computed.

"Distributed systems already exist for a long tThe software industry has been writing distributed systems for several decades. Two examples include The US Department of Defense ARPANET (which eventually evolved into the Internet) which was established back in 1969 and the SWIFT protocol (used for money transfers) was also established in the same time frame [Britton2001].

Nevertheless, In 1994, Peter Deutsch, a sun fellow at the time, drafted 7 assumptions architects and designers of distributed systems are likely to make, which prove wrong in the long run - resulting in all sorts of troubles and pains for the solution and architects who made the assumptions. In 1997 James Gosling added another such fallacy [JDJ2004]. The assumptions are now collectively known as the "The 8 fallacies of distributed computing" [Gosling]:
  1. The network is reliable
  2. Latency is zero
  3. Bandwidth is infinite
  4. The network is secure
  5. Topology doesn't change
  6. There is one administrator
  7. Transport cost is zero
  8. The network is homogeneous
..."

While in the process of designing a new distributed information system, it a good idea to check how it position itself regarding these 8 fallacies.

The network is reliable

DIS uses TCP which was designed to be reliable and robust. Reliable means that data is transmitted uncorrupted to the other end and robust means that it may resist to a certain amount of errors. There is however a limit to the robustness of a TCP connection, and in some conditions connection to a remote service may even not be possible.

DITP, the communication protocol of DIS, is of course designed to handle connection failures. Higher level and distributed services will have to take it in account too.

Making a distribute information system robust implies to anticipate connection failures at any stage of the communication. For instance, a flock of servers designed to synchronize with each other may suddenly be partitioned in two or more unconnected flocks because of a network failure, and be connected back together later.

The latency is zero

Latency was a major focus in the design of the DITP protocol because DIS is intended to be used for World Area Network (WAN) applications. DITP reduces latency impact by supporting asynchronous requests. These requests are batched and processes sequentially by the server in the order of emission. If a request in the batch is aborted by an exception, subsequent requests of the batch are ignored. This provides a fundamental functionality to support transactional applications.

In addition to this, DIS may also support the ability to send code to be executed by a remote service. This provides the same functionality as JavaScript code embedded in web pages and executed by browsers, allowing to implement powerful and impressive web 2.0 applications.

With DIS, remote code execution is taken care by services made available by the server manager if he wants to support them. The services may then process different types of pseudo-codes: JavaScript, Haxe, JVM, Python, ... Many different pseudo-codes services may then coexist and evolve independently of DIS. Such functionality is of course also exposed to security issues. See the secure network fallacy for an insight on how DIS addresses it.

Bandwidth is infinite

This fallacy is the rational of the Information Data Representation (IDR) design. It uses binary and native data representation. In addition to be very fast and easy to Marshall, it is also very compact.

DITP supports also user defined processing of transmitted data so that compression algorithms may be applied to them. DITP is also multiplexing concurrent communication channels in the same connections, allowing to apply different transmitted data processing to each channel. By choosing the channel the user may decide to compress transmitted data or not. 

The network is secure

A distributed system designed for a world wide usage must obviously take security in account. This means securing the transmitted data by mean of authentication and cyphering, as well as authenticating communicating parties and enforce access or action restriction rules.

Communication security is provided by the DITP protocol by mean of the user specified transmitted data processing. As data compression, these can also handle data authentication and cyphering. Different authentication and cyphering methods and algorithms can coexist in DIS and may evolve independently of the DITP protocol.

Authentication and access control may use conventional passwords methods as well as user identification certificates. But instead of using x509 certificates, DIS uses IDR encoded certificates corresponding to instances of certificate classes. Users may then derive their own certificates with class inheritance. They may extend the information carried in the certificate or combine different certificate types together.

An authentication based on password checking or user identity certificate matching doesn't scale well for a world wide distributed system because they need to access a reference database. With distributed services, accessing a remote database introduces latencies and replicating it (i.e. caches) weakens its security by multiplying the number breach points.

The authentication mechanism favored in DIS uses member certificates. These certificates are like club or company member access cards. When trying to access a service, the user present the corresponding certificate and the service needs simply to check the certificate validity.

With such authentication mechanism, the service can be scattered all over the Internet and remain lightweight as is required for embedded applications (i.e. smart phones, car computers, ...). The authentication domain can also handle billions of members as well and easily as a few ones. Member certificates may be extended to carry specific informations and connection parameters.

Topology doesn't change

The ability to handle network topology changes initiated the conception of DIS in 1992. It is thus designed from the start to address this issue in a simple, robust and efficient way. It is not a coincidence that the DIS acronym resembles the one of DNS. DIS is a distributed information system as the DNS is a distributed naming system. DIS uses the proven architecture of the DNS and applies it to generic information with additional functionalities like allowing to remotely manage the information. The DNS is known to be a corner stone of the network topology change solution, as will be DIS.

There is one administrator

As the DNS, DIS supports a distributed administration. Information domain administrator have full liberty and authority in the way they organize and manage their information domain as long as the interface to DIS respects some standard rules. As for the DNS, there will be a central administration that defines the operational rules and control their application. If DIS becomes a broadly adopted system, the central administration will be composed of members elected democratically and coordinated with the Internet governance administration if such structures happens to be created.

Transport cost is zero

The transport cost is indeed not zero but most of it is distributed and shared by the users. There remains however a residual cost for the central services and administration for which a revenue has to be identified. The DIS system will allow to obtain such a revenue and there is a rational reason why it ought to.

Imposing a financial cost to some domains or features of DIS which are limited or artificially limited resources provides a mean to apply a perceptible pressure on its misbehaving users (i.e. spam).

The network is homogeneous

DITP is designed to support different types of underlying transport connections. The information published in DIS is treated like an opaque byte block and may be of any type as well as its description language. It may be XML with its DTD description, binary with C like description syntax, python pickles or anything else. Of course it will also contain IDR encoded information with its Information Type Description.

Conclusion

The conclusion is that DIS, DITP and IDR have been designed without falling on any of the common fallacies. This is partly due to the long maturation process of its conception. While this may be considered as a shortcoming, it may also be its strength since it allowed to examine all aspects wisely with time.
4 Comments

DIS development roadmap

11/12/2008

0 Comments

 

The following figure shows the kernel components of the Distributed Information System, the road map and how far I am today. The items in black are implemented and operational and the items in gray still needs to be implemented. Progress is going clockwise :).

OID An OID is to DIS what the URL is to the web. It is a unique, binary encoded and non reusable reference to an information published in the distributed information system. It was the first tile I designed and implemented. Its simplicity is inversely proportional to the time and effort required to invent it because I had to explore and compare many different possible and existing solutions.

IDR It is to DIS what HTML or XML is to the web. IDR is the Information Data Representation used in DIS. It is a stream oriented encoding with support of object serialization and exceptions. The prototype implementation is currently being fully rewritten. It still miss the ability to specify the encoding version or a formalization of data description. The later is required to be able to display data in a human readable format or to automatically generate data manipulation functions or containers mapped to different programming languages.

DITP It is to DIS what HTTP is to the web. It is the protocol used to exchange information or invoke remote actions in DIS. It is very simple, modular and extensible through the use of dynamically configurable data processing tasks. Support of compression, authentication or encryption is then provided by some kinds of plugins. The protocol use the object oriented model with remote method invocation. The current prototype does not yet support concurrent asynchronous method invocation.

DIS DIS stands here for Distributed Information Service and is not to be confused with Distributed Information System. It is fundamental to DIS, so a confusion is not really a problem. This service combines the properties of DNS and LDAP and would be a new kind of service on the Internet. I can't disclose more  on it because it is still in development. A first prototype has been implemented unfortunately proving the need to support data description.

SEC This part covers authentication and access control in DIS. It requires a functional DIS service. An interesting feature is that it is designed to scale up so that a service could cope with millions of different users without having to keep track of million accounts and passwords.

IDX It is a service simply mapping human readable UTF8 strings to OID references. It is equivalent to the list of named entries in a directory. Like any other services, its access is controlled by ACL and can thus be modified remotely with appropriate privileges. An index may be huge with multiple alternate entry point, exactly like the DNS but exclusively as a flat name space. The OID associated to the UTF8 string is stored in an object so that polymorphism allow to associate images (icons) and other informations to entries by extension.

DIR It is a graph of IDX services with one root entry. Services or information published in DIS can then be referenced by a humanly readable path in the IDX graph relative to the root.



It is an ambitious project but, I am convinced, its added value is worth the effort. I wish I could work full time on this project with the help of some other developers, but this would require funding I don't have access to for now.

An application would help demonstrating the added value of the system. I'm still looking for one with an optimal balance in development effort and success potential.

0 Comments

Making DITP flexible, versatile and simple

9/4/2008

0 Comments

 

DITP is flexible, versatile and simple because it uses the inter-object communication model. Not only for the user needs to communicate with its service, but also to setup and configure the connection data processing (i.e. authentication, encryption, compression, logging, tunneling,...).


Inter-object communication

By adopting the inter-object communication model users can create any type of service (remote object) they want. They can also extend or refine their capabilities by using inheritance with the polymorphism property and preserve backward compatibility at the same time.

This makes DITP versatile and flexible, but any other inter-object communication protocol could claim the same.

Configuring and setting up the connection

What makes DITP different is that the connection configuration and setup are also performed by using the object oriented model. The different algorithms used are controlled by specific services and the client controls them by invoking the appropriate methods.

This makes the protocol very flexible and versatile since the algorithm can be combined and configured in any way. It is easy to add support for new algorithms and there is no constrain on the transaction polka required to configure them.

This design choice basically factorizes and parameterizes the protocol. What is left for DITP to define is how to open a connection, how to exchange messages between client and services and how to setup a new client-service binding.

Opening the DITP connection

Opening a DITP connection implies a very simple transaction where the client and the service side exchange a four byte message. If both message contain the expected value, the connection is considered opened. It can hardly be made simpler.

When the connection is opened the client and service implicitly attach a channel control service to the connection. This service has very few methods. One is used to close the connection and the other to request the attachment of another service whose type and identity are given as argument.

That is all it takes to have an operational DITP server. The exchanged messages have also a very simple structure, but will be described in another note because they have another original feature allowing to minimize latency.

Once the connection is opened, if the client wants to secure the connection by adding authentication or encryption, he request the attachment of the corresponding services and configure them by calling their methods.


This is why I claim that DITP is versatile, flexible and simple.

0 Comments

Optimizing DITP connection open

8/25/2008

0 Comments

 

The DITP protocol has been designed to minimize the time required to setup an operational connection. This is achieved by a simple method which is made explicit in the following figure.

In common protocols, like HTTP and SMTP, the server is expected to send a greeting before the client can respond and proceed by sending its first request.

The time the client has to wait for this greeting message is usually dominated by the round trip time. In a LAN the round trip time is less than a millisecond, but on Internet it will take many tens of milliseconds and sometime hundreds of millisecond if the server is on another continent.

By simply swapping the DITP open transaction orientation, we save one round trip time delay before the client can sent its first transaction request. Another advantage of this method is that the server can use a very narrow timeout window for the arrival of the DITP connection setup request. This protects against some type of DOS attacks.

There are two additional things we can observe from the previous figure.

1.- The HTTP or SMTP protocol could be optimized by allowing the client to send its first data without having to wait for the server greeting.

2.- The round trip time due to TCP could be avoided if we could combine it with the DITP connection set up and the two first requests. There is clearly room for improvement on this layer, but this is out of scope regarding this project.

0 Comments

Time value encoding in DIS

5/26/2008

0 Comments

 

One fundamental question is the encoding of a time value. A time value has two types of use. One is as time stamp and the other is just as a general time reference.

Requirements

On one hand, a time stamp has the requirement to have a well defined and controlled precision, while the covered time span can be limited (i.e. +/- 200 years).  On the other hand, a general time reference needs to be applicable to a very large time span, with less constrains on the precision limit.

Options

For the time reference value one could use a double precision float representation with seconds as units. All arithmetic operations are provided right out of the box and generally hardwired in the processor. Conversion to calendar time is trivial since one simply has to extract the integer part of the value and convert it to a time_t value. From there one can use the common calendar time conversion and formatting functions.

For time stamps, using integers seems preferable. But we still have a choice between a split encoding like the timeval structure, a 64bit fixed point encoding, or an integer with very small time unit (i.e. nanoseconds).

Discussion

There is not much to discuss about the absolute time. Using a double precision float is an optimal solution. For time stamps however we have three different solutions.

From my experience, I've seen that split time encoding like the timeval structure is not convenient to use when dealing with time arithmetics. It is even error prone if the user has to program the operations himself.

I also tried to implement a fixed point time encoding class with the decimal point between bit 29 and 30. But this is tricky to get right and some operations are not trivial to implement correctly. This is because fractional computation requires normalization and optimal rounding errors handling.

A 64bit  integer using  nanoseconds as time units is apparently the most simple and straightforward time stamp encoding. Converting to seconds is done with a simple 64bit integer division which is also hardwired in most recent processors. Conversion to other time units like microseconds, milliseconds, days or week is as accurate and simple. Multiplication or division with decimal scalar values is also trivial.

Another advantage of the 64bit integer nanosecond values is that there is no need of special functions to do the conversions or operations. A programmer can easily figure out what to do and use conventional arithmetic operations.

With a 64 bit signed integers with nanosecond units, the covered time span is over +/- 292 years range. One can thus afford keep the current time_t January 1970 epoch and push back the wrapping limit far away. 

Conclusion

In DIS, we'll thus use a double precision float for general time reference value and a 64bit integer with nanosecond units for time stamps and delays encoding.

Note: I've seen the use of a double precision float for time encoding in some Windows operating system API. I still have to see the use of a 64bit signed integer with nanosecond units. It would make sense as an upgrade of time_t which is required since we are getting close to the wrapping limit.Update : It has been brought to my attention that Java stores time values in a signed 64bit integer with milliseconds as time units relative to January 1, 1970. The covered time span is thus +/- 290 million years. I'll stay with the nanosecond units for time stamps.

0 Comments

How to correctly define a standard...

3/19/2008

0 Comments

 

The "Martian headset" is a long but very interesting article on software standards published on the Joel on Software blog.

I've learned that it is not enough to publish a standard specification document. At least one reference implementation is required. Java did this with its compiler and managed by that to ensure interoperability. It could even resist attempts to break the standard.

Lesson learned !

0 Comments

Network latency

1/21/2008

1 Comment

 

Minimizing latency was one of the main aspects driving the design of the DITP protocol. You may check wikipedia for a definition of latency in the networking context.

While network speed has still allot of room to increase, network latency doesn't. Today 80GB/s backbone fiber links are getting common, and we know that there is no limit to reach Tera Byte/s speed or even above. We can easily achieve this by frequency multiplexing or adding more parallel fibers. So we have plenty of room to extend the number of Bytes/s we can send. We can expect a network speed gain of a factor of 100 or maybe even 1000 in the next 25 years.

On the other hand we have a hard and very close limit in transmission latency. This is all the fault of the speed of light limit (~300.000 Km/s). See the numbers. The distance between Paris (France) and New York (USA) is ~ 6000 Km. So it takes 20ms for a single bit to reach the other end and there is no way to lower this time unless we find some "worm hole" in our knowledge of physics.

We are still far from this lower bound limit, but only by a factor 10 or less. Thus a protocol designed for WAN (World Area Network) application should really care about latency.

Here is how I learned the lesson. I designed a protocol to transmit 2MB blocks between computers. The protocol was trivial and worked very well in LAN (Local Area Network) applications. We then had the opportunity to test it on a leased 6GB/s long distance connection between CERN (Geneva in Switzerland) and the university of Alberta (Canada). The surprise was that the bandwidth usage never exceeded 2%. We found out that it was caused by the network latency which was ~500ms. In this context, the  handshake time dominated the transmit time, something we didn't see in a LAN. So our protocol had to be redesigned ! Since that day I understood how important and critical network latency can be.

The first lesson we may learn from this analysis is that when designing a modern protocol for potential WAN usage, minimizing network latency is much more important than encoding concision.

The second important lesson is that inter-object communication will be impaired by the network latency of long distance connections. In such context I thus expect that the agent model will be more efficient. In this model, it is a piece code, a program or even a virtual robot that is sent to the remote host. And this is exactly what is already happening today with JavaScript code in web pages. I assume this tendency will develop and extend in the next 10 years. 

DITP is ready for this since it can be used as the transport layer of such agent transmission. All we need is a special remote service object acting as an agent host. This is on top of the DITP communication layer so that many different types of agents can coexist. The technology can thus also evolve and preserve backward compatibility. There are other reasons why DIPT has a good potential for such usage model, but it is still a bit early to expose them.

Network latency is an often disregarded parameter, but things might change in a near future when we'll get closer to the hard limit !

1 Comment

Progress status

12/18/2007

0 Comments

 

Progress on the design and the prototype implementation is going on. I now have a working prototype for the inter-object communication system. This helps me testing and refining the design. I also regularly review and update the specification documents.

On DITP, the current points of focus is to find a good way to manage the PDU (Protocol Data Units) processing like compression, authentication or enciphering. The user must be able to select and set them up in a snap while keeping it as versatile and flexible as possible.

On IDR, the current point of focus is a refinement of signed information encoding. A straightforward implementation is to simply append to signature to the signed information. But this annihilates all the benefits of the stream oriented encoding. Beside, invalid signature or data must be detected as early as possible. A solution has been identified, but fitting it nicely with the current encoding requires some more investigation.

A design process is a difficult task because we have zillion of decisions to make. The more complex the design, the more decisions there is to make, and likely we can make a mistake somewhere. The two heuristics I use to minimize this risk is first to keep the design as simple as possible and second to minimize the constrains on usage. The former is popular, the later much less.

0 Comments

Communication model

12/7/2007

0 Comments

 

DIS is based on the object model. DITP, the communication protocol used by DIS, is thus an inter-object communication protocol: it makes it possible to invoke methods of an object hosted in another process.

Client and service

The most simple API to do so, is to have a dummy object on the client side with exactly the same interface as the remote object. The methods of the dummy object, we'll call client, forward the call to the remote object we'll call service. Forwarding a method call means packing arguments into a message, send it to the service that process the call and sent back a message containing the result. From the user point of view there is no difference with calling a method of a local object.

In DITP the client object has an exclusive relationship with the remote service object it is connected to. The service is thus in fact an agent acting on behalf of the client. It is however still under control of the hosting process who can modulate its behavior according to client credentials or specific context. This design has also the benefit to associate a state to the service that may be transient or persistent (restored in a new connection).

Shared services are services that may be accessed by multiple clients simultaneously, Such services are implemented by a dedicated object supporting concurrent method invocations from the different services (agents).

Sequential and concurrent method invocation

Sequential method invocation is the most simple to implement and is also expected to be most frequent use case. The service process incoming method invocation requests one by one in the order of arrival, processing the next one only when the result of the previous one has been sent back. Adding a timeout control system will ensure that the system will never block.

Concurrent method invocation requires that each method invocation is processed by a dedicated thread. The service has to be thread safe and a congestion avoidance system is required in addition to the timeout control. It is thus more complex to design and implement.

DITP is designed so that simple communication models are simple to implement and use, and that more complex communication models can be implemented by composition.

Thus a client-service connection, called a channel, will only support sequential method invocation. Since DITP supports channel multiplexing and in both directions, supporting concurrent method invocation is implemented by encapsulating a pool of parallel client-service connections inside of the client and the service. Callbacks may then be implemented by multiplexing reverse client-service connections. .

0 Comments

Security

6/24/2007

2 Comments

 

A modern communication protocol must be secure. And to do it right, security must have been integrated in the design from the very start. Here is a short list of security requirements for DITP:

   - authenticate peers
   - support exchanged data authentication and encryption
   - provide access control on accessible services, objects and methods
   - support single and multi-signed information of any kind
   - signed information supporting polymorphism and aggregates
   - allow anyone to verify any signature with minimal knowledge

Multi-signed information is when more than one people sign a given information, (i.e. a contract).

With a stream oriented encoding this all imply that we are able to apply a hash function (i.e. SHA) on transmitted data while it is encoded or decoded.

This is what I am currently implementing. Unfortunately, a server crash, monopolized all my time this week. Murphy's law revenge... 

2 Comments
<<Previous

    Author

    Christophe Meessen is a  computer science engineer working in France.

    Any suggestions to make DIS more useful ? Tell me by using the contact page.

    Categories

    All
    Business Model
    Database
    Dis
    Ditp
    Dvcs
    Git
    Gob
    Idr
    Misc
    Murphys Law
    Programming Language
    Progress Status
    Startup
    Suggested Reading
    Web Site

    Archives

    December 2017
    November 2015
    September 2015
    February 2013
    December 2012
    November 2012
    May 2012
    February 2012
    March 2010
    October 2009
    September 2009
    July 2009
    June 2009
    May 2009
    February 2009
    January 2009
    November 2008
    September 2008
    August 2008
    July 2008
    May 2008
    April 2008
    March 2008
    February 2008
    January 2008
    December 2007
    October 2007
    August 2007
    July 2007
    June 2007
    May 2007

    RSS Feed

    Live traffic feed
    You have no departures or arrivals yet. Wait a few minutes and check again.
    Powered by FEEDJIT
Powered by Create your own unique website with customizable templates.