Distributed Information System (DIS)

The "Martian headset" is a long but very interesting article on software standards published on the Joel on Software blog.

I've learned that it is not enough to publish a standard specification document. At least one reference implementation is required. Java did this with its compiler and managed by that to ensure interoperability. It could even resist attempts to break the standard.

Lesson learned !

As a follow up to the previous blog note, I found this blog note of Paul Buchheit very enlightening.

The Drake equation provides an estimation of the probability of extraterrestrial life existence.  Note that XKCD has a joke about it here, the point being that attempting to quantify the probability is pushing the logic too far.

The value of the Drake equation is in identifying the relevant parameters so that if any of these parameter is demonstrated to be zero, then extraterrestrial life can be for sure excluded. If it is not null, there is a very high probability that we are not alone in the universe even if  the probability is extremely small. See my note on Murphy's Law to understand why.

Transposing this to startups, it would be very tempting to elaborate an equivalent equation to estimate a startup success probability.

Following Y Combinator news, I found today an interesting post that summarizes the fundamental questions provided in the marketing book "Selling the Invisible" by Beckwith helping to identify the profile of a startup. A book I will surely read.

My impression so far is that these questions are identifying the key parameters of a possible Startup equation, the equivalent of the Drake equation. I refactored them here below:

1. who are you ?
2. what do you do ?
3. who do you serve ?
4. who are your competitors ?
5. what makes you the best ?
6. how do you monetize ?

There are probably other parameters. Though I think these are the most important one because they tell you if the business is sound. Take care to examine these questions with a very broad view and open mind. Be also very accurate and objective.

For instance, regarding monetizing, I would strongly recommend reading the article "Free! Why $0.00 Is the Future of Business" of Wired to make sure you address this question with a modern view. As I understand it, the fundamental logic here is to identify indirect monetizing sources.

When answering the question "what do you do" make sure to explore beyond the basic facts. If your activity is to provide a service, you are also involved in a trust relationship with your users. What do you do on this aspect ? Don't miss these because they could be the "killing" difference with the competitors.

Here is a short anecdote on this last point. I'm buying my computer stuff through Internet from one big French company. One day, there was a problem and I contacted them to resolve the issue. At one point during the email discussion I said that what they sell was in fact trust and not computer goods.

I recognize that I pushed it a bit far but apparently it seeded some thoughts because in the next month they added various insurance offers for buyers. A kind of premium fee for priority help in case of trouble. You know, something that turns a client into a VIP. So they basically added a panel of virtual anti-anxiety pills to their catalog !

Keep thus a very broad view on what and how to monetize.

This is a 3 minute presentation of Richard St. John at the february 2005 TED conference presenting the 8 keys to success he identified after interviewing 500 successful people.

This is a talk of Steve Jobs at Stanford  in 2005. It's really worth  viewing.

All along the development of DITP and DIS, the question of which business model to apply taps my mind. Here is a blog note I found really enlightening on this issue. 

"Better Than Free" by Kevin Kelly

In some discussion group we just settled, we were asked to write down what would be the most important for us in our lives. And to our big surprise 90% youngsters in the group used the words "recognition" or "social recognition". There wasn't any significant discussion before that could have biased this informal survey. It was surprising enough to catch my attention.

20 years later, here is how I explain this to myself. It is known for many years now that our brain has special centers responsible of fundamental drives in our behavior. These centers have been detected and localized by changes in people's behavior when altered by an accident or a disease. The drives are related to hunger, libido, aggressivity/love, etc.

Assuming that mankind is the product of an evolution driven by natural selection, the existence of such drive centers makes sense. It ensures perennity of our species from the biological and the physiological perspective. Maybe the social perspective did not get the same attention.

To me, the existence of a drive for social recognition would also make sense because it pushes toward social coherence and cohesion and thus also to stability while being flexible enough to preserve the capability of our society to evolve and adapt to context changes.

This social recognition drive is thus equivalent to sexual drive, with all the possible related pathology. Social recognition compulsion can for instance lead to mythomania or eccentric behavior.

So my impression, if you allow me to push the logic to its extrema, is that social network sites are a variant of porn sites. They just tickle the brain centers driving our behavior.

I would thus say that social networking is directly related to the social recognition drive, which is itself a kind of attractive magnetic force required to get social cohesion and coherence. If this is true, animals would show the same fundamental drive, and why not, ETs too.

Minimizing latency was one of the main aspects driving the design of the DITP protocol. You may check wikipedia for a definition of latency in the networking context.

While network speed has still allot of room to increase, network latency doesn't. Today 80GB/s backbone fiber links are getting common, and we know that there is no limit to reach Tera Byte/s speed or even above. We can easily achieve this by frequency multiplexing or adding more parallel fibers. So we have plenty of room to extend the number of Bytes/s we can send. We can expect a network speed gain of a factor of 100 or maybe even 1000 in the next 25 years.

On the other hand we have a hard and very close limit in transmission latency. This is all the fault of the speed of light limit (~300.000 Km/s). See the numbers. The distance between Paris (France) and New York (USA) is ~ 6000 Km. So it takes 20ms for a single bit to reach the other end and there is no way to lower this time unless we find some "worm hole" in our knowledge of physics.

We are still far from this lower bound limit, but only by a factor 10 or less. Thus a protocol designed for WAN (World Area Network) application should really care about latency.

Here is how I learned the lesson. I designed a protocol to transmit 2MB blocks between computers. The protocol was trivial and worked very well in LAN (Local Area Network) applications. We then had the opportunity to test it on a leased 6GB/s long distance connection between CERN (Geneva in Switzerland) and the university of Alberta (Canada). The surprise was that the bandwidth usage never exceeded 2%. We found out that it was caused by the network latency which was ~500ms. In this context, the  handshake time dominated the transmit time, something we didn't see in a LAN. So our protocol had to be redesigned ! Since that day I understood how important and critical network latency can be.

The first lesson we may learn from this analysis is that when designing a modern protocol for potential WAN usage, minimizing network latency is much more important than encoding concision.

The second important lesson is that inter-object communication will be impaired by the network latency of long distance connections. In such context I thus expect that the agent model will be more efficient. In this model, it is a piece code, a program or even a virtual robot that is sent to the remote host. And this is exactly what is already happening today with JavaScript code in web pages. I assume this tendency will develop and extend in the next 10 years. 

DITP is ready for this since it can be used as the transport layer of such agent transmission. All we need is a special remote service object acting as an agent host. This is on top of the DITP communication layer so that many different types of agents can coexist. The technology can thus also evolve and preserve backward compatibility. There are other reasons why DIPT has a good potential for such usage model, but it is still a bit early to expose them.

Network latency is an often disregarded parameter, but things might change in a near future when we'll get closer to the hard limit !

Progress on the design and the prototype implementation is going on. I now have a working prototype for the inter-object communication system. This helps me testing and refining the design. I also regularly review and update the specification documents.

On DITP, the current points of focus is to find a good way to manage the PDU (Protocol Data Units) processing like compression, authentication or enciphering. The user must be able to select and set them up in a snap while keeping it as versatile and flexible as possible.

On IDR, the current point of focus is a refinement of signed information encoding. A straightforward implementation is to simply append to signature to the signed information. But this annihilates all the benefits of the stream oriented encoding. Beside, invalid signature or data must be detected as early as possible. A solution has been identified, but fitting it nicely with the current encoding requires some more investigation.

A design process is a difficult task because we have zillion of decisions to make. The more complex the design, the more decisions there is to make, and likely we can make a mistake somewhere. The two heuristics I use to minimize this risk is first to keep the design as simple as possible and second to minimize the constrains on usage. The former is popular, the later much less.

DIS is based on the object model. DITP, the communication protocol used by DIS, is thus an inter-object communication protocol: it makes it possible to invoke methods of an object hosted in another process.

Client and service

The most simple API to do so, is to have a dummy object on the client side with exactly the same interface as the remote object. The methods of the dummy object, we'll call client, forward the call to the remote object we'll call service. Forwarding a method call means packing arguments into a message, send it to the service that process the call and sent back a message containing the result. From the user point of view there is no difference with calling a method of a local object.

In DITP the client object has an exclusive relationship with the remote service object it is connected to. The service is thus in fact an agent acting on behalf of the client. It is however still under control of the hosting process who can modulate its behavior according to client credentials or specific context. This design has also the benefit to associate a state to the service that may be transient or persistent (restored in a new connection).

Shared services are services that may be accessed by multiple clients simultaneously, Such services are implemented by a dedicated object supporting concurrent method invocations from the different services (agents).

Sequential and concurrent method invocation

Sequential method invocation is the most simple to implement and is also expected to be most frequent use case. The service process incoming method invocation requests one by one in the order of arrival, processing the next one only when the result of the previous one has been sent back. Adding a timeout control system will ensure that the system will never block.

Concurrent method invocation requires that each method invocation is processed by a dedicated thread. The service has to be thread safe and a congestion avoidance system is required in addition to the timeout control. It is thus more complex to design and implement.

DITP is designed so that simple communication models are simple to implement and use, and that more complex communication models can be implemented by composition.

Thus a client-service connection, called a channel, will only support sequential method invocation. Since DITP supports channel multiplexing and in both directions, supporting concurrent method invocation is implemented by encapsulating a pool of parallel client-service connections inside of the client and the service. Callbacks may then be implemented by multiplexing reverse client-service connections. .