On Messages and Files, and the Rest of the Future of Data

Mark Nottingham's questions on messages and files...

Why do messages — which in developers’ minds, inevitably means “short-lived” — have an advantage over files? Making your OS message-based seems to just add complexity, not make data more portable and ubiquitous.

...have me thinking about many things which I will boil down to a small hypothesis.

When we think of messages, we should say "messages are documents, typically shorter in length and shorter in duration". Then "message passing" is like "document passing". When we think of longer lived documents and messages, we should think merely of "passing" them to a service that has the responsibilty for the life of that document. Of course this should be independent of OS or file system. (Those are implementation details only. Our applications should rise above them.)

From working on systems in a half dozen domains I have come to the conclusion there are three simple patterns we should lean on for making documents and messages persistent.

• Tuples Spaces (think Ruple Forums) for in-progress, state machine-like transactions and collaboration.
• Versioned document trees (think Subversion's file system) for long-lived, shared, document editing.
• Star Schema-based storage (think Sybase IQ, a simple, low-maintenance, scalable database technology) for read-only transaction history and analysis.

I deeply believe (it is my hypothesis) that these are the only patterns you need, and the implementations going forward can become far simpler and adaptable than all of today's cruft. As for "messaging" (the gerund, i.e. SOAP and its ilk), WS-xxx is a band wagon approach toward more complexity. Messaging protocols are a means to an end (i.e. getting a message into one of these persistent "end point like" locations). All collaboration can take place via these "persistence" mechanisms. They are actually coordination mechanisms, and persistence is a by-product since collaboration is frequently different-time and/or different-place, and even when computing is solo or collaboration is same-time and same-place, you often would like a history.

This is the rest of the future, so to speak. The present vision of rest is blinded by HTTP. Although HTTP is an application layer mechanism, it does not address application semantics in a deeply meaningful way. Consider that any of the above persistence mechanisms, and many others, can be defined with or without HTTP. Saying HTTP is sufficient is like saying the verbs "go", "stop", "turn left", and "turn right" are sufficient for defining any children's playground activity. The real meaning is in the decisions of when to take these actions and how to interpret the response of the collaborators in the activity.

These three persistence mechanisms match the needs of the kinds of decisions our systems make for us, and the kinds of decisions we wish to make with them. You can tell me how crazy I am on this wiki page.