REST and SOAP

Don Box on REST vs. SOAP...

In hopes I never have to address this debate again...

The debate seems just to be getting started and has a long way to go. I hope Microsoft has not made up their minds yet.

The following design decisions are orthogonal, even though people often conflate two or more of them...

1. Whether one uses SOAP or POX (plain-old-XML).
2. Whether or not one publishes an XML schema for their formats.
3. Whether or not one generates static language bindings from an XML schema.
4. The degree to which one relies on HTTP-specific features. That stated, screw with GET at your peril.
5. Whether one adopts a message-centric design approach or a resource-centric design approach.

Some of the decisions (specifically 5) are architectural and sometimes philosophical.

What does "philisophical" mean? Does "philisophical" mean "does not count"?

Some of the decisions (specifically 1-2) are simple business decisions that are determined by who your target audience is.

What does "simple business decision" mean? (Let's ignore the "simple" part and just consider "business".) I can understand the decisions could have a value and a cost associated with them. But which ones would not?

1. If you want a great experience for .NET/Java devs, you’ll typically publish schemas (through wsdl) and support SOAP.
2. If you want a great experience for LAMP folks, you’ll support POX messages and will provide a non-XSD description of your formats.
3. If you want to reach both audiences, you’ll do both #1 and #2.
4. If you want to reach both audiences before your competition does, you'll avoid indulging in religious debates and ship something.

Starting with (1), what does "great experience" mean? Great *initial* experience? And does that slash mean dotnet *or* java? Or does it mean dotnet *and* java together?

My recent experience (in which I was shocked how little interoperability, not to mention functionality, has been accomplished in the WSDL and SOAP world over the last several years) tells me that if the target is either dotnet *or* java then your chances are significantly better than targetting both. (But then if separate, the questions becomes again *why* WSDL and SOAP when a proprietary solution could do better still.

Doing (3) seems like a lot of work when perhaps (2) would work well for all. Again my recent experience supports that. I think (1) works for vendors whose tools have been aimed at (1) but only *initially*. What little experience I have in this area has even so raised huge warning flags that the initial joy of a dotnet or java SOAP and WSDL toolkit very soon runs out of gas (i.e. I have not seen quantifiable benefits for choosing WSDL and SOAP over real HTTP), yet introduces problems of its own (complexity, lack of interop, little support for versioning, magic instead of repeatability, little in the way of testability).

That last one about shipping something is interesting. In a recent experiment a small group and I found ourselves wrestling a lot with WSDL and SOAP (this was a dotnet *and* java experiment rather than a dotnet *or* java experiment). Wanting to quickly put that behind us, we decided to run a different experiment... address the same business problem but with just HTTP. Very soon we were spending all our time talking about business functionality and messages rather than infrastructure headaches.

And yes, we could have at that time very easily thrown in LAMP, Lisp, Smalltalk, Erlang, etc. Anything that can do simple HTTP could be a full participant. The interfaces were narrowed and became a non-issue. The messages and the funcationality became the concern -- what is the business functionality? what is the message description? (We used xsd's and so I can attest to the above statement about schemas being independent of SOAP, WSDL, etc.)

Mark Baker has additional useful comments and pointers to other recent items (here and here for example) about this debate. To repeat myself, this seems far from over, and neither should it be. There is a lot more to discuss. The lists above from Don seem oversimplified. From his accompanying text, he seems too eager to conclude with this list and live by it. I don't see how that would help anyone.

Update: Mike Champion makes an analogy between messaging technologies (SOAP/WSDL and HTTP) and road vehicle types (trucks and cars). Unfortunately this is an arbitrary analogy. That is, saying that SOAP/WSDL is best "to haul a lot of heavy stuff securely and reliably, use a truck" does not make it so. The question is how to make an objective determination.

Update Day 3: As Chris Ferris says, the saga continues. Chris plays the "if only you knew what I know" card. That could be a good card to play, as I am only reporting what little I know. That is not much, but at least it is based on hands-on experiments.

But Chris appears to advocate SOAP/WSDL over HTTP on the basis of reliable messaging. Here's my (limited) experience... several popular SOAP/WSDL toolkits fail to easily interop over HTTP using unreliable messaging. Should I come back in a few years to see where they stand on reliable messaging? Chris admits...

I will concede that it is taking longer than I might like to drive the WS-* specifications through the standards process.

Yet the standards that are in place already seem open to arbitrarily incompatible interpretations. And the toolkits I have seen appear to do the darnedest things to generate complicated code and require programmers to jump through obscure hoops without a net.

Chris -- promises of paradise sometime in the future may come true, but I don't see that as a reason to invest in the mess we're forced to deal with today.

Stefan Tilkov writes...

If you’re talking about SOA at an enterprise level... whether you end up doing POX over HTTP in a RESTful or unRESTful way, or use WSDL and SOAP with a code-first or contract-first approach, is not the sort of thing that matters...

But he seems to agree there is a significant technical debate. I'm not sure I agree the technical debate does not matter at the enterprise level. The most significant enterprise IT issue I am aware of is the inability to get out from under undesirable technical legacies. My fear is the current SOAP/WSDL, etc. conglomeration is not a wedge for loosening older legacies, but could turn out to be its own significant layer of cruft that drains more resources from focusing on pure business value.

On reliability, security, etc. within the enterprise, Mike Champion in a comment on his blog writes...

Typical enterprise IT shops tend see the Web as a necessary evil to communicate with customers, and must invest heavily in tools and expertise to get acceptable security / reliability; they generally use databases or enterprise message oriented middleware (e.g. MQ) to handle internal communications securely and reliably.

And I agree with this observation. I can see a place within the enterprise for HTTP as well as for some other API or protocol-based database and messaging systems. I don't see HTTP as being the only protocol, especially within an enterprise. However I don't see SOAP/WSDL at *this* point adding value to any of these. Experience shows they are complicated, vague, and underdeveloped (relative to what the adopted and in-progress specifications might lead you to believe.)

Each has their place... but where? Kurt Cagle adds to the discussion. He makes a statement I have heard before, "both have their place". I don't recall seeing useful information about where these appropriate places are for SOAP/WSDL. Some people think they belong between enterprises. Others believe they belong within an enterprise. Kurt appears to uphold the latter at least in part, but I am not sure.

My sense from limited hands-on experience is that the more one's interfaces depend on SOAP and WSDL the more difficult they will be to change. I would be interested in contrary experiences. The last thing an IT shop needs is more cruft that is difficult to change.

Update Day 4: Dare Obasanjo makes constructive observations until he falls back to the apparently default advice...

If you know the target platform of the consumers of your service is going to be .NET or some other platform with rich WS-* support then you should use SOAP/WSDL/WS-*. On the other hand, if you can't guarantee the target platform of your customers then you should build a Plain Old XML over HTTP (POX/HTTP) or REST web service.

Again the "rich" support appears to hold if one does not care about interop. But then the question is, why use SOAP/WSDL for a non-interop scenario? Certainly there are better solutions when confined to just C# or java.

Concurrency-Oriented Programming

Peter Williams needs a look at Erlang and its take on Concurrency-Oriented Programming (pdf slides, pdf paper)...

My gut tells me that none of the approaches I know for apparent concurrency are going to work well for a highly concurrent application on highly concurrent hardware. If that is the case we will see something new and different as soon as the cool hardware gets into circulation. (via James Robertson)

On the other hand I don't see everyone cutting rope and heading over to Erlang. Probably some languages will evolve concurrency mechanisms. Those that are able to evolve most easily will be the simpler dynamic languages like Lisp, Smalltalk, Python, and Ruby.

On the way I would expect simpler extralingual concurrency mechanisms to gain popularity, e.g. JavaSpaces, esp. in the form of cross-language tuple spaces.

Where We Stand

From SDForum via xmlgrrl...

Harold: Web services is where we would have been 10 years ago if MSFT had joined OMG.

Andrew: You’re giving us too much credit.

Oh, was that a compliment?

Structured Speech

Given that I have listened to no more than three podcasts and have found no more than one useful, I would have to agree with Werner Vogels' desire for more navigational structure.

I believe my main problem with the format is you are supposed to use it linearly. I love to read articles, papers, books, etc., but I am often a non-linear reader. I will scan back and forth for interesting pieces. The fact that you cannot build a hierarchical model of a podcast for selective drill down is pretty annoying to me...

It is not that I want everything written down... I believe more structure would be absolutely helpful.

I would probably pay more attention to podcasts if they were written down. 8^(

The Secret

Blaine knows the secret...

Accidental architectures arise from information exposure. Knowing what to expose and what to hide takes thought and planning in design. It just doesn't happen.

This applies not just to an isolated software application. This applies to an entire enterprise information system. Eliminating unnecessary dependencies among components is the key to managable evolution of software systems.

Abstracting Over Time

Motz writes about a recent talk by Richard Gabriel, including better abstractions for time...

there is nothing like abstracting over time (mp3) which leads to a system that - compared with the real world - would mean that "if you go to the beach, you would need a place in your brain that let you remember that you went to the beach".

Faith and Reason

Sam Harris, author of "The End of Faith", from a talk on CSPAN...

"In the west we are standing on the shoulders of dwarves, contemplatively speaking."

From his web site...

Harris argues that in the presence of weapons of mass destruction, we can no longer expect to survive our religious differences indefinitely. Most controversially, he maintains that “moderation” in religion poses considerable dangers of its own: as the accommodation we have made to religious faith in our society now blinds us to the role that faith plays in perpetuating human conflict. While warning against the encroachment of organized religion into world politics, Harris draws on insights from neuroscience, philosophy, and Eastern mysticism in an attempt to provide a truly modern foundation for our ethics and our search for spiritual experience.

Linux and Windows (and Mac OSX)

Here's a funny report on Linux from Microsoft. Not funny "ha-ha" but funny "sad".

First of all I did not even have to read the report to laugh, or cry as it were.

I have a couple of computers that are about 5 years old. They are set up for dual boot to either Windows or Linux. The only facts I need to know about them are these: I can run the latest Linux on them. Every time there has been a new version that I wanted, I was able to upgrade with no problem.

On the Windows side, I am still running Windows 2000. I am unable to get XP running on them. Oh, maybe someone will respond with a statement like, "All you have to do is X." But any value of "X" that I have run into appears to be at best an unknown amount of time and possibly money.

Let me say right now, I'd like to run XP on them just because the other Windows systems I have around do run XP. I could not tell you the reason why XP is better than 2000 other than there are probably some significant security differences. I am actually fine running Windows 2000 when I need them to run Windows at all, which just further makes the point of incompatibility absurd. Did they really screw up Windows NT and then Windows 2000 so much that a significantly incompatible rewrite of the OS was required? Or just an opportunity for more revenue?

I can also run Apple's latest Mac OSX on my six year old Macintosh very well. I have all the facts I need. I have no expectations that anything will get better over the next few years with "Vista". Apparently Microsoft is only looking forward to getting more revenue and not looking at the real value for the customer. To date, that has worked out well for them because they either don't have as bad a reputation as I think they deserve, or they don't have a customer base that cares about bad reputations. And that's a pretty darn big customer base, so I would have to go with what works if I were them, too. Will it catch up at some point? I think it has to, but I have never been successful at predicting these things.

Constant Flow of Ice Cream In

I am watching this Microsoft video to find out what The Roches have to do with sound on Windows Vista. It turns out they have Robert Fripp doing some sounds and in the recording evokes some essence of his work with The Roches. I saw the Roches perform at the Berkeley School of Music in Boston in 1985 (around the time Another World was released).

They are three amazing sisters for getting all kinds of harmonies into their music. (If you have not heard them perform the Hallelujah Chorus, well.) I was looking forward to some harmonies on the Windows Vista OS. Oh, well.

It turns out The Roches web site has a *lot* of video from Soundstage, SNL, the Late Show, etc. I leave you with some favorites of mine...

• Love to See You
• Mr. Sellack (it is good to see Johnny Carson again)
• Another World (I don't feel that great today, there's got to be another world, oh yeah, ooh ooh ooh baby)
• The Largest Elizabeth in the World
• I can't stop watching their videos...
• Married Men
• Keep On Doing/Jerks On the Loose
• The Hammond Song
• And not on their site, unfortunately, their spectacular version of...
• On The Road To Fairfax County (Real Audio) (or Windows Media)
• I Fell In Love (Windows Media) (listen to the following lyrics in three-part harmony... "You're a slick, affected, switchblade-flashing, motorcycle freak... You walk that mean and tough-guy walk of all the other guys who hang around...")
• The haunting "The Scorpion Lament (Windows Media)" and "I'm Losing You (Windows Media)"

Why don't you listen to my little pep talk? Come in off the ledge.

RSS as an Operating System Component?

This is a sign of sure doom.

Problem is, I don't know whose doom it is a sign of.

Have you seen this man?

Wondering if this post will land me on the "no fly" list, I now relay a funny story. Not funny as in "ha-ha" but funny as in "that's funny. hmm."...

The author of a critical biography of Karl Rove ended up on a no-fly list. James Moore, an Emmy-award winning journalist and author of Bush's Brain was recently barred from boarding a flight from Texas to Ohio.

An airline employee gave Moore an 800 number to check on his no fly status. Moore posted this excerpt from their conversation at the Huffington Post:

"Mam, I'd like to know how I got on the No Fly Watch List."

"I'm not really authorized to tell you that, sir," she explained after taking down my social security and Texas driver's license numbers.

"What can you tell me?"

"All I can tell you is that there is something in your background that in some way is similar to someone they are looking for."

"Well, let me get this straight then," I said. "Our government is looking for a guy who may have a mundane Anglo name, who pays tens of thousands of dollars every year in taxes, has never been arrested or even late on a credit card payment, is more uninteresting than a Tupperware party, and cries after the first two notes of the national anthem? We need to find this guy. He sounds dangerous to me."

"I'm sorry, sir, I've already told you everything I can."

"Oh, wait," I said. "One last thing: this guy they are looking for? Did he write books critical of the Bush administration, too?"

Haskell and Erlang

Joel Reymont has an interesting experience report from using Haskell and Erlang. He also has an article published on "Writing Low-Pain Massively Scalable Multiplayer Servers".

Windows Switcher -- I'd Love to Switch, Oh, Yes!

Wow, a Three-Dee view of windows to allow me to switch among them. How... well, oh boy.

Microsoft, do you know what I would really like? I spent an hour or so putting drawings in a Word document. They looked fine yesterday. But today I opened up the document and all the drawings are squished. Expanding the drawing area does not also expand the drawings. Big area, squished objects.

Maybe by sometime in 2006 you'd think Word could embed a drawing and retain the layout correctly?

May Apple, Google, or whoever, run over you with a steam roller... as long as they provide a document processor that actually works.

Microformat => Microprotocol?

I just did a search for "microprotocol" and was surprised I did not find what I expected. Well, I found a couple of citations related to Atom from Bill de hÓra's site.

Shouldn't there be as much attention paid to microprotocols as to microformats? Or nearly so. I understand HTTP GET can be used to provide an XHTML document with multiple embedded instances of microformats. I understand Atom Publishing Protocol can be used to provide even more functionality with other HTTP verbs.

But what about WebDAV and CalDAV? These are fairly complex application protocols with out a clear purpose from what I can tell. Shouldn't this functionality be provided by more conventions on HTTP and APP? Right? The nice thing about microformats is they do not extend XHTML, just tell you ways to use it. The nice thing about APP is it does not extend HTTP, just tell you a way to use it.

To get beyond mere APP, would you want to extend HTTP like WebDAV and CalDAV? Why?

Am I missing where people are discussing ways to *use* HTTP (as opposed to extend it) for calendaring and other microprotocols? Otherwise how many protocols do we think we need? Is 2006 the year for microprotocols?

Glitter on the Highway

Happy New Year, drive carefully...

Hop in my Chrysler, it's as big as a whale and it's about to set sail!
I got me a car, it seats about 20
So hurry up and bring your jukebox money...
Can't move. Resting...
Huggin' and a kissin', dancin' and a lovin' at the love shack

A (young) driver (without a license) in a borrowed car (without insurance) smacked into my car yesterday. He tried crossing two lanes of traffic from a side street and did not make it.

He hit my left front fender. If I was a few more feet into the intersection he would have hit my door. I bumped my head a little bit, but neither of us had significant injuries. Another good thing.

Drive carefully out there, and have fun.

Type Pecked

Brian Foote used the term "type-pecked [languages] like C++ and... Java". I had not heard that before. And so I chuckled.

An Acronym With Meaning

My vote for best acronym in the SOA world...

WSOL

SCA: EJB Redux?

Steve Loughran writes...

A component architecture from a group of vendors including IBM, Oracle and BEA. The last time I saw one of those it was called Enterprise Java Beans... I am so excited I could lie down

I was a participant early (pre-1.0) in that process. It was *truly* frightening. Literally a Jeckyll and Hide experience. I would say a "ping-pong" experience as the spec bounced around, but more scary.

Following the Rest of You

Mark Baker, Jeff Schneider, and others are going around on the WS / Rest debate. (Hey, has there been a live debate on this anywhere? Saved to the web?)

Mark says WS is unnecessary, that HTTP will do. Jeff says, yes but all kinds of things will do. It seems to me in this argument, Jeff should get down to one selection or at least begin to lay out his criteria for when to choose one or another. But...

The funny thing is, I'm a REST fan - I just can't stomach this one-sided bullshit.

But if Jeff is a REST fan then *when* is he a REST fan and when not? Jeff thinks Mark's stance is BS, but Mark is consistently towing the line on his choice. Exactly how does Jeff propose choosing one or another approach in any given setting?

As for my own semi-informed, partially-formed opinion, aren't the constraints of REST in the form of HTTP providing a more *broad* set of capabilities? i.e. an HTTP-based resource-representation-centric approach can be used by all systems that understand that architecture, which includes all the software written for the web already. (That's pretty substantial. 8^)

However if you take some other approach, say WS, then which WS standards do various vendors support? Big problem from what I can see. Then suppose you want to leverage all that existing web stuff. So you have a WS to get some resource in some representation and you want the representation to include a form that will POST requests to update the resource. From what I can see you still need to implement the REST model using HTTP anyway to do this. So why the WS?

So is the question just this: is the representational state transfer approach going to form the core of your architecture or not? If so, use REST. If not, then *what* will your architecture be?

For some situations I could see adopting some other architecture in conjunction with REST. Within some boundary, say an internal set of IT data centers, I could see using various architectural approaches that are more along the lines of what you put *behind* the REST-ful applications. Here you need a request dispatching and approval architecture, a caching architecture, a query architecture, a systems management architecture, and so on.

When I listen to vendors talking about Enterprise Service Busses this is what I think I am hearing -- how do you implement the data center itself? GET should be fairly ubiquitous. Then there would uses of POST, PUT, DELETE, etc. but maybe not so ubiquitously. i.e. eventually there are databases of various kinds: relational databases, file systems, and tuple spaces may form the core, if not completely in their current form.

There are two architectural questions it seems to me -- what do you present as an abstract programming model? And what do you use to implement that model? I am not sure there are enough constraints around a general WSDL model. More constraints are needed to guide the specific model. What do you think they should be?

RDF, REA, MRAM, etc.

On the topic of representing persistent formats for RDF, I came across another favorite paper from some years ago. Reading through this paper in 2005 (Adaptive Framework for the REA Model), there would appear to be some correspondence between RDF triples and REA, which also has a kind of "triple" model: Resources, Events, and Agents. I can't say what that correspondence is except at my superficial level of understanding of either.

The paper's approach to incremental computation results in a kind of graph model that would correspond to a graph of RDF triples representing the same information. i.e. I think RDF could be used in a straightforward manner to represent the fine-grained elements of REA models. Both models have some scenarios where connected graphs would be useful and this paper points out one of them... incremental computation, esp. for aggregating information and computing results not represented directly in the model itself.

This is an interesting approach and I wonder if the authors or others have pursued it since 1998. (Aside: the creative approach is yet another to come out of the Smalltalk community, in particular Ralph Johnson and his U.Illinois squadron.) So if you are not familiar with the history of ideas that have streamed out of the Lisp and Smalltalk communities, start your search now because the list is long.

Also -- this would seem to be an interesting solution for throwing hardware at a problem rather than the typical way low level engineering of computation. (Disclaimer -- I work for a hardware vendor, so you might conclude I have a personal interest in this approach. Actually I am generally interested in simple software approaches that can be accelerated with the evolution of hardware. I recognized this several times over the last 20+ years working for varying employers on varying problems. Through it all, reasonable solutions in Lisp, Smalltalk, etc. became faster simply by migrating the hardware. Meanwhile the industry generally migrated their languages to take on more features of these earliest dynamic languages born in the laboratories of the 1960's.

On another aspect of hardware, unfortunately for soloutions like the one in this paper, I was (now unrealistically) thrilled by the advances in Magnetic RAM (MRAM). This hardware seemed to be on the cusp of a price and density breakthrough which would have virtually eliminated the need for secondary disc-based storage for even very large persistent data sets. That does not seem to be panning out on the rosy schedule presented a few years ago. There appear to be some real limitations that will be difficult to overcome. The run-time / persistence mapping problem will be with us for a while. All the more reason to strive for simple run-time models as well as simple persistence models.

Not Unexpected

You scored as The Amazing Spider-Man. After being bitten by a radioactive spider, Peter Parker was transformed from a nerdy high school student into New York's greatest hero. Peter enjoys the thrill of being a super hero, but he struggles with the burdens of leading a double life. He hopes someday to win the heart of his true love Mary Jane, the woman he's loved since before he even liked girls. Right now, he just wants to make it through college and pay his bills. The Amazing Spider-Man 63% Neo, the "One" 63% Maximus 50% Lara Croft 46% El Zorro 42% Batman, the Dark Knight 42% Captain Jack Sparrow 38% The Terminator 33% William Wallace 33% Indiana Jones 33% James Bond, Agent 007 17% Which Action Hero Would You Be? v. 2.0 created with QuizFarm.com

Google: "All your Base are belong to us"

Via Mark Baker, apparently Google says, "All your Base are belong to us."

At least if you make your base Google Base then Google will prevent that content from being openly searchable on the web.

Also from Mark's same post, apparently Google Base is supporting the most base of purposes. As I wrote regarding another new Internet capability, oh, dear. That seems to be the way of the world.

Which Internet, again, did Google climb to the top of? Is it safe?

MIMO - Super G Wireless

My treat this season is a NetGear RangeMax MIMO wireless Super-G router. Seven antennas and 108 mbps. Actually it was a treat for my youngest who just got his own first PC. He is gaming wirelessly reliably from his room with a very strong signal and 108 mpbs.

I just have an 802.11g card in my laptop so I'm still at 54 mbps. But where the signal from the far corner of my house had been low, now it is very strong. The other son wants a Super-G card now too. I've got my eye on the new round of wireless products for network storage, etc.

'Tis the season I suppose.

Logically Out of the Blue

I did not see this coming. I was in a Barnes and Noble book store the other day and a new book jumped off the shelf into my hands. (I was shopping for gifts, honestly. I don't know what drew me over to the computer section.)

The relationship between Lisp and logic programming goes back to the 1960's and was a precursor to Scheme being invented in the first place. Steele created Scheme as a vehicle to understand Carl Hewitt's actor language which was derived from his Planner language which Hewitt developed about the same time as Alain Colmerauer was developing Prolog. Planner implemented a backtracking capability (for planning, get it?) similar to Prolog's. Scheme has since been used to implement many kinds of programming languages, including several kinds of Planner-like and Prolog-like languages.

And so The Reasoned Schemer brings it all back home to the student of programming and programming languages. It follows the Q&A-with-food-themes style of books also written by Dan Friedman with various authors for learning Lisp and Scheme themselves. (They even have a Java book in this style.) Some people like the style, others do not.

Programmers interested in Lisp, Scheme, functional programming, and/or logic programming will be interested in this book.

Representing RDF

Bill de hÓra explores the upside and down of using an RDBMS for a persistent representation of RDF. This reminds me of several times I've encountered a generic attribute/value pair relational data model. As Bill points out regarding the "disenabling" this does to Rails and Django (as they are currently designed), the generic model moves the domain vocabulary out of the relational schema and into the data itself.

I understand one of the keys to RDF is that this is a conceptual model for the semantic web and not necessarily a recommended persistent physical model. That doesn't mean it couldn't be, but I wonder what other models might make sense.

An advantage Bill points out to this approach is the RDBMS schema itself would change little if any over time. Everything becomes data stored using one simple, generic model. This tells me we need a database, independent of RDF or any other conceptual model, that can recognize more about the domain-specific data and relationships while supporting "agile data refactoring". Some databases are pretty good about this for star schema models. Star schemas kind of aggregate related RDF triples into another conceptual model that is only somewhat more complex than the RDF concept itself.

A star schema relates MxM things to each other through a common "fact". The simplest fact is a statement that such a relationship exists. So when some relationship is established at some point in time, say an employee is assigned a manager, the fact that this exists is established by a fact relating the employee (e.g. a row in the "employee" table) to the manager (e.g. another row in the "employee" table) via a fact (e.g. a row in the "management" table) along with other related "triples" vectoring through the same table (e.g. the "calendar" table for the start date and the "department" table for the organization being managed, perhaps the "role" table for the row of the role the employee is playing).

Then other triples even more tightly associated with each other are those "dimensions" themselves. So the relationship of the employee to their name, address, pay scale, and other vital statistics may simply be represented as columns in the same employee row. The result is a normalized fact table bringing together an array of denormalized dimension tables.

The kinds of changes to this model are simpler than those made to a "fully" normalized model. Columns are typically added to a dimension but are rarely removed. Columns infrequently change data type, but could. Dimensions added to the model, new facts are added, and new relationships between facts and dimensions.

A database not based on 25 year old approaches in the typical RDBMS are pretty efficient about representing this model and refactoring. Sybase IQ is one, e.g. in the Sybase Dynamic Operational Data Store. Efficient in this case means the space used in the database is typically *less* than the space used to represent the same data in a flat file. Very few databases have this capability and at the same time *reduce* the maintenance effort.

This model would be useful for RDF data sets that need to do a good bit of computation. For example when the "facts" of the relationships include numerical measurements of some kind (payments, temperatures, velocities, occurrences (e.g. attendance, page hit, etc.), etc.)

When the model is not so computation intensive I would suspect the queries would be more like searches and path navigation. The structure would not benefit from being in a more general attribute/value exploded RDF-like schema. Some kind of graph model would be better to support path navigation around large networks.

The three activities I think would come up frequently are more or less free-form search (text and other indexing independent of conceptual model), path navigation (across a large network of related RDF triples), and numeric computation (on measurements of some kind related to the things of the triple).

Jini/JavaSpaces/Rio... and Jython?

This one slides out off my sphere of awareness and then it's not too long before something pops it back inside. Are JavaSpaces still the best thing going for Java and yet still the least adopted capability?

I'm thinking about code replacement, etc. to keep a distributed system evolving. Erlang does it explicitly. Smalltalk does it well, too.

Java per se has class loaders, etc. But the still-too-secret-weapon that Java really has going for it is JavaSpaces (and the surrounding capabilities like Jini, including Rio).

JavaSpaces provides a simple distributed-shared-memory model for Java objects. While class loaders, etc. support dynamic loading, the mechanisms inside the language and VM per se do not support evolving interfaces very well. JavaSpaces is that "dynamically typed" (if you will forgive the phrase) surface in which "objects" [1] can actually change entire interface definitions over time. Not much more care is needed for this than for, say, a Lisp or Smalltalk system.

I'd rather be using Smalltalk, but JavaSpaces deserve much more attention. I wonder how well JavaSpaces and Jython fit together?

[1] I use "object" in quotes because I do mean to imply that a POJO will change its entire interface. Rather the POJO that is playing the role of some conceptual object will go away over time, to be replaced by some other POJO that is playing the same or some evolved role. This is like the observation that over time the cells of our bodies go away, being replaced by new cells. After some time and degree of replacement, do we have the "same" body that we had earlier? This is the core essence of a survivable complex system.

A Bag on the Side of the Eclipse

I was recently reminded of a favorite book: Tracy Kidder's "The Soul of a New Machine". I went to Data General developing CAD software and met some of these guys a couple years after the book was published.

A couple of phrases stick out... Tom West saying, "Not everything worth doing is worth doing well." and building exactly what Ed deCastro said he did not want, "a bag on the side of the Eclipse".

This is a bag on the side of my blog. I have not been around a computer much outside of work lately. Hopefully over the holidays I will get back to finding out what this AJAX thingy is.

I am currently engrossed in another book though, "Wicked: The Life and Times of the Wicked Witch of the West". As John Updike is quoted on the cover, it is an "amazing novel". It reminds me of Marion Zimmer Bradley's "The Mysts of Avalon" in its imaginative telling of a new story about an old story. Both books creatively incorporate real issues about society, culture, religion, and politics.

Dabble DB

I've not dabbled yet, but looking at the videos I agree with the consensus... the Smallthought team has put together a nice, simple system in a short amount of time via Smalltalk and Seaside.

This blows Ruby on Rails a bit out of the water. Maybe it will inspire RoR and other systems to become even more radically simple.

The Dabble design has borrowed a number of good ideas from OVAL and Lotus Agenda. They've also thrown some other good ideas into the mix. The mix looks good, great even. Because they've built on what I consider some under-explored fundamentals (simple mechanism to combine semi-structured information) and the simpler interactivity enabled by Seaside, Dabble could continue to go a long way for some time. This kind of thing, if not Dabble per se, should be a fundamental building block of those office applications you wish you had but would cost too much to develop.

The exciting thing is there are even more good ideas that have not been incorporated. I guess the unexciting thing for the developer community is that Dabble appears to be a commercial venture. But I guess that's good for them, because they have a winner on their hands.

I think it will be easier to dabble than it is to jot.

Don't Lose Any More

From Mike Warot at the IT Garage...

This is the 21st Century... don't lose my work any more!

SAP -- Behind the OSS Curve and Loving It?

From FTPOnline, quoting SAP's Shai Agassi on open source software and intellectual property...

Open source will fail to deliver innovation and is more likely to break applications...

First, open source is good at "opening" mature software patterns. For example, about the time Unix was 30 years old, Linux became relatively easy to create. There are some innovations, but for the most part Linux has opened access to the Unix operating system approach. GCC did the same for compilers. Open Office for desktop applications. GCC actually became a vehicle for compiler research. Linux and Open Office less so, to my knowledge.

Second, there is actually a good bit of innovation in OSS. Look at the Java industry. EJB is a disastrous combination of innovation by entrenched IP vendors. As the market realized that, individuals stepped up with better approaches. The Java industry is more rich and I would suggest has had its life extended through this innovation. Not to mention Ruby on Rails, Seaside, etc. beyond the Java industry per se.

It is ironic that these comments are coming from SAP rather than BEA or IBM. SAP's J2EE environment is noticably behind the industry in general and would be advanced greatly by adopting some of the OSS Java innovations.

Third, it is unclear to me that IP goes away with OSS. Google is a fairly successful company based on leveraging OSS to get its IP into the market. Back to SAP, why should SAP invest large resources into a proprietary Java environment when its clear advantage is in its business capabilities, i.e. its IP? SAP has a long history of innovation in virtual machines and server utilization, that is one reason. But 80 percent of their Java environment could probably be OSS and in doing so SAP would approach and begin to surpass the leading J2EE vendors in their innovativeness.

I would consider myself an OSS "realist" in some ways... I agree with comments that many companies are underwriting OSS because they can leverage it for other revenue streams. OSS has to be funded somehow, and I have questions about how sustainable or broadly OSS models can be applied. I for one am happy that Cincom is profitable with Smalltalk. They are a clear example of having met OSS on their own terms. In a position to choose software tools for one purpose or another, there are several instances where I would choose Cincom's non-OSS Smalltalk product. There are others where I would choose OSS tools like Erlang, Ruby, Python, or Squeak Smalltalk.

"But if you look at the most innovative desktop today, Microsoft's Vista is not copying Linux, it is copying Apple."

This seems like kind of a silly quote in the context of OSS and IP. Mac OSX is based on Unix, BSD, and Mach. Darwin, the underpinning of Mac OSX with this BSD/Mach heritage, is itself OSS. Linux is just another implementation of Unix. Less innovative than Mach but unarguably successful and innovative in its industry impact.

And so the other apparent absurdity in this statement: MSFT's Vista has not innovated significantly beyond Mach. Has it? I guess we could argue about its "significance" and its "innovation". I think Vista neither exceeds Mach at the OS level nor other OSS innovations at the user level by an order of magnitude no matter how you measure it. Microsoft is near the top of software IP companies and yet they have not demonstrated innovation well beyond the OSS community.

At least, they have not created innovative software so "compelling" (to use their own adjective) that I believe I *have* to have it, or it *has* to be part of a software infrastructure that I would be responsible for.

Shai -- get busy on your own environment which is well behind the technology curve. And a good way to do this would be to leverage OSS.

Beyond Java: A little Dabble?

From the article "Moving Past Java" an interview with the author of "Beyond Java"...

Bruce Tate: Keep in mind that I'm suggesting Java will be dead like COBOL, not dead like Elvis...

Madhu Siddalingaiah:I agree with you that innovations are beginning to appear outside of Java.

Don't worry. Madhu will kind of correct himself in just a few paragraphs...

Madhu Siddalingaiah: Ruby is a dynamic language, but there are many others, such as LISP and Smalltalk. Are there lessons we can learn from the one of oldest and one of the most innovative of languages?

Bruce Tate: Sure. Ruby is actually a nice fusion of Smalltalk and Lisp. I think the biggest lesson is that you need a catalyst. Ruby has Rails. None of the other dynamic languages got a catalyst.

Maybe a little Dabble do ya as a catalyst for Smalltalk, building on Seaside.

Then Bruce nails the essence of the problem...

Bruce Tate: Java just doesn't express data well. Other languages make it easy to put data into a hash map. Not so with Java, so we just drop down into XML.

They wrap it up like this...

Madhu Siddalingaiah: Do other leaders in the open source community share your opinions about Java?

Bruce Tate: You can see people ramping up on Ruby on Rails in a big way. David Geary, JSF expert group member, and one of the greatest Java authors, is writing a Ruby on Rails book with me. James Duncan Davidson, inventor of Tomcat and Ant, is now working with Rails. There are many, many others.

Beyond Java

From the Java community, looking beyond Java. High praise for so called dynamic languages and how they have resulted in more productivity with Seaside (Smalltalk) and Rails (Ruby), etc.

Oh Dear

Questions about the Video iPod...

Uncertainty surrounding the level of demand for the product, the sources for compelling content, and its possible use to view pornography seems to leave more question marks than exclamation points...

Brian Milburn, president of Solid Oak Software in Santa Barbara, Calif., which sells software that filters out objectionable content on the Internet, says he's certain video iPods will quickly become a favorite tool to download and view pornography.

By Christmas, he predicts, the ability to download pornography "will be one of the main attractions for getting a video iPod.... These kind of technological advances can have a lot of unintended consequences...

Yes, we've certainly done the job of keeping porn off of computers. Now on to the iPod I guess. Since when has video technology *not* been driven by the porn industry? (I think there may be a lesson, or two, there.)

Of course I live in a fairly relaxed climate for such things. Here's a description of my fair city (link may not be safe for work)...

With more strip clubs per capita than anywhere else in the United States, Portland, Oregon had become known as a hot bed for erotic dancing. Add to that an extremely accepting public, a growing sex worker movement and a liberal constitution that protects the right to dance nude, and you will find a city where an oft-marginalized subculture has gained a degree of acceptance almost unknown anywhere else in North America.

Termite

No, I don't know what's up with Termite. I like the idea of Scheme being an UNCOL.

But what I really want is an Erlang environment up and running for me. So I decided not to wait for Termite to mature or to rewrite Termite using Gambit's new mailbox mechanism.

We are now in the postmodern, dare I say, Web 2.0, era. I'm back to using Erlang after several years and loving it. It has so much good stuff packed in, from Mnesia to simple distibuted messages. It's not my favorite sequential language, but it seems like my favorite concurrent language.

WS-Interop-In-The-Small

I left a comment on James Robertson's blog re: his item on CORBA and WS. I agree with his point about port 80. (Although CORBA was addressing that just as the SOAP community accelerated.)

Still, as far as port 80 and people working hard on the basic interop capabilities go... the result I think is still essentially that SOAP and WS-* interop is about at the level XML-RPC was five years ago. Is it really much beyond that today?

The data passed via SOAP is arguably more expressive, or at least more widely recognizable as XML, than the XML-RPC syntax. Once you get off of port 80 though and stop using HTTP as a transport, then WS-* seems to quickly devolve into the same old proprietary vendor race to lock in customers.

JBI is an attempt to avoid this but has little momentum that I know of. The ESB is still promoted as the key to deep success of integration within the enterprise. We had this five years ago in proprietary form. Today we are no further from the vendor lock-in that it takes to move in this direction. Outside the enterprise there's still no advantage over REST in sight.

This is still interop-in-the-small... I think even CORBA was beyond this point five years ago.

The Cathedral is the Bizarre (Too)

Jon Udell points out that the cathedral and the bazaar have much in common, perhaps most of all would be their respective ongoing evolution. They are equally bizarre.

To the point about Sed and Awk... I am not sure the cathedral or today's bazaar is more coherent. I am not sure they need to be. We've been postmodern for some time and we should not expect that to change. Embracing the chaos is almost the goal itself.

Innovation implies chaos. Sometimes the cathedral can be too limiting for innovation, or at least timely innovation. Look at the back and forth on schedules and dependencies among the new work coming out of Microsoft's dotnet and longhorn team. When was WinFS avalable again? What platforms? How does that line up with the LINQ query capability again?

In the bazaar there is much less control over the chaos. I'm fine with that because generally people find or make ways to get from one place to another within the bazaar. Here's some postmodern glue I have been working with lately...

I wanted to use Erlang with the Berekeley XML database as well as with the Clips rules engine. After a little consideration of my options for integration, for my purposes the easy answer was Python. Running SWIG for Python on Erlang's C-based erl_interface creates the glue that gets me into Python from Erlang (with all the desired Erlang node management capabilities). From there Python already has Berkeley XML DB integration and the PyClips interface to Clips. Plus I can send Python code from Erlang over the distributed process connection and have it executed dynamically in those Python "agents". On the front end, Erlang sends JavaScript, XML, JSON, and CSS to the browser. These combinations make Sed, Awk, and various shells look downright homogenous.

Bizarre combinations gathered from the bazaar. But when you begin to piece together all the different MSFT components you get something equally as bizarre.

Innovation or Standard: Choose One

On BPEL standardization...

the market is fairly immature as to what is the right way to take something like that. What it also shows then is you're getting a lot of innovation in the standards body. Innovations in standards bodies typically aren't good things.

Comments on Processes, Etc.

Le Roux Bodenstein writes in email...

Some comments on some of your recent posts: You were talking a lot about scaling to more processors, app servers becoming the new operating systems, etc.

Aren't app servers just big processes that manage large amounts of threads? Don't multiple processes scale more easily because of "share nothing"? What's wrong with operating systems? They are basically (among a few other things) very efficient process schedulers. (modern unixes, at least)

"Shared nothing" is important. If the multiplexor does not enforce shared nothing then convention should. The new multiplexor (e.g. the web/app server) is lighterweight than the older OS multiplexor. Shared nothing lightweight processes are the best of both worlds.

I think if anything we should work on making inter-process communication easier. Processes should be tied closer into languages and inter-process communication should be abstracted. Even more than some newer languages did with threads.

I agree. But we need to lose the distinction between local and remote processes. e.g. Erlang processes us the same simple communication abstraction whether they are co-located or remote. So do HTTP processes of course, and XMPP, and... Perhaps the OS is the last bastion of the local/remote distinction? (The now defunct Apollo Domain OS being one of the rare exceptions.)

I think "lightweight processes and threads" will lose popularity. Multiple cores and processors effectively make the "heavy-weight overhead of processes" a mute point. There's a lot less context switching when things run on their own processors and once a process is started (which is as far as I know quite expensive compared to starting a thread, but I am not an expert) it can just keep running until it is needed.

I agree heavyweight processes will perform better, but lightweight processes will perform that much better. The OS process approach can and I am sure will be improved but making improvements there is probably a longer road than taking the Erlang approach in a specific web/app server or language runtime implementation.

Anyway... my argument is I don't think app servers will help much on operating systems with decent kernels. In fact - they will just be in the way. Or an extra layer. Or what little use they have can be added to the os.

I think this is a fine way to approach the problem but as I said I think it is a longer road. Hopefully the OS designers will move in this direction, having taken some best practices from the web/app server and language runtime designers. (For that matter hopefully the web/app server and language runtime designers will take some best practices from their peers.)

I don't think we need new programming languages either. I think Python, Ruby, etc. can probably just be "fixed" to make multiple-process development a lot easier...

We continue to be in agreement.

The New Ribbon

I have not tried the new MSFT Office Ribbon, so I will withhold judgement for now. Others are noting the amount of real estate the ribbon requires.

The only thing I want to point out here is we have gone full circle from ribbons to menus and back. The first GUI applications I developed were CAD tools. Back in the day, those tools were expensive and ran on expensive hardware. One had to go to a special (usually dimly lit) room to use the CAD systems.

Those systems had very expensive, and large, screens. The GUI was essentially an area to present as much of the schematic as possible surrounded by, yes, ribbons. We developers had ongoing discussions to determine how to minimize the ribbon space and maximize the capabilities offered in the ribbons. CAD designers did not want to waste time clicking through all kinds of icons to get to the right function.

Soon after the Mac caught on, then X and Motif. Menus and dialog boxes became the thing. Ask me sometime how this specific transition almost brought down one of the top 3 CAD vendors.

Language Wars

What do language wars mean in the age of services and post-modern software development?

YURLs

Tyler Close has been contributing some useful exposition in a series of messages on capability-based URLs to implement authority in the REST Yahoo group.

More on the Future of Multiprocessing

ACM Queue has good explanations of where hardware is going and how to scale software into the multiprocessing era.

The introduction of CMP systems represents a significant opportunity to scale systems in a new dimension. The most significant impact of CMP systems is that the degree of scaling is being increased by an order of magnitude: what was a low-end one- to two-processor entry-level system should now be viewed as a 16- to 32-way system, and soon even midrange systems will be scaling to several hundred ways.

Software Education

Phil Windley writes about software education...

I've determined that I'm no longer convinced that software engineering, at least as it's commonly discussed and taught, is what I want to prepare students to do. I try to focus them on being innovative, entrepreneurial, and working with dynamic languages on networked applications.

The New Application Architecture

Phil Windley looks at new hardware architectures and their forces on future software systems...

Application servers (like jBoss or Weblogic) support a development model in which programmers develop threadless code and the app server manages the threads. That simplifies the point too much, perhaps, but I think having that much parallel processing power on a single chip might make app servers much more important for developing applications. There are continuing debates about whether app servers add more complexity than their worth, but that might be because we haven't met many problems large enough to require them–yet. In the early 60's programmers scoffed at the idea of operating systems as being "needlessly complex;" that idea is ludicrous today.

I think application servers are the new operating system. (Ten years ago they were the new transaction monitor, but now we know we all need one. We all need more than one.) We need to be able to write smaller "applications" and have them interact more easily. Look at the advice for programming web applications, web services, EJB's, as well as applications in Erlang. They are all very similar. Erlang applications tend to be more simple than the rest because the available framework takes this point to the extreme.

Those other systems are built by and large on heavy-weight process technology. Maybe Erlang is the new Lisp. We need a Lisp for lighter-weight process interaction, as Jon Udell pointed out.

I should point out that I equate "web server" and "app server". That there is a distinction is an artifact rather than an inherent reason. An application server is inherently a "process monitor" with various drivers. HTTP drivers, SMTP drivers, XMPP drivers, etc. I don't think the new "superplatform" belongs to Microsoft, IBM, or BEA. The superplatform is one based on the standard application protocols. The best of these platforms will combine support for these application protocols (and more... IMAP? WebDAV?) and abstract the complexity. Most of us should be writing sequential code, rule-based code, constraint-based declarative code, etc. But we'll have to plug into these protocols.

Phil continues...

It's tough to see how you'll use that much parallelism on the desktop. I just counted the number of processes running on my Powerbook: 76... there are things we've hardly been able to imagine.

This is the result of the languages we have been using to think with. As Phil points out, these new hardware architectures will give us more reasons to think differently.

CORBA Redux

Roger Sessions is a bit of a character. Witness his back-and-forth with Terry Coatta in ACM Queue around CORBA and Web Services. Sessions continues to point out that CORBA worked when CORBA was on both ends of the wire. Well, what else would you expect? Does HTTP work when HTTP is not on both ends? For some reason Sessions seems to believe that WS-* does not require WS-* on both ends.

For years now Sessions has promoted DCOM and now he promotes WS-*. A continuing theme of these promotions is that CORBA has failed. Certainly DCOM failed worse than CORBA. Moreover, as Coatta points out...

It looks to me like CORBA is more of a success than Web services.

The CORBA community learned many lessons and would have been even more of a success had that organization incorporated the web sooner and more naturally. WS-* usurped CORBA on the premise that the result would be simpler and better, yet years later both claims remain suspect.

Later in the exchange Sessions says something incredible. Upon admitting the WS-* specs are *more* complex than CORBA's, Sessions suggests that is OK...

I agree that the Web services standards are harder to understand than most of the CORBA specifications, but there’s one fundamental difference between these specifications and the CORBA ones. The CORBA specifications had to be understood by developers. The Web services standards don’t. Nobody needs to understand the Web services standards except for Microsoft and IBM because these standards are about how Microsoft and IBM are going to talk together, not about how the developer is going to do anything... These standards have no relevance to Joe or Jane Developer, none whatsoever.

This is ironic since a few minutes previously Sessions complained that these vendors are in danger of making WS-* too transparent...

In some sense, the transparent ability to make something a Web service is not really a good thing, because making an effective Web service requires a much more in-depth understanding of what it means to be a Web service.

Coatta catches this apparent contradiction...

It sounds like what you’re saying is that the tools that automatically supply Web services interfaces are, in fact, absolutely necessary because they’re that insulation between the developers and the underlying protocols. At the same time, they’re the downfall that’s making it possible to generate poorly architected systems. Two-edged sword?

Sessions has little to defend himself with other than to suggest that although these tools don't prevent bad architectures, just think how bad the architectures would be without those tools.

I'd give Coatta the victory in this debate. Too much confusion from Sessions who has not made up his mind whether tools, protocols, or APIs are good or bad...

The big difference between Web services and CORBA is that the Web services people said right from the beginning: there is no API.

So there is no API, because those are bad. But the protocols are damn near impossible to understand. But that is good, because we only need two vendors and they can provide tools. Well, as long as they don't provide APIs, 'cause that would be bad.

The only thing more confusing than the mess of WS-* specs is the explanation Sessions is trying to proffer.

Spiral Staircase: Going Up or Down?

Jon Udell asks a great question...

In the realm of service-oriented design and business-process modeling, what are the modern counterparts to Lisp and Smalltalk?

First, who says Lisp and Smalltalk are not "modern"?

Second, I don't know the answer. Neither Lisp nor Smalltalk offer more than any other tools for these purposes. I don't think we have much in these spaces yet.

An argument could be made that for service-oriented design the counterpart seems to be HTTP. Whether or not HTTP is the best we can do is moot. Lisp and Smalltalk were allowed to evolve in elite laboratories for a decade or more before widespread adoption. HTTP is simple with demonstrated success, but I'm not sure a successor will emerge from an MIT or a PARC anytime soon.

As for process modeling... we are in worse shape. I developed electronic design and manufacturing software (e-CAD, CAM) for many years, 1983-1993. This work included developing schematic editors as well as software that consumed schematic data. In those days there were few standards for this kind of software, interchange formats, or protocols.

I see several parallels to todays BPM software. Standards like BPEL are a start, but there is a long way to go on the front and back ends for these systems.

Back to services... I think we have a long way to go here as well. Most services I am aware of (whether they are REST or WS-*) are built on languages that emphasize the "inward view" of the system (e.g. the arrangement of code and data within the process) rather than the "outward view" (e.g. the service interface and contract). Sure there are "declarative metadata" schemes in various languages for denoting some function should be a web service, etc. But these schemes are bolted on to previous generation languages. Accessing semi-formal data from multiple sources and "mashing" these together appear to be another capability in demand.

What other capabilities should such a language provide? What about features for security? Availability? Distribution?

Comments are gone

I have republished the blog without comments. There will be no more comments until I can prevent the spam attacks. I have been all but unaffected until now. The losers lose for us all.

Feel free to send email directly to me in lieu of comments on any topic.

Threads, Processes (and Shared Memory)

James Robertson quotes from several interesting pieces on threads, processes, and programming models. I am squarely in the multi-process camp too. I remember my absolute shock about a decade ago when I learned Java (at that time aimed at being a simple web programming language) had a shared-memory with monitors concurrency model.

The key phrase there is "shared-memory". Threads in and of themselves are not the problem. The shared-memory model is the real problem.

With a shared-nothing model then threads take on much more of a "process" feel. Languages like Erlang and Termite implement lightweight shared-nothing processes above the level of the OS. The benefits of sharing lightweight processes within a single address space provide a "best of both worlds" model. Sharing can take place under the abstraction of the language... the runtime benefits of sharing but the developer-time benefits of isolation. Whether an associated process is local, or in another OS process, or on another node, is irrelevant in many respects. Performance, reliability, etc. are still concerns so the distribution problem is not completely abstracted away.

Smalltalk Leading the Way

David Buck writes about an enticing upcoming meeting of the Ottawa Smalltalk User Group...

Smalltalk and Smalltalk developers have led the way to the modern techniques and practices that [David] teaches in Simberon's courses... even now, the future of software development is being led by Smalltalk whether or not the rest of the world cares to notice.

Happy Goldfish in a Bowl

Having dabbled on the edge of AI for a number of years, I enjoy the technology as well as the philosophy that surrounds the subject. Jon Udell writes of the so-called Singularity...

That's us: just goldfish to the post-human super-intelligences... True machine intelligence was what the advocates of strong AI wanted to hear about, not the amplification of human intelligence by networked computing. The problem, of course, is that we've always lacked the theoretical foundation on which to build machine intelligence. Ray Kurzweil thinks that doesn't matter, because in a decade or two we'll be able to scan brain activity with sufficient fidelity to port it by sheer brute force, without explicitly modeling the algorithms.

Now's time for this goldfish (me) to refresh himself with a healthy dose of Terry Winograd and Fernando Flores. I don't consider myself religious in the least. However I think that neither "intelligence" nor "emotions" are in any way mechanical (i.e. transferable to a machine). I do think they are *biological abstractions*. They are the names we give to our interpretation of biological (and so, chemical, and so, physical) processes.

Back to Jon on the current understanding of human vision...

We are, however, starting to sort out the higher-level architecture of these cortical columns. And it's fascinating. At each layer, signals propagate up the stack, but there's also a return path for feedback. Focusing on the structure that's connected directly to the 14x14 retinal patch, Olshausen pointed out that the amount of data fed to that structure by the retina, and passed up the column to the next layer, is dwarfed by the amount of feedback coming down from that next layer. In other words, your primary visual processor is receiving the vast majority of its input from the brain, not from the world.

We can manipulate biological and psychological processes. We can mimic them mechacinally to increasing degrees. But we cannot "make them" out of parts and we cannot "tansfer" them to such a machine. What would that mean? The simulation of the hurricane is not and never will be the hurricane even if the simulation actually destroys the city. Something destroys the city, but it is not a hurricane.

A machine that has some representation of my "brain" may continue to develop a further simulation of what my brain would be like had it undergone similar stimula. But in no way is that machine now "my brain" and in no way does that machine "feel" things in the way my "real brain" feels things. Those feelings are interpretations of abstractions of biological processes. The machine is merely a simulation of such things. If you care about me and then observe further simulations of such a machine you may further interpret your observations to be "happy" or "sad" and you may even interpret that machine to be "happy" or "sad". But that machine is in fact not happy or sad in any biological or psychological sense.

Let's face it, even my dog "loves" me because I feed it, and when it comes down to it my kids do too. We are abstract interpreters of biological processes. One interpretation of that may be "loneliness" but so? Here is where I note that my dog gets as excited to go out and pee as it does to see me come home from a week on the road. Fortunately my kids have a finer scale of excitement.

Back to Jon for more about augmentation rather than "strong" AI...

I had the rare privilege of meeting Doug Engelbart. I'd listened to his talk at the 2004 version of this conference, by way of ITConversations.com, and was deeply inspired by it. I knew he'd invented the mouse, and had helped bring GUIs and hypertext into the world, but I didn't fully appreciate the vision behind all that: networked collaboration as our first, last, and perhaps only line of defense against the perils that threaten our survival. While we're waiting around for the singularity, learning how to collaborate at planetary scale -- as Doug Engelbart saw long ago, and as I believe we are now starting to get the hang of -- seems like a really good idea.

Seems like a really good idea. "Strong" AI is fascinating, but "augmentation" is useful.

Streamlined Concurrency Modeling

I like this book Blaine refers to. It's been sitting on my shelf for a couple of years where I've picked it up and dusted off a few pages here and there. Recently I have been following it a bit in practicing *concurrent* programming as opposed to object programming per se.

Concurrent processes can be seen as "active objects". Not every object in your system should be concurrent (not even in a concurrent programming language like Oz, Erlang, or Gambit Scheme). But the significant ones almost certainly should.

An active object (concurrent process) in these systems are like smallish databases and application-specific protocols for accessing them. Each process has state and essentially implements a loop of selecting the next most important message, interpreting it, and updating its state (its little possibly-persistent "database"). These message-passing protocols are not unlike the protocols of messages sent to (sequential) objects.

More later. If you have any favorite references for concurrent programming leave a comment.

Nintendo Revolution

The long-rumored controller for the Nintendo Revolution has been revealed. What a winner.

My understanding is Nintendo was the early innovator of the common features of current controllers. This is another revolutionary advance from the looks of the video from the Tokyo Game Show.

I have been telling my kids for years that once a controller along these lines gets us away from the unnatural button combinations I'll be able to kick their butts. They have a physical memory for modern controllers I have never invested enough time in to become competent. That is why I am competitive with them on games like Mario Kart Double Dash (simple controls), but not on most games.

Now we shall soon see if my words will hold up. This controller looks to make all sorts of actions more directly translatable into the game.

I don't care what else is in the Revolution. I am not a big gamer at all but I cannot wait to try out this controller. Cell wha? 360 wha? Ho hum. I could become more of a gamer with this controller. My 13 year old showed me the video last night. At first I thought, what, the controller is a TV remote?

But watch the video of it in action. I hope they have the virtual drum set ready to go in March!

Finally Integrated Query

Looks like a popular production language design finally will provide a query capability that disregards the kind of data (persistent or not, objects or not, XML or not) being queried. Visual Basic just became somewhat appealing for this and some other new features. I still have reasons not to use it, but this is an appealing step that will benefit many developers for some time to come.

Kudos to Visual Basic. Does C# have *all* these same features? I've not read all the PDC news yet.

Gambit Scheme 4.0 beta 15

Gambit Scheme 4.0 beta 15 is available. Changes include the incorporation of some Termite capabilities, in particular each thread has a mailbox for receiving asynchronous messages.

Beware Dichotomies

Paul Graham's latest essay is making the rounds. My initial reaction reflects my general approach to socio-political issues: beware dichotomies. Really, beware those who would present any complex issue as a dichotomy. Paul's opening paragraph has to be the worst in all his essays unless he was *intending* to present to us a straw man.

By the end of the essay though he makes at least one excellent point: transparency matters. Yes, wealth is power and I think Paul fails to recognize how closely wealth and power are related, let alone to begin recognizing the good and bad of that.

If global capitalism is going to work (I currently give it a mixed review but that's another story) then we need to remember that the theory itself is founded on "perfect knowledge". The closest things we have to perfect knowledge are democracy and transparency.

Why (not) Smalltalk?

A coincidence that on Sam Ruby's blog and on the Extreme Programming Yahoo Group on the same day there are conversations about using Smalltalk. Both conversations are along the lines of...

OK, Smalltalk is very good. So why use a language influenced by Smalltalk? Why not use Smalltalk itself?

Sam's quote is so good...

Reinventing Smalltalk One Decade at a Time

Collapse

What happened to New Orleans is terrible. But as James Robertson points out, it was not unanticipated.

The situation reminds me of our increasing understanding of earthquakes where I live in the Pacific Northwest. A similar hell could break out at any time here.

Jared Diamond's book "Collapse" addresses this recurring theme. A societal collapse is often triggered by an environmental collapse. These days the signs of a coming disaster are well known if not widely acknowledged, let alone acted upon effectively.

What Google Gets

Steve Gillmor has a great characterization of where Google could be going. Rather than trying to build the platform for everybody, Google seems to want everybody to build the platform for them or rather, with them. Google could be engineering something at the scale of the Internet, not just producing a suite of somewhat related products. If others continue to join in, the competition could be dizzying. Steve called it "iterative competitive development", "overlapping interoperability" rather than "lock-in" or "lock-out".

Anyone not taking this kind of approach probably is locking themselves out.

Threading Eras

James Robertson points out the difference between "native" OS threads and threads implemented in the language runtime. Some people (as quoted) see mapping language-level threads to OS threads as a good thing.

In part this may come from Java's first implementation of threads (aka "green" threads, a term Jim also applies to VW Smalltalk's threads). These had their share of implementation problems early on and so before long the Java community seemed to gravitate toward an anti-green-thread attitude at the conceptual level. Only now for example is BEA's JRocket JVM *experimenting* with a high-performance non-native thread implementation they call "thin threads" (wisely avoiding the "green" label).

As VW Smalltalk, Gambit and PLT Scheme, Erlang, and the Oz language demonstrate, highly concurrent programming requires much lighter-weight mechanisms than OS threads. (For the typical OS.)

Apparently most Common Lisp systems have heavy thread implementations and so the ErLisp implementation may not scale up as well as these other systems either.

A File System For (or To) the Rest of Us

Update 2: comments on WinFS capabilities...

Across the network you could still use it as a WinFS if you have WinFS installed. If you don't, it's just a file system, so you're not locked to it. It might be better just to use a common, standard, protocol; the idea that it's appropriate to degenerate to file system behavior is just retro

Update 1: Michael Lucas-Smith has another useful take on WinFS. I am not as thrilled by the synchronizing capabilities in these local pseudo-databases as he seems to be. I have a different interpretation of the Network being the Computer. Michael suggests a Topic Maps-like capability on top of databases or files or whatever might be more useful than WinFS. I agree with this, and I agree having to rewrite apps is a problem. Those seem like two different issues though that each need to be addressed.

I do agree with one vision of WinFS... I think we need easier structuring as well as semi-structuring mechanisms. A relational database is a structuring mechanism. A topic map is a "semi-structuring" mechanism. Big old-fashioned databases are good at structuring, it's just they suck at ease of use and ease of operations.

Michael points out some capabilities of SQL Server (and Oracle). For example they *can* act as file systems. But who uses them as file systems? Which goes more to the point: These more mature databases are too cumbersome. However file systems are too limited. WinFS is trying to hit somewhere in between. I think they are aiming at the wrong target.

Hopefully Microsoft is heading toward better support for unstructured data. But if WinFS is the best they can do for *structured* data... they should try again. A "local" filesystem/database with some proprietary synchronization capabilities is just not what I think we need. If we're going to rewrite systems for the formal aspects of structuring data then we can do better than a system which appears to be something less than a Lotus Notes database at best.

The bottom line for me is: whatever WinFS *is*, well, it should be *that* for the network. WinFS should not be something for me if it happens to be running on my specific box, something else for you if it happens to be running on some other specific box.

End Update 1

A telling quote in the Channel 9 piece on WinFS...

"WinFS -- to the rest of the world it's important that it looks like a file system"

A better philosophy would be this:

To the rest of the world it is important that it looks like WinFS.

But apparently the WinFS team has not entered the age of the network is the computer. I think that is what is important to "the rest of the world." Why degrade capabilities over the network?

The WinFS team continues to be stuck in outdated thinking.

Piracy

A graph showing the inverse relationship between global temperature and pirates. Actually I think there may be as many pirates as ever. So this *could* very well turn out to be bad science.

Trusted Computing and Digital Rights Management

Someone once said, "Follow the money." The aphorism is broadly applicable.

Note that the opinions expressed here are my own. In the interest of full disclosure, I currently am paid by Intel Corporation to do something akin to real work and I own Intel stock, but my work for them has nothing to do with Trusted Computing, nor do I have any privileged insights into that technology.

So here's my thought: I find irony that the Internet continues to be victimized through its own orifices, especially those on the common endpoints, while an appropriate solution, Trusted Computing, is getting a bad rap because some media companies want to use it to prop up their derelict business model (see Digital Rights Management (DRM)).

The digital rights problem seems to me to be a social problem in the market, not a technology problem. I think by associating Trusted Computing too closely with DRM. Tim Bray writes...

OK, if there’s ever a place where DRM is appropriate, it had better be open and non-monopolistic and all that.

And this is exactly right... the technology can be used broadly to support capability-based security down to the hardware. All software can be made more secure if this problem can be addressed effectively. The media companies are just a mountain in the road toward safer computing generally. We need market solutions to solve that problem, but the technology should be considered as a potential solution to the fact that insecure office and back office software are still exposing our orifices on the net to whomever wishes to pay an unwanted visit.

Trusted computing *and* greasemonkey might make a terrific combination. (With a fair bit of reengingeering unfortunately.)

Cairo looks cool

A little reminder to myself to try cairo (lower case) 1.0, cross-platform structured 2d graphics.

Oz Fest

Hop, skip, and jumping around... I'm taking another look at Oz, a "multiparadigm" language I took a brief look at a few years ago. Along with the book "Concepts, Techniques, and Models of Computer Programming" my first look was brief and did not I did not appreciate everything about them.

I was originally put off by the shear number of capabilities in the language and presented in the book. On the other hand those capabilities fit together very well now that I am actually working through them.

OK, so the only real oddity is this: unary minus is represented as a tilde. And so...
~10
Should not read as "about ten" but rather "negative ten". This is the same as
0 - 10
using the Oz binary substraction operator, but -10 is invalid in Oz. I guess if that's the oddist thing...

WikiWYG

WYSIWYG in a Wiki. Nice feature. Horrible acronym.

It's a Wiki with better in-line editing.

P2P

And by P2P I mean Point to Point. From Loosely Coupled, appropriately enough...

The deeply ingrained point-to-point mindset of mainstream computer systems design is hampering recognition of some of the most basic concepts required for successful deployment.

Google Map of Site Visitors

(via the lispmeister himself)

My google map of visitors.

Google Talk and Jabber

Update, google sez:

We want to make IM as open as the web or email...

I think that the ultimate destination is to come up with a system that is as distributed and decentralized as SMTP/email while not falling prey to the spam problems that SMTP has. We are going to test those waters first off by starting federation with a few partners and then moving on from there.

End Update

More advances into messaging, voice, and other desk-side goodies from Google...

Independent journalist Om Malik on his blog at http://gigaom.com/ pointed to technical clues that suggest Google is preparing to run an instant messaging service based on an open-source system known as Jabber. Jabber technology would allow Google instant message users to connect with established IM systems that also work with Jabber, including America Online's ICQ and Apple Computer Inc.'s iChat, Malik said.

Multi-multi-core

The future of concurrent computing...

Come 2010, expect to see chips containing ten or more processing cores, he added.
-Paul Otellini at the Intel Developer Forum

The key to building future systems more easily is in languages for making concurrent programming as easy as object-oriented programming has made sequential programming. Objects (and higher-order functions, etc.) make the organization of names and sequential statements more manageable. Good concurrency models make the organization of space and time more manageable.

A programming language's concurrency model will have to support programmers conceptualizing tens and hundreds of thousands of processes the way languages today support programmers conceptualizing object models of tens and hundreds of thousands of objects.

Polar Opposite

Have you ever been dumbstruck by a polar opposite opinion that is perfectly valid just nothing you could image yourself.

Then Apple fell into a huge hole. It started with "Hypercard". It was a ridiculous way to build programs. I could never understand it.

Hypercard is about the most fun I've ever had with a computer.

Cheap Gas -- more fun with google maps

(via Mark Baker)

... the web wins again. Find cheap gas.

Portland has typically been more expensive than non-left-coast locations in the US. At least Portland does not seem to be increasing as quickly as some other locations.

SAP Trex Search Engine and Python

Interesting comment from Andrew Barnard...

SAP uses Python as an integral part of the TREX product and has done so for many years.

Perhaps Shai is not aware of the extent to which open source products are used by SAP?

Oh, that OODB

Guess what? Transparent persistence has sex appeal but gotchas. (de hÓra and Thornhill punch) Look before you leap. Things are not always what they start out to be.

Emacs of the Web

Bill de hÓra contributes to the "one click subscription" discussion, or what he dubbed "clicksub"...

I don't use browsers much anymore and will be using them even less next year. Aggregators are so much better than browsers for following content. Really, if you have to read stuff on the web and are using a browser for that, you should try an aggregator. And then, what's the browser good for?...

To be honest, next year's browsers need to be aggregators, else I don't see the point in using them...

Instead this would be great: at some point weblogs flip over and the HTML website bits will become secondary fluff to the XML content, like how PDFs are secondary web fluff to HTML today. The frontpage would be the feed, the archives would be Atom entries...

Honestly, permalinking to a html file is starting to look more and more like a bug. Why not point to the XML entries?

I was going to blah blah about some analogies with Emacs. Hard-core emacs users find themselves snickering about artificial boundaries around "applications" -- there is very little concept in the world of Emacs that "this thing is a word processor" but "that thing is an email reader" and "some other thing is a directory explorer". And the list goes on for outliners, class browsers, PIMs, etc.

The underlying problem may be there is no (not yet) Emacs of the Web. But Bill what I get from Bill's post is my Emacs perspective is mostly a "front-end perspective". There is a whole back end perspective on the future of feeds, pages, formats that is in its infancy.

Emacs was born as a set of flexible front-end parts in the 1970's. Even in the world of objects and MVC, a significant number of programmers were just starting to grasp the same concepts in the late 1980's and early 1990's. By and large, looking at our software landscape, we still have not grasped the concept. In fact, the browser may be the closest thing we have to this on a widespread basis!

Not only does Bill show this is not seriously the case, by now we really should have moved on to the back end of resources, formats, and hyperlinks. The head spins. We have a lot of catching up to do.

When to create syntax in Lisp?

From the gambit email list, I respond to this question about writing macros...

In your opinion, is it appropriate to use a macro to abstract away repetitive boiler-plate code? Or is this better done in a procedure?

This is almost always a procedural abstraction rather than syntax, especially for beginners with Lisp... better to spend a lot of time with just procedural abstraction, higher-order functions, etc.

Syntactical abstraction I use for controlling the order of evaluation and sometimes to put a pretty syntax around use of lambda.

As an example an old Lisp control structure is called PROG1. This structure takes a sequence of statements, evaluates each in order, and returns the result of the first statement after the last has been evaluated. This is can be expressed as a Gambit macro, but I'll call it begin1 to be more like Scheme's begin than the old Lisp's progn.

? (define-macro (begin1 first-statement . remaining-statements)
    (let ((result-var (gensym 'first-result)))
      `(let ((,result-var ,first-statement))
         , at remaining-statements
         ,result-var)))
? (begin1 1 2 3)
1
> (begin 1 2 3)
3
? (define result-var 5)
? (begin 1 2 result-var)
5
? (begin result-var 1 2 3)
3
? (begin1 result-var 1 2 3)
5
? (begin1 0 (display result-var) (newline) 1 2 3)
5
0

The begin1 is an example of control abstraction.

An example of the latter use of pretty syntax... consider a scenario where you are using a resource and you want some "before" and "after" actions. The base level way to implement this is with higher-order procedures. But who wants to write (lambda () ...) all the time? So on top of this build some pretty syntax. [Note that a better lambda notation like Smalltalk's block syntax would reduce the need for these situations. (pdf)] For example...

(define (call-when-ready procedure)
  (wait-until-ready time-out)
  (if (not (ready?))
      (call-when-ready procedure)))

Use it like this...

(call-when-ready 
  (lambda () 
    (display "I am glad this is finally ready!")
    (newline)
    (do-something)))

This is fine when someone else is generating the code for you. Normally you might want to abstract the procedure as a sequence of statements...

(when-ready
  (display "I am glad this is finally ready!")
  (newline)
  (do-something))

And so when-ready is defined as a macro that calls call-when-ready...

(define-macro (when-ready . body)
  `(call-when-ready
     (lambda ()
       ,@body)))

Will Farr added a neat example in the same thread...

Making little sub-languages for specialized processing e.g.

(with-vectors (v1 v2 v3) (v1 <- (+ v2 v3)))

for summing up the vectors v2 and v3 and storing it in v1. (I'm not going to put this macro up because it's long---the code is buried in this post: http://wmfarr.blogspot.com/2005/06/bigloo-macros-and-repl.html .)

[This] involves changing the evaluation rules for a piece of code (the vector assignment is evaluated once for each index with the corresponding variables bound to elements of a vector).Procedures would not work for either of them; you have to have a macro.

Because

Because the world is round...

Identities, Associations, and Behaviors

Doc Searls has the lead article in the September Linux Journal on "Identity". This is yet another topic I have barely understood. My interest is growing though with the concept of "user centric identity".

The title of Searl's article is "Independent Identity" which points out an irony of sorts that's been on my mind. My impression is that this is not about "identity" at all. Rather it is about "association".

What use is an independent identity? I have to associate my identity with some other body's identity to accomplish anything. The crux of the "user centric identity" effort is how to limit the ramifications of that association.

The other thought that's been brewing is the relationship between capabilities and identity. We know our current approach to permissions is not sufficient to limit or even audit the results of association. On the other hand if I can reliably limit the available behaviors of my associates, and they mine, then we can mutually benefit from our association.

Identity without behavior does not lead to associations. Since we are after beneficial associations, I'm assuming we need to focus on how "identity management" ultimately translates into appropriate behaviors and reliably excludes inappropriate behaviors. A capability system requires identity management to communicate with the outside word. I also assume an identity management system requires capabilities or something equally secure to ultimately implement trustworthy associations.

I'm looking forward to Phil Windley's new book on Digital Identity to get a better handle on some of these ideas.

DZ

and now, daily zen...

By sitting alone all day long
I clear my mind of a thousand thoughts.
To speak of this is beyond our words;
Only by sitting under the quiet forest
Can we ever understand.

- Fa Yen (885-958)