On Pattern Matching

An aspect of Erlang that is often mentioned but seldom highlighted, and fairly unrecognized for its contribution is pattern matching. Languages implementors that wish to achieve Erlang-like concurrency in their own language's runtime will also have to come to terms with Erlang's simple message selection mechanism based on pattern matching.

Again this is where an implementation like Termite, based on Lisp (Scheme) can work as well as Erlang. Other languages not so much, but some creative use of syntax and meta-programming in Smalltalk, Python, and Ruby might do the job.

Other languages not so much. Especially those modern curly braced ones, which will likely have pattern matching piled-on in their next set of features, this time to seem more Erlang-like. Either Java or C# will add this first, then the other language implementors will be compelled to meet or exceed the first.

Either way, they've all got a long way to go. I'd bet on several Lisps and on Smalltalk to keep up most easily.

On Sequential Languages and Concurrency

Ralph Johnson wrote about Erlang this week, taking multiple angles worth reading, "Erlang, the next Java". One of them being how difficult it may be to update other sequential languages to Erlang's level of support for concurrency and distribution. Another being the relative importance (or not) of Erlang's sequential constructs being mostly functional.

Joe makes too much of functional programming because he says that lack of mutable state implies no locks. However, it is really lack of SHARED state that implies no locks. You could write processes in Basic, perl, or C. I'm sure that lots of people will look at Erlang and say "we can add that to our language". In my opinion, it is the concurrent programming aspects of Erlang that make it special, along with its mature implementation and powerful library designed for concurrency and reliability.

I do not believe that other languages can catch up with Erlang anytime soon. It will be easy for them to add language features to be like Erlang. It will take a long time for them to build such a high-quality VM and the mature libraries for concurrency and reliability. So, Erlang is poised for success. If you want to build a multicore application in the next few years, you should look at Erlang.

The best candidate runtime in my opinion is Gambit Scheme. Gambit's already been used to achieve Erlang-like levels of concurrent threads, it's been used to implement an Erlang compiler, and it's been used to implement Termite a mostly-functional Scheme-like language with many Erlang-like concurrency features. (A point I've written about several times.)

At one point I wrote "parsers" for very small subset of Ruby and of Javascript. So small I need to use quotes. The parse trees were simply lists in Scheme ("s-expressions") and then some simple macros and functions to turn those trees into executable Scheme s-expressions. Since Gambit generates native code by way of compiling to C and it's runtime is compiled efficiently to native code, this approach essentially results in, for example, a Ruby-to-Gambit-to-C-to-Native code compiler.

I am convinced this approach would work well for "real" development, allowing the various language translators to run interpreted and compiled, just as Gambit itself is. Moreover, the translators could support sequential, shared-nothing implementations of various languages, relying on Gambit's threads and mailboxes to implement concurrent Erlang-like processes with asynchronous message passing among multiple languages running in the same Gambit OS process.

But then there's the rub. Ralph writes, "it is the concurrent programming aspects of Erlang that make it special".

This does make it special, but it's the simple, mostly-functional sequential aspects of Erlang that make it work so well. As soon as the sequential language is one with assignment and mutable memory, several unnecessary "issues" have to be dealt with. Even though the language is defining a shared-nothing, single-threaded process, these new issues overtake the simplicity of the concurrent message passing.

For example, what about "identity"? What about mutable lists, not to mention mutable, large object graphs? Erlang can play some tricks under the covers when "passing" immutable data among lightweight processes in the same OS process, potentially even in shared-memory among nodes on the same "physical" memory.

So a mutable sequential language gums up the concepts for the application programmer and gums up the mechanisms for the systems programmer. Javaspaces handles this fairly well for Java by keeping the mechanisms simple, yet the rules and the mechanisms (and copying) are significantly more complex and in the way of application developers. (Also pattern matching in Java is just cumbersome and not so nice to look at.)

These more "traditional" imperative languages (with or without "objects") were not designed for Erlang-like concurrency and are simply not as good a fit. Termite recognizes this and so provides a simpler, more-functional Scheme than Scheme R5RS.

Joe Armstrong writes in his HOPL III presentation (pdf):

Robert and I were convinced that one day we would have to add destructive operations to the language, but that we would wait until a problem that could not be solved with pure operations turned up.

This never happened.

Erlang is special because its combination of sequential and concurrent features combine so well. Most languages have piled sequential feature on sequential feature. As Ralph Johnson notes:

Unlike Java or Smalltalk, where you only write threads/processes when you want concurrency, Erlang programmers use processes for modularity, reliability, and reuse. Then they get concurrency for free.

San Francisco "Erlang Lounge" at Thoughtworks Office

Ryan Tecco posted the following to the erlang mail list...

The San Francisco Bay Area ErlLounge will be held on Wednesday, August 15th at 7:00 at the Thoughtworks office (410 Townsend St.) in downtown San Francisco. Thoughtworks is less than block away from the 4th and King Caltrain station for those who might be coming up from the South Bay.

Pizza and beer will be provided as well as wi-fi and whiteboards, so bring your laptop if you are so inclined.

RSVP with me off-list so that I can get a rough head count.

Cool.

Erlang at History of Programming Languages III

Here is the PDF from Joe Armstrong's talk (n.b. it's a PDF) at the recent ACM History of Programming Languages III event.

What is Erlang?

• Concurrent (processes belong to Language – NOT OS)
• Very light-weight concurrency (lighter than threads)
• “Share nothing” process semantics
• Pure asynchronous message passing
• Core language is a simple dynamically typed FPL
• Non-pure extensions (ets) for implementing databases.
• Mechanisms for in-service code upgrade
• Large set of libraries (OTP)(Unix <-> C <==> OTP <-> Erlang)

What Rails (and the Web) Did

Expanding on a reply I just wrote to a comment on last night's Erlang post...

What Ruby and especially Rails did, though, is break IT free of the curly braces sociology. Used to be languages that did not resemble C would get skewed looks. When they'd receive attention they would be cast as odd-ball curiosities, far from viable in the mainstream.

Just think of all those Lisp and Smalltalk pieces in Dr. Dobbs and Byte over the years.

More generally than Ruby and Rails, I guess, is that the *web* did this. The web, being a large, dynamic system, demanded more than the traditional mainstream could supply, having been designed primarily for small, single-user systems that don't actually do all that much.

Categories and Systems

Probably early humans benefited more from thinking in terms of categories than in terms of systems. "Is that food, or not?" and "Is that my friend, or not?" being two important categorizations.

And so we still seem to define categories and compartmentalize more than we make connections and think about systems, let alone "systems of systems".

The "interesting" events in the financial world are an example of categorizing more than systematizing. Many institutions have been playing the game of "subprime" lending for years. The way these games are presented is in terms of "markets" where a market is treated more as a category than as what it really is: a system. And the relationships *between* markets is repeatedly overlooked. The Japanese financial crisis of a decade ago or so hit Portland, Oregon somewhat hard, but only after the local business analysts declared the result would be otherwise because Japan is "over there" and Portland is "over here".

For quite a while now the headlines have been presenting the problems with subprime loans and defaults on them. The subprime market (category) was in trouble. "No problem. So what if some low-wage earners can't pay their debts and some lower-tier lenders take a bath? I'm not in that category."

Recently the problems secondary effects on other markets have been presented as problems of "confidence". That's probably true, since all markets are ultimately based on confidence. But that's not the real problem, since all markets are ultimately related, and in many cases far more closely related than the institutions would want investors to believe.

"Our current system of levered finance and its related structures may be critically flawed..." Uh-huh. From the International Herald Tribune...

The result has been a freezing up of markets for many securities that, it turns out, were critical to the free flowing of credit in recent years.

"Our current system of levered finance and its related structures may be critically flawed," said Bill Gross, the chief investment officer of Pimco, a mutual fund company. "Nothing within it allows for the hedging of liquidity risk, and that is the problem at the moment."

The basis of the system has been a belief that securities backed by bad credits could be very safe - so long as there were other securities that would suffer the first losses that came from defaults in pools of subprime mortgages, or of loans to highly leveraged companies.

Even the people playing the system are finding out they've been playing a misconception of the true system. The true crisis of confidence is now about finding out what the true system is, and how it works, in order to find some solid ground for building up confidence. No confidence: no credit. No credit: no business.

One of the bases of confidence has been the credit rating system. That turns out to be based on a system that does not truly exist. Or at least not in the form used to establish the bases.

"The complete evaporation of liquidity in certain market segments of the U.S. securitization market has made it impossible to value certain assets fairly, regardless of their quality or credit rating," the French bank said.

Added to the problem is that the questionable securities now are widely owned, and sometimes have been repackaged to form the basis of other securities.

Gross compared the problem confronting investors to a game of " 'Where's Waldo' - Waldo being the bad loans and defaulting subprime paper."

Suddenly, some fear Waldo is everywhere. European banks and funds own paper tied to subprime mortgages, and it is not clear who else does, or how investors will react.

"You have to believe that in the hedge fund and mutual fund complexes, there is a decision that is building that says, 'I want to hold some Treasuries to have a cushion if I see redemptions,' " said Robert Barbera, chief economist of ITG.

We need better tools and more awareness of the systems we live in. This crisis is a problem due to lack of systems thinking. We've enjoyed the credit ride fueling the economy for the last seven years or so, flying high, smooth sailing.

Meanwhile, hold on. Monday could get a bit bumpy, and there could be a fair bit of turbulence ahead during the search for solid ground. And we may be running out of gas.

If the current panic is just that - unreasoning fear - then... cash infusions may be able to let the new financial system weather the storm. Money can be lent to those owning the dubious securities, obviating the need for them to sell them. As they eventually turn out to be good, the loans can be repaid and all will be happy.

On the other hand, if many of those securities turn out to be as bad as people now fear, some of those loans will not be good, and there may be more financial failures.

But the central banks still confront significant challenges. The new financial system, with its credit expansion through securitization, made it possible for questionable borrowers to keep borrowing at very low rates even as the Fed was tightening.

Now, the combination of market reaction and new regulations on mortgage lending have drastically tightened credit on many borrowers...

The new financial system is not the one the Fed was created to deal with, but it is the one it must try to handle.

Dive Into Erlang?!

Wow. Are things hopping in the world of Erlang, or what? Several high-profile people have blogged about recent explorations. Also just strolling through del.icio.us turns up all kinds of interesting bits. Like this from Dive Into Erlang ("A Rubyist dabbles in today's most exciting programming language"!)...

I felt naturally comfortable with the Erlang way of doing things. This is how I've been writing programs all along. It's feels so nice to finally find a language that abstracts away all the complex behaviors of concurrent network processing. Furthermore, distributed Erlang and the OTP framework are beyond my wildest dreams in terms of what you can build once the underlying problems are all abstracted away.

Now I can finally stop trying to build an ad hoc, informally-specified, bug-ridden framework, and start focusing on what I've actually cared about all along: the function.

I would qualify that "underlying problems are all abstracted away" by pointing out the problems are still there, and you still have to deal with them. But OTP provides convenient ways to deal with them more easily, and the Erlang literature explains the problems and the solutions well.

But what's happening? In the last six months or so HTTP overthrows WS-Deathstar far more rapidly than at least I imagined and now Erlang is basking in far more sunshine than I imagined would appear over the next several years.