"I have a mind like a steel... uh... thingy." Patrick Logan's weblog.

Search This Blog

Loading...

Sunday, May 18, 2008

More Ted

This is part two of my response to Ted Neward's latest misunderstanding of Erlang. Ted wrote to Steve Vinoski...

Sorry, Steve-O, but I think you're out in left field on this one. I'm happy to argue it further with you over beer, but if you want the last word, have at it, and we'll compare scores when we run into each other at the next conference.
Everything I've read of Steve's tells me he's well informed by actual experience. Ted's writing exhibits the most basic misunderstandings. I wonder why Ted didn't spend a little beer time with some experts before attempting to write with supposed authority.

More from Ted...

any time we incorporate something directly as part of the language, there's all kinds of versioning and revision issues that come with it. This, to my mind, is one of Scala's (and F#'s and Lisp's and Clojure's and Scheme's and other composite languages') greatest strengths, the ability to create constructs that look like they're part of the language, but in fact come from libraries.
Most of Erlang's power comes from libraries, but Ted wouldn't know that. Erlang the language is very small and expressive, but Ted wouldn't know that. Erlang's not really suffered from revision issues over the nine years I've been following the language, but Ted wouldn't know that. And Ted apparently didn't bother to ask anyone or read anything that might inform himself.

I agree with Ted's observation that extensible language syntax is desirable. Erlang's extensibility is cumbersome and not-too-beautiful. That's not the goal of Erlang though.

So you have to decide whether the benefits of Erlang outweigh issues like this. By the way, Gambit Scheme has Erlang-like concurrency performance, although shared-everything. But since Scheme can be used to define new languages such as Termite, it can be used to create syntactically-extensible Erlang-ish systems.

If you decide to go with Erlang the decision should be based on what Erlang is now, and not expect Erlang to change much. It's been largely stable as a language for years, as opposed to the arms race in Java and Java-like worlds. Language extensibility should not be as important to you as the simple Erlang programming model. This is a reasonable approach to take. Erlang is a known quantity, well proven. Not every language has to be as extensible as Lisp, in fact most are not.

Beware of languages like Scala and Clojure that intend to be safer concurrently than Java - they still run on the JVM and integration with Java code. So by definition their safety only can go so far.

I do criticize Erlang for having its own VM (though I think it's not a VM, it's an interpreter, which is a far cry from an actual VM)
Maybe we should criticize Java for having its own JVM? Why doesn't it use Smalltalk's or Erlang's which pre-date the JVM? (Trick question. The JVM, and Hot Spot in particular, is based on ideas developed in Smalltalk, etc. Essentially the JVM is a dynamic language VM plus layers of cruft.)

Not much digging would have told Ted that Erlang compiles to native code or to an intermediate compiled code. Ted's misleading claims are heading into the dangerously ignorant. Why should I believe anything Ted writes about any topic? When Ted writes "I think that..." I think that we should read it as, "I'm guessing that..."

The JVM and the CLR have (literally) thousands of man-months sunk into them to reach high levels in all those areas. Can Erlang claim the same?
Well, yeah. I don't know how many "man months" it stacks up to be, but Erlang has been around for decades in some very demanding situations. Did Ted bother to look into any of these VM questions?
Best part is, the IT department doesn't have to do anything different to their existing Java-based network topology to start taking advantage of this. Can you say the same for Erlang?
If the architecture is going to change from "big, honkin' J2ee behemoths" to highly concurrent, lightweight interaction, then the topology should change. This is true whether moving from J2EE to Erlang or J2EE to JRuby/Rails or J2EE to Project Zero or J2EE to anything reasonably resembling modern best practices.

Moving to Erlang can take advantage of existing Java bits using the various network protocol standards or CORBA or SOAP or the JInterface class library which makes a Java application act like an Erlang node.

Frankly, whether the application you're monitoring hooks into the monitoring infrastructure is not really part of the argument, since Erlang doesn't offer that, either. I'm more concerned with whether the infrastructure is monitoring-friendly... if Erlang ties into SNMP out of the box with no work required by the programmer, please tell me where that's doc'ed and how it works!
Apparently Ted's world of monitoring begins and ends with Java and dotnet's JMX and WMI. How would you monitor both Java and dotnet? SNMP likely. Both JMX and WMI can be hooked into SNMP. SNMP is the internet standard, more mature, and extends beyond Java and dotnet.

I've never used it, but just doing a quick check shows that Erlang has a pretty extensive SNMP api and Reference Manual, and User Guide. Hmm. Did Ted even bother to look?

Yes, the JVM could easily adopt the multi-process model if it chose to. (Said work is being done via the Java Isolates JSR.) The CLR already does (via AppDomains).
I addressed Erlang vs. AppDomains in my previous post. These are clearly two different mechanisms with two different intents: AppDomains provide application isolation; Erlang processes provide finer-grained isolation of concurrent activities ("actor"-like) within an application.

Within an application neither the JVM nor the JVM-ish CLR provide Erlang-like processes. Scala and Clojure go a good bit of the way, but the advantage of Erlang is in its simplicity and combination of concurrency programming _and_ management. This goes well beyond a little bit of syntax around Java threads within a single JVM.

Barry Kelly has a really good blog post explaining more about these various VMs and Scala's Lift framework and implementation of Actors.

9 comments:

Steve Jenson said...

Scala actors don't have to be mapped directly onto Java threads, they can be mapped onto event loops (that run in Java threads) so you can have hundreds of thousands of Scala Actors running at the same time.

Patrick Logan said...

"you can have hundreds of thousands of Scala Actors running at the same time"

According to the information Barry Kelly blogged about, Scala may support that many actors, but not nearly so many connections.

Steve Jenson said...

You can definitely handle lots of connections with Scala just like you can with Erlang (and Java honestly) but you can't tie a Java thread directly to a connection just like you can't tie a pthread or unix process to a socket and expect to handle significant load. It'd be great if Java threads could be easily suspended and resumed but the Scala community moved on and came up with a nice workaround in Event-based Actors, I think, and the Java networking geeks have come up with some nice tools like Mina and Grizzly built around NIO.

I talked to David Pollak (he's my neighbor) about how his load testing for Buy-a-Feature went and he said things were humming along and that he stopped at 2k because that's the load he was expecting and he needed to move on to other coding tasks for launch.

Nat said...

On Linux you can have hundreds of thousands of Java native threads running at the same time without problem.

In fact, that's the biggest problem with Java's threading and I/O model. The performance trade-offs differ between platforms. On Windows it's faster to use NIO and async I/O. On Linux its faster to use native threads.

The language should have defined a high-level threading and I/O model and the VM should have mapped that down to async I/O or native threads as appropriate for the platform.

Rich Hickey said...

"Beware of languages like Scala and Clojure that intend to be safer concurrently than Java - they still run on the JVM and integration with Java code. So by definition their safety only can go so far."

Should people also beware of Erlang because it can have deadlocks?

All those 9s don't simply fall out of using Erlang - they reflect best practices by experts. Clojure, when used as recommended, with the immutable data structures and concurrency primitives provided (which, BTW, are deadlock-free), is a safe as Erlang.

"This goes well beyond a little bit of syntax around Java threads within a single JVM."

The above is a slight - Clojure's agents and STM are more than a bit of syntax around Java threads.

Steve Vinoski said...

@Rich Hickey: so you're saying that Clojure is a language that completely prevents programmers from writing incorrect concurrent applications? Sign me up!

Ah, back to reality. Did you even read the paper you referenced? The paper does not prove, as you're trying to imply, that Erlang primitives can deadlock. Rather, the paper presents possible solutions to helping with the fact that any concurrent message-passing system can deadlock when multiple threads/actors/processes are programmed incorrectly such that they're all waiting on each other, or if one dies but the rest wait for it as if it's still there, again due to programmer error. I'd really love to hear how Clojure prevents such problems, as you seem to claim.

Rich Hickey said...

@Steve - You missed my point - we agree. Systems have to be used correctly in order to behave as advertised. I felt Patrick was holding Scala and Clojure to a different standard (condemned by their mere association with Java) and wanted to point up how even Erlang too can go awry when "programmed incorrectly" as you say. I asked a rhetorical question to which I thought the obvious answer was - "No, the ultimate responsibility for a correct system lies with the developer". Neither Erlang nor Clojure nor Scala should be avoided just because they are capable of misuse.

I didn't say, nor mean to imply, anything at all about the integrity of Erlang's primitives, that's why I linked to the paper (which I have read) rather than misstate something. Personally, I think Erlang is terrific and inspiring, and wish Scala and Clojure hadn't been dragged into this unproductive dialogue. Like you with Erlang, I hate to see them misrepresented. I fail to see why Erlang, Clojure and Scala can't peacefully coexist, providing solutions at different points along the various reliability, ease-of-use, performance, scalability, type-safety, extensibility, applicablity, complexity etc spectra.

Steve Vinoski said...

@Rick: my apologies for misinterpreting you. I mistakenly took your statement about Erlang and deadlocks together with your comment about deadlock-free Clojure primitives as a statement that Erlang suffers from deadlocks while Clojure doesn't. It was my error, and thanks for correcting me on it.

I can understand why you'd rather that Clojure and others were not dragged into this debate, but at the same time, I can understand why Patrick chose to do so: there is definitely a crowd out there who feel that if it's not on the JVM, it's not real software, and it's just not worth looking at. The reality I'd prefer to see, though, is just what you expressed in your final sentence above, and I'm pretty sure Patrick agrees with that as well.

Rich Hickey said...

@Steve - No problem. I agree there are biases all around, and it is a fair amount of work just to correct misconceptions.

Towards that end, I meant what I said - Clojure's STM and agent mechanisms are deadlock-free. They are not message-passing systems with blocking selective receive. The STM uses locks internally, but does lock-conflict detection and resolution automatically, and the agent system performs atomic, non-nestable actions with no in-action ability to wait on another agent. To Patrick's point, their proper operation does require avoiding certain Java facilities while in a transaction/action. For instance you must avoid side-effects during transactions and there is no way for Clojure to enforce that.

Blog Archive

About Me

Portland, Oregon, United States
I'm usually writing from my favorite location on the planet, the pacific northwest of the u.s. I write for myself only and unless otherwise specified my posts here should not be taken as representing an official position of my employer. Contact me at my gee mail account, username patrickdlogan.