"I have a mind like a steel... uh... thingy." Patrick Logan's weblog.

Search This Blog


Friday, September 29, 2006

I'm An Isolationist - You Probably Are Too

I rec'd an email about my recent thing to do with the "shared nothing" vs. "shared memory" models, JRuby, et al. being crammed into the JVM, CLR, Parrot, and so on.

The point of the email is well taken. Essentially JRuby is a good thing because the JVM is widely adopted and accepted in enterprises. This is the vehicle that will allow JRuby to be widely adopted. I agree with that approach, and do think JRuby, Jython, IronPython, and related languages are all very good things to have.

I'd rather have a widely adopted shared-nothing environment. That is the model of the future. The shared-memory model *is* the current model but there is plenty of evidence we've been done with it for some time. Another five years or so will be necessary for this to really sink in.

But we have JRuby now. Yeah, I'll program in JRuby today. At the same time I will push for something better, and expect to get it about 2010 or so.

Let's be clear that the JVM has been lurching toward a shared-nothing model already for well over five years.

The servlet model, the EJB model, the Jini model... these are all in various ways attempting to provide semi-shared-nothing models so we can each contribute our own isolated parts to the whole. Unfortunately they're each anachronistic, one-off, partial solutions. JSR 121 is attempting to make this even more general for all kinds of pojo scenarios.

Given the isolation model that just comes for free with Erlang, the vast majority of the complexity and variation of these lurches toward isolation for Java would just go away. (Erlang itself has a *little* cruft but that's nitpicking.)

I've been through the garbage collection wars, the dynamic typing wars. I think we'll get better isolation more easily but so far it remains a subtle point in the community at large.

Iceman: "You like to work alone. I've heard that about you."

Update: A Fellow traveller writes...

Yes, you can layer a shared nothing architecture on top of a language that does not hand-hold you towards a shared nothing design, but the scope for error is huge. Remember the days of malloc anyone? Same idea.
I don't think it is quite that drastic. Yes, I lean toward semi-functional languages like Scheme and Erlang, mostly Scheme because you can more easily write all kinds of languages in Scheme. Gambit Scheme is a good candidate platform for building mult-language shared-nothing runtimes.

But even "heavily-imperative" languages like Smalltalk, Ruby, and Python can take much better advantage of a shared-nothing runtime model. These implementations just need to be able to run more than one environment within an OS process. Kind of like doing a "fork" that forks an isolated process within the current OS process. Now you have two or more Smalltalks, Rubies, Pythons, Javas, etc. running in that one OS process, isolated one from another.

When these are truly isolated and efficient then nothing should prevent an OS process from mixing these languages "sub-processes" within the same OS process. These are not new ideas, but have never really been fully explored. Most of the bits and pieces are in place. It's just around the corner.


Ian Bicking said...

I think this pragmatic approach might be an important part of Sugar -- we set up the archetype of a Python process (including importing core modules) when the machine is started up, and fork off from that to start new processes. Processes aren't as light as Erlang, but it does make processes somewhat light (and that's very important in the environment Sugar is intended for). Then we use DBus to do IPC from there.

It's nothing all that new, but it will be interesting to see how far we can take it.

Michael Sparks said...

I recently took a look at Erlang and discovered their model is almost identical to Kamaelia's. However, Kamaelia's model is based essentially on asynchronous hardware, which limits you to a default of shared nothing. It's interesting to see the convergence however.

However, Kamaelia is implemented as python generators (similarity to erlang microthreads), pushed into a class to give them named inboxes and outboxes (which I viewed as equivalent to stdin/out/err, but are equivalent to erlang mailboxes), and we run these using a scheduler.

However our target is something understandable by the average user (rather than force people to change language if they don't want to), and to simplify maintenance of concurrent systems. This has the side effect of making it easier to create these systems in many cases. (regarding forcing people to use a language/not, we've got a basic translation into C++ as well, and the ideas should apply in ruby happily as well)

One of our summer of code students has written a short article introducing our latest release, which has some screenshots from the visual builder we're able to have for free as well.

Our communications method and process model is significantly lighter weight than Dbus, and we can happily run thousands of these microprocesses. We're also able to have threaded components and easily shell out to the unix environment, and using python makes it easy to reuse existing libraries meaning we now have direct support for OpenGL, Pygame, Speex, Vorbis, Dirac, and a few other interesting bits and bobs.

We're also building useful systems using it such as Kamaelia Macro - which is a system designed for recording all TV broadcast by the BBC, transcoding and making it available. We use this as a record of transmission, but in the UK it's fine as a method for timeshifting as well.

(You'll see from the Macro page that we've extended the normal unix pipeline metaphor to the concept of a graphline as well)

As an idea of what some kamaelia systems look like, the best place to start is in the cookbook, and the fundamentals behind it (though massively simlified) we use as a tutorial to help people to understand the core.

The fun thing is we also tend to be able to use any component with any other (though sometimes you need to write an adapator of course), meaning we can do fun things like play video in 3D (screenshot), and spin it round in a handful of lines.

Based on your comments, I'd encourage you to take a look at Kamaelia - we've been working on this for some time now, and rather than being at the "this may or may not have legs" stage, we're very much at the "we're finding this a very useful stage".

Isaac Gouy said...

Michael Sparks wrote However our target is something understandable by the average user (rather than force people to change language if they don't want to) ...

"Finally, we have seen that programmers – even those with no previous exposure to functional languages – learn Erlang quickly, and become productive within a matter of weeks."
Four-fold Increase in Productivity and Quality p24

Blog Archive

About Me

Portland, Oregon, United States
I'm usually writing from my favorite location on the planet, the pacific northwest of the u.s. I write for myself only and unless otherwise specified my posts here should not be taken as representing an official position of my employer. Contact me at my gee mail account, username patrickdlogan.