"I have a mind like a steel... uh... thingy." Patrick Logan's weblog.

Search This Blog

Monday, October 22, 2007

1958 / 1964 - So What

Update: shmul recommends in the comments Bill Evan's "Sunday at the Village Vanguard". Oh. My. Yeah, that is a fantastic recording. It's probably another heavy link to Evans' addictions. Evans was always in search of his ideal bassist, and found Scott LaFaro. They developed their sound together, just as Evans did with Davis. A few days after these recordings, LaFaro was killed in a car accident. Evans was devastated.

Previously: Now with the correct links.

Compare a 1958 recording of So What (via Steve Dekorte) with a 1964 recording. What happened? Free jazz. And the Beatles.

Coltrane was on the edge of 1964 already in 1958. Miles Davis' sound for "Kind of Blue" was settling in. They could not have recorded it earlier or later than 1959. There's nothing else like it.

It could not have been recorded without Davis of course. Also indispensable: Bill Evans and Coltrane. Probably indispensable: Cannonball Adderley for the way he and Coltrane interact.

Bill Evans was still playing his sound he found around 1959, in 1979, just about dead from addictions that began in Mile's group, playing under incredible pressure from black audiences pissed off by a white pianist in the group. Davis was an icon in the black community in the 1950s. Incredibly successful and refusing to take any shit from powerful white men. By standing up for Evans, Davis showed he'd take no shit from black fans either: Davis was in awe of Evans' sound and made that sound the centerpiece of Kind of Blue even though the piano itself is not a centerpiece. The two of them came together perfectly influencing each other's sound.

(Aside: unlike Evans, Davis was able to hole himself up at his parent's place and kick his habit before this time. Evans would hole himself up at his family home subsequently, but never could kick all of his addictions. He slowly eroded his body's chance of ever recovering over 20 years. The pain from earlier damage fueled more drug use. Read his biography -- not as dramatic as other addicted musicians -- just incredibly slow, painful erosion.)

Anyway, Evans played on every piece on Kind of Blue except Freddie Freeloader. Listen to his notes on those pieces... unbelievably the right note in the right place at the right time. But Kind of Blue would be a snoozer if the only sounds were Davis' and Evans'. The saxophones and everything else keep the whole thing from falling over from inertia.

Kind of Blue is still my favorite album from any genre, any era. I bought my first copy as a teenager about 1977. I don't know how many hours I've listened to it, but it is a lot.

QCon SFO:

Fuzzy and I are all signed up for QCon in San Francisco early November. In particular we'll be at the track: "Connecting SOA and the Web: How much REST do we need?".

What a great line-up of people to talk with. Just in time with the work we're doing. Ed couldn't make it, something about a class for the baby on the way that Thursday.

Saturday, October 20, 2007

Programming with Streams

Michael Lucas-Smith illustrates programming with streams and a helpful "functional" protocol added by the ComputingStreams package from Cincom Smalltalk's public repository. Michael makes the case that more code should be based on streams instead of collections, and use a common protocol like ComputingStreams instead of custom, one-off methods that won't play well with others. Streams are a better abstraction than collections because they inherently "delay" and scale up to and including infinitely long streams.

Streams are interesting creatures, fundamental to programming if you've got them available. See Structure and Interpretation of Computer Programming: "Streams Are Delayed Lists" and Richard Waters' Series package for Common Lisp for more examples.

Also this Smalltalk package is similar to Avi Bryant's Relational Object Expressions. ROE uses objects using the standard Smalltalk collections protocol to delay "realizing" complete, in-memory collections, enabling the original collections to be enormous numbers of rows in tables on disk. In the case of ROE the objects are delaying the eventual composition and execution of SQL, including the conditional expressions that may have narrowed the size of the selection.

Tuesday, October 16, 2007

Closures and Objects

I am not sure what Steve Dekorte is saying...

A language that uses objects on the bottom can use them for everything, but a language with closures on the bottom needs other types or "atoms" for things like numbers, lists, etc and then scatter around functions for operating on those other types. If you care about simplicity, consistency and organization, this is a big difference.
If you have objects, you still need compiler support for closures.

But if you have closures, you need no compiler support for objects.

Either way you need compiler support for other literals like numbers.

Unless you want to program numbers in the lambda calculus. And then too closures win.

Your name is different, but you're really a Unix system too, aren't you?

Microsoft's Unix from back in the day, Xenix...

Sunday, October 14, 2007

Simplified Javascript: Cruft Reduced

A bunch of comments just showed up, June 21, 2008.

Like it or not (and FWIW, I like it quite a bit), JSON has made an impression on networked information exchange. Douglas Crockford, who started the whole JSON thing, has a chapter in Beautiful Code on Pratt-style parsing. His example parser is written in, and happens to parse, something he calls "simplified javascript".

Simplified JavaScript is just the good stuff, including:
  • Functions as first class objects. Functions in Simplified JavaScript are lambdas with lexical scoping.
  • Dynamic objects with prototypal inheritance. Objects are class-free. We can add a new member to any object by ordinary assignment. An object can inherit members from another object.
  • Object literals and array literals. This is a very convenient notation for creating new objects and arrays. JavaScript literals were the inspiration for the JSON data interchange format.
Implementing some of the currently popular "scripting" languages, in particular "full" Javascript, Python, and Ruby will make evident the amount of cruft they have for no apparent reason. Lisp traditionally has had some cruft, but, arguably, nothing like what can be found in these languages. Such cruft makes these languages significantly more difficult to implement than implementing Lisp or Smalltalk (two amazingly "cruft-reduced" languages given their age).

Simplified Javascript would make a decent base for a scripting language. I pointed this out to a couple people looking at full Javascript over the last couple of months for an Erlang-based server and now for Croquet.

People already in the more or less "end user scripting" space tend to have some Javascript knowledge. This subset has good functional and OO capabilities, comes with a simple parser implementation already(!), and from implementing a bit of interpretation code already, seems amenable to a simple elisp/emacs-level of performance, which is more than sufficient for an interactive GUI system. Compiling to a native or byte code would not be too difficult for better performance.

Simplified javascript not only eliminates the uglier and more difficult to implement bits. It also eliminates a lot of bad security problems.

And so on designing a new scripting capability, combining Simplified Javascript with a capability-based approach to authority as with the E programming language could have some benefits. And JSON for "data"... go ahead and use "eval" to implement the JSON parser in such an environment.

Well, that would be a heck of a lot better than the current browser/javascript situation which is horrendous. Now we've got a truly undesirable "legacy" situation(s) in the browser. Apparently Mark Miller or one of those E/Capability folks is at Google now and their looking at something like the browser those folks built for DARPA. (Found in this powerpoint if you don't mind opening those up in OpenOffice.)

Innovation

From Business Week on improving what already works vs. finding new things that work better...

As once-bloated U.S. manufacturers have shaped up and become profitable global competitors, the onus shifts to growth and innovation, especially in today's idea-based, design-obsessed economy. While process excellence demands precision, consistency, and repetition, innovation calls for variation, failure, and serendipity.

Indeed, the very factors that make Six Sigma effective in one context can make it ineffective in another. Traditionally, it uses rigorous statistical analysis to produce unambiguous data that help produce better quality, lower costs, and more efficiency. That all sounds great when you know what outcomes you'd like to control. But what about when there are few facts to go on—or you don't even know the nature of the problem you're trying to define?

"New things look very bad on this scale," says MIT Sloan School of Management professor Eric von Hippel, who has worked with 3M on innovation projects that he says "took a backseat" once Six Sigma settled in. "The more you hardwire a company on total quality management, [the more] it is going to hurt breakthrough innovation," adds Vijay Govindarajan, a management professor at Dartmouth's Tuck School of Business. "The mindset that is needed, the capabilities that are needed, the metrics that are needed, the whole culture that is needed for discontinuous innovation, are fundamentally different."

Planned RESTful Services

Stu says stuff about reuse...

From this viewpoint, "build it and maybe people will use it later" is a bad thing. SOA proponents really dislike this approach, where one exposes thousands of services in hopes of serendipity -- because it never actually happens.

Yet, on the Web, we do this all the time. The Web architecture is all about serendipity, and letting a thousand information flowers bloom, regardless of whether it serves some greater, over arching, aligned need. We expose resources based on use, but the constraints on the architecture enables reuse without planning. Serendipity seems to result from good linking habits, stable URIs, a clear indication of the meaning of a particular resource, and good search algorithms to harvest & rank this meaning.

This difference is one major hurdle to overcome if we are to unify these two important schools of thought, and build better information systems out of it.

The early "mashups" were certainly serendipitous. I would expect most service providers on the web today are planning for machine-to-machine to a much greater extent than even a couple of years ago.

Is it really "serendipitous" when an organization follows those good web practices and enters formal-enough agreements with participants, say in an intra-enterprise-services situation? No, that seems very much "planned". The biggest hurdle may be the addition, perhaps, of a stronger demand to negotiate terms for support and upgrades.

This is necessary of any kind of service, networked or otherwise, that has significant risks to stakeholders. If you buy SAP systems and bring them in-house, or if you network out to Salesforce, you negotiate terms of service with in-house IT, with the vendors, with the ISPs, etc.

Saturday, October 13, 2007

Scalability Requirements

Stefan Tilkov and Dare Obasanjo debate the number of big sites that would benefit from giving up on the relational database as a centerpiece of their systems. The parameter used in the discussion is scalability.

A number of the lessons learned from these big web sites could apply to smaller data center designs as well. The biggest problem I've seen in the typical data center, and from comparing notes, seems fairly common, is the ability evolve components relatively independently.

Relational databases can be in the mix and still evolve, just as more supposedly "loosely coupled" mechanisms can actually tie components tightly together. For example some systems may use asynchronous messaging to decouple with respect to time, but then pass to each other data that exposes implementation details. And so these systems are coupled to each other with respect to maintenance and enhancements: changing the implementation of component C demands corresponding changes to every other component that consumes component C's implementation-specific messages.

But the lessons are worth following for more reasons than just scalability.

Content Unbecoming

Sam Ruby comments in his own blog...

I will merely point out that I’ve dealt with formats and protocols which are “explicitly not intended for human readability” and I’ve learned to avoid them.
And Bill de hÓra comments further down...
Atompub/Atom shines light on the fact we have serious issues around sharing and “understanding” structured content. Complaining about the source of light isn’t sensible.

So, to me, this is like the web services debates rehashed half a decade later. The only interesting difference in this thread is the level the argument is happening - around the content/payload/mediatype instead of wire/transfer...

Maybe it’s not as obviously random at this time to argue that per silo data access formats are a good idea...

Fwiw, “technically”, I don’t see why facebook don’t serve class laden XHTML and document the attributes.

And along the way James Snell compares to Yaron's example...
<entry xmlns="http://www.w3.org/2005/Atom">
  <id>tag:example.org,2005/someuser/profile</id>
  <title>Some User's Profile</title>
  <updated>2000-01-01T00:00:00Z</updated>

  <author><name>Profile System</name><author>
  <content type="xhtml">
    <div xmlns="http://www.w3.org/1999/xhtml">
      <div class="profile">
        <div class="section" id="professional">

          ...
        </div>
        <div class="section" id="personal">
          ...
        </div>
        <div class="section" id="clothingPreferences">
          ...
        </div>

      </div>
    </div>
  </content>
</entry>
Looks good. Damn. Even *this* xml-hater (yours truly) has had trouble realizing all that one-off stuff is bogus.

Don't Fear the Programmer

Not to beat a dead horse, but Robert Cooper adds an important point to the enterprise development tool conversation...

And here is where it breaks down. All these unusable drag and drop tools, and “easy” XML programming languages aren’t targeted at programmers. They are targeted to suits who can buy into the idea that some non-techy is going to orchestrate these services and modify business rules. These products are unworkable because they are based on the idea that “You won’t need programmers anymore!” at least at a core level. Once you make that assumption you start building things that get in programmers way, and still include enough abstract programming concepts that no non-programmer is ever going to be able to work with it proficiently
I've not heard this lately, but in previous situations this point was widely held: BPM tools should be used to get the programmer out of the loop. I always enjoyed bringing testing into the conversation. The current crop of BPM tools I've seen up close are terrible at incremental development, testing, and release. And yet at least some IT shops have seen them as *the* way to get from under long software development cycles.

Those shops I know of also happen to be historically lacking good automated testing and release practices. Most of their causes for long development cycles have almost *nothing* to do with programmers, save for the fact that most programmers historically have not paid much attention to good testing either. Of course that's changed dramatically over the last five years or so.

Managers: if you have long development cycles, backlogs of feature requests and bugs, and "legacy code" that seems impenetrable, then please embrace your programmers' desires to be more productive and effective. There are many things you can do today to make big improvements. Buying tools that will reduce the need for programmers is a pipe dream for now.

Friday, October 12, 2007

Sad IP

So let me get this straight -- Novell and Redhat are being sued over a patent some mangy IP shop bought that originated at Xerox PARC (I suppose Rooms?). And the patent is about making the same window show up in multiple workspaces?

This is the best IP some mangy techno-lawyer can pin on *Linux*?

Em, wow. Crawl back under your rock until you can make it *really* interesting... Next.

And note to Steve Ballmer: stop being such a wuss. You're really just appearing pretty sad these days. Can't you get a ride in space or something?

Monday, October 08, 2007

Friday, October 05, 2007

Bungie Cuts the Cord, So to Speak

Wow. This is huge. This speaks volumes between the lines and in the lines themselves. Just as Halo 3's $300MM USD is ballyhooed all over the press.

Just when there's all this buzz about how even enterprise applications are should become more game-like, and just as Microsoft must be getting desperate to show they can innovate, Bungie obviously takes off in order to be more innovative with game development than they can be as part of Microsoft.

Todd Bishop's Q&A is great. Maybe he could interview U.S. presidential candidates?

Say it Once Again

Pete Lacey: What is SOA?. Nicely done.

The definition of SOA is still an argument after all these years. This is a darn good sign that we should not care to use the term at all.

Equivocate: To avoid making an explicit statement. See Synonyms at "lie".

Mark Baker wisely points out...

There are, of course, a very large number of automated solutions to a particular problem.

...if you don’t feel disdain towards a lot (97%?) of these solutions, then you simply don’t understand what it means to be a software architect.

There is too much equivocation in our field. It's hard enough building software when you've got good tools at hand.
e·quiv·o·cal (-kwv-kl)
adj.
  1. Open to two or more interpretations and often intended to mislead; ambiguous. See Synonyms at ambiguous.
  2. Of uncertain significance.
  3. Of a doubtful or uncertain nature.

Thursday, October 04, 2007

Oh Dear, Part VII

Arnon Rotem-Gal-Oz assumes the position...

One thing that is missing from "HTTP variety of REST" implementation is reliable messaging. Location transparency is harder to solve with HTTP etc...

The solution should match the problem, that's probably the primary reason why we need architects after all.

Please Don't Pick on the Donkey

More Steve...

REST is unproven. Sigh. I can’t decide if people say this because they’re just trying to stir up an argument, or they’re so heavily biased toward their non-REST approaches that they just can’t even consider that there might be viable alternatives, or they really have no clue about what REST is, how it works, and why it works, and they’re not interested in learning it, so they just react badly whenever they hear it, or all of the above. If you’re anti-REST or REST-ignorant, and you haven’t read RESTful Web Services, then don’t even talk to me about REST. The book is absolutely wonderful, and its explanations and answers are extremely clear. If you consider yourself informed and capable when it comes to distributed systems and integration, but you don’t know REST, then there’s simply no way you can read that book and not have it lead you to seriously question your core beliefs regarding how you think such systems should be built, unless you’re completely close-minded of course...

What do mono-language programmers have to do with ESBs? It’s all part of the same culture, the “one size fits all” approach, where you have answers looking for problems rather than the other way around, and where people intentionally wear blinders to less expensive, more productive, and far more flexible and agile approaches because “it’s just not the way we do things around here.”...

At the end of the day, if you want to ignore my advice on using REST and dynamic languages, that’s your own problem.

And where his previous post received a comment, apparently from a Neudesic employee (ever heard of Neudesic???)...
Sonic Software, BEA, IBM, IONA, TIBCO, webMethods, Cape Clear, Fiorano, Oracle, Software AG, Neudesic, And how many open source projects????

All deliver ESB’s. It seems highly unlikely that all of these organizations, with plenty of smart folks, could be incorrect. Moreover, Gartner, Forester, ZapThink, and Burton, all speak loudly about ESB’s being an enabling technology on the road to building workable SOA’s.

Oh, dear. Steve responds best...
Making a list of smart companies to prove a point, well, doesn’t actually prove anything. I bet there are some actual smart people out there who have locked themselves out of their car, for example, or deleted whole directories they really wanted to keep but never backed up, or even paid actual money for a Zune. What does that prove?...

Unlike most of the commenters on this posting, I’ve personally built at least one of everything I’m talking about, on the ESB side and on the REST side, so I’m extremely confident in my opinion. I’d therefore much rather get specifics from you.

The View From Here Is... Grisly?

Joe Wilcox on MSFT Vista, speaking of big software the world is leaving behind...

The results, while arguably anecdotal, are grisly...

"Everyone—every single person—that I put on Vista has switched back to XP," he said. "It's too complicated."

From an administrator's perspective, my VAR buddy likes some Vista deployment tools, but he viciously complained about poor driver support, networking changes and end user complaints about UAC (User Account Control) and Internet Explorer 7 security popups.

In this ringing Vista endorsement, one Microsoft Watch commenter claimed: "I'm an IT consultant and I'm proud to announce I've formatted 450 Windows Vista machines back to Windows XP to date. I have also prevented at least 1,000 Windows Vista sales."

...the relevance of Windows is in decline. Microsoft's desktop operating system is rapidly becoming a commodity.

LAMP

Her royal pingdom lists...

OS: Linux 7 - Windows 2
Web server: Apache 7 - IIS 2 - Lighttpd 2
Scripting: PHP 4 - Perl 4 - ASP.NET 2 - Python 1 - Java 1
Database: MySQL 7 - SQL Server 1 (possibly 2)
And you should not be investing in some reasonable approximation to LAMP, again, because...?

Protected Game Preserves

That's how Bob Warfield refers to how Adobe, Microsoft, and SAP are approaching Software as a Service (SaaS). There's something different about Adobe relative to the other two.

Microsoft and SAP are figuring out how to transition gigantic cash cows into a world that is rapidly leaving them behind. Not so much for Adobe.

Adobe bought BuzzWord. They now have an early beta of an interesting word processor. They also have a few more apparently kickass flash/flex developers and creative software designers.

As opposed to MSFT's and SAP's situations, the fact that BuzzWord doesn't compete with other current Adobe products is probably an advantage. BuzzWord as a product may or may not ever matter.

Adobe's strengths in the new world appears to be (1) graphics, especially as they become more web-friendly, e.g. Thermo is hot.

And (2) their own decent runtime from Macromedia, something relatively easy to build upon compared to java or dotnet, from a company with decent developer relationships.

I think considering the company as a whole, Adobe has some good stuff to take into the new world. MSFT and SAP have bigger transitions to make, product-wise and company culture-wise. Not that I would bet much against them, because they are f'ing big and know how to sell software to vice presidents.

Properly Striking a Balance Between Shared Agreement and Decentralized Execution

Stu Charlton of BEA says stuff on the ESB brouhaha, but then he says some really crucial stuff we all need to consider...

I like ESBs if they're used properly -- to me, a good ESB is actually just a scripting language with supporting infrastructure. AquaLogic Service Bus, to me, is a Pipeline language, an XQuery language, and the supporting management & interoperability infrastructure, for example. But an ESB is not mandatory, and sometimes can slow you down if you're not looking to make legacy systems interoperate.
Wow. Equating Aqualogic with a scripting language. That's bold.
I agree with Steve Vinoski's The ESB Question. Don't use an ESB if it's going to slow you down. ESBs are a decent, productive tool for SOA (and even REST!) when applied properly, but so are Python or Ruby. ;-)
I want to like what Stu's saying, but his sprinkling of "properlies" just seems like equivocation.
BEA tries really hard to make Jython supported in WebLogic and AquaLogic, for example. The whole JMX object model is scriptable, and you can do some amazing things with it. And we have PHP running on WebLogic Server.
OK. I kicked the tires on Aqualogic over a year ago. I did not appreciate it any more than other ESBs, and found Jini/Javaspaces significantly more productive.

But I like the parts about Jython and PHP.

What gives? Perhaps two things:
  1. software has long term residual value but our investment decisions are always short term
  2. we intertwingle what changes from what doesn't change
We've fixed #2 with hardware platforms. SOA was supposed to fix it for software, but perhaps not (yet).
Yeah, across the industry we pretty much have #2 at the level of a high art.

As for SOA, well. It's well known...

SOA is the only thing Chuck Norris can't kill...

Its too bad you can't afford it.

Now we get to where I really like what Stu writes (and it has nothing to do with ESBs, unsurprisingly)...
I've seen many fast-moving IT departments, particularly in investment banking, but even some in telecom, continue to use the "build it to throw it away" mantra with reasonable success. This works when your organization is compartmentalized. But when information sharing is of primary importance, such as the intelligence or defense establishment, for example, we get to a case where these huge organizations need a different approach. Layers upon layers of bureaucracy make it hard to abandon any pet project.

This is partly why I think Enterprise Social Computing, the Web's architecture (or some successor of REST), etc. are very exciting; they seem much more likely to strike a balance between shared agreement and decentralized execution.

Every branch of software I know of must build an organization that is good at building software that is easy to throw away. If your organization can do this, it is at the master level and should be highly valued. Very few organizations can throw away much of anything in their data centers. This is *horrible*.

Stu nails that. But moreover he latches onto what should remain relatively constant in the data center: the information (web) architecture. That seems to be a good way, as he writes, "to strike a balance between shared agreement and decentralized execution". If you do that properly, you win.

That BEA Enterprise Social Computing thing looks interesting. It's encouraging to see BEA reaching out in that direction.

Making Great Decisions

From the Lean Development yahoo group...

Ford has spent the last thirty years moving all its factories out of the US, claiming they can't make money paying American wages. Toyota has spent the last thirty years building more than a dozen plants inside the US .

The last quarter's results: Toyota makes 4 billion in profits while Ford racked up 9 billion in losses. Ford folks are still scratching their heads

I viewed a Google Tech Talk recent on "Making Great Decisions". The presenters have a book by the same name. The talk and the book a very similar. The book has more content, of course, but the format is better as a book for me. It's the kind of book that I can pick up here and there and read a few of their stories. Nothing truly astounding has jumped out to grab me, which may be good: the stories are just common sense reminders for the most part, with a few fundamentals of analysis thrown in, e.g. considering your biased viewpoint when analyzing a trend.

By and large we tend to stop thinking even when we appear to be "doing".

Apollo

I've enjoyed reading Steve's posts and articles for a while. His latest is somewhat of a shockwave. An entertaining shockwave, the best kind...

The ESB becomes like one of those tools you see on late-night TV, where it’s a screwdriver and a hammer and a wrench and fishing reel and a paint brush, and plus you can flip it over and it can make you a grilled cheese sandwich. Of course, such tools always end up not doing any of those things particularly well.
But more than that he's former Apollo, which is way up on my list of lost treasures. Aegis / Domain/OS was a joy...
AEGIS was distinctive mainly for being designed for the networked computer, as distinct from its competitors, which were essentially standalone systems with added network features. The prime examples of this were the file system, which was fully integrated across machines, as opposed to Unix which even now draws a distinction between file systems on the host system and on others, and the user administration system, which was fundamentally network-based.

Wednesday, October 03, 2007

Yes, IM

From "discipline and punish" this...

Now every application isn't complete until it can send/receive IMs.
For whatever reason this came to mind the other day...
Send an instant comment to me,
Initial it with loving care
Don't surround
Yourself.

'Cause it's time, it's time in time with your time and its news is captured...
I'll have to go back and listen to the recording. I always thought it was "instant comment" but most lyric sites have "send an Instant Karma to me". Hmm.

OK. So in case anyone is interested, I did listen to the music this morning, and it does sound like he's saying "send an Instant Karma to me". Oh well. That's probably more Yes anyway.

Delay Tactics

See over on OreillyNet...

Well-intentioned people often add risk to their projects when they make hard decisions too early.

Carry On

The comments to Cedric Otaku's "verdict" on Erlang are a better read than the post itself. A smattering of choice words...

It is amusing to watch people attempt to tear apart and deconstruct Erlang, trying to refute its benefits, while Erlang coders continue on their merry way creating incredible software. Carry on, Java programmers!
and
What you're saying in effect is that despite the fact that the Erlang guys have already proven this stuff in practice time and again with actual, working, reliable, and long-running telecommunication systems (and systems in other fields as well), your gut feelings are a much better proof and indicator of what actually works in practice and what doesn't, so the reader should believe you, not them. Wow.
and
The real difference here is that it is much simpler (i.e. you don't even have to think about it) to run the supervisor process on an entirely separate node in your cluster. This is paramount for real fault tolerance in a running system, and you don't even have to think about it in erlang. I don't know how easy or hard this would be to accomplish in Java, but it is stupid simple in erlang.
and
In fact, the "post message to mailbox" operation has historically been the most tightly tuned hand-written algorithm in the entire erlang VM, for exactly that reason... but go ahead and assume it sucks. :)
and
Someone suggested Java can't compute 4 digit factorials. Use BigInteger - and it comes back in a snap for the factorial of 1000.
(You gotta love *that* one, don't you!) And...
Man, I'm sooo over language wars.
Finally from the post itself...
The only way a process can modify the state of or talk to another process is by sending a message to that process. Regardless of how lightweight the implementation of this message passing is, there has to be a lock on all these inboxes to guarantee that when dozens of processes send messages to your own process, these messages are delivered and treated in the order they are received from.
Erlang makes no such ordering guarantees. The messages from any process A to any process B will arrive in order, but independent of the interleaving of messages to B from processes C to Z or beyond.

And like Mark Baker has said about designing Restful HTTP systems: nobody said it would be *easy* just more simple and rational than other approaches. I don't know anyone who said Erlang made complex, distributed programs *easy*. But Erlang makes some aspects simple, gets you thinking in the right way, and has a body of knowledge and community easily available to help you build large systems.

Earlier in the year the team I am on took on building the kernel of a potentially large system using Jini/Javaspaces. That is the most "erlang-like" tool for building large, distributed systems in Java. (No, ESBs are not even in the running.)

The software was fine. The mechanisms worked. The biggest lack was the community. It would take a good deal of effort to build up our own knowledge base. There were bits and fragments and people like Dan Creswell we could call on to get advice here and there. But Dan moved on from J/J for the most part into another full-time gig. The rest of the community just wasn't there.

That community and knowledge base seems to be pretty much in place for Erlang development. That in itself is a huge advantage. Carry on.

Tuesday, October 02, 2007

Flex Builder Alpha for Linux

Flex Builder alpha for Linux. Great. Well, I use emacs, but still, great for most people.

Dang, I can hardly wait for AIR to beta, or even alpha, for Linux. Did I miss something yet???

By the end of the year, we can hope?

Sunday, September 30, 2007

Many-Node has many more implications than Many-Core

ManyCoreEra refers to a post on the Parrot VM and various concurrency mechanisms.

These shared-memory mechanism discussions continue to miss the point about the Many Core Era...

The many-core era will also be a many-node era. You will not have a C: drive except for those legacy systems running in their little jars of formaldehyde.

You will have a lot of computing power "out there" and several kinds of displays that are not directly attached to "your computer". You probably will not be able to locate an "application" as being installed on your C: drive and running on the CPU on the mother board that has a ribbon running out to the disk that has C: formatted on it.

Hardware will be too cheap to be organized this way. We need to begin reorganizing our software now or five years from now, we'll be really bad off, without any excuses.

If your current programming models distinguish between "these threads threads" and "those other processes over there" then it's almost certainly the wrong model for today, not to mention tomorrow.

Update:

The Java Server Side site has a lengthy discussion based on this lil'post of mine. A number of different views there.

More on the ESB

Cross-referencing fuzzy's cross reference to my cross reference -- we may create a vortex that implodes on itself. But anyway, in my URI post I said the ESB subject per se is a topic for another post, which turns out to be this one. Mike's hit my main impression with ESBs over the years. I wonder what Ross Mason or other ESB advocates have to say about my experience. First Mike's note on his recent exploration with Mule...

I have more learning on Mule to do. I was a little overwhelmed in working with it yesterday (lots of jar files, lots of XML config, examples that didn't help with what I wanted to do, docs that didn't answer my questions - typical newbie stuff). I did succeed in getting it configured to do what I wanted it to do after going down a number of rat holes & now have something to build on this week.
In my experience with various middleware tools (some of which predate the term "ESB" but all of which tend now to place themselves somewhere in the ESB territory) these turn out to be excercises in installation and configuiration by dialog box and/or by XML config file. I dislike both.

ESBs seem to me to be lumps of things useful for integration, but do not form any kind of coherent shape out of the individual lumps. If you need XML transformation, email, HTTP, file drop, etc. then why not just use the simplest dang library for the one or two of those needed for any given situation? If you need more than two of those in a single situation, then that's a symptom rather than a need.

Look at the list of "adapters" or whatever an ESB advertises. In each case, one can locate a fairly simple library, probably already on their machines, that implement that one feature well enough. ESB's say they can "wire" all that stuff together.

Such wiring usually appears to be more complicated than the original problem. They just have never appealed to me. But never say never, I guess.

I am from Missouri when it comes to ESBs. Show me. I'll give you the benefit of the doubt, if you can just show me, why it makes more sense to use such an all-in-one, configuration nightmare. :-/

I may be biased already, but I've not seen where these things pay off over a little code, some tests, and a couple simple libraries that do exactly what's needed. Assuming your experience differs from mine, please show me.

Update from comments: Yuen-Chi Lian's comment to this post responds to Dan Creswell's. First, Dan Creswell's... (Dan's connected to Jini/Javaspaces, in particular to the Blitz Javaspace implementation and some significant uses of J/J in production situations)

Of course the more point-to-point type integration solutions can also get out of hand but as you say it's simpler, more contained and easier to get to grips with.....
And now Yuen-Chi Lian's response to that... (he's connected to Mule, says google, so presumably has some good case studies in applying ESBs)
That's the point. And when it comes to a system with more and more enterprise applications connecting to it -- different vendors, different language, etc. You need a centralized mechanism, a universal connectivity, a message bus, an ESB. To keep messy things at a manageable level.

Also, programming in the large is simply a different (not better) development model to follow. But there are still fundamental concepts as its core which we have learned throughout the years in "traditional" development -- "high abstraction" and "loose-coupling".

I've got some experience with the Tibco RV bus, JMS, and Javaspaces. I understand how to use pub/sub, queues generally, and tuple spaces to reduce the NxM connectivity problem. I understand how to define "canonical messages" so that the messages on the bus are not just propagating the internal implementation details of the participants. And so this to me seems like appropriate "high abstraction" and "loose coupling".

I've never been a fan of the big tools sets that have grown up around tools like RV. All the point-and-click, configure the xml, "wire-up-the-bubbles" tools just don't do much for me. Writing a little bit of code to read/write messages on a bus seems just fine.

And so to me "message bus" and "enterprise service bus" seem to be two different animals. One is a simple animal (basic bus capabilities), the other a mangy mutt (ESB). The ESB is like a grab-bag of mechanisms that form no coherent whole. And they tend to have WS-Deathstar infused in their being, lacking simplicity and simply defined terms.

So give me a bus, give me a tuple space, give me simple mechanisms like that. But xml config'ing a zillion jar files to "mix-n-match" all the ESB thingies -- who really does that? Successfully? And what does the *architecture* of that look like? I remain unconvinced and nearly uninterested.

Today I am drawn to the simple web approach as opposed to the simple bus approach. The web, via atompub and the atom format, seems to incorporate enough "bus-like" behavior to meet many needs. Where it falls short then simple bus mechanisms can be employed as I'd turn to before. But I'd look to do that *without* hurting the web aspects. All that NxM connectivity stuff should have a web woven around the bits. They can all use the web as the simple, unifying architecture.

ESBs do not form an architecture in and of themselves. They are mechanisms that can provide some plumbing. I'm not sure at all that they add much to the simple plumbing mechanisms I've had success with already. I have trouble understanding how to approach them, and nothing I have seen to date has eased that for me.

So really good case studies would be appreciated. Give me your best shot, and I'll try to consider it.

Leaning

Fuzzy again...

The Poppendiecks are my favorite methodologists. I have yet to be disappointed with Lean thinking.
(Get their books if you have not yet. And read their other articles too.)

Mike has some good quotes and I'll pull some others out here. Another nice article from that pair. They provide a setting for having discussions with the rest of the business, less tied to the jargon and rigidity that has overtaken a true "agile" approach to software development. Here they are quoting Taiichi Ohno of Toyota...

"Years ago, I made them hang the standard work documents on the shop floor. After a year I said to a team leader, 'The color of the paper has changed, which means you have been doing it the same way, so you have been a salary thief for the last year.' I said 'What do you come to work to do each day? If you are observing every day you ought to be finding things you don't like, and rewriting the standard immediately. Even if the document hanging there is from last month, this is wrong.' At Toyota in the beginning we had the team leaders write down the dates on the standard work sheets when they hung them. This gave me a good reason to scold the team leaders, saying 'Have you been goofing off all month?'

"If it takes one or two months to create these documents, this is nonsense. You should not create these away from the job. See what is happening on the gemba and write it down."

Years ago I attended several weeks of CMMI training from a couple of really good instructors from SEI. They taught us essentially this. Start where you are, make small, incremental improvements.

I was the "agile" person there, and of course identified with this immediately. Most of the other people there were from "the CMMI team" whose job it was to "improve" everyone else. Unfortunately after years of discussion, they never really understood how to make improvements. To a large degree this was due to the fact that none of them ever developed much software, ever made many improvements, and were determined to force a made-up, "ideal" process on all the developers and project managers, who did nothing but resent it.

And so it goes. Now we have "agile" groups acting just as rigidly about their crown jewels.

Saturday, September 29, 2007

Happy 80th

John McCarthy turned 80 recently. Happy birthday!

In 1961, he was the first to publicly suggest (in a speech given to celebrate MIT's centennial) that computer time-sharing technology might lead to a future in which computing power and even specific applications could be sold through the utility business model (like water or electricity).

Enterprise URIs

I saw the page in our wiki where fuzzy was working out some uri's. That *is* cool...

I'd rather bet on the URI than any ESB vendor...

With a URI naming scheme that doesn't change you get a very different, very simple view of your systems. Sure, there may be madness today behind those URIs, but over time that madness will hopefully start to go away. But your URIs will stay the same.

Rather than selling proprietary ESB middleware, "integration" vendors should be developing and selling "information architecture" expertise as well as experience in planning and developing simple tools, that don't hurt the web, around the awful messes every organization has and would like to move away from over time.

Update: Ross Mason comments...

I agree that URI design is very important, but I'm not sure I understand what that has to do with ESB vendors. In the Mule we use URI to express service endpoints, but this isn't limited to http but also Jms queues, data repostiories even IM and EJBs. However, in all these cases, we, the vendor, don't control how the URIs are defined above ensuring that they are valid. Can you elaborate a little on your comments about the ESB vendor involvement?
Good question. Over the last few years, in a couple of different IT shops, I have been directly involved in "SOA" efforts. At the same time I have been a fairly close observer of other similar efforts. In each of these efforts a lot of time has been devoted to products in the ESB category.

My primary concern over the last 25 years of software development has been the inability for the software developers to make a change they would otherwise choose to make, except for the fact that making that change is inordinately too difficult. The difficulty in almost all circumstances were due to implementation details of one component being expressed as dependencies by another component. In order to change A, you also have to change B, and usually worse than that.

As a result the *apparently* most cost-effective choice *in the moment* usually is, "don't change anything". And so these problems compound with more problems over time until the entire data center is in a development deadlock. Nothing significant can be improved because everything depends on everything else. I've been involved in a couple of situations where the big changes *were* funded and they are nightmares.

To avoid, or to climb out of, such massive technical debt, an enterprise must invest in separation of concerns, abstractions that hide details as much as possible, "wedges", if you will, between the components that should be able to change independently. The WS-* proponents envisioned WSDL-based interfaces as such "wedges". The message bus proponents (and I was one of those for a long time) envisioned "messages" as such wedges. The distributed object proponents (and I was one of those for a long time too) envisioned "objects" as such wegdes.

I was never a WSDL/WS-* fan, but I still believe reasonable solutions could be implemented with messages and/or objects in many data center situations. But taking a step further back, I can now see how identifying the significant "resources" of an organization, and how to identify (and "locate") those can be even more abstract than messages or objects. Atom, for example, can be used to identify, locate, manipulate, and announce changes to resources over time. Nothing at that level must be said about the lower-level techniques (e.g. messages and/or objects) used to do so.

I would like to say as little as possible at the enterprise level about how I implemented the enterprise. An ESB product may or may not fit in somewhere behind the scenes (that's another discussion). I would like products and technologies to come and go as desired without unduly affecting the rest of the enterprise.

An ESB, or any kind of "bus" or object system is not an architecture, per se. They may help to realize an architecture. Right now it looks like "the web" is a pretty good architecture for the enterprise.

Third parties, such as ESB vendors, over time could have more real success with enterprises by helping them implement web-like architectures rather than helping them install and configure ESBs. Mule may play a role here or there behind the scenes, but enterprises need to learn how to build more lasting structures ("information architectures" may be a decent name for it) and focus less on the plumbing that they've been sold by the ESB vendors.

Thursday, September 27, 2007

When it rains, it pours i/o all over Erlang

klacke adds to the erlang i/o discussion as he did with the regexp discussion, with a faster library...

Originally at Bluetail, we had some serious problems with high performance file I/O, especially line oriented such.

I then wrote a portable (yes win32 too) linked in driver for fast FILE I/O. It's based on an old and hacked version of the BSD FILE* interface. It's called bfile and we've been using it in pretty much all projects during the past 8 years. I've prepared a tarball of it at

http://yaws.hyber.org/download/bfile-1.0.tgz

2> bfile:load_driver().
ok
4> {ok, Fd} = bfile:fopen("Makefile", "r").
{ok,{bfile,#Port<0.98>}}
5> bfile:fgets(Fd).
{line,<<10>>}
6> bfile:fgets(Fd).
{line,<<10>>}
7> bfile:fgets(Fd).
{line,<<97,108,108,58,32,10>>}
14> bfile:fread(Fd, 10000).
{ok,<<10,10,105,110,115,116,97,108,108,58,32,97,108,108,10,9,40,99,100,32,99,95,115,114,99,59,32,...>>}
15> bfile:fread(Fd, 10000).
eof

More on Erlang's i/o Rationale

From Ulf Wiger of Ericsson on the erlang-questions email list, regarding the performance of some of the i/o functions...

One reason is that file IO in Erlang has traditionally been tuned in order to be as unobtrusive as possible, in massively concurrent systems. For example, Mnesia's log dumps usually run in the background at low priority in such systems, and the more important IO is the signaling to/from the network. In these systems, writes to disk are uncommon, and reading large volumes of data from disk only occurs at restarts (which are - hopefully - exceedingly uncommon).

While we've noticed for a long time that Erlang's IO generally sucks in benchmarks that test raw sequential speed on one large file or one socket, it hasn't been clear that this adversely affects the key products using Erlang.

I'm sure that we can find ways to speed up such IO without adversely affecting the characteristics of massively concurrent IO. As Erlang is spreading more into other application areas, this is bound to be a major issue.

Solid State

(via Steve Dekorte)

Fusion io's flash storage card. Neat.

...the cards will start at 80 GB and will scale to 320 and 640 GB next year. By the end of 2008, Fusion io also hopes to roll out a 1.2 TB card...

...the card has 160 parallel pipelines that can read data at 800 megabytes per second and write at 600 MB/sec. He even proved it by running a Linux drive I/O benchmark. But for large corporations running busy databases, operations per second is a much more important number than bandwidth.

Flynn set the benchmark for the worst case scenario by using small 4K blocks and then streaming eight simultaneous 1 GB reads and writes. In that test, the ioDrive clocked in at 100,000 operations per second. “That would have just thrashed a regular hard drive,” said Flynn.

Five years from now will be fun, running not-so-little data centers in a pizza box.

How are you going to justify running your operations on a mainframe in 2012?

Moore's Law is changing the hardware landscape orders of magnitude more quickly than the software community can track. We have not even grasped the difference between today and tomorrow because we're still way back before yesterday in the way we think about software.

How long before someone gets rid of that artificial disk driver sitting between the processor/caches/memory and the "disk"?

Tuesday, September 25, 2007

Apparently Fast Erlang File Read and Regexp

Claes Wikstrom sent a link to the erlang-questions list to his faster regexp library. No report yet on speed...

...the only fast way today to process a large file line/by/line is to
  1. file:open(Filename, [read, raw])
  2. In a loop {ok, Bin} = file:read(Fd, BufSize)
  3. Use a binary regex matcher such as...
http://yaws.hyber.org/download/posregex-1.0.tgz

(I don't know the state of the regex lib in OTP today, last time I looked it sucked bigtime though)

/klacke

Here's an example of its use. Note it uses Erlang's binary representation (a sequential hunk of memory) of strings instead of its list-of-characters representation...
Erl Interface to posix regular expressions by klacke@emailaddress.xyz
LICENSE: BSD style, free, use,  molest and rewrite

To build, make and sudo make install

To use:

1. Compile your regexp.

4>  {ok, RE} = posregex:compile(<<"abc.*foo">>, [extended]).
{ok,#Port<0.101>}

Try to match something 

7> posregex:match(RE, <<"abc mre text here foo">>, []).
ok

If it doesn't match 

9> posregex:match(RE, <<"abdc mre text here foo">>, []).
{error,nomatch}

Try to match and find out where the match occured

10> posregex:exec(RE, <<"abc mre text here foo">>, []).  
{ok,[{0,21}]}

Free memory occupied by the compilation (or exit process since
RE is an erlang port)

11> posregex:free(RE).
ok 

Monday, September 24, 2007

Beer Riot

Oh and BeerRiot.com is also done in Erlang and ErlyWeb. Gulp.

I gotta try that Sam Adams Imperial Pilsner. (Sam Adams? Imperial? The irony.)

Vimagi

Vimagi was built with Erlang and the ErlyWeb framework. (And Flash apparently.)

Check out an example.

Pier Port for Cincom Smalltalk

Via James Robertson, Pier has been ported to Cincom Smalltalk. What's that mean?

Pier may be the best open source application built on the open source Seaside framework. (The best commercial application being DabbleDB of course.)

Pier may be the best extension of Ward Cunningham's "wiki" concept. And it is built on Magritte, which may be the best self-describing meta application system on... well, on earth.

And Cincom Smalltalk may be the best OO dynamic language system, probably the best such commercial system, and has all the openness that Smalltalk systems have had going back to the early 1980s. In fact CST's lineage goes all the way back. (Why would you use Ruby when you could use Smalltalk???)

Cincom has or will soon have support for Seaside. Try out Pier. It's cool.

Regular Expression Matching Can Be Simple And Fast

Speaking of regexp. Interesting analysis of approaches to regexp design. Rob Pike shows up again in this. Those Unix guys had something going back then.

This is a tale of two approaches to regular expression matching. One of them is in widespread use in the standard interpreters for many languages, including Perl. The other is used only in a few places, notably most implementations of awk and grep. The two approaches have wildly different performance characteristics...

Notice that Perl requires over sixty seconds to match a 29-character string. The other approach, labeled Thompson NFA for reasons that will be explained later, requires twenty microseconds to match the string. That's not a typo. The Perl graph plots time in seconds, while the Thompson NFA graph plots time in microseconds: the Thompson NFA implementation is a million times faster than Perl when running on a miniscule 29-character string. The trends shown in the graph continue: the Thompson NFA handles a 100-character string in under 200 microseconds, while Perl would require over 1015 years. (Perl is only the most conspicuous example of a large number of popular programs that use the same algorithm; the above graph could have been Python, or PHP, or Ruby, or many other languages. A more detailed graph later in this article presents data for other implementations.)

It may be hard to believe the graphs: perhaps you've used Perl, and it never seemed like regular expression matching was particularly slow. Most of the time, in fact, regular expression matching in Perl is fast enough...

Today, regular expressions have also become a shining example of how ignoring good theory leads to bad programs. The regular expression implementations used by today's popular tools are significantly slower than the ones used in many of those thirty-year-old Unix tools.

This article reviews the good theory: regular expressions, finite automata, and a regular expression search algorithm invented by Ken Thompson in the mid-1960s. It also puts the theory into practice, describing a simple implementation of Thompson's algorithm. That implementation, less than 400 lines of C, is the one that went head to head with Perl above. It outperforms the more complex real-world implementations used by Perl, Python, PCRE, and others. The article concludes with a discussion of how theory might yet be converted into practice in the real-world implementations...

While writing the text editor sam in the early 1980s, Rob Pike wrote a new regular expression implementation, which Dave Presotto extracted into a library that appeared in the Eighth Edition. Pike's implementation incorporated submatch tracking into an efficient NFA simulation but, like the rest of the Eighth Edition source, was not widely distributed. Pike himself did not realize that his technique was anything new. Henry Spencer reimplemented the Eighth Edition library interface from scratch, but using backtracking, and released his implementation into the public domain. It became very widely used, eventually serving as the basis for the slow regular expression implementations mentioned earlier: Perl, PCRE, Python, and so on. (In his defense, Spencer knew the routines could be slow, and he didn't know that a more efficient algorithm existed. He even warned in the documentation, “Many users have found the speed perfectly adequate, although replacing the insides of egrep with this code would be a mistake.”) Pike's regular expression implementation, extended to support Unicode, was made freely available with sam in late 1992, but the particularly efficient regular expression search algorithm went unnoticed. The code is now available in many forms: as part of sam, as Plan 9's regular expression library, or packaged separately for Unix. Ville Laurikari independently discovered Pike's algorithm in 1999, developing a theoretical foundation as well.

Finally, any discussion of regular expressions would be incomplete without mentioning Jeffrey Friedl's book Mastering Regular Expressions, perhaps the most popular reference among today's programmers. Friedl's book teaches programmers how best to use today's regular expression implementations, but not how best to implement them. What little text it devotes to implementation issues perpetuates the widespread belief that recursive backtracking is the only way to simulate an NFA. Friedl makes it clear that he neither understands nor respects the underlying theory.

Regular expression matching can be simple and fast, using finite automata-based techniques that have been known for decades. In contrast, Perl, PCRE, Python, Ruby, Java, and many other languages have regular expression implementations based on recursive backtracking that are simple but can be excruciatingly slow. With the exception of backreferences, the features provided by the slow backtracking implementations can be provided by the automata-based implementations at dramatically faster, more consistent speeds.

Companion articles, not yet written, will cover NFA-based submatch extraction and fast DFA implementations in more detail.

Just goes to show that various benchmarks are relative, and there's likely a good bit of low-hanging fruit in Erlang's implementation.

Steve Vinoski on Tim Bray and Erlang

Not only is Steve Vinoski blogging again, he's blogging about Erlang. Not only is he blogging about Erlang, he's written some code like Tim Bray's but parallelized it on two-core and eight-core machines, with ease, as a relative newbie to Erlang...

Reading between the lines, it seems that Tim was hoping to take advantage of Erlang’s concurrency to put his multicore machines to work analyzing his logs...

I decided to take a crack at it myself...

The way this solution works is that it uses multiple Erlang processes to convert chunks of the input file to lists of strings and process them for matches...

The best I got on my MacBook Pro after numerous runs was 0.301 seconds with 2400 processes, but the average best seems to be about 0.318 seconds. The performance of this approach comes pretty close to other solutions that rely on external non-Erlang assistance, at least for Tim’s sample dataset on this machine.

I also tried it on an 8-core (2 Intel Xeon E5345 CPUs) 64-bit Dell box running Linux, and it clocked in at 0.126 seconds with 2400 processes, and I saw a 0.124 seconds with 1200 processes. I believe this utilization of multiple cores was exactly what Tim was looking for.

If you’re a Java or C++ programmer, note the ease with which we can spawn Erlang processes and have them communicate, and note how quickly we can launch thousands of processes. This is what Tim was after, I believe, so hopefully my example provides food for thought in that area. BTW, I’m no Erlang expert, so if anyone wants to suggest improvements to what I’ve written, please feel free to comment here.

Very cool. There are still the benefits to be gained from improving Erlang's I/O and regexp libraries for doing the sequential aspects of Tim's work. But this shows the real value of Erlang (and Erlang-like capabilities if they show up in other language systems) for the increasingly multi-core, multi-node world.

Sunday, September 23, 2007

Intel C/C++ STM Compiler

James Reinders, who lives just down the road a bit, announces a prototype C/C++ compiler with Software Transactional Memory...

We have a lot to learn before we can decide whether STM offers some relief from locks (they are NOT going away) and offers help for programming, or for tools which compose programs automatically. We think that the existence of a C/C++ compiler supporting Software Tranactional Memory (STM) would be a great help. So... Today, we released a prototype version of the Intel C/C++ Compiler with support for STM. It is available from Whatif.intel.com. The Intel STM Compiler supports Linux and Windows producing 32 bit code for x86 (Intel and AMD) processors. We hope that the availability of such a prototype compiler allows unprecedented exploration by C / C++ software developers of a promising technique to make programming for multi-core easier.
That's a healthy attitude. Have fun with it.

If you *are* interested in STM (well, I'm not), then you might consider how a system like Gambit Scheme, which compiles to C, could use this new C compiler. (You'd also have to consider how the Gambit Scheme interpreter does the same.)

Postmodern I/O

Update: As it turns out, get_line is intended primarily for writing interactive tty character-by-character apps. On the erlang-questions list, I think, someone on the implementation team announced they're updating the performance guides. And now I assume the i/o and regexp implementations. End.

Tim Bray's note to Erlang...

I like you. Really, I do. But until you can read lines of text out of a file and do basic pattern-matching against them acceptably fast (which most people would say is faster than Ruby), you’re stuck in a niche; you’re a thought experiment and a consciousness-raiser and an engineering showpiece, but you’re not a general-purpose tool. Sorry
I've never had to do a lot of really fast I/O in Erlang, so Tim's excercise (and Steve Loughran's a while back) have been useful for me.

Fortunately for Erlang, making improvements to the I/O and perhaps the regexp libraries should be a fair bit easier than making concurrency and distributed system improvements in other languages.

If I had to do a lot of really fast I/O and Erlang did not pan out for that, I would probably turn to Gambit Scheme. In fact Gambit can do really nice Erlang-like concurrency as well as really fast I/O. It just doesn't have all the OTP libraries that Erlang has.

Maybe Gambit will get there some day. Maybe someone will port Ruby to Gambit, so Ruby can run really fast and have really fast I/O too. And have really nice concurrency. All without building a new virtual machine from scratch or building on top of the JVM.

(If you're interested in that, let me know. I just don't care enough for the cruft in Ruby to write an implementation of it for my own use. There're a helluva lot of good things that fall out of this approach, like secure multiple application spaces per OS process, multiple languages with message-passing integration per OS process, "engines" that can be metered, pet-named secure access to resources, etc. OK -- that's its own blog post.)

If I had to do a lot of really fast I/O in the context of a reliable, scalable distributed system, I would probably do the really fast I/O in Gambit Scheme connected to an Erlang external port. Or this could be Python or Ruby if you wanted something from those systems.

For example instead of implementing a rule engine in Erlang, I'd integrate from Erlang to PyClips (a really nice integration of Python and the Clips rules engine) like this. That seems like the way to develop postmodern systems... use good tools for appropriate situations, especially if they are built to be integrated easily. Programming today of any size probably leads to multiple languages, by its very nature.

Meanwhile Tim's I/O results have been taken to the erlang-questions list. With those numbers there seems to be some low hanging fruit that may or may not require some underlying systems coding. Already solutions are pouring in just using a different approach than the apparently slow get_line.

Saturday, September 22, 2007

PDXFunc Wiki

The PDXFunc, the Portland functional programmers group has a wiki. Nothing there yet, but there it is, and it's a wiki.

Actually it seems to be password protected, so I could not edit the sidebar to include a link to the PDXFunc google group. Coming soon, I hope.

Biztalk Services In The Cloud, Or: Why Not Jingle All The Way

Jon Udell mentioned Biztalk's Internet Service Bus a while back. I'm just getting caught up with that.

Curiously, in this video they mention using WCF and Biztalk Services to implement a chat that can traverse firewalls, etc.

Have they considered not hurting the web and using IETF standards like XMPP? XMPP defines an HTTP-based client. And the Jingle (XEP-0166) proposal, em, which actually has some implementations, extends all *that* to accommodate higher-bandwidth, out-of-band, protocols for VOIP, includes firewall negotiation, etc. to figure out if they need to go through a server.

Ah, but, yeah, we could just use this Microsoft code and go non-standard all the way. Maybe I'm missing something. Or maybe you're happy in your little Microsoft corner of the internets.

Early Diagnostics

Tim Bray and Cédric Beust each make good points about Erlang's error reporting. My first reaction was: apparently they've not used Haskell. But coming back around, I do agree that the messages can be confusing, especially to beginners, especially to beginners whose mindset is in the typical assignment-oriented, imperative C/C++/Java model of computing.

Pattern matching errors can result in a "bad match" message. When you thought you were doing assignment, and you forgot you forgot you cannot "re-assign" different values to the one some variable already has, then having the system tell you "bad match" is going to cause some grief.

Learning anything new, and especially "re-learning" something you thought you already knew, will take time. Grant it, that effort may not always pay off. Maybe it's just not your cup of tea. On the other hand what I learned about programming in Erlang is that a few error messages pop-up with some regularity, and I now am fairly quick to identify the cause.

Good practices to aid diagnosing errors are:

  • Write a little code at a time.
  • Test-drive that code using an xUnit framework or just using the erl command line.
  • Use pattern matching and guards to narrow the acceptable parameters to a function.
That last one is a key: catch the error asap, and rely on the language itself to catch it for you. In Tim's example, the error message is correct - it's a bad argument. But the top-level function did not catch it. Rather the recursion was fairly deep before some other function caught the problem.

That top-level function looks as though it should expect a list, and not an atom. So this would have helped greatly...

scan(Arg) when is_list(Arg) ->
  ...
Or maybe better...
scan([]) ->
  %% Empty list, do the default thing...
  ;
scan([Head | Tail]) ->
  %% Do something with Head then recursion on the Tail...
  ...
  scan(Tail).
Cédric also says Erlang "feels old"... so do I sometimes, but I still have something left every now and then. :-D

Erlinguistics: Counting with a Functional Accumulator

Tim Bray though Erlang weird, which it is, but he's pushing on with it, which is fun and educational, hopefully, for everyone following along.

Tail recursion is really a glorified goto statement, and after a while it starts to feel like an elaborate fake-recursive lie.
Once this style is adopted though, solutions tend use less code and become more tractable. At least code comes out more factored and there's less of it in one place. i.e. you know those long stretches of looping code that can last for a couple of screenfuls? You tend not to find those in code written in a functional style. Assignments don't change the state of a variable 17 lines into a 49-line iteration. So in a sense there is no "lying" in a recursive function, but sometimes some unwinding is needed to get a sense of what's happening.

On counting...

I thought of two ways to count things. Every Erlang process has a process dictionary, i.e. a hashtable or content-addressable store or whatever, with get() and put() calls. This is the kind of thing that the Ruby code above depends on; the square brackets are a dictionary lookup. The problem is, Joe Armstrong severely disses the use of the dictionary... [Tim's analogy elided]

There’s also a hashtable package called ets, of which Joe expresses less disapproval.

Another way to count things, which I suspect would make Erlinguists happier, is to spawn a process for each thing you need to count

Yes, using a process to maintain changeable state is generally a good thing. Think of each such process as a "little database" and a "little messaging protocol" for accessing that database. Counting and inspecting the count would be the degenerate example. This approach usually would be reserved for more complicated kinds of "little databases".

More preferable in this simple counting situation might be to include one or more "accumulators" in the recursive function. Instead of passing around a Counter process id, the actual "count" could be passed around. For example, the original "morse codes in erlang" uses an accumulator that is not a count, but it is an accumulator of a list of results. As the results are accumulated, those results are passed around until the recursion is terminated, and then those results are returned as, well, the results.

A simpler counter kind of accumulator could look like the code below, which is close enough to Tim's original example. I'm not sure what his process_match is doing, but in this case, handle_match is playing that part, and here simply printing information about the match. The count is maintained in the Count "accumulator" which is passed around with the recursion. The top-level function "seeds" the accumulator with 0.

-module(foo).
-export([count_matches/2]).

-import(file, [open/2]).
-import(io, [format/2, get_line/2]).
-import(regexp, [match/2]).

count_matches(Filename, Pattern) ->
  {ok, In} = open(Filename, read),
  count_matches(In, get_line(In, ""), Pattern, 0).

count_matches(_In, eof, _Pattern, Count) ->
  Count;
count_matches(In, Line, Pattern, Count) ->
  case match(Line, Pattern) of
    {match, Start, Length} ->
      handle_match(Line, Start, Length),
      count_matches(In, get_line(In, ""), Pattern, Count + 1);
    nomatch ->
      count_matches(In, get_line(In, ""), Pattern, Count)
  end.

handle_match(Line, Start, Length) ->
  format("Found a match in ~s from ~w for ~w~n", [Line, Start, Length]).
Taking this approach to its limits results in functions like lists:foldl and lists:foldr which "fold" some value over a list of elements, starting from the left or the right of the list, respectively. It's all a variation on "map/reduce", essentially. The above code could be made more general to "fold" over the lines in a file from top to bottom for some given functional argument.

Wednesday, September 19, 2007

Atom and RelaxNG

(Update: Links now work, thanks to Sjoerd Visscher.)

I'm fairly new to the details of Atom format and brand-spanking new to the details of RelaxNG. So here's what I did, and if you have any comments, then add them here or send an email to patrickdlogan at gmail dot com. I also did not find any explicit examples like this on the internets, so this may help someone else -- it's not complex, and that makes me feel good about atom.rnc and relaxng in general.

My objective was to use relaxng to validate an atom entry where the content is inline'd xml. The validation should be for the atom stuff per se, as well as for the specific xml content. A second objective was to apply the relaxng schema for that specific xml to that content separate from the atom stuff. So the xml content should stand alone or be inline'd in an entry and one schema definition should be usable for both cases.

I grabbed atom.rnc, the compact relaxng definition for atom. I grabbed the jing validator. Applying just this schema worked. Good. Good.

Then I spent some time looking at that schema and reading through various relaxng combination mechanisms. The combination takes place at the definition of atomInlineOtherContent in atom.rnc. This is where the inline'd xml content goes.

An interleaved combination did not work, as I found out, because that definition as well as the definition of my own xml includes "text", which causes relaxng trouble. Although I don't fully understand why this should cause trouble if my schema is more specific.

The combination mechanism that did work is to redefine atomInlineOtherContent in my own schema...

namespace atom = "http://www.w3.org/2005/Atom"

# Usage:
# java -jar jing.jar -c atom_policy.rnc example_insurance.atom

# Note: this includes an unmodified atom.rnc grammar from the ietf
# standard. This grammar here redefines the inline content pattern to
# specify that such content should include appropriate attributes from
# the atom spec, e.g. type="application/xml", but the content itself
# should be a  tree of elements.

include "policy.rnc"

include "atom.rnc"
{
  atomInlineOtherContent =
     element atom:content {
        atomCommonAttributes,
        attribute type { atomMediaType }?,
        amitPolicy
     }
}
This looks just like the original definition except the actual content has to match the definition of amitPolicy as defined in policy.rnc. (I should probably redefine the attributes to require a specific type of "application/xml" or something, but this is close enough for now.)

The example entry with policy content validates using this more specific grammar. More goodness.

I met the second objective with just a little bit of cruft. I'd like to use policy.rnc to validate just some xml that starts with amitPolicy and has no atom elements at all.

I could not apply policy.rnc directly because I'd have to add a "start" to that grammar, which then gets the validator bogged down when using that grammar with atom, two starting points, not combined correctly.

Instead I wrote another small schema to define the starting point for a standalone policy...

include "policy.rnc"

# This currently exists to support validating policy xml content
# without being part of an atom entry. There may be another way to
# avoid having this separate little grammar.

start = amitPolicy
There may be another way to do this without defining another schema like this.

Tuesday, September 18, 2007

Erlang Interview

André Pang was interviewed at builder.au re: Erlang...

Our very first server was written in C++, but we realised that we had a problem if the server crashes. There might be 20 VFX houses connected to this server, and when it goes down, they all go down with it. So we looked at using an Erlang server, by writing custom plug-ins to ejabberd, which is a popular XMPP (Extensible Messaging and Presence Protocol) platform -- so we layered all of our code above that.

In the end we found that to be a good solution, in fact the only time I can remember it crashing is when we forgot to allocate swap space for the server and it ran out of memory. Due to the way you can set up ejabberd and Erlang in general is that it really suits a distributed system without a single point of failure. It makes it easy to set up different nodes which all replicate as each other, for instance, there were plenty of times when the primary server went down, but users were automatically connected to the secondary server instead. With a bit of smarts in the client it was possible to reconnect to a secondary server in the middle of the session and only lose a small amount of information in the process.

We had three or four servers around the world, and it was incredibly cheap to deploy, because with the Erlang runtime you don't need any special hardware or anything. It was very reliable and turned out to be a very good decision.

Sunday, September 16, 2007

AMQP, XMPP

Stefan Tilkov suggests...

Personally, I see AMQP on the one side and AtomPub/XMPP on the other side of a fence — AMQP internally, addressing the same problem domain as current, proprietary queueing solutions, and AtomPub and XMPP over the Internet.
I'd like to understand more about his reasoning.

What would be the drivers for this dichotomy? Why are two different messaging systems necessary? What would limit AMQP from being used on the internet scale? What would limit XMPP from being used on the intranet scale?

Without some reasoning behind this dichotomy, it just feels to much like the old WS-* on the intranet, HTTP on the internet. I don't recall seeing a sufficient rationale for that.

Now it seems fairly clear that HTTP fits well on both scales, no?

So is HTTP sufficient for all kinds of "message transfer"? HTTP and XMPP? HTTP, XMPP, and AMQP? Or... how many kinds of messaging are enough?

As an aside, our viking friend suggests Apache can go beyond the traditional as well. I think we have a lot to learn still about how simple the enterprise can eventually be.

Emacs Tag Magic

Tim Bray on programming in C using Emacs...

I’ve been using Emacs, and I seem to recall that it has all sorts of navigation magic.
That would be the "tags" mechanism. Tags work with non-emacs tools as well, but emacs is especially good with them. See the manual.

Tag databases can be created for all kinds of languages.

Those IDE's still have *nothing* on emacs. :-D

Nuke-you-lair

On nuke-you-lair power...

How effective is long term storage of nuclear waste? Stewart's answer was typically provocative. As I recall it, he said something like this: "We don't know, but our framing of the question shows a failure of long term thinking. We've all been imagining that we have to solve the nuclear waste problem for all time to come. In fact, we only have to solve it for a few hundred years. Either by then technology will have advanced sufficiently that it will no longer be a problem, or we will have regressed so far that a few nuclear waste dumps in out of the way places will be the least of our worries."
In other words, we're screwed. Welcome to your post-apocalyptic future.

Saturday, September 15, 2007

Sweet Linux Air

From the RIA Cowboy himself. Yeehaw. While we're waiting for Air to be officially available on Linux...

*** WARNING - THIS IS TOTALLY UNSUPPORTED, UNENDORSED, AND A COMPLETE HACK ***

...

All I need is ADL - the testing tool for AIR applications. So I gave it a try on a Salesforce.com project I’m working on:

jamesw@dos:~/projects/mavericks/examples/air/AccountTracker/bin$ wine ~/flex_sdk-3_b1/bin/adl.exe salesforceTest-app.xml
And to my total surprise the AIR application loaded and ran on Linux! Sweet!
The one component that truly needs Windows or Mac right now can run on Wine on Linux. The degree the API is mapped to the Linux environment is probably limited, but for at least some things, developers can currently use Linux for Air.

Friday, September 14, 2007

Squeak by Example

(via James Robertson)

A nice looking new book for getting started with Squeak Smalltalk, pdf is free online, or you can order it print-on-demand from lulu. They also accept donations for the pdf.

Co-authored by local Portlander Andrew Black.

The book seems comprehensive for getting started, covering the development environment (including the Monticello package tool) as well as the language, the GUI classes, "meta" classes, etc.

This is a good year to learn Smalltalk and Erlang, with two of the best programming language books ever now available for two of the most distinguished programming languages. Every programmer would be better at whatever they do by knowing a good bit about these languages.

Thursday, September 13, 2007

ONLamp: an Introduction to Erlang

From Gregory Brown at ONLamp...

I consider myself someone with a middling grasp of concurrency at best, and though I'd likely be scratching my head trying to write this in many other languages, I hacked this together in about 20 minutes with only a beginner's level of experience in Erlang. This certainly says a lot for the possibilities of the language to make parallel programming very easy.

A Scalable Distributed Data Structure for P2P

Google Tech Talk

Tuesday, September 11, 2007

Robust Composition: Unified Access and Concurrency Control

Mark S. Miller's PhD dissertation, "Robust Composition: Towards a Unified Approach to Access Control and Concurrency Control"...

When separately written programs are composed so that they may cooperate, they may instead destructively interfere in unanticipated ways. These hazards limit the scale and functionality of the software systems we can successfully compose. This dissertation presents a framework for enabling those interactions between components needed for the cooperation we intend, while minimizing the hazards of destructive interference.

Great progress on the composition problem has been made within the object paradigm, chiefly in the context of sequential, single-machine programming among benign components. We show how to extend this success to support robust composition of concurrent and potentially malicious components distributed over potentially malicious machines. We present E, a distributed, persistent, secure programming language, and CapDesk, a virus-safe desktop built in E, as embodiments of the techniques we explain.

Performance of Selective Receive

Pascal Brisset explains (on erlang-questions some time ago) a scenario where Erlang's selective receive can fall behind...

The system is dimensioned so that the CPU load is low (say 10 %). Now at some point in time, the backend service takes one second longer than usual to process one particular request. You'd expect that some requests will be delayed (by no more than one second) and that quality of service will return to normal within two seconds, since there is so much spare capacity.

Instead, the following can happen: During the one second outage, requests accumulate in the message queue of the server process. Subsequent gen_server calls take more CPU time than usual because they have to scan the whole message queue to extract replies. As a result, more messages accumulate, and so on.

snowball.erl (attached) simulates all this. It slowly increases the CPU load to 10 %. Then it pauses the backend for one second, and you can see the load rise to 100 % and remain there, although the throughput has fallen dramatically.

Here are several ways to avoid this scenario...

...

Add a proxy process dedicated to buffering requests from clients and making sure the message queue of the server remains small. This was suggested to me at the erlounge. It is probably the best solution, but it complicates process naming and supervision. And programmers just shouldn't have to wonder whether each server needs a proxy or not.

I'm not sure how it really complicates naming and supervision so much. I think it is the best solution. The problem is not in selective receive, per se, which has a lot of benefits that outweigh this specific scenario. Especially wrong would be to gum up the Erlang language and simple message passing mechanisms just for this.

The problem in this scenario is *coupling* too closely the asynchronous selective receive with the backend synchronous service. This is not an uncommon scenario in all kinds of "service-oriented architectures" and the solution, generally, should be the one quoted above.

A programmer should legitimately wonder whether some kind of a "proxy" is needed when they see this kind of a combination.

This is related to the blogs going round not so long ago among fuzzy, Bill de hÓra, Dan Creswell, and others.

Receptionists for Shared Resources

Carl Hewitt, et al.'s Linguistic Support of Receptionists for Shared Resources (MIT AI Memo 781 -- pdf).

exmpp

Back in December, Mickaël Rémond wrote on the erlang-questions list...

Jabberlang for now rely on ejabberd. You need ejabberd to use it. The shared object you are mentionning comes from ejabberd. That said Jabberlang is a client library that rely on a server implementation which seems strange. That's why we are currently doing some code refactoring. Our target is to have:
  • exmpp: A common XMPP library that is used by ejabberd and Jabberlang
  • Jabberlang: Relying on exmpp.
  • ejabberd: Relying on exmpp.
We are in the middle of this refactoring currently.
Recently, elsewhere, he updated:
Jabberlang will be replaced soon by a much better and easier to use library called exmpp. Stay tuned. It will be published here: http://www.process-one.net/

The Killer App for Multi-Node

From Many Core Era, the question: what's the killer app for many-core?

The true "many core" era will have arrived when we realize this is the wrong question.

By the time we get to a significant number of cores we shouldn't care much about cores any longer.

Monday, September 10, 2007

So You Flip The Common Torch Around

This google tech talk by Van Jacobson on networking is great.

Fun game: the video has closed captions in english. Watch for the phrase around 28 min. into it, where the captions that reads "so you flip the common torch around" and catch what Jacobson actually says. (Answer below.)

Was the caption written by a human or a machine? Was it proofed and edited?

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Answer: "so you flip the combinatorics around"

Blog Archive

About Me

Portland, Oregon, United States
I'm usually writing from my favorite location on the planet, the pacific northwest of the u.s. I write for myself only and unless otherwise specified my posts here should not be taken as representing an official position of my employer. Contact me at my gee mail account, username patrickdlogan.