The Lightning Sessions Session

Another favorite session was "Lightning Sessions," at AONW this week. Jon Marshall brought this idea to the conference. This is another example of people bringing the "extreme" back to "agile".

Here is how it works: within the hour and a half session, the participants continuously create new 60-second sessions. The only rules:

1. Think of something interesting to say.
2. Limit yourself to 60 seconds.
3. You must begin your talk with the last sentence of the previous person.

There is no predefined order for speakers, you speak when you're ready. Or not.

Over the hour and a half we had at various times eight to ten people participating. That last rule makes the effort much more interesting: your concentration on the current speaker intensifies. For one thing, you never know which sentence will be that speakers last one. For another, if you've got something to say that seems really appropriate for that specific moment, you are kind of in competition with the others to claim the conch.

The result was really productive. The first several lightning sessions were based on recollections from sessions the speakers attended earlier in the conference. About halfway through the topics evolved to be more in the moment and innovative, less about the prior sessions.

Jon attended last year in Portland. There were many faces from last year, and several people from Portland and Seattle who I'd not seen in a while. The conference was full of the familiar and the new, regarding conversations and people. I hope we can gather many of us back next time.

That one will likely end up back in Portland. I wonder where? Last year we were at the Kennedy School, and it worked out really well. But maybe it is too soon to return there.

I had a thought on the drive back home... maybe "offsite" somewhere like Skamania Lodge or the Edgefield. We have a good set of sponsors and a very positive cash flow to help keep the participants' costs down.

I was also wondering whether we could find a way to specifically reach out to the open source community to mix up more folks with that influence into the AONW "community".

(Pictures are now posted.)

This week's conference was at the Seattle Center in the Northwest Rooms. One of our rooms, Alki, turned out to be a great social area, but too much echo for the sessions. Overall a good setting, though, and a good bit of sun brightened everything up. I especially appreciated the windows in the Alki room. And the food from Gretchen's Shoebox Express was really good.

Extremities

I'd like to thank my friend, and Portlander, Arlo Belshee (of Promiscuous Pairing fame) for once again making "agile" unsafe and going all "extreme" on us. His session at AONW on Tuesday has been my favorite so far.

The sessions have all be great. Agile clearly has a beach head in (nearly?) all software development organizations of any size. (And at least one fantastic non-software, "outdoor start-up", http://trackersnw.com)

So while adoption can still be challenging in many organizations, clearly "agile" is now "safe". So much so that sometimes it feels unsafe to use the term "extreme". I was not crazy about that term in the beginning. Now I kind of miss it.

Arlo's never let go of the ideal: push yourself and your team to its extremes. Can your knobs go to eleven? Arlo's can because he continuously works at it.

His session on "Working Tiny" was excellent, and not just because of him: the group made it great. Arlo made it unsafe enough in that corner of the conference to once again think, and talk, "extreme".

Pictures of the conference are now posted. Track the wiki for more... http://agileopennorthwest.com/wiki

Cross Platform Performance

Ted Patrick makes some important points about the meaning of "cross platform", especially when it comes to browsers and performance...

It is great that things work on different operating systems but performance needs to be similar. It is a poor user experience for an application to run 50% faster on a specific platform than another. I know for a fact that lots of engineering work goes into making sure Flash Player performs similarly across different operating systems and browsers. Actually we implemented incremental garbage collection in Flash Player because browsers differ wildly in how and when they allocate memory to a plug-in. On some browsers memory is handed out like candy while on others it is given in 256Kb chunks up to a ceiling of 10MB total. Regardless of the browser in use Flash Player tries to make sure all application behave the same both in start-up time and during use.

Founded on AIR

Ted Patrick observing that the reach of Adobe AIR has a wider reach across Microsoft operating systems than Microsoft's own "presentation foundations"...

Writing a single application that works across all version of Microsoft® Windows® is often an impossible task. Worse Microsoft deprecates support of certain operating systems as they push new operating systems, tools, and dependencies. For example, WPF is only supported on Microsoft® Windows® XP Service Pack 2 (Must add .NET 3.0) and Microsoft® Windows® Vista® (native) leaving the many XP and 2000 customers without few options for using the latest software. Even Microsoft® Silverlight only supports up to Windows XP Service Pack 2 or higher for browser based applications.

Enter Adobe AIR!

Science Debates 2008

A group of scientific-minded people recently held a press conference urging the presidential candidates to participate in science-oriented debates.

Computer Power and Human Reason

I read "Computer Power and Human Reason" 25 years ago, just as the AI boom was getting underway. The author wrote the book at least partly as a result of observing people using his ELIZA program.

In 1962, he published a comparatively simple program called ELIZA which demonstrated natural language processing by engaging humans into a conversation resembling that with an empathic psychologist. The program applied pattern matching rules to the human's statements to figure out its replies. (Programs like this are now called chatterbots.) It is considered the forerunner of thinking machines. Weizenbaum was shocked that his program was taken seriously by many users, who would open their hearts to it. He started to think philosophically about the implications of Artificial Intelligence and later became one of its leading critics.

Joseph Weizenbaum died on March 5 of this year. As a Lisp programmer doing some AI programming, but not really being an "AI guy", his book and ELIZA helped me understand not only AI, but AI researchers!

Clojure

Christophe Grand pointed to the Clojure programming language in a comment on my recent Erlang post.

I'd heard the name but never investigated. My loss. Clojure is a mostly-functional, concurrent Lisp dialect for the JVM. That definitely warrants an in-depth look for me.

Here's some more information I quickly found.

Erlang Criticism Not Unwarranted

Eric's post on the recent Erlang criticism and his take on the subsequent trail of mail list discussions is fairly accurate in my opinion. I posted a long response on his blog and midway realized it's a blog post of its own. Here.

I'm not going to argue too strongly for "sequential Erlang" because it's really "concurrent Erlang" that stands out. The Erlang syntax is ok by me, but then I still prefer Lisp syntax to anything else I've seen anywhere. Simple.

And I can say nothing for the erlang list, except I've been skimming it too, with not so different reactions as yours.

Anyway...

"in development, the behavior of that existing function often changes thereby affecting all the calling functions"

As with all dynamic languages (especially those unlike Lisp and Smalltalk that do no have good refactoring editors), you need good unit test coverage. Which you are doing, right?

"defining dozens of one-line functions has made flow control difficult to follow in some circumstances."

This is pretty much the same for all functional languages, and even Smalltalk, which tends to have many small methods, where other OO language programmers tend to write overlong methods.

Good tools, which again Lisp and Smalltalk tend to have, and others don't, are indispensable. Otherwise the only thing that tends to help IMHE are again many good tests to read, "Oh, so this is the thing to call, and how to call it."

I still think that's better than really long methods Java or other languages that tend not to be reusable without a good deal of pain. (For one thing, they tend not to be really well tested.)

"It shouldn't take a committee to help you write good file handling code."

I don't disagree completely except that Wide Finder pushed Erlang and its community into a new I/O situation. So most of what happened by the end of it was a core Erlang member adding some new core capabilities to the language. Then also what happened was the exposure of a development process for how to evolve concurrent algorithms generally in Erlang or maybe any concurrent system. I would think both, especially the new core I/O capabilities, will help down the road without requirement the full committee.

"the language is a DSL for networking gear."

Which essentially what it is. CouchDB, Yaws, ErlyWeb, etc. will help, but mostly Erlang and its surrounding code is not much more than what the telecom programmers were using in the mid-1990's. that is its upside and its downside.

"I can write a Scala function to make an HTTP POST in four lines of code and have it run screamingly fast"

Two points:

• Yeah, *but* then you also have to deal with all of Scala's type system. "Oh, you need an existential type here because a mere parameterized type gets the type system all confused, even though it's fairly obvious what you are trying to do."
• Yeah, *and* what the rest of the world should learn from Erlang is "concurrent Erlang" and how to apply *those* lessons in a distributed, multi-language way. You shouldn't need to use Erlang to get the benefits of things like OTP. OTP and basic message passing with pattern matching is missing from pretty much every system but Erlang. Scala and Java do not have an OTP although Scala addresses the latter.

(And now you see why I thought so highly of Jini and Javaspaces, because it also attempts to bring these to Java. It's the closest thing Java has to Erlang/OTP. But now, where did that go again? Dang it.)

Like you wrote...

"actors are so important, it will be amazing when the programming public realizes why we all need to do this"

You could also try Gambit Scheme, which I think is better as a sequential language (Lisp) but also has pretty much Erlang-scale thread/mailbox performance on a single CPU. Not much in the way of distributed or multi-core support yet. You get more of those, "shared nothing" threads, and functional data structures if you use the Termite package on top of Gambit. Multi-core is bubbling to the top of Gambit now that a decent package system has been released.

Buyer Beware: Corrosion Alert

Asus predicts their new Windows XP-based Eee PC will outsell their current Linux-based PC. But buyer beware.

First, it's XP. Will the typical Eee PC consumer understand this is not "Vista" and will likely not continue running newer and newer versions of Windows software for long?

Linux on the other hand will do the job the Eee PC was intended to do, very well. Probably XP does the same job just as well. Until it doesn't.

I have simply found after years of running Linux and all kinds of Windows systems (except Vista, thank you!) that Linux systems tend to just run for years with little corrosion and little maintenance.

I am forever "managing" the rest of my family's Windows-based PCs, currently Windows 2000 and Windows XP. As the Eee PC's XP begins to decay (usually within six months), what then?

Is the Err PC intended to be a simple computer that "just runs" or will the consumer understand that they'll need a Windows technician to clean up the registry, uninstall software correctly, and continually ghost and/or rebuild the OS completely every year?

Thanks

Can I just say if there is any kind of a supreme being in need of thanking for spring emerging in Oregon these days...

Consider yourself thanked. Well done you.

gas and heartburn

Joe Gregorio on the price of gas...

A doubling or tripling of the relative price of gas is inevitable, and is probably the best long-term thing that could happen to the country, fully realizing that in the short term it's going to be painful.

Carter set a goal in the 70s of importing no more foreign oil than the levels at that time. He was belittled, and then we got Reagan. Reagan took Carter's solar panels down from the White House. No oil company bag man could be seen with those.

Now here we are 30 years later, *subsidizing* the oil companies while they "earn" record profits.

I don't know all that much about Carter, but now I think as a president there are a few things he's under-appreciated for. I voted for John Anderson in 1980, my first presidential election. I was the typical Anderson voter -- a socially left-leaning college student, from a fiscally right-leaning family.

Had America the leadership with the foresight and fortitude back then, and since then, we may have been able to have gradually increased the tax on oil to fund strategies toward Carter's vision of more, cleaner, energy self-sufficiency. Ultimately there are many powerful, near-sighted, self-centered forces that interfere with that kind of grand vision and strategies.

Maybe the only way to get to such a future is to stumble into it. "When the student is ready, the teacher appears."

Concurrency Summit

Tim Bray reports on the 2008 O'Reilly Concurrency Summit a bit. Hopefully there will be more from Tim or others. The summit looks like an "open space" conference, scheduled on the spot by the attendees. Apparently the topics included REST and Erlang, and so the discussion must have branched out into distributed systems, not just concurrency per se.

Anyhow, two subjects that came up were REST (which is concurrent at the largest possible scale), and, unsurprisingly, Erlang. And it struck me that they’re kind of like each other.

Well, in REST, a message sender has to know the receiver’s address, there’s no global data, the receiver has to be waiting for the message, and the receiver will typically dispatch on its contents. Just like Erlang.

Yeah, I think it's helpful to compare and contrast the two (HTTP and Erlang's own message passing). Also comparing and contrasting XMPP and Erlang's message passing is useful. They are three different, but related, points in the spectrum. (I wonder if XMPP came up at the summit? Is it bright enough on the radar yet?)

e.g. XMPP's "presence" capability is similar to Erlang's node linking, but more rich.

Neither HTTP nor XMPP have the selective, pattern-matching, message receive capability of Erlang. But they both would support multiple "channels", which can be used for similar purposes.

The Wiki Way

After all these years, Ward's wiki is still the simplest, and because of that, the best.

Nicely done.

We Did It For The Children

A grand justification for the greed that's bringing down the mortgage industry and wreaking havoc on credit generally...

As you might expect, they are now self-righteously blaming those, such as Countrywide, who did what they were told.

Golly, we were just trying to help the kids. Think of the children.

Uh-huh. That criticism is just a leee-tle bit "off balance".

Where your time goes when you're not on the web

We have a simple REST web service running on a dev server. The url is typical, e.g. http://domain/thingies/thingy5. Use it anywhere you like. We don't care. And it's really easy.

The other day I came across another team's wiki page with one of our urls in it. The page had some text discussing some code for working with the information about these thingies. This url showing up should be no surprise -- it's just the web.

But at the same time this week we're working on some other REST web services. The underlying implementation of these has to work across multiple legacy systems, several of which are accessed via their own proprietary SOAP services.

Looking at the WSDL for one, I see it's an end point definition for an interface defined in another WSDL somewhere else. But getting to that other WSDL is a royal pain -- if this is a *web* service, why can't I just *click* on something to get it?

Instead I copy and paste the pieces of text I think will form a url to the desired WSDL. Nope. Is it just gone, or did I copy and paste incorrectly?

Meanwhile next to me and in Belfast other team members are running into familiar problems with several SOAP toolkits generating code against these WSDLs. Then SoapUI makes the connection, and a co-worker cuts and pastes this into his application code instead of using one of the high and mighty SOAP toolkits. It works!

Another co-worker then suggests that maybe our app should just screen-scrape SoapUI! Brilliant! 8^)

This is the nth time in my career I've used SOAP, only to find each one a time sink more than an order of magnitude less productive than simply using HTTP. But go ahead and argue best as you can why that's a good idea.

I'll just respond, "Please don't hurt the web." Or my productivity.

Agile Open Northwest 2008 - a few spots still open

We're down to about a half dozen available spots for people interested in attending Agile Open Northwest 2008 in Seattle. Like last year, we'll have between 110-120 people.

If you are going and you want to figure out a ride share, room share, etc. see the following wiki page... http://c2.com/cgi/wiki?AgileOpenNorthwest

More Cloud Data Services

The competition for cloud services is picking up with Microsoft's SQL Server Data Services. I guess that name is a nice ad for their database product, but from glancing at the page for the service, has no bearing on the service itself.

i.e. nothing about SQL Server that I've seen so far appears in the service interface. It's an HTTP-based service.

See? I like writing positive things about Microsoft.

Too Big To Fail

A NY Times article on the credit crisis...

Over the last two decades, few industries have lobbied more ferociously or effectively than banks to get the government out of its business and to obtain freer rein for “financial innovation.”

But as losses from bad mortgages and mortgage-backed securities climb past $200 billion, talk among banking executives for an epic government rescue plan is suddenly coming into fashion...

Surprisingly, the normally free-market Bush administration has expressed interest.

Bush? Free-market? Right.

Here's the surprise: "free-market" is a euphemism. Rarely has it meant what you think it means. And that is intentional. This has nothing to do with whether or not a "free-market" would work. Nobody with any current leverage wants a free-market. Why should they? Just look at this very deal being worked out, right in the open. This isn't even a back-room deal.

It reminds me of the Bush tax cuts, more than anything else.

Nomenclature

From ...

Yahoo! recently launched what we believe is the worlds largest Apache Hadoop production application. The Yahoo! Search Webmap is a Hadoop application that runs on a more than 10,000 core Linux cluster and produces data that is now used in every Yahoo! Web search query.

Hadoop.net?

Seriously, what organization, other than Microsoft itself, would afford the equivalent running all Microsoft software?

The Difference Between Theory and Practice

Can we face the reality that "core" states have sponsored, continue to sponsor, and likely will always sponsor state terrorism?

There is very little moral high ground on this planet. What little there is can be occupied by a small number of individuals, and precious few organizations of any size.

Linux AIR

Finally some news on AIR for Linux. Adobe is looking for "pre-beta" evaluators. Didn't that used to be called "alpha"?

I can't wait for the beta!

Multicore Squeak

Via the Weekly Squeak, a multicore implementation of the Squeak VM. Apparently the work was done by Qwaq to improve the performance of running Croquet. Open source.

SQL

I barely escaped the Raleigh, NC, airport yesterday for a 6:08am flight. I'd never seen such long security lines so early, even on a Friday morning. But I made it, and there were others on my plane later than me. Just as the cold and rain was hitting NC, I'm back in OR on a sunny Saturday. We'll get more cold and rain, but we're definitely on the upswing into spring here.

Anyway...

Steve Dekorte writes about SQL...

The motivation for SQL queries being declarative was that you could just tell the database "what" you want... If something fails at it's primary reason for existence, is it time to reconsider it's existence?

I don't want to defend SQL too much -- even the declarative expressions have their faults -- but most of the blame for so much "imperative" SQL has to do with the database implementations themselves. The first 15-20 years of the relational database market were aimed at OLTP on hardware with significantly less capabilities than we have today. There was relatively little emphasis on "relations" in the OLAP sense.

And so SQL had to become more imperative, account for explicitly created temporary tables, and incorporate "stored procedures" to perform the explicit, step-by-step operations.

If you look at database systems like Teradata (which is a highly concurrent, shared-nothing architecture), or column-oriented databases like SybaseIQ or C-Store then you will find the ability to program SQL queries as originally intended, in a functional style. I wrote some SQL for Teradata ca. 2001-2003 and helped others improve their SQL (who had been using primarily SQL-Server and T-SQL) because I had a good bit of Lisp and functional programming experience.

Programming truly declarative SQL and the more common procedural SQL are very different experiences. But it is possible, if the underlying database architecture will support decent performance for declarative queries.

Bad Design

DeveloperWorks has a new article published on REST. Unfortunately I was able to find several significant problems with it in a matter of minutes.

The design is awful: methods in the URLs.

The "logic" on when to apply REST vs. "heavyweight" SOA is simply unfounded. Please: "As your application environment grows, it's likely you'll abstract away from the REST implementation details more and more... It shouldn't be too hard to extract the actual business logic behind the services and rewrap it in a SOAP package in the new environment."

Huh? That and the discussion at the beginning about large vs. small organizations using heavyweight SOA or REST. The author should follow his own advice:

"Take some time to think about this: it'll pay for itself in the long run!"

Integrity and Dignity

These are qualities I look for in an elected leader. Too bad they're rare traits in politicians.

Agile Open Northwest: Seattle

I'm helping organize the Agile Open Northwest conference again this year. Last year was my first "open" format conference, as an attendee or organizer. I was amazed how well the conference self-organized into a number of useful discussions and activities.

It seemed to happen effortlessly, but really the other organizers knew what they were doing. Can you tell from this list which of us is not a consultant? 8^)

Better digs than last year's at the Kennedy School in Portland will be tough to achieve. But this year we'll be in Seattle, at the Seattle Center, and how can you go wrong when you have a Space Needle?

Accounting for Free Markets: the Euphemism

Via Joe Gregorio, on petroleum, gas mileage, and free markets...

Moody-Stuart, who is currently chairman of the mining group Anglo American, says he is a great fan of the free market, "but like most things, they have a failing. Without regulation to channel their power, markets will not deliver things which are of no immediate benefit to the individual making his or her choice, even though they may be beneficial to society."

The problem is worse than whether a free market can work here or not. The problem is "free market" is a ghastly oversimplification for this situation. A truly free market would make obvious the true cost of gasoline. This would mean the cost would account for all the military, all the lobbying, etc. that's involved in that industry.

"Free market" is often a euphemism spouted by the primary beneficiaries of the market itself, which in fact is highly controlled. People have been moving many, large levers behind the scenes of Big Oil for well over a century. That market is *anything* but free.

Not that I disagree with the whole "free market fairy" concept -- any significantly valuable market is going to have a lot of powerful operators using every hidden lever they can get their hands on. This is called "working the system". People will always attempt to "work" to their advantage whatever system confronts them, no matter how large or small that system. Some people are better than others at doing so, and then using that leverage to continue on. Some people have very low ethics in deciding what levers to pull.

Welcome to planet earth.

Microsoft!

This could turn out to be the best thing to happen to Google in five years.

Yeah, but otherwise the browser is a great platform for applications...

(via James Robertson, who could tell you about Seaside)

News from the Blackberry

From Joe Wilcox at Microsoft Watch...

"Forrester expects at least half of the 42 percent of enterprises that
say Web 2.0 is not on their priority list to add it by year's end."

Oh, and Ray Ozzie will be speaking in March.

2008.

Testing Trumps Design

Reaching back a year or so ago to a point Ralph Johnson made about practicing test-driven development. The big lesson: every little bit helps. You don't have to be perfect by any stretch, just stretch yourself more than where you are currently, and then a bit more...

For some years, I've taught a software engineering course that used both XP and RUP. Students had group projects, and half were XP and half were RUP. The projects are not run in as controlled a fashion as those in the paper, so perhaps I am missing something. However, in general I have not seen a big difference in results between the two. The main indicator of success is previous experience. Groups that have a couple of experienced people do better than those without.

However, one group of people consistently did better with XP than with RUP. This is a group with little programming experience. RUP did not work for them because they had nobody who could act like an architect. They would produce UML diagrams, but they were always wrong, so it was a waste of time. However, when they followed XP, they produced (usually poor) working code that had regression tests. Eventually they started to refactor it. Because they always kept their tests working, they always made progress, even if it was slow. XP worked much better for these groups than it did for average groups.

I'd take a reasonable, automated test suite over a great design any day. The one can lead to the other more easily. I'm not sure I'd recognize a "perfect" test suite if it hit me in the face, but a team that tries to improve it's testing has the best chance of success.

Javascript and Smalltalk

Joe Gregorio writes...

Alan Kay had a vision for the web... Ten years later we have the Lively Kernel...

JavaScript, it's the new Smalltalk.

I can go along with the Javascript part. There are aspects of Javascript I like more than Smalltalk, and the other parts are not so bad either. Javascript does have some cruft that makes it more complicated than Smalltalk, oh well.

I can go along with the HTML and SVG parts too. But the Lively Kernel part has a way to go. Too bad it's built on the browser, which is a much worse medium than the Smalltalk image. Hopefully that will improve, but browser progress as an application development medium is depressing.

The web makes a helluva "image file". Morphic is a great UI approach, but the browser is not ready to fully realize it.

The current browser is not even up to the level of Smalltalk-72 running on a DG Nova, pushing that analogy a little over the top. It's kind of sad to see Dan Ingalls mucking around with LivelyKernel when he could be working on such a better medium. It's kind of like when Ward Cunningham was at Microsoft and they just couldn't figure out what a creative mind they had available to tap.

Wiimote and Your Computer

(Via James Robertson) Cool, cheap touch controls on your computer using a wiimote. Good YouTube videos of what's possible now for the "maker" types, and should be fairly cheap out of the box for everyone else in the near future.

And now for something...

Steve Dekorte on another of my favorite subjects...

If you're looking languages or concurrency tools that will scale to the high core count desktop machines of the near future, I wouldn't put stock in MISD oriented solutions such as transactional memory or fancy functional programming compiler techniques. Shared memory systems simply won't survive the exponential rise in core counts.

Deeper Dynamics

Oh dear. This can't go on for long. I've been in the middle of these muddles before. Say something like, "Dynamic languages have great tools and can be used to build large, long-lived systems." And you will find yourself on the receiving end of questions like the following which someone just sent me.

I can indulge these for a minute or two. The problem is they are rarely asked by open-minded people. Usually they come from the statically convinced, who just need to find a way to catch you, to allow themselves the comfort that their static languages and their tools could be the only possible methods for large systems. They need to assure themselves of being on the righteous path, and you, poor fellow, are one step away drowning in Styx.

Before I proceed let me state: I know large systems can be built statically. I've done it in more than one language. I've also done it dynamically in more than one dynamic language. Have you? Have you been part of failures in each? Yes, I have. They almost never have anything to do with the languages and the tools themselves.

That said, I prefer dynamic languages because they ease the pain when used well. On the other hand nothing eases the pain of bad projects.

"What happens in Smalltalk projects where you grow larger? Where you want to divide a system up into components, and some of those components don't exist yet?"

Well, what happens in other projects? You have a need; you have an idea to meet the need; you try it; you grow it. I guess I don't understand the question.

"How are interfaces between library and user code specified?"

Interface notations have been attempted for various dynamic languages, including Smalltalk and Lisp. There is a reason they've not really caught on, yet large systems are still developed in these languages.

Semi-formal notations, tests and other examples, documentation, good tools -- these all contribute to success. But if you are looking for some kind of a "static declaration" as they saving mechanism for large systems -- it may work for you, but I doubt you can prove they are necessary and sufficient. I have counter-examples.

"Can such interfaces be browsed in the same way if the code can't be called?"

Your question betrays your bias. "There must be some such mechanism as the one I am used to," he thinks.

Most large systems developed with dynamic languages have not used the kind of mechanism you seek. Look elsewhere.

Can we go back to arguing over WSDL vs. REST now? Face it. The history of programming is one of gradual adoption of dynamic mechanisms.

Dynamic Languages: Should the Tools Suck?

A comment to my previous "oh my god" post just came through. It is the typical "static typing is good because it enables good tool support like refactoring and finding methods". Mmmm.

First let me agree that tool support for Python and Ruby truly suck. They do. I still love the languages and would use them in a heartbeat before any static language. But the tool support sucks. But not because the languages are "dynamic".

To make my point a bit more concrete, here is a snapshot of a Smalltalk image I just created...

The aqua colored window is a "method finder". I entered the text "#(1, 2, 3, 4). 2. true", which is an array of numbers, the number 2, and the boolean true.

The method finder then found five methods that when applied to the array and the number, return true. It found those for me by considering potential candidate messages and executing the method code for me. The matches are listed in that window's pane just below the text I entered.

I then selected from that list the method "windowReqNewLabel:" and the method finder brought up the green window which is a system browser. It set the browser to display the code for that selected method. (By the way the browser has a complete kick-ass refactoring engine available. But that's for another post maybe.)

In the system browser for that method I clicked on "senders". This brought up a menu consisting of that method's selector as well as all the other message selectors that are used in that method. I chose that method's selector because I hypothetically want to know all the places this application sends that message. (Otherwise I could find all the senders of messages that this method also sends.)

Doing so brought up the front, light blue, window. This is the "Senders of windowReqNewLabel:" window. This window lists the four places that message is sent in this application. You can see I have selected the third occurrence, and the code for that sender is displayed in the bottom pane of that window.

As you can see, tools for dynamic languages only suck when they have not been implemented. On the other hand everything you see in this screenshot has been in Smalltalk for decades.

Decades of better tools than you can find for Java, C, Scala, you name your favorite statically typed language.

I am sick and tired of people whining about how dynamic languages cannot support useful tools. They can, and they are *better* than yours. This example is the tip of an iceberg I just captured in a matter of seconds. That first method finder "search by example" completes in a split second.

Can your static tools do that? Maybe, which is why I won't go around claiming that they can't, because I am ignorant of that, just as you are ignorant of what dynamic language tools can do.

Begin Update

An anonymous commentator points out that one of us is being obtuse. How can a dynamic language possibly refactor the name of a message when there are three implementations of that method but only one of them should be renamed.

I will quote from the documentation for the very first usable refactoring tool ever. (This very first one ever was written in Smalltalk and is the one that still is used by Smalltalkers.) The quote...

One of the key features of Smalltalk that allows us to perform the sorts of analyses and source code transformations necessary for refactoring is that it has reflective facilities, that is, the ability for the language to examine and modify its own structures. For instance, the Refactoring Browser examines the context stacks of executing programs to determine runtime properties such as callers of a particular method or cardinality relationships between components and aggregates. Refactoring can be performed in the absence of reflective facilities, but then requires a separate, metalanguage that can be used to manipulate the original program. Having a single language simplifies the process...

Some preconditions of refactorings are not simple to compute from the static program text. With Smalltalk's dynamic typing, determining the type of a particular variable can be extremely difficult and time consuming. Moreover, there are analyses that defy static analysis in any language, such as cardinality relationships between objects.

Since, performing these types of analysis statically is difficult or impossible, the Refactoring Browser computes some of the preconditions by performing the analysis dynamically rather than statically. By observing the actual values that occur, the Refactoring Browser can perform correct refactorings. As an example, consider the rename method refactoring. To correctly rename a method, all calls to that method must be renamed. This is difficult in an environment that uses polymorphism to the extent that Smalltalk does. Smalltalk also allows dynamically created messages to be sent via the perform: message. If an application uses this approach, any automatic renaming process has the potential of failure. Under these conditions, guaranteeing the safety of a rename is impossible.

The Refactoring Browser uses method wrappers to collect runtime information. These wrappers are activated when the wrapped method is called and when it returns. The wrapper can execute an arbitrary block of Smalltalk code. To perform the rename method refactoring dynamically, the Refactoring Browser renames the initial method and then puts a method wrapper on the original method. As the program runs, the wrapper detects sites that call the original method. Whenever a call to the old method is detected, the method wrapper suspends execution of the program, goes up the call stack to the sender and changes the source code to refer to the new, renamed method. Therefore, as the program is exercised, it converges towards a correctly refactored program.

The major drawback to this type of refactoring is that the refactoring is only as good as your test suite.

Except that good test suites are never a major drawback. They are actually the only thing that can save your application over time, dynamic or static.

And so you see good refactoring tools, good (dynamic) languages, and other good things go hand-in-hand with good tests and good test coverage. Working incrementally helps you make only small changes at a time, testing well points out just the few things you just broke, good coverage makes sure you find everything, and refactoring tools introspecting the code that your good tests cover help automate those changes and bring them up to a higher level of expression for you.

No single tool or practice is in itself sufficient. They work together for the good of the whole. Should I assume your refactoring tool for your static language works so well as this? I believe method wrappers, etc. take a fair bit more machinery and end up not so expressive in static languages. At best they require more code than with dynamic languages.

You do not have to compromise your very simple language just so your tool can work. Dynamic languages and good tools work for you. Static declarations -- they appear to work for you until they don't. They're not a compromise, they are a bloody surrender, somewhat quoting Guy Steele from some long ago Lisp memo I don't fully recall.

Oh my god here I go again. I know you love your static languages and type theories. Good on you. I don't.

End Update

Experiment, Programmers and Engineers

This summary is not available. Please click here to view the post.

Math and Software Development

Via Reg Braithwaite, Ravi Mohan questions the arrogance of software developers calling themselves "engineers". Ravi's argument is they do not do math...

If you don’t use mathematics in your day to day work, you aren’t an engineer...

The title “Software Engineer” is (most of the time) a particularly deceptive one.

Certainly "logic" is a form of math and is a central aspect of software development.

Algorithms.

Processes.

Cost / benefit trade-off analysis and assessment, i.e. kinds of "forecasting", "budgeting", "accounting".

These are all mathematical, and significant aspects of software development.

Not that I care. I'm "just a programmer".

Oh my god here I go again

I quit participating in "static language" vs. "dynamic language" debates for the most part a few years ago. A few pieces have been getting my goat lately, though. There was that piece on "gradual typing" (I won't find it for you).

Now this linked from Lambda the Ultimate...

When scripts in untyped languages grow into large programs, maintaining them becomes difficult. A lack of types in typical scripting languages means that programmers must (re)discover critical pieces of design information every time they wish to change a program. This analysis step both slows down the maintenance process and may even introduce mistakes due to the violation of undiscovered invariants. This paper presents Typed Scheme...

The problem is I've seen large programs in all kinds of languages become difficult. Two things can preserve large programs... many good tests, and familiarity. I've seen that work.

Maybe good type systems can help maintain large programs. I don't know, since I've never participated in developing a large program with a *good* type system.

I do know that a good type system is *not necessary* for maintaining large programs. I suspect it would not help as much as many good tests and familiarity.

I await the comments showing up in my inbox. No promises to publish them.

Premature Capitulation

The new thing is "gradual typing"...

I always like to know the origin of a name. In this case, gradual refers to the idea that in a gradually typed language, you can gradually add type annotations to what is otherwise a dynamically typed program.

Sounds like you need better tools (they pretty much stink for Python and Ruby, try Smalltalk or Lisp) and perhaps a better compiler (ditto). Doubtful you need "gradual typing".

More on "stable layers" regarding efficient dynamic languages

Another data point on the idea of languages implemented in layers, consider this... ("why you should care as a programmer about the study of programming languages")... (pdf)... Dan Friedman's talk at UNAM, Mexico, on April 27, 2001... quoting his student Jonathan Sobel:

“That was the amazing part: I had produced a program that I could not have written, and in any case would not have wanted to write.”

Sobel wrote a program in a simple, direct style using Scheme and then applied several transformations, from Scheme to Scheme, each a fairly simple transformation. The final transformation was ready to be translated into a C program using another simple transformation.

The efficient C program does not have to be maintained. Just the original, simple Scheme program. Probably programming in C itself (or Java or Scala) is a kind of premature optimization.

On the concept of a "stable layer"

Ola Bini writes about the concept of a "stable layer" for language implementations, especially (I gather) where the "stable layer" is underneath a "dynamic language" used by application programmers. I am using quotes because these terms do not have to my knowledge widely accepted, rigorous definitions. That's OK, coming to a common understanding of these terms is part of it.

I interpret his use of "stable layer" as perhaps an "implementation layer", what is used to implement the language one truly wants to use. Maybe they are the same language. If you want to program in Java or in Scala, then it would seem in these terms, the stable and the "top" (?) layer are one and the same.

CPython, CRuby, Jython, and JRuby have as "stable layers": C, C, Java, and Java, respectively. But this arrangement is not necessary for dynamic languages generally. My understanding is PyPy is implemented in Python and Rubinous is implemented in Ruby. (Excuse me if I messed that up a bit.)

As a couple of other data points on suitable languages for a "stable layer" consider the implementations of Squeak Smalltalk and Gambit Scheme. Both of these languages are implemented in subsets of themselves. Both of those subsets compile to C primarily to avoid implementing special translators for each instruction set architecture they run on.

Those subsets (of Smalltalk and Scheme) are essentially as efficient as C. But they are each "dynamic languages". And the stability of both implementations can be measured now in *decades*, regarding how long they've been around, used in "real" systems. Gambit in particular having very good performance because *any* application code can be compiled this way, and Squeak could take this same approach if desired.

BASIC and the IBM 5100

Douglas Crockford's humor...

Before video terminals became cheap, Basic was usually accessed with Model 33 Teletype machines that could only print a subset of the ASCII character set. They couldn't handle lower case or some of the special characters. I called that subset Half-ASCII.

My first paid programming gig was in BASIC on an IBM 5100.

See that red switch between the tiny screen and the streaming tape drive? That determined whether you were programming in BASIC or APL!

Tivo and XMPP

(via stpeter) Tivo wanted to send messages from their servers to your Tivo and so they are doing so with XMPP.

The Tivo Lovers Blog says...

Yep, TiVo is basically using instant messaging for real-time communication. Now when the TiVo server has a new recording to schedule, it will IM the TiVo to tell it. Or if there is a download to pull, it will IM the TiVo to tell it to do so. This is a much more efficient system and it eliminates latency. It is really a clever idea.

"Shoes is a monkey on LSD"

Gregory Brown and Brad Ediger write about some Rubyware from why the lucky stiff, et al....

If Merb is a paragon of professionalism and class, Shoes is a monkey on LSD. Shoes, by why the lucky stiff, is an incredibly compact cross-platform GUI toolkit for Ruby, but it looks nothing like the other cross-platform toolkits out there. For one thing, it is lightweight. Shoes lets you build GUIs in Ruby whose code actually looks like Ruby, not XML or Java. Shoes is under heavy development right now, but it will eventually form the basis for the new Hackety Hack, _why's programming environment for kids.

So, what are a web framework and a GUI framework doing together, you might ask? We are going to build a pastebin as a repository for our own code snippets and pieces of text we want to save. We'll build a GUI frontend using Shoes, and connect it to a Merb backend that will handle the database...

I love that by page three of why's Shoes book, the crab is toasting us on our adventure with a glass of champagne. I hope I am not giving anything away with that.

Should an SMTP (or HTTP) server require an intermediary?

Michael Nygard tilts a little too much toward the enterprisey in his recent post on outbound email.

Some of his complaints seem to be about the JavaMail API specifically. e.g. "one connection per message" is not a limitation of SMTP. I suppose his concern is that the JavaMail API has a lot of setup and teardown cost for each message.

But regarding his concern about holding up request-handling threads and recovering mail messages when connecting to the mail transfer agent fails... he goes on to recommend using a message bus/queue like JMS or an ESB, for goodness sake, in *addition* to the SMTP server. An intermediary for the intermediary??

Rather than doing running some other kind of server, why not make sure you're running a fairly stable SMTP server? He writes...

...keep in mind that SMTP servers aren't always 100% reliable...

But I thought the point of his book is that *no* system is always 100% reliable. (btw is there such a thing as "sometimes 100% reliable"? :^)

And so even with a bus/queue or an ESB in between your system and the SMTP server there are still opportunities to fail. In fact, there seem to be *more* opportunities to fail.

Another quote from Michael...

Yes, you can log an exception when your connection to the MTA fails. But that only lets the administrator know that some amount of mail failed. It doesn't say what the mail was!

But if you can log the exception, certainly you can log the message that would have been sent, and then either automatically or "manually" resend it.

How would you handle this situation?

This seems like a general problem, and not specific to SMTP. For example, let's say instead of SMTP you'd like to publish a message to a collection on an Atompub HTTP service. Would you recommend a bus/queue or an ESB as a stepping stone between your system and the HTTP server?

How is this similar or dissimilar to the SMPT scenario?

Push Here

A handy quote...

The time taken to learn things isn't as important as how useful they are once learned.

-Ricky Clarkson

Io Buzzzzing

I guess people are out looking for that new language they're going to learn in 2008, or something.

I like seeing the Io language get some attention. I've been reading Steve Dekorte's blog for a long time because I enjoy his posts on architecture and design, generally, as well as his policital/economical/sociological posts.

I only played with Io once, for an afternoon, several years ago with a friend who was getting into it at the time. He walked me through the basics of Io's reflection and "meta" capabilities as I typed his instructions into the repl.

What strikes me about all these aspects of Steve is the common theme of minimalism. Io certainly holds the "minimalist" point on the range of language options.

The Processing Around Restful Web Services

JJ Dubray asks and Stefan Tilkov responds...

Is your solution also about implementing complex processes that are likely to change on sophisticated business objects such as an insurance policy or claim?

I’ll happily agree that if you’re after a BPM-style solution, REST is not the first choice... clearly there’s no equivalent of BPEL for plain RESTful HTTP.

I think this is another false dichotomy. The most complicated business process could still incorporate GET, POST, PUT. Why on earth not?

As it happens, I work on Restful Web Services for insurance, including policies and claims. So if JJ or anyone wants to work out some details, that collaboration could be educational for both of us.

Update: I created a wiki for this collaboration.

Scala Buzzzzings

All the buzz about Scala again, suddenly.

Via Stefan Tilkov, from Doug Pardee...

I’d long felt that combining a powerful static type system with functional coding would be almost ideal. Now that it’s been done—and done quite well—I find that it’s “not all that” after all.

The documentation is weak... A book is in the works... I haven’t read any of it, but I don’t have much hope...

I’ve already discussed the complexity of its type system... But with Scala’s abstract types, a variable or parameter or whatever can be declared to be of a type that is defined somewhere in the inheritance tree… and not just superclasses, it could be defined in a subclass as well, including the possibility of an anonymous subclass. And since Scala allows classes to be defined darned near anywhere, tracking down all of the possibilities is not fun...

def sum(l: List[int]): int = (0/:l){_+_} Isn’t that lovely?

I agree with Doug's complaint about weak, out of date information. But I have the current book draft and have read a good deal of it already. The book seems a good place to start to learn functional programming, especially for someone with Java or C# experience. There are some good introductory books for Haskell as well, and this books seems about as good. Unfortunately about twice the size as the Haskell books, though, as this book attempts to explain most of the Scala language, while the Haskell introductions I've been through do not cover all of Haskell.

I've written several times already how I am at best unused to thinking with more modern function type systems. This dynamic language programmer is used to sending messages to objects and applying any number of any kind of argument to a function without a darn compiler insisting that it must have a valid type theory for what's bounding around in my head.

As for that stretch of Scala syntax above, I do think sometimes Scala, like Haskell, goes way over the top with its cute, concise notation. I can handle this ok, since I am used to the concepts like "fold left" and "fold right". If your a Smalltalker, think "inject:into:". Then once those little bits of syntax can be translated into a familiar vocabulary, it's not so bad. But not great for beginners, I'd agree.

DRM Disasters: The Song Remains The Same

Jimmy Guterman describes a DRM / installation "catastrophe" with Adobe's Creative Suite on Windows. (I'm not sure if the problem is with MSFT, or Adobe, or both).

He concludes...

Yet another case in which copy protection and productivity don't go hand-in-hand. The problem is solved, but the great error message lives on.

Reading this brought up memories of the security dongle fiascos back in the 1980s.

The Good Examples Would Help

Update: Stu has a good response in the comments.

Stu says there are some good things in ws-* that could or should be translated into the Rest world. That may be true, but I cannot tell just from his descriptions. The missing pieces that would help are really the missing pieces that had led to my doubts in ws-* all along.

Those pieces are really good examples.

I can see that WS-Security, WS-Trust, WS-SecureConversation, and Security Assertions Markup Language specify a *lot* about security and trust, very little of which (admittedly) I do not have much of a grasp of. Why do I need them though? Are there really clear examples of real world usage that I can follow to understand why these were used over other capabilities, and how they were used in a complete path to production, and evolution in production over time, and what else was needed to provide all the security and trust of the complete solution?

Multiply that by the rest of Stu's list: transactions, "coordination", "choreography", BPEL. That's a whole lot of stuff that needs explaining. I am a reasonably bright person. I can even curiously dig into things I don't understand and begin to make sense of them, when I put my mind to it.

Years after first being turned off of the seemingly impenetrable ws-* specs, and years after first (of many times) being told that as an application developer, I simply don't need to understand much of the specs because the vendors understand them on my behalf, I still cannot rationalize why I would need anything on Stu's list, let alone anything on the rest of the ws-* heap (because ws-* is way messier than a "stack" from my observations).

Lists of capabilities are insufficient for me. Maybe that's just me, but oh well, it still frustrates me because I feel as a fairly experienced developer I keep getting the sense that I *should* at least understand these things better even if I am still opposed to them.

Good examples would help me. I can find all kinds of them in the Rest world. I can figure out how to write my own without too much time or effort. Why can I not say the same thing about even Stu's abbreviated list?

That Hardware / Software Analogy Applied to Closed / Open Source Software: Bzzt from this vantage point

Via Stefan Tilkov, some observations on closed vs. open source software from Jaron Lanier...

A closed-software team is a human construction that can tie down enough variables so that software becomes just a little more like a hardware chip—and note that chips, the most encapsulated objects made by humans, get better and better following an exponential pattern of improvement known as Moore’s law.

Yeah, and we're seeing so many innovations occurring on closed software today, compared to open software, it is frightening. (insert "sarcastic" emoticon here)

Even the iPhone is based on an open source Unix. I don't think Jaron is being fair to compare the iPhone to Linux. Maybe compare the iPhone to Android-based systems in a few years. That would seem fair. Even then I would expect some of them to have various "closed" aspects.

And let's not take that hardware analogy too far. I think it was Edsger Dijkstra who observed about 30 years ago or so that CPUs iterate over the same design but different process technologies. That's more or less true still, but where it is less true today what also is true is that the design complexity and design approaches are more like software than they are like the hardware of yesterday.

Penultimately, let me note from personal experience that with either closed or open source, the challenges of getting "better and better following an exponential pattern of improvement" in software is damn hard. I've not seen many systems that do that well, but the best candidates at this point seem to be open in various ways.

Finally Jaron's entire debate does not seem worth having. Questioning whether closed or open source is "the right" model is a false dichotomy, a source of info-pollution. The answer is clearly, from observation, somewhere in the middle. Sometimes a little this way, other times a little that way.

CouchDB Finds a Home

Damien Katz is going to IBM to work on CouchDB full time. CouchDB is now intended to become an Apache project.

That's a great development. Good for IBM picking it up, too.

Not knowing CouchDB too deeply, but the basic gist of it, I wonder how well the approach would translate into a cloud-based system, not too unlike Simple DB?

The database world is suddenly (to me at least) looking significantly different than even a year ago, given CouchDB, Simple DB, and the research at MIT on C-Store, H-Store, etc.

Last year at this time my co-workers and I were diving into a serious look at Jini/Javaspaces for some intra-data-center messaging and persistence needs. The community was what we decided was the biggest lack, but the complexity was high on the list, and the Java-centric nature was ok, but not... (alright Jini community... I know it can be used with/from other non-Java systems...)

Given systems like those above, I wonder which direction we would have favored back then.

Scalagain

Respectable people (1, 2) are still looking at the Scala programming language for the JVM. I took a reasonably deep dive on it back in the summer. As I wrote back then, I've tinkered back in the day with SML, Haskell (and a bit of other functional languages like Clean). But being significantly deeper into "dynamic" languages like Lisp, Smalltalk, and now somewhat Python and Ruby, I never really got over the curve of "thinking in types".

There is a difference between "type-driven programming" and "test-driven" (or in Lisp, repl-driven, or in Smalltalk, workspace-driven). I'm not sure what "type-driven" feels like in real settings. And so after a week of kicking Scala's tires off and on, I decided it's nice but not enough in my wheelhouse to get excited about.

Reading folks blogging about their own Scaladventures and reading that a new book (400+ pages!! That could be good or bad) is available (currently as a draft), I purchased the book and will tinker again, at least for a few more days, with Scala.

I just got my access key to Amazon's Simple DB, so maybe I'll dream up something interesting there with Scala. Integrating the dynamic, semi-structured nature of systems like Simple DB with the compile-time rich type checking of a language like Scala is an ongoing area of, em, interest. Dynamic languages never get in my way with things like Simple DB.

MSFT and the Clouds

Bob Warfield writes about Microsoft and their "rift with the web"...

While I am again a whole-hearted advocate, giving your software over to a hosted environment and all of its attendant API’s (RESTful or not) is a big step towards lock-in, no matter how you look at it. This is again an area where Microsoft’s old school monopolist behavior won’t serve it well. There will be fear, perhaps unreasonable, that Microsoft will take unfair advantage if handed the keys to your kingdom by hosting on their cloud infrastructure. The problem is that they’ve failed to conduct themselves as the Swiss do in matters of banking. They are voracious competitors and seemingly always will be. It isn’t enough for them to win, others must lose.

Initial reaction: avoiding, or at least reducing the amount of, lock-in with any service provider will be important. My additional reaction to a MSFT service would be, yeah, especially with MSFT. But that's probably largely irrational. If the service is something like the ones we've seen so far, with comparable pricing, who cares if it is MSFT as the provider?

This is just an opportunity to bring a couple of related thoughts that have been bouncing around my head lately...

One is that the pricing of Amazon's services, and their payment services, really usurp what I think has been a long-held dream at MSFT to institute some kind of micro-transactional payment mechanism. Apparently now they have a model to emulate, so we should start seeing these kinds of services from Microsoft.

But wait, isn't one of their recent services a sort of "Biztalk in the Clouds"? Maybe they will completely rift with the web. Astoria is also referred to as "ADO.Net Data Services" but they say it has a Rest api and can be used from AJAX front ends. I assume this means it can be used by any HTTP client, but the documentation is Word and I dislike having to download that just to find this out.

But all this leads to another question: can anyone *other* than MSFT afford to write a cloud service using MSFT products? I assume Amazon and others use primarily open source tools and products, and when they do not then they are still Unix-based.

MSFT obviously has no licensing issues around building a large cloud based on windows and dotnet. Would anyone else choose to do so?

Big IT and the Clouds

Tying a couple folks blog posts together. James Robertson on SimpleDB, etc....

Bypassing IT just went from being somewhat painful to being a line item in your budget.

And Bill de hÓra on pretty much the same subject...

This seems to me like the desktop computer revolution of the eighties or the email/web of the ninties - consumer tech that infects the enterprise.

Yep. Like the Apple II and Viscalc just showing up in the office because they could get a job done. Like the Mosaic browser just showing up in the office because... oh, wait, that was for porn. But like the sales dept signing their own deal with Saleforce because someone's IT could not get a job done.

I suspect large enterprises will find their way to the clouds in fits and starts, mostly from outsiders being able to get a job done and that job can be paid for on a meter. At some point there will be pressure from Sales, or from Marketing, or even Accounting, to integrate with the cloud.

Certainly by that time CEOs or someone should be asking IT why their costs are so damn high and progress so awfully slow compared to all those new systems that run in the clouds. In 2007 we could finally see that ws-* will end sooner rather than later. Now we are starting to see that the big IT data center will also be shorter lived than at least I imagined a year or two ago. It won't be soon, and it won't be painless, but at least now we have the end in our sights and we can start taking aim for that goal.

Sociology

Quote from someone named Jerry back in 2003...

It is for sure a VHS world when people flock to XML, XML namespaces, XML parsing, and DOM and then flinch at a parenthesis.

No, this is not related to any predictions for 2008.

Milemeter: Buy Your Insurance By the Mile

Brady Forrest writes about Milemeter, a new insurance startup. First, the words "insurance" and "startup" seem to form an oxymoron. But I don't see them listed, so I read on.

An interesting aspect of insurance to me is that the "product" is completely information-based (most of my career has been in software for the design and manufacture of tangible, electronic products; I currently work in the insurance industry). Unlike some other information-based products in financial services, though, insurance seems to be more heavily regulated, and the period between events is long (days, weeks, months). The most successful sale of a product (i.e. no claims, no changes to the coverage) has one significant event a year, (re-)issue, and some minor billing/payment events.

Compare this to financial portfolios, which can be wide ranging and nearly unregulated (cf. the current sub-prime crisis and obscure bundling as supposedly secure instruments), and the period between trading events can be measured in sub-seconds and/or distributed around worldwide markets.

On the one hand there does not seem to be a lot of pressure to change the way information technology works in insurance. On the other hand all these aspects seem to open up new opportunities for change.

Brady observes...

As you may have guessed, they are built on AWS (you can see a video discussing their usage on their blog). They are also using Ruby on Rails with Postgres...

This is what I want to see, large, black-box industries being taken down and made consumer-friendly. (Can the health system please be next?) I don't really know what I pay for with my current insurance, but with Milemeter I'll have a much better understanding.

The internet will sooner or later affect all these industries. (Amazingly much of the current B2B transactions in insurance takes place over proprietary networks, that is, when they are automated at all.)

The established insurance IT has to get its cost of change significantly lower. The best way to do this is to copy the way software is developed for the internet. As Milemeter demonstrates, this will come from the "outside" whether or not the "inside" is ready for it.

I had a chance to visit with Steve Loughran and some of his local friends, when Steve was in Oregon last week. We had a good talk about all these changes, where they are trending, and which kinds of organizations are doing what along those trend lines.

There is no doubt the cost of change is the limiting factor in established organizations from following those trends as aggressively as possible. The opportunities are there and the pressure to change will increase.

Chris Gay of Milemeter notes in the video, linked above, "Amazon Web Services is a pay-as-you-go infrastructure and Milemeter is a pay-as-you-go insurance provider". The ability to use nimble infrastructure(s) will aid the product itself to remain nimble.

Patterns of Change

Reg Braithwaite knows...

Of course I recommend reading the original. But may I add, please do not get sucked into arguing whether Design Patterns are good, or whether IDE refactorings really work, or any of the other technical points that are so much fun to rehash for the millionth time.

Instead, consider the cultural forces at work. Cultural problems cannot be solved with technology. If you are an advocate for change, ask yourself what sort of cultural change is needed, not what sort of technical problems need to be solved.

RubyCamp 2008 in Vancouver

This announcement came around...

RubyCamp 2008 in Vancouver on Saturday, January 26th.

RubyCamp is an one-day gathering for Rubyists and Railers.

When and Where:

WorkSpace in downtown Vancouver, B.C., Canada
January 26th, 2008 from 9:00 to 5:00

We've been getting into the JRuby here, and a couple days in Vancouver might be nice. This may go on the list of game-time decisions.

Where's the Beef?

Subbu Allamaraju shows that a significantly more RESTful api than Amazon's SimpleDB is not so difficult to conceive.

The SimpleDB API is neither resource oriented nor HTTP friendly. Having said that, how should such an API be designed in a resource-oriented manner? Here is my take, a version-0.1 of a RESTful SimpleDB. In the design below, I tried to keep the semantics of this version as close as possible to the official SimpleDB API

The Wayback Machine: Internet Archive

James Robertson and Robert Scoble lament the loss of one's data one old sites they themselves do not own. But the Internet Archive's Wayback Machine does a reasonable job of rescuing this. (Could wayback rescue your facebook if that were to disappear?)

e.g. my old Radio Weblog is no longer around: http://radio.weblogs.com/0100812/

However the Wayback Machine can find it: http://web.archive.org/web/*/http://radio.weblogs.com/0100812/

Note that the wayback machine continues to archive the "not found" page. Actually whatever service is running the old Radio is returning a 300, "multiple choices", rather than a 404, "not found".

So using the archive you can actually see when my Radio went off the air, sometime between December 7, 2003 and April 5, 2004.

It looks like the archive has not been able to retrieve at least all the longer essays, which Radio stored under a "story" URL.

Also without exhaustively searching the archive, the earliest criticism of WS-* that I have found of mine is August 25, 2002. See "Protocols, Documents, and Transactions"

Thud

How did Amazon allow this design out into the wild? Apparently they've already had a private beta period with non-Amazon developers. No one suspected this use of HTTP GET to be a poor choice?

It's 2007. There are all kinds of basic REST reference and getting started materials. Didn't the whole net go through that "Google is pre-caching everything it can GET" episode a couple years ago. That's the episode where the Rails folks learned that GET should not have side effects.

Wow. Hopefully Amazon will POST a fix asap.

Subbu Allamaraju calls this "SOAPy REST". Didn't even the SOAP folks learn to avoid these kinds of GETs a couple years ago?

IBM published an article recently on using HTTP to access DB2 and IDS. The URLs they provide need improvement, embedding the operation in the URL. This despite that they do seem to be distinguishing between GET and POST. From the information at that site and another bit I could get behind their developerWorks login, it's not clear to me yet whether they allow the use of GET with a URL that includes a destructive operation.

Maybe the RESTful Web Services book will begin to turn these things around. But that Amazon did this in 2007 is a bit of a shock.

Update: Steve Loughran nominates SimpleDB for a new award...

I nominate it for the 2007 Restless awards, in the much contested category of

"things that claim to be RESTful but do side effects in their GETs" along with the ever popular "SOAP endpoint in disguise" category

I know this mailing list has not, historically, had such awards, but now is as good a time to start as any.

Yeah, maybe there should be annual (or more frequently, on Internet-time, awards for "RESTless" and "RESTful" web services that degrade or enhance, RESTpectfully, the web.

Column-Oriented RDF Storage

A couple years ago it occured to me that a database with column-oriented storage, such as Sybase IQ, might make a reasonable database for storing RDF data, where a star schema can be seen as a way to represent related tuples together.

Now it turns out some folks have been working on such a thing... C-Store is an open source, column-oriented databse. The paper "Scalable Semantic Web Data Management Using Vertical Partitioning" (pdf) discusses using C-Store for RDF...

Efficient management of RDF data is an important factor in realizing the Semantic Web vision. Performance and scalability issues are becoming increasingly pressing as Semantic Web technology is applied to real-world applications. In this paper, we examine the reasons why current data management solutions for RDF data scale poorly, and explore the fundamental scalability limitations of these approaches. We review the state of the art for improving performance for RDF databases and consider a recent suggestion, “property tables.” We then discuss practically and empirically why this solution has undesirable features. As an improvement, we propose an alternative solution: vertically partitioning the RDF data. We compare the performance of vertical partitioning with prior art on queries generated by a Web-based RDF browser over a large-scale (more than 50 million triples) catalog of library data. Our results show that a vertical partitioned schema achieves similar performance to the property table technique while being much simpler to design. Further, if a column-oriented DBMS (a database architected specially for the vertically partitioned case) is used instead of a row-oriented DBMS, another order of magnitude performance improvement is observed, with query times dropping from minutes to several seconds.

Release It Again

Pete Lacey -- what he said, about Release It and Michael Nygard's blog.

If you find yourself in a panic over some centralized resource, wonder if the full costs are accounted for. What alternatives might exist for decentralizing, and how do the cost/benfits really add up over time?

The cost of operations is dropping. The cost of change is still too high for many to take advantage of that though. By the time we can get our systems onto more budget-friendly architectures... well, I guess heading in that direction puts you on the path toward even better things.

Depending on the business, if one extrapolates out from one's current position, through the point where more open/available/scalable systems are in use... well, then is this evidence that for most of us, our ultimate position is out in "software as a service/utility" land? Exactly who should be in the data center business five to ten years from now?

Another data point shows up. (Oh, and speaking of cost of change, it's in Erlang and it provides a REST api available from any language.)

From the SimpleDB FAQ...

Q: Where is my data stored?

Amazon SimpleDB stores your data in our multiple data centers in the United States. We anticipate adding other geographies over time.

Q: Does Amazon store its own data in Amazon SimpleDB?

Yes. Developers within Amazon use Amazon SimpleDB for a wide variety of projects. Many of these projects use Amazon SimpleDB as their authoritative data and query store and rely on it for business-critical operations.

So, "business-critical" seems kind of reliable. Not just scaled out databases, but scaled out data centers. Those are even more expensive to operate on your own.

Yahoo Flex Skin, Other Flex News

Among other recent Flash/Flex/Air/Adobe news, Yahoo released an open source skin for Flex.

Elsewhere new versions of Flash/Flex/Air from Adobe have been released and in some cases newly opened up.

I don't feel compelled to use the Flex Data Services, but for those who do, or when I do, today's news should be encouraging. You can use their open source implementation, or use their open specification for an alternative, e.g. in some other non-JVM language.

Embedded Lightness

I was out with a long-time friend and former co-worker last night. He's been working the last year or so on software for managing blade servers. Technically this is an "embedded system". Some veteran embedded system developers on his team have been learning just how much their field has changed.

Essentially the system consists of linux, lighttpd, php, and sqlite. Where the veterans were set on writing a lot of C and custom protocols for handling events in the server (blades coming and going, for example), they've been able to do everything using the web's architecture. They wrote some C to enhance the php interface to snmp.

They had some pre-existing C code that has a memory problem after some number of days. No problem, they just kill and restart the process well before that. When that code becomes the most important problem, they'll look at it.

Job Opportunity

Mike's got a job posting up at his new gig. You'd be working with a smart, funny, curious (in the good sense!) project manager/developer, on an interesting problem, and a high-powered executive team.

You don't have to live in Portland, although why would you want to live anywhere else?

Safety Dance

Douglas Crockford at XML Conference 2007. You can dance if you want to...

The current browser implementations have problems because they share all of the information between the current sessions (problems with cookie stealing, replay attacks, and chrome changes). That’s the dangerous web 1.0. Now, we’re trying to intentionally mashing stuff up (which we’d always tried to prevent when unintentional).

Meet the new Reg...

Paul Fremantle on WSO2's registry work...

For a while I've been thinking that the SOA registry space has been a little overcomplicated...

So fundamentally the approach we have taken is to build a registry/repository based on REST concepts. And as we looked at the REST space, we kept noticing how close the Atom Publishing Protocol (APP) is to our needs, so we've made that the public remote API to access the repository. Of course, if you are just browsing the registry, you only need a browser - APP is mainly there to support updating resources.

Of course, using Atom and APP gives some really nice benefits too - like being able to subscribe a feed of new resources that meet your search criteria.

Glen Daniels was talking about this registry in the hallway at QCon a few weeks ago. Nice. You could definitely see Glen's gears turning during the sessions and discussions.

Reg

Some folks have been quoting Reg Braithwaite lately. (Steve Vinoski did just a bit ago.) Here's another...

I have never met someone who desperately wanted to be great but failed to be at least decent.

That's a good observation. And almost Goethe-like.

The name of this band is Talking Heads

I was poking around youtube for the Late Night with David Letterman show (the one back on NBC), the episode back around 1983-84, where the entire screen rotated clockwise over the course of the hour.

I've not found a clip yet. But I've been watching other clips, like this one...

Amazing. There was a time as an early 20-something when we *had* to watch Late Night and talk about it the next day. The current Letterman is more mild, but still bits like, "Is this anything" and "Will it float" provide glimpses back to the original show, which at the time seemed like "anything goes".

I wish I could find a clip of that episode with the screen rotating slowly over the hour.

Andy Kaufman and Elayne Boozler

Andy Kaufman and Elayne Boozler...

Boozler I think has gone under appreciated over the years. She keeps up with Kaufman here. Painfully hilarious improv.

"http is like air"

Quote of the day from an IBM exec on the importance they are placing on rest, http, atom being everywhere.

And Then There Were Three... Then Five Again

Mike? He gone.

He formed the band, a little over a year ago, chartered to launch an open source project within a large "vertical". That was four charters, a CIO, one viking, and Joanne ago.

Don't ask. That's the nature of large "verticals" and being a small team, unlike any other, weaving in and out of the established structures. We got pretty good at writing charters on the wiki.

But in between we've done some pretty good work across a couple tables of old Dells running the Ubuntu server. And we're still being asked to be creative, agile, and infectious, which is not often the nature of large "verticals".

This fourth charter could stick. We're back to a full complement of five. And a real budget. And simple tools for restful web services.

Good luck, Mike. You made a lot of good things happen and made them a lot of fun. And took more than one for the team. Now your original charter has come around... nice.

Bird of prey.

Learning Scalable Internet Services

Via Planet Trapexit, and this interesting post from RightScale on the network performance of Amazon EC2 and S3, there is this fascinating page from a UC Santa Barbara course...

"CS290F - Scalable Internet Services"

The project consisted of building a transactional dynamic web site in Ruby on Rails and running on Amazon's Elastic Compute Cloud (EC2). Each site had to hold >100'000 database records that could be searched and explored, have user accounts, and include some form of transaction, such as a shopping cart check-out.

Each project then had to be deployed on multiple servers on EC2 and the groups had to use httperf to demonstrate that they could scale the performance of their site by running a front-end load balancer server, a database server, a memcached server, and up to 10 application servers. All this had to fit into a 10-week quarter, with none of the students knowing either Ruby or Rails at the outset!

More information from the instructor's RightScale blog. The wiki link there is broken, but at least pieces are still around, including the course material.