Too Big To Fail

A NY Times article on the credit crisis...

Over the last two decades, few industries have lobbied more ferociously or effectively than banks to get the government out of its business and to obtain freer rein for “financial innovation.”

But as losses from bad mortgages and mortgage-backed securities climb past $200 billion, talk among banking executives for an epic government rescue plan is suddenly coming into fashion...

Surprisingly, the normally free-market Bush administration has expressed interest.

Bush? Free-market? Right.

Here's the surprise: "free-market" is a euphemism. Rarely has it meant what you think it means. And that is intentional. This has nothing to do with whether or not a "free-market" would work. Nobody with any current leverage wants a free-market. Why should they? Just look at this very deal being worked out, right in the open. This isn't even a back-room deal.

It reminds me of the Bush tax cuts, more than anything else.

Nomenclature

From ...

Yahoo! recently launched what we believe is the worlds largest Apache Hadoop production application. The Yahoo! Search Webmap is a Hadoop application that runs on a more than 10,000 core Linux cluster and produces data that is now used in every Yahoo! Web search query.

Hadoop.net?

Seriously, what organization, other than Microsoft itself, would afford the equivalent running all Microsoft software?

The Difference Between Theory and Practice

Can we face the reality that "core" states have sponsored, continue to sponsor, and likely will always sponsor state terrorism?

There is very little moral high ground on this planet. What little there is can be occupied by a small number of individuals, and precious few organizations of any size.

Linux AIR

Finally some news on AIR for Linux. Adobe is looking for "pre-beta" evaluators. Didn't that used to be called "alpha"?

I can't wait for the beta!

Multicore Squeak

Via the Weekly Squeak, a multicore implementation of the Squeak VM. Apparently the work was done by Qwaq to improve the performance of running Croquet. Open source.

SQL

I barely escaped the Raleigh, NC, airport yesterday for a 6:08am flight. I'd never seen such long security lines so early, even on a Friday morning. But I made it, and there were others on my plane later than me. Just as the cold and rain was hitting NC, I'm back in OR on a sunny Saturday. We'll get more cold and rain, but we're definitely on the upswing into spring here.

Anyway...

Steve Dekorte writes about SQL...

The motivation for SQL queries being declarative was that you could just tell the database "what" you want... If something fails at it's primary reason for existence, is it time to reconsider it's existence?

I don't want to defend SQL too much -- even the declarative expressions have their faults -- but most of the blame for so much "imperative" SQL has to do with the database implementations themselves. The first 15-20 years of the relational database market were aimed at OLTP on hardware with significantly less capabilities than we have today. There was relatively little emphasis on "relations" in the OLAP sense.

And so SQL had to become more imperative, account for explicitly created temporary tables, and incorporate "stored procedures" to perform the explicit, step-by-step operations.

If you look at database systems like Teradata (which is a highly concurrent, shared-nothing architecture), or column-oriented databases like SybaseIQ or C-Store then you will find the ability to program SQL queries as originally intended, in a functional style. I wrote some SQL for Teradata ca. 2001-2003 and helped others improve their SQL (who had been using primarily SQL-Server and T-SQL) because I had a good bit of Lisp and functional programming experience.

Programming truly declarative SQL and the more common procedural SQL are very different experiences. But it is possible, if the underlying database architecture will support decent performance for declarative queries.

Bad Design

DeveloperWorks has a new article published on REST. Unfortunately I was able to find several significant problems with it in a matter of minutes.

The design is awful: methods in the URLs.

The "logic" on when to apply REST vs. "heavyweight" SOA is simply unfounded. Please: "As your application environment grows, it's likely you'll abstract away from the REST implementation details more and more... It shouldn't be too hard to extract the actual business logic behind the services and rewrap it in a SOAP package in the new environment."

Huh? That and the discussion at the beginning about large vs. small organizations using heavyweight SOA or REST. The author should follow his own advice:

"Take some time to think about this: it'll pay for itself in the long run!"

Integrity and Dignity

These are qualities I look for in an elected leader. Too bad they're rare traits in politicians.

Agile Open Northwest: Seattle

I'm helping organize the Agile Open Northwest conference again this year. Last year was my first "open" format conference, as an attendee or organizer. I was amazed how well the conference self-organized into a number of useful discussions and activities.

It seemed to happen effortlessly, but really the other organizers knew what they were doing. Can you tell from this list which of us is not a consultant? 8^)

Better digs than last year's at the Kennedy School in Portland will be tough to achieve. But this year we'll be in Seattle, at the Seattle Center, and how can you go wrong when you have a Space Needle?

Accounting for Free Markets: the Euphemism

Via Joe Gregorio, on petroleum, gas mileage, and free markets...

Moody-Stuart, who is currently chairman of the mining group Anglo American, says he is a great fan of the free market, "but like most things, they have a failing. Without regulation to channel their power, markets will not deliver things which are of no immediate benefit to the individual making his or her choice, even though they may be beneficial to society."

The problem is worse than whether a free market can work here or not. The problem is "free market" is a ghastly oversimplification for this situation. A truly free market would make obvious the true cost of gasoline. This would mean the cost would account for all the military, all the lobbying, etc. that's involved in that industry.

"Free market" is often a euphemism spouted by the primary beneficiaries of the market itself, which in fact is highly controlled. People have been moving many, large levers behind the scenes of Big Oil for well over a century. That market is *anything* but free.

Not that I disagree with the whole "free market fairy" concept -- any significantly valuable market is going to have a lot of powerful operators using every hidden lever they can get their hands on. This is called "working the system". People will always attempt to "work" to their advantage whatever system confronts them, no matter how large or small that system. Some people are better than others at doing so, and then using that leverage to continue on. Some people have very low ethics in deciding what levers to pull.

Welcome to planet earth.

Microsoft!

This could turn out to be the best thing to happen to Google in five years.

Yeah, but otherwise the browser is a great platform for applications...

(via James Robertson, who could tell you about Seaside)

News from the Blackberry

From Joe Wilcox at Microsoft Watch...

"Forrester expects at least half of the 42 percent of enterprises that
say Web 2.0 is not on their priority list to add it by year's end."

Oh, and Ray Ozzie will be speaking in March.

2008.

Testing Trumps Design

Reaching back a year or so ago to a point Ralph Johnson made about practicing test-driven development. The big lesson: every little bit helps. You don't have to be perfect by any stretch, just stretch yourself more than where you are currently, and then a bit more...

For some years, I've taught a software engineering course that used both XP and RUP. Students had group projects, and half were XP and half were RUP. The projects are not run in as controlled a fashion as those in the paper, so perhaps I am missing something. However, in general I have not seen a big difference in results between the two. The main indicator of success is previous experience. Groups that have a couple of experienced people do better than those without.

However, one group of people consistently did better with XP than with RUP. This is a group with little programming experience. RUP did not work for them because they had nobody who could act like an architect. They would produce UML diagrams, but they were always wrong, so it was a waste of time. However, when they followed XP, they produced (usually poor) working code that had regression tests. Eventually they started to refactor it. Because they always kept their tests working, they always made progress, even if it was slow. XP worked much better for these groups than it did for average groups.

I'd take a reasonable, automated test suite over a great design any day. The one can lead to the other more easily. I'm not sure I'd recognize a "perfect" test suite if it hit me in the face, but a team that tries to improve it's testing has the best chance of success.

Javascript and Smalltalk

Joe Gregorio writes...

Alan Kay had a vision for the web... Ten years later we have the Lively Kernel...

JavaScript, it's the new Smalltalk.

I can go along with the Javascript part. There are aspects of Javascript I like more than Smalltalk, and the other parts are not so bad either. Javascript does have some cruft that makes it more complicated than Smalltalk, oh well.

I can go along with the HTML and SVG parts too. But the Lively Kernel part has a way to go. Too bad it's built on the browser, which is a much worse medium than the Smalltalk image. Hopefully that will improve, but browser progress as an application development medium is depressing.

The web makes a helluva "image file". Morphic is a great UI approach, but the browser is not ready to fully realize it.

The current browser is not even up to the level of Smalltalk-72 running on a DG Nova, pushing that analogy a little over the top. It's kind of sad to see Dan Ingalls mucking around with LivelyKernel when he could be working on such a better medium. It's kind of like when Ward Cunningham was at Microsoft and they just couldn't figure out what a creative mind they had available to tap.

Wiimote and Your Computer

(Via James Robertson) Cool, cheap touch controls on your computer using a wiimote. Good YouTube videos of what's possible now for the "maker" types, and should be fairly cheap out of the box for everyone else in the near future.

And now for something...

Steve Dekorte on another of my favorite subjects...

If you're looking languages or concurrency tools that will scale to the high core count desktop machines of the near future, I wouldn't put stock in MISD oriented solutions such as transactional memory or fancy functional programming compiler techniques. Shared memory systems simply won't survive the exponential rise in core counts.

Deeper Dynamics

Oh dear. This can't go on for long. I've been in the middle of these muddles before. Say something like, "Dynamic languages have great tools and can be used to build large, long-lived systems." And you will find yourself on the receiving end of questions like the following which someone just sent me.

I can indulge these for a minute or two. The problem is they are rarely asked by open-minded people. Usually they come from the statically convinced, who just need to find a way to catch you, to allow themselves the comfort that their static languages and their tools could be the only possible methods for large systems. They need to assure themselves of being on the righteous path, and you, poor fellow, are one step away drowning in Styx.

Before I proceed let me state: I know large systems can be built statically. I've done it in more than one language. I've also done it dynamically in more than one dynamic language. Have you? Have you been part of failures in each? Yes, I have. They almost never have anything to do with the languages and the tools themselves.

That said, I prefer dynamic languages because they ease the pain when used well. On the other hand nothing eases the pain of bad projects.

"What happens in Smalltalk projects where you grow larger? Where you want to divide a system up into components, and some of those components don't exist yet?"

Well, what happens in other projects? You have a need; you have an idea to meet the need; you try it; you grow it. I guess I don't understand the question.

"How are interfaces between library and user code specified?"

Interface notations have been attempted for various dynamic languages, including Smalltalk and Lisp. There is a reason they've not really caught on, yet large systems are still developed in these languages.

Semi-formal notations, tests and other examples, documentation, good tools -- these all contribute to success. But if you are looking for some kind of a "static declaration" as they saving mechanism for large systems -- it may work for you, but I doubt you can prove they are necessary and sufficient. I have counter-examples.

"Can such interfaces be browsed in the same way if the code can't be called?"

Your question betrays your bias. "There must be some such mechanism as the one I am used to," he thinks.

Most large systems developed with dynamic languages have not used the kind of mechanism you seek. Look elsewhere.

Can we go back to arguing over WSDL vs. REST now? Face it. The history of programming is one of gradual adoption of dynamic mechanisms.

Dynamic Languages: Should the Tools Suck?

A comment to my previous "oh my god" post just came through. It is the typical "static typing is good because it enables good tool support like refactoring and finding methods". Mmmm.

First let me agree that tool support for Python and Ruby truly suck. They do. I still love the languages and would use them in a heartbeat before any static language. But the tool support sucks. But not because the languages are "dynamic".

To make my point a bit more concrete, here is a snapshot of a Smalltalk image I just created...

The aqua colored window is a "method finder". I entered the text "#(1, 2, 3, 4). 2. true", which is an array of numbers, the number 2, and the boolean true.

The method finder then found five methods that when applied to the array and the number, return true. It found those for me by considering potential candidate messages and executing the method code for me. The matches are listed in that window's pane just below the text I entered.

I then selected from that list the method "windowReqNewLabel:" and the method finder brought up the green window which is a system browser. It set the browser to display the code for that selected method. (By the way the browser has a complete kick-ass refactoring engine available. But that's for another post maybe.)

In the system browser for that method I clicked on "senders". This brought up a menu consisting of that method's selector as well as all the other message selectors that are used in that method. I chose that method's selector because I hypothetically want to know all the places this application sends that message. (Otherwise I could find all the senders of messages that this method also sends.)

Doing so brought up the front, light blue, window. This is the "Senders of windowReqNewLabel:" window. This window lists the four places that message is sent in this application. You can see I have selected the third occurrence, and the code for that sender is displayed in the bottom pane of that window.

As you can see, tools for dynamic languages only suck when they have not been implemented. On the other hand everything you see in this screenshot has been in Smalltalk for decades.

Decades of better tools than you can find for Java, C, Scala, you name your favorite statically typed language.

I am sick and tired of people whining about how dynamic languages cannot support useful tools. They can, and they are *better* than yours. This example is the tip of an iceberg I just captured in a matter of seconds. That first method finder "search by example" completes in a split second.

Can your static tools do that? Maybe, which is why I won't go around claiming that they can't, because I am ignorant of that, just as you are ignorant of what dynamic language tools can do.

Begin Update

An anonymous commentator points out that one of us is being obtuse. How can a dynamic language possibly refactor the name of a message when there are three implementations of that method but only one of them should be renamed.

I will quote from the documentation for the very first usable refactoring tool ever. (This very first one ever was written in Smalltalk and is the one that still is used by Smalltalkers.) The quote...

One of the key features of Smalltalk that allows us to perform the sorts of analyses and source code transformations necessary for refactoring is that it has reflective facilities, that is, the ability for the language to examine and modify its own structures. For instance, the Refactoring Browser examines the context stacks of executing programs to determine runtime properties such as callers of a particular method or cardinality relationships between components and aggregates. Refactoring can be performed in the absence of reflective facilities, but then requires a separate, metalanguage that can be used to manipulate the original program. Having a single language simplifies the process...

Some preconditions of refactorings are not simple to compute from the static program text. With Smalltalk's dynamic typing, determining the type of a particular variable can be extremely difficult and time consuming. Moreover, there are analyses that defy static analysis in any language, such as cardinality relationships between objects.

Since, performing these types of analysis statically is difficult or impossible, the Refactoring Browser computes some of the preconditions by performing the analysis dynamically rather than statically. By observing the actual values that occur, the Refactoring Browser can perform correct refactorings. As an example, consider the rename method refactoring. To correctly rename a method, all calls to that method must be renamed. This is difficult in an environment that uses polymorphism to the extent that Smalltalk does. Smalltalk also allows dynamically created messages to be sent via the perform: message. If an application uses this approach, any automatic renaming process has the potential of failure. Under these conditions, guaranteeing the safety of a rename is impossible.

The Refactoring Browser uses method wrappers to collect runtime information. These wrappers are activated when the wrapped method is called and when it returns. The wrapper can execute an arbitrary block of Smalltalk code. To perform the rename method refactoring dynamically, the Refactoring Browser renames the initial method and then puts a method wrapper on the original method. As the program runs, the wrapper detects sites that call the original method. Whenever a call to the old method is detected, the method wrapper suspends execution of the program, goes up the call stack to the sender and changes the source code to refer to the new, renamed method. Therefore, as the program is exercised, it converges towards a correctly refactored program.

The major drawback to this type of refactoring is that the refactoring is only as good as your test suite.

Except that good test suites are never a major drawback. They are actually the only thing that can save your application over time, dynamic or static.

And so you see good refactoring tools, good (dynamic) languages, and other good things go hand-in-hand with good tests and good test coverage. Working incrementally helps you make only small changes at a time, testing well points out just the few things you just broke, good coverage makes sure you find everything, and refactoring tools introspecting the code that your good tests cover help automate those changes and bring them up to a higher level of expression for you.

No single tool or practice is in itself sufficient. They work together for the good of the whole. Should I assume your refactoring tool for your static language works so well as this? I believe method wrappers, etc. take a fair bit more machinery and end up not so expressive in static languages. At best they require more code than with dynamic languages.

You do not have to compromise your very simple language just so your tool can work. Dynamic languages and good tools work for you. Static declarations -- they appear to work for you until they don't. They're not a compromise, they are a bloody surrender, somewhat quoting Guy Steele from some long ago Lisp memo I don't fully recall.

Oh my god here I go again. I know you love your static languages and type theories. Good on you. I don't.

End Update

Experiment, Programmers and Engineers

This summary is not available. Please click here to view the post.