Programming with Streams

Michael Lucas-Smith illustrates programming with streams and a helpful "functional" protocol added by the ComputingStreams package from Cincom Smalltalk's public repository. Michael makes the case that more code should be based on streams instead of collections, and use a common protocol like ComputingStreams instead of custom, one-off methods that won't play well with others. Streams are a better abstraction than collections because they inherently "delay" and scale up to and including infinitely long streams.

Streams are interesting creatures, fundamental to programming if you've got them available. See Structure and Interpretation of Computer Programming: "Streams Are Delayed Lists" and Richard Waters' Series package for Common Lisp for more examples.

Also this Smalltalk package is similar to Avi Bryant's Relational Object Expressions. ROE uses objects using the standard Smalltalk collections protocol to delay "realizing" complete, in-memory collections, enabling the original collections to be enormous numbers of rows in tables on disk. In the case of ROE the objects are delaying the eventual composition and execution of SQL, including the conditional expressions that may have narrowed the size of the selection.

Closures and Objects

I am not sure what Steve Dekorte is saying...

A language that uses objects on the bottom can use them for everything, but a language with closures on the bottom needs other types or "atoms" for things like numbers, lists, etc and then scatter around functions for operating on those other types. If you care about simplicity, consistency and organization, this is a big difference.

If you have objects, you still need compiler support for closures.

But if you have closures, you need no compiler support for objects.

Either way you need compiler support for other literals like numbers.

Unless you want to program numbers in the lambda calculus. And then too closures win.

Your name is different, but you're really a Unix system too, aren't you?

Microsoft's Unix from back in the day, Xenix...

Simplified Javascript: Cruft Reduced

A bunch of comments just showed up, June 21, 2008.

Like it or not (and FWIW, I like it quite a bit), JSON has made an impression on networked information exchange. Douglas Crockford, who started the whole JSON thing, has a chapter in Beautiful Code on Pratt-style parsing. His example parser is written in, and happens to parse, something he calls "simplified javascript".

Simplified JavaScript is just the good stuff, including:

• Functions as first class objects. Functions in Simplified JavaScript are lambdas with lexical scoping.
• Dynamic objects with prototypal inheritance. Objects are class-free. We can add a new member to any object by ordinary assignment. An object can inherit members from another object.
• Object literals and array literals. This is a very convenient notation for creating new objects and arrays. JavaScript literals were the inspiration for the JSON data interchange format.

Implementing some of the currently popular "scripting" languages, in particular "full" Javascript, Python, and Ruby will make evident the amount of cruft they have for no apparent reason. Lisp traditionally has had some cruft, but, arguably, nothing like what can be found in these languages. Such cruft makes these languages significantly more difficult to implement than implementing Lisp or Smalltalk (two amazingly "cruft-reduced" languages given their age).

Simplified Javascript would make a decent base for a scripting language. I pointed this out to a couple people looking at full Javascript over the last couple of months for an Erlang-based server and now for Croquet.

People already in the more or less "end user scripting" space tend to have some Javascript knowledge. This subset has good functional and OO capabilities, comes with a simple parser implementation already(!), and from implementing a bit of interpretation code already, seems amenable to a simple elisp/emacs-level of performance, which is more than sufficient for an interactive GUI system. Compiling to a native or byte code would not be too difficult for better performance.

Simplified javascript not only eliminates the uglier and more difficult to implement bits. It also eliminates a lot of bad security problems.

And so on designing a new scripting capability, combining Simplified Javascript with a capability-based approach to authority as with the E programming language could have some benefits. And JSON for "data"... go ahead and use "eval" to implement the JSON parser in such an environment.

Well, that would be a heck of a lot better than the current browser/javascript situation which is horrendous. Now we've got a truly undesirable "legacy" situation(s) in the browser. Apparently Mark Miller or one of those E/Capability folks is at Google now and their looking at something like the browser those folks built for DARPA. (Found in this powerpoint if you don't mind opening those up in OpenOffice.)

Innovation

From Business Week on improving what already works vs. finding new things that work better...

As once-bloated U.S. manufacturers have shaped up and become profitable global competitors, the onus shifts to growth and innovation, especially in today's idea-based, design-obsessed economy. While process excellence demands precision, consistency, and repetition, innovation calls for variation, failure, and serendipity.

Indeed, the very factors that make Six Sigma effective in one context can make it ineffective in another. Traditionally, it uses rigorous statistical analysis to produce unambiguous data that help produce better quality, lower costs, and more efficiency. That all sounds great when you know what outcomes you'd like to control. But what about when there are few facts to go on—or you don't even know the nature of the problem you're trying to define?

"New things look very bad on this scale," says MIT Sloan School of Management professor Eric von Hippel, who has worked with 3M on innovation projects that he says "took a backseat" once Six Sigma settled in. "The more you hardwire a company on total quality management, [the more] it is going to hurt breakthrough innovation," adds Vijay Govindarajan, a management professor at Dartmouth's Tuck School of Business. "The mindset that is needed, the capabilities that are needed, the metrics that are needed, the whole culture that is needed for discontinuous innovation, are fundamentally different."

Planned RESTful Services

Stu says stuff about reuse...

From this viewpoint, "build it and maybe people will use it later" is a bad thing. SOA proponents really dislike this approach, where one exposes thousands of services in hopes of serendipity -- because it never actually happens.

Yet, on the Web, we do this all the time. The Web architecture is all about serendipity, and letting a thousand information flowers bloom, regardless of whether it serves some greater, over arching, aligned need. We expose resources based on use, but the constraints on the architecture enables reuse without planning. Serendipity seems to result from good linking habits, stable URIs, a clear indication of the meaning of a particular resource, and good search algorithms to harvest & rank this meaning.

This difference is one major hurdle to overcome if we are to unify these two important schools of thought, and build better information systems out of it.

The early "mashups" were certainly serendipitous. I would expect most service providers on the web today are planning for machine-to-machine to a much greater extent than even a couple of years ago.

Is it really "serendipitous" when an organization follows those good web practices and enters formal-enough agreements with participants, say in an intra-enterprise-services situation? No, that seems very much "planned". The biggest hurdle may be the addition, perhaps, of a stronger demand to negotiate terms for support and upgrades.

This is necessary of any kind of service, networked or otherwise, that has significant risks to stakeholders. If you buy SAP systems and bring them in-house, or if you network out to Salesforce, you negotiate terms of service with in-house IT, with the vendors, with the ISPs, etc.

Scalability Requirements

Stefan Tilkov and Dare Obasanjo debate the number of big sites that would benefit from giving up on the relational database as a centerpiece of their systems. The parameter used in the discussion is scalability.

A number of the lessons learned from these big web sites could apply to smaller data center designs as well. The biggest problem I've seen in the typical data center, and from comparing notes, seems fairly common, is the ability evolve components relatively independently.

Relational databases can be in the mix and still evolve, just as more supposedly "loosely coupled" mechanisms can actually tie components tightly together. For example some systems may use asynchronous messaging to decouple with respect to time, but then pass to each other data that exposes implementation details. And so these systems are coupled to each other with respect to maintenance and enhancements: changing the implementation of component C demands corresponding changes to every other component that consumes component C's implementation-specific messages.

But the lessons are worth following for more reasons than just scalability.

Content Unbecoming

Sam Ruby comments in his own blog...

I will merely point out that I’ve dealt with formats and protocols which are “explicitly not intended for human readability” and I’ve learned to avoid them.

And Bill de hÓra comments further down...

Atompub/Atom shines light on the fact we have serious issues around sharing and “understanding” structured content. Complaining about the source of light isn’t sensible.

So, to me, this is like the web services debates rehashed half a decade later. The only interesting difference in this thread is the level the argument is happening - around the content/payload/mediatype instead of wire/transfer...

Maybe it’s not as obviously random at this time to argue that per silo data access formats are a good idea...

Fwiw, “technically”, I don’t see why facebook don’t serve class laden XHTML and document the attributes.

And along the way James Snell compares to Yaron's example...

<entry xmlns="http://www.w3.org/2005/Atom">
  <id>tag:example.org,2005/someuser/profile</id>
  <title>Some User's Profile</title>
  <updated>2000-01-01T00:00:00Z</updated>

  <author><name>Profile System</name><author>
  <content type="xhtml">
    <div xmlns="http://www.w3.org/1999/xhtml">
      <div class="profile">
        <div class="section" id="professional">

          ...
        </div>
        <div class="section" id="personal">
          ...
        </div>
        <div class="section" id="clothingPreferences">
          ...
        </div>

      </div>
    </div>
  </content>
</entry>

Looks good. Damn. Even *this* xml-hater (yours truly) has had trouble realizing all that one-off stuff is bogus.

Don't Fear the Programmer

Not to beat a dead horse, but Robert Cooper adds an important point to the enterprise development tool conversation...

And here is where it breaks down. All these unusable drag and drop tools, and “easy” XML programming languages aren’t targeted at programmers. They are targeted to suits who can buy into the idea that some non-techy is going to orchestrate these services and modify business rules. These products are unworkable because they are based on the idea that “You won’t need programmers anymore!” at least at a core level. Once you make that assumption you start building things that get in programmers way, and still include enough abstract programming concepts that no non-programmer is ever going to be able to work with it proficiently

I've not heard this lately, but in previous situations this point was widely held: BPM tools should be used to get the programmer out of the loop. I always enjoyed bringing testing into the conversation. The current crop of BPM tools I've seen up close are terrible at incremental development, testing, and release. And yet at least some IT shops have seen them as *the* way to get from under long software development cycles.

Those shops I know of also happen to be historically lacking good automated testing and release practices. Most of their causes for long development cycles have almost *nothing* to do with programmers, save for the fact that most programmers historically have not paid much attention to good testing either. Of course that's changed dramatically over the last five years or so.

Managers: if you have long development cycles, backlogs of feature requests and bugs, and "legacy code" that seems impenetrable, then please embrace your programmers' desires to be more productive and effective. There are many things you can do today to make big improvements. Buying tools that will reduce the need for programmers is a pipe dream for now.

Sad IP

So let me get this straight -- Novell and Redhat are being sued over a patent some mangy IP shop bought that originated at Xerox PARC (I suppose Rooms?). And the patent is about making the same window show up in multiple workspaces?

This is the best IP some mangy techno-lawyer can pin on *Linux*?

Em, wow. Crawl back under your rock until you can make it *really* interesting... Next.

And note to Steve Ballmer: stop being such a wuss. You're really just appearing pretty sad these days. Can't you get a ride in space or something?

RFC 5023

Nicely done. RFC 5023.

Bungie Cuts the Cord, So to Speak

Wow. This is huge. This speaks volumes between the lines and in the lines themselves. Just as Halo 3's $300MM USD is ballyhooed all over the press.

Just when there's all this buzz about how even enterprise applications are should become more game-like, and just as Microsoft must be getting desperate to show they can innovate, Bungie obviously takes off in order to be more innovative with game development than they can be as part of Microsoft.

Todd Bishop's Q&A is great. Maybe he could interview U.S. presidential candidates?

Say it Once Again

Pete Lacey: What is SOA?. Nicely done.

The definition of SOA is still an argument after all these years. This is a darn good sign that we should not care to use the term at all.

Equivocate: To avoid making an explicit statement. See Synonyms at "lie".

Mark Baker wisely points out...

There are, of course, a very large number of automated solutions to a particular problem.

...if you don’t feel disdain towards a lot (97%?) of these solutions, then you simply don’t understand what it means to be a software architect.

There is too much equivocation in our field. It's hard enough building software when you've got good tools at hand.

e·quiv·o·cal (-kwv-kl)
adj.

1. Open to two or more interpretations and often intended to mislead; ambiguous. See Synonyms at ambiguous.
2. Of uncertain significance.
3. Of a doubtful or uncertain nature.

Oh Dear, Part VII

Arnon Rotem-Gal-Oz assumes the position...

One thing that is missing from "HTTP variety of REST" implementation is reliable messaging. Location transparency is harder to solve with HTTP etc...

The solution should match the problem, that's probably the primary reason why we need architects after all.

Please Don't Pick on the Donkey

More Steve...

REST is unproven. Sigh. I can’t decide if people say this because they’re just trying to stir up an argument, or they’re so heavily biased toward their non-REST approaches that they just can’t even consider that there might be viable alternatives, or they really have no clue about what REST is, how it works, and why it works, and they’re not interested in learning it, so they just react badly whenever they hear it, or all of the above. If you’re anti-REST or REST-ignorant, and you haven’t read RESTful Web Services, then don’t even talk to me about REST. The book is absolutely wonderful, and its explanations and answers are extremely clear. If you consider yourself informed and capable when it comes to distributed systems and integration, but you don’t know REST, then there’s simply no way you can read that book and not have it lead you to seriously question your core beliefs regarding how you think such systems should be built, unless you’re completely close-minded of course...

What do mono-language programmers have to do with ESBs? It’s all part of the same culture, the “one size fits all” approach, where you have answers looking for problems rather than the other way around, and where people intentionally wear blinders to less expensive, more productive, and far more flexible and agile approaches because “it’s just not the way we do things around here.”...

At the end of the day, if you want to ignore my advice on using REST and dynamic languages, that’s your own problem.

And where his previous post received a comment, apparently from a Neudesic employee (ever heard of Neudesic???)...

Sonic Software, BEA, IBM, IONA, TIBCO, webMethods, Cape Clear, Fiorano, Oracle, Software AG, Neudesic, And how many open source projects????

All deliver ESB’s. It seems highly unlikely that all of these organizations, with plenty of smart folks, could be incorrect. Moreover, Gartner, Forester, ZapThink, and Burton, all speak loudly about ESB’s being an enabling technology on the road to building workable SOA’s.

Oh, dear. Steve responds best...

Making a list of smart companies to prove a point, well, doesn’t actually prove anything. I bet there are some actual smart people out there who have locked themselves out of their car, for example, or deleted whole directories they really wanted to keep but never backed up, or even paid actual money for a Zune. What does that prove?...

Unlike most of the commenters on this posting, I’ve personally built at least one of everything I’m talking about, on the ESB side and on the REST side, so I’m extremely confident in my opinion. I’d therefore much rather get specifics from you.

The View From Here Is... Grisly?

Joe Wilcox on MSFT Vista, speaking of big software the world is leaving behind...

The results, while arguably anecdotal, are grisly...

"Everyone—every single person—that I put on Vista has switched back to XP," he said. "It's too complicated."

From an administrator's perspective, my VAR buddy likes some Vista deployment tools, but he viciously complained about poor driver support, networking changes and end user complaints about UAC (User Account Control) and Internet Explorer 7 security popups.

In this ringing Vista endorsement, one Microsoft Watch commenter claimed: "I'm an IT consultant and I'm proud to announce I've formatted 450 Windows Vista machines back to Windows XP to date. I have also prevented at least 1,000 Windows Vista sales."

...the relevance of Windows is in decline. Microsoft's desktop operating system is rapidly becoming a commodity.

LAMP

Her royal pingdom lists...

OS: Linux 7 - Windows 2
Web server: Apache 7 - IIS 2 - Lighttpd 2
Scripting: PHP 4 - Perl 4 - ASP.NET 2 - Python 1 - Java 1
Database: MySQL 7 - SQL Server 1 (possibly 2)

And you should not be investing in some reasonable approximation to LAMP, again, because...?

Protected Game Preserves

That's how Bob Warfield refers to how Adobe, Microsoft, and SAP are approaching Software as a Service (SaaS). There's something different about Adobe relative to the other two.

Microsoft and SAP are figuring out how to transition gigantic cash cows into a world that is rapidly leaving them behind. Not so much for Adobe.

Adobe bought BuzzWord. They now have an early beta of an interesting word processor. They also have a few more apparently kickass flash/flex developers and creative software designers.

As opposed to MSFT's and SAP's situations, the fact that BuzzWord doesn't compete with other current Adobe products is probably an advantage. BuzzWord as a product may or may not ever matter.

Adobe's strengths in the new world appears to be (1) graphics, especially as they become more web-friendly, e.g. Thermo is hot.

And (2) their own decent runtime from Macromedia, something relatively easy to build upon compared to java or dotnet, from a company with decent developer relationships.

I think considering the company as a whole, Adobe has some good stuff to take into the new world. MSFT and SAP have bigger transitions to make, product-wise and company culture-wise. Not that I would bet much against them, because they are f'ing big and know how to sell software to vice presidents.

Properly Striking a Balance Between Shared Agreement and Decentralized Execution

Stu Charlton of BEA says stuff on the ESB brouhaha, but then he says some really crucial stuff we all need to consider...

I like ESBs if they're used properly -- to me, a good ESB is actually just a scripting language with supporting infrastructure. AquaLogic Service Bus, to me, is a Pipeline language, an XQuery language, and the supporting management & interoperability infrastructure, for example. But an ESB is not mandatory, and sometimes can slow you down if you're not looking to make legacy systems interoperate.

Wow. Equating Aqualogic with a scripting language. That's bold.

I agree with Steve Vinoski's The ESB Question. Don't use an ESB if it's going to slow you down. ESBs are a decent, productive tool for SOA (and even REST!) when applied properly, but so are Python or Ruby. ;-)

I want to like what Stu's saying, but his sprinkling of "properlies" just seems like equivocation.

BEA tries really hard to make Jython supported in WebLogic and AquaLogic, for example. The whole JMX object model is scriptable, and you can do some amazing things with it. And we have PHP running on WebLogic Server.

OK. I kicked the tires on Aqualogic over a year ago. I did not appreciate it any more than other ESBs, and found Jini/Javaspaces significantly more productive.

But I like the parts about Jython and PHP.

What gives? Perhaps two things:

1. software has long term residual value but our investment decisions are always short term
2. we intertwingle what changes from what doesn't change

We've fixed #2 with hardware platforms. SOA was supposed to fix it for software, but perhaps not (yet).

Yeah, across the industry we pretty much have #2 at the level of a high art.

As for SOA, well. It's well known...

SOA is the only thing Chuck Norris can't kill...

Its too bad you can't afford it.

Now we get to where I really like what Stu writes (and it has nothing to do with ESBs, unsurprisingly)...

I've seen many fast-moving IT departments, particularly in investment banking, but even some in telecom, continue to use the "build it to throw it away" mantra with reasonable success. This works when your organization is compartmentalized. But when information sharing is of primary importance, such as the intelligence or defense establishment, for example, we get to a case where these huge organizations need a different approach. Layers upon layers of bureaucracy make it hard to abandon any pet project.

This is partly why I think Enterprise Social Computing, the Web's architecture (or some successor of REST), etc. are very exciting; they seem much more likely to strike a balance between shared agreement and decentralized execution.

Every branch of software I know of must build an organization that is good at building software that is easy to throw away. If your organization can do this, it is at the master level and should be highly valued. Very few organizations can throw away much of anything in their data centers. This is *horrible*.

Stu nails that. But moreover he latches onto what should remain relatively constant in the data center: the information (web) architecture. That seems to be a good way, as he writes, "to strike a balance between shared agreement and decentralized execution". If you do that properly, you win.

That BEA Enterprise Social Computing thing looks interesting. It's encouraging to see BEA reaching out in that direction.