"I have a mind like a steel... uh... thingy." Patrick Logan's weblog.

Search This Blog

Friday, December 14, 2007

Column-Oriented RDF Storage

A couple years ago it occured to me that a database with column-oriented storage, such as Sybase IQ, might make a reasonable database for storing RDF data, where a star schema can be seen as a way to represent related tuples together.

Now it turns out some folks have been working on such a thing... C-Store is an open source, column-oriented databse. The paper "Scalable Semantic Web Data Management Using Vertical Partitioning" (pdf) discusses using C-Store for RDF...

Efficient management of RDF data is an important factor in realizing the Semantic Web vision. Performance and scalability issues are becoming increasingly pressing as Semantic Web technology is applied to real-world applications. In this paper, we examine the reasons why current data management solutions for RDF data scale poorly, and explore the fundamental scalability limitations of these approaches. We review the state of the art for improving performance for RDF databases and consider a recent suggestion, “property tables.” We then discuss practically and empirically why this solution has undesirable features. As an improvement, we propose an alternative solution: vertically partitioning the RDF data. We compare the performance of vertical partitioning with prior art on queries generated by a Web-based RDF browser over a large-scale (more than 50 million triples) catalog of library data. Our results show that a vertical partitioned schema achieves similar performance to the property table technique while being much simpler to design. Further, if a column-oriented DBMS (a database architected specially for the vertically partitioned case) is used instead of a row-oriented DBMS, another order of magnitude performance improvement is observed, with query times dropping from minutes to several seconds.

Thursday, December 13, 2007

Release It Again

Pete Lacey -- what he said, about Release It and Michael Nygard's blog.

If you find yourself in a panic over some centralized resource, wonder if the full costs are accounted for. What alternatives might exist for decentralizing, and how do the cost/benfits really add up over time?

The cost of operations is dropping. The cost of change is still too high for many to take advantage of that though. By the time we can get our systems onto more budget-friendly architectures... well, I guess heading in that direction puts you on the path toward even better things.

Depending on the business, if one extrapolates out from one's current position, through the point where more open/available/scalable systems are in use... well, then is this evidence that for most of us, our ultimate position is out in "software as a service/utility" land? Exactly who should be in the data center business five to ten years from now?

Another data point shows up. (Oh, and speaking of cost of change, it's in Erlang and it provides a REST api available from any language.)

From the SimpleDB FAQ...

Q: Where is my data stored?

Amazon SimpleDB stores your data in our multiple data centers in the United States. We anticipate adding other geographies over time.

Q: Does Amazon store its own data in Amazon SimpleDB?

Yes. Developers within Amazon use Amazon SimpleDB for a wide variety of projects. Many of these projects use Amazon SimpleDB as their authoritative data and query store and rely on it for business-critical operations.

So, "business-critical" seems kind of reliable. Not just scaled out databases, but scaled out data centers. Those are even more expensive to operate on your own.

Yahoo Flex Skin, Other Flex News

Among other recent Flash/Flex/Air/Adobe news, Yahoo released an open source skin for Flex.

Elsewhere new versions of Flash/Flex/Air from Adobe have been released and in some cases newly opened up.

I don't feel compelled to use the Flex Data Services, but for those who do, or when I do, today's news should be encouraging. You can use their open source implementation, or use their open specification for an alternative, e.g. in some other non-JVM language.

Saturday, December 08, 2007

Embedded Lightness

I was out with a long-time friend and former co-worker last night. He's been working the last year or so on software for managing blade servers. Technically this is an "embedded system". Some veteran embedded system developers on his team have been learning just how much their field has changed.

Essentially the system consists of linux, lighttpd, php, and sqlite. Where the veterans were set on writing a lot of C and custom protocols for handling events in the server (blades coming and going, for example), they've been able to do everything using the web's architecture. They wrote some C to enhance the php interface to snmp.

They had some pre-existing C code that has a memory problem after some number of days. No problem, they just kill and restart the process well before that. When that code becomes the most important problem, they'll look at it.

Job Opportunity

Mike's got a job posting up at his new gig. You'd be working with a smart, funny, curious (in the good sense!) project manager/developer, on an interesting problem, and a high-powered executive team.

You don't have to live in Portland, although why would you want to live anywhere else?

Monday, December 03, 2007

Safety Dance

Douglas Crockford at XML Conference 2007. You can dance if you want to...

The current browser implementations have problems because they share all of the information between the current sessions (problems with cookie stealing, replay attacks, and chrome changes). That’s the dangerous web 1.0. Now, we’re trying to intentionally mashing stuff up (which we’d always tried to prevent when unintentional).

Meet the new Reg...

Paul Fremantle on WSO2's registry work...

For a while I've been thinking that the SOA registry space has been a little overcomplicated...

So fundamentally the approach we have taken is to build a registry/repository based on REST concepts. And as we looked at the REST space, we kept noticing how close the Atom Publishing Protocol (APP) is to our needs, so we've made that the public remote API to access the repository. Of course, if you are just browsing the registry, you only need a browser - APP is mainly there to support updating resources.

Of course, using Atom and APP gives some really nice benefits too - like being able to subscribe a feed of new resources that meet your search criteria.

Glen Daniels was talking about this registry in the hallway at QCon a few weeks ago. Nice. You could definitely see Glen's gears turning during the sessions and discussions.

Sunday, December 02, 2007

Reg

Some folks have been quoting Reg Braithwaite lately. (Steve Vinoski did just a bit ago.) Here's another...

I have never met someone who desperately wanted to be great but failed to be at least decent.
That's a good observation. And almost Goethe-like.

Saturday, December 01, 2007

The name of this band is Talking Heads

I was poking around youtube for the Late Night with David Letterman show (the one back on NBC), the episode back around 1983-84, where the entire screen rotated clockwise over the course of the hour.

I've not found a clip yet. But I've been watching other clips, like this one...

Amazing. There was a time as an early 20-something when we *had* to watch Late Night and talk about it the next day. The current Letterman is more mild, but still bits like, "Is this anything" and "Will it float" provide glimpses back to the original show, which at the time seemed like "anything goes".

I wish I could find a clip of that episode with the screen rotating slowly over the hour.

Andy Kaufman and Elayne Boozler

Andy Kaufman and Elayne Boozler...

Boozler I think has gone under appreciated over the years. She keeps up with Kaufman here. Painfully hilarious improv.

Friday, November 30, 2007

"http is like air"

Quote of the day from an IBM exec on the importance they are placing on rest, http, atom being everywhere.

Wednesday, November 28, 2007

And Then There Were Three... Then Five Again

Mike? He gone.

He formed the band, a little over a year ago, chartered to launch an open source project within a large "vertical". That was four charters, a CIO, one viking, and Joanne ago.

Don't ask. That's the nature of large "verticals" and being a small team, unlike any other, weaving in and out of the established structures. We got pretty good at writing charters on the wiki.

But in between we've done some pretty good work across a couple tables of old Dells running the Ubuntu server. And we're still being asked to be creative, agile, and infectious, which is not often the nature of large "verticals".

This fourth charter could stick. We're back to a full complement of five. And a real budget. And simple tools for restful web services.

Good luck, Mike. You made a lot of good things happen and made them a lot of fun. And took more than one for the team. Now your original charter has come around... nice.

Bird of prey.

Learning Scalable Internet Services

Via Planet Trapexit, and this interesting post from RightScale on the network performance of Amazon EC2 and S3, there is this fascinating page from a UC Santa Barbara course...

"CS290F - Scalable Internet Services"

The project consisted of building a transactional dynamic web site in Ruby on Rails and running on Amazon's Elastic Compute Cloud (EC2). Each site had to hold >100'000 database records that could be searched and explored, have user accounts, and include some form of transaction, such as a shopping cart check-out.

Each project then had to be deployed on multiple servers on EC2 and the groups had to use httperf to demonstrate that they could scale the performance of their site by running a front-end load balancer server, a database server, a memcached server, and up to 10 application servers. All this had to fit into a 10-week quarter, with none of the students knowing either Ruby or Rails at the outset!

More information from the instructor's RightScale blog. The wiki link there is broken, but at least pieces are still around, including the course material.

Sunday, November 18, 2007

Classic

From Jean-Jacques Dubray...

I am a bit saddened by the Open Source community mainly siding behind REST. REST will not get you there, I hope you guys know what you are doing, because this could be a historical mistake. I don't see any serious open source Composite Application Platform.
BTW, many of the links are broken on your site.

Classic.

JRuby and JXPath

Final Update:

Bottom line: for the current implementation of JRuby implementing this does not seem possible. A near-future release is supposed to improve on the Java/Ruby integration. In this case what is needed:

  • A Java class that is the superclass of all JRuby classes. Similar to the PyInstance class (a Java class) in the Jython implementation.
  • A Java class that corresponds to a class defined in JRuby such that when instantiated via newInstance() in Java also creates a corresponding JRuby instance. Similar to the JythonPropertyHandler class I defined in Jython, corresponding to the JRubyPropertyHandler class I attempted to define in JRuby, below.
End Update

Sometimes I drink to forget, and sometimes I just forget. Several years ago I blogged about using JXPath to query Jython objects. Sometime since then I forgot all about JXPath. I remember it now being fairly simple, and working with all kinds of structures. (And so JXPath should be a fairly simple way to use XPath to query JSON objects in Java, BTW.)

I started playing with JRuby recently to see how it's doing. When I came across the Jython code I just mentioned, I thought I'd try the equivalent in Jruby. Here's what I have so far, but I have two question marks preventing it from running. I sent them to the JRuby users list, but if anyone reading this has an answer or a hint, I'd appreciate that.

class JRubyPropertyHandler 
  include org.apache.commons.jxpath.DynamicPropertyHandler

  def getPropertyNames(robject)
    return robject.instance_variables.to_java
  end

  def getProperty(robject, property)
    if robject.instance_variables.include?(property)
      robject.instance_variable_get(property)
    else
      nil
    end
  end

  def setProperty(robject, property, value)
    robject.instance_variable_set(property, value)
  end

  def JRubyPropertyHandler.register()
    introspector = org.apache.commons.jxpath.JXPathIntrospector
    introspector.registerDynamicClass(???, ???)
  end
end
It's that call to registerDynamicClass (javadoc) that has me stumped. The first argument is a class of all the instances that should be handled by JRubyPropertyHandler.

The second argument ideally should just be JRubyPropertyHandler. i.e. not an instance of the handler, but the handler class itself.

I tried calling...

    introspector = org.apache.commons.jxpath.JXPathIntrospector
    introspector.registerDynamicClass(Object, JRubyPropertyHandler)
But these two arguments are each instances of org.jruby.RubyClass, not java.lang.Class as needed.

In Jython the two arguments to register a handler are PyInstance, the Java class that is the superclass of all Jython instances, and the handler class itself. The code for my working Jython implementation is on my original blog post.

Update: from the jruby user's mail list, Nick Sieger drops a good hint, and I have more questions until I can get back to a jirb prompt...

> Perhaps this snippet helps?
>
> $ bin/jruby -S irb
> irb(main):001:0> class Foo
> irb(main):002:1> include java.lang.Runnable
> irb(main):003:1> end
> => Foo
> irb(main):004:0> Foo.new.java_class
> => $Proxy7

Thanks. Maybe. I'll look into it when I can get back to my jirb prompt.

That will work for the second argument *if* it implies that when the
jxpath java code creates a new instance of $Proxy7 then what actually
happens is a new jruby instance of Foo is created.

But what does this say about all jruby objects?

i.e. in jruby, they are all instances of the class Object, but if I do
the following...

o = Object.new
j = o.java_class

...then can I assume that all jruby objects are (in java-land)
instances of a java class that inherits from "j"?

Or is the java_class $ProxyN class more of an on-demand, dynamically
generated, bridge to the other world?

VW: An Efficient Dynamic Runtime

Cincom VisualWorks Smalltalk is an extremely mature, efficient implementation of a dynamic language runtime. James Robertson points to one of the payoffs, running Seaside, on VW Smalltalk...

I tested: Seaside 2.8 in Squeak (using the "one click experience" image), Seaside 2.7 on VW, and Seaside 2.8 on VW - the latter required the under development release that's coming. Here's a summary of what I got:

PlatformSessionsAvg Sessions per SecondAvg Pages per Second
Seaside on VW 7.5971.6217.9
Seaside 2.8 on Squeak3706.1748
Seaside 2.8 on VW 7.65829.779.4

...

Also relevant is this: In Seaside 2.7 on VW, pages per second started off at 34, and then dropped to 10 by the end of the one minute test. Squeak dropped from 50 to 45.5, which is pretty stable. VW 7.6 with Seaside 2.8 started at 81, and dropped to 79.5 - which is even more stable.

If you are looking at Ruby on Rails or a similar dynamic language web framework, there should be several reasons to put Seaside and VW Smalltalk on your list of options.

Update: James updates the tests and brings in Ruby on Rails for comparison. Seeing an "engineering shootout" shape up among these variations would be a fun time. Let various teams engineer the bits out of their own stacks on fairly similar functionality. (Of course a Ruby on Rails on the VW virtual machine would be fun too. :-)

Gemstone Seaside guru, Dale Henrichs, reminds me in the comments:

Patrick, don't forget that there's another mature dynamic runtime system out there that provides transparent persistence along with pretty good performance that even scales across multiple cores... benchmark.
Sorry, Dale!

Get hooked on Seaside, then get hooked on Smalltalk.

Resources and the Kimball

I have not brought out the Kimball in a while. Recently Bill de hÓra linked to the collection of Kimball articles, but without context. The thing about Kimball's design approach is I've found applications of it, at least significant aspects of it, to several systems beyond data warehouses. Understanding the essence of Kimball's approach should be a fundamental part of a software developer's education.

One reason for this is the Kimball approach is a form of domain-driven design. Another reason is the technical aspects are relatively simple. And so the result is not a be-all and end-all solution to everything, but it is a tool with legs the go beyond the original intent.

The Kimball approach has influenced how I think about objects, data, and (from this 2003 blog post) integration...

His keys to an effective virtual database (or data bus architecture in his words) is conformed dimensions, smallest grain facts, etc. Most of these principles apply to the largest data warehouses or the smallest databases, and would benefit any Enterprise Information Integration effort.
What is the web, but a kind of large "virtual database"?

At the time I was in the middle of a large project integrating several systems with a new installation of the SAP FI-CO general ledger module and those systems with a new Teradata data warehouse. We more directly applied these ideas to the warehouse, but generally to information exchange.

Today with groups looking at Restful web services centered around resources, representations, and relationships/links among them, the Kimball approach also applies. It is domain driven, concerned about the entities and activities of the enterprise.

Until we get more analysis patterns from real-world, machine-machine Restful web service applications, the Kimball approach to, and examples of, information management should probably be a key ingredient of this kind of design. We should, of course, expect some similarities in the domain view of information across systems integration, data analysis, and the web of documents and events.

Tuesday, November 13, 2007

Mest

Erik Johnson (via Stefan Tilkov) wonders what's so bad about the "process this" notion. Jim Webber, a coiner of Mest, spent time in his recent QCon talk on this idea before moving on to a nice exposition of HTTP hyperstuff.

I think I have a reasonable handle on what the HTTP methods generally mean. I've not applied them to a wide variety of situations, and I wonder sometimes where that line and whether some specific use is crossing it too far.

More than that though is I've been playing with XMPP a bit again. Where is that line between pushing HTTP too far and jumping over that line into XMPP for "Mest", ad hoc, "process this", I'm not counting on any of the benefits of HTTPness?

These lines are probably fairly blurry on close inspection, when the programmer does not have a clear set of constraints leaning one way (e.g. GET's benefits) or the other (e.g. presence/status). but I've certainly found running an XMPP server and connecting from various systems at least as easy as doing the same with an HTTP server. And XMPP (over HTTP) runs through firewalls, for better or worse.

Anyway... if you are interested in that "one method to process them all" then why not use XMPP for that, and leave HTTP for more specific duties? Or not. If you are in a Mest then are you saying you have no clear understanding of your resources and constraints and so HTTP POST *is* as good as anything?

Just noodling...

Sunday, November 11, 2007

Relaxing

Relaxing

Making It Real

James Snell's story highlights the point I just made in the previous post. The conversation about SOA / WS-* / REST should move forward by experience reports not from vendors or implementors but from people developing real applications...

Those who are familiar with my history with IBM should know that I was once a *major* proponent of the WS-* approach. I was one of the original members of the IBM Emerging Technologies Toolkit team, I wrote so many articles on the subject during my first year with IBM that I was able to pay a down payment on my house without touching a dime of savings or regular paycheck, and I was involved in most of the internal efforts to design and prototype nearly all of the WS-* specifications. However, over the last two years I haven’t written a single line of code that has anything to do with WS-*. The reason for this change is simple: when I was working on WS-*, I never once worked on an application that solved a real business need. Everything I wrote back then were demos.

Now that I’m working for IBM’s WebAhead group, building and supporting applications that are being used by tens of thousands of my fellow IBMers, I haven’t come across a single use case where WS-* would be a suitable fit. In contrast, during that same period of time, I’ve implemented no fewer than 10 Atom Publishing Protocol implementations, have helped a number of IBM products implement Atom and Atompub support, published thousands of Atom feeds within the firewall, etc. In every application we’re working on, there is an obvious need to apply the fundamental principles of the REST architectural style. The applications I build today are fundamentally based on HTTP, XML, Atom, JSON and XHTML...

Are average developers and architects able to design ANY system correctly? I think if you look at the history of software development as a whole, you’d really have to stop and wonder about the answer to this question. The fundamental challenge comes down to this: developers get paid for coming up with solutions that work; doing so means learning just enough about a technology so address the immediate need so they can move on to the next line item in their list in order to meet the deadline; this will quite often mean that average developers and architects aren’t even going to bother designing and implementing solutions “correctly”; nor should we ever actually think that they will. The best we can do as tool providers is educate users better and provide excellent tooling that makes it easier to do the right thing. Most of the time the tool developers can’t even get it right tho.

Blog Archive

About Me

Portland, Oregon, United States
I'm usually writing from my favorite location on the planet, the pacific northwest of the u.s. I write for myself only and unless otherwise specified my posts here should not be taken as representing an official position of my employer. Contact me at my gee mail account, username patrickdlogan.