When I first started test driving the Magic Mouse, I happened to be in iTunes, and happened to need to adjust the volume – which is a horizontal slider in the UI. I thought to myself, “This mouse has the ability to do scrolling in any direction – I wonder what happens if I hover over the volume control and swipe right?” Bam. Of course it worked. It’s Apple, and they think of *everything.*

I was truly impressed. And, being an engineer and naturally inquisitive, I took notice that I was impressed – particularly by such a “little thing.” I then realized that I was impressed not despite the fact that this was a little thing, but because it was a little thing. The sort of thing that other user interface and hardware designers might well have missed.

Which brings me to the moral of this story. I work in a consulting firm, and our goal is not merely to meet our customers’ expectations, but to surpass them. In short, to impress them. I think that sometimes we find this hard to do. We need to remind ourselves that sometimes it’s just the little things we do – things that others might have missed – that will make the biggest impression.

See, technology is my day job.  When I’m not working (or walking the dog, or schlepping kids around town), I usually have some sort of musical instrument in my hands. or in front of me. Guitar, mandolin, bass, keyboards, drums and trumpet, for now.  Hope to add to the collection soon – I think I need a bouzouki.  Or maybe a dobro.  But I digress…

When I have a block of free time and a little peace and quite in the house, I like to break out my recording gear and see what I can throw together.  And, occasionally, share it with a few friends to see what they think.  Or even send a track to a friend who’s a real drummer to lay down a drum track for me to mix in later.

In the past, sharing music online has been inconvenient at best.  Oh, there are music sharing sites like SoundClick, or social sites like MySpace that cater to musicians.  But they all have some pretty serious limitations (like having to actually log into their service to find and listen to tracks).

SoundCloud is different.  It’s a seriously open, cloud-based music storage and publishing service.  I just signed up an hour ago, and within minutes I was uploading my first track.  (I love, by the way, that you specify the file to upload first, and it automatically starts uploading while you fill in the track info form.)

But the really cool part is the openness of the service.  There’s no need for anyone to go to their site to hear my music.  I can email it to them, or drop a widget into Facebook or a blog entry (as I’ve done here), and voila.  Music.  Where I want it.  No strings attached.

And now, if you’ll excuse me, I have a recording project to dive into…

Remember Shimmer on Saturday Night Live? Dan Aykroyd and Gilda Radner? “It’s a dessert topping and a floor wax!” If you’re as old as I am or enough of an SNL fan to have caught that old episode on DVD, you’re probably chuckling a little. Humor has many triggers in the human brain, and absurdity is certainly one of them.

That’s why my initial reaction to something that happened at work the other day was to laugh – a reaction which, thankfully, I was successful in suppressing. Not only because laughing would have been unprofessional, but because, as is often the case, my first impression was superficial and, ultimately, off the mark.

The topic of discussion was the Continuous Integration build at a client. Let’s see if I can recreate the scene. Names, of course, have been changed to protect those who may not want their names to appear here…

Jeff the Consultant: I see that there are 800-odd unit tests in our code base, and I can run them locally, but the CI build only runs 150 or so of them. Is something wrong?

Bob the Builder: No, this is correct.

Jeff the Consultant: Come again?

Bob the Builder: Yes, we do this intentionally.

Jeff the Consultant: Okay, I have to ask. Why? Does it take too long to run all of the tests?

Bob the Builder: No, not really. The build with all the tests runs in 10 minutes or so.

Jeff the Consultant: Then why do we run a just a subset of them on code checkin?

Bob the Builder: Well, we run the full set of tests in a different build that runs nightly.

Jeff the Consultant: Hmm… Doesn’t running the tests at night make it difficult to track down whose code submission is responsible for tests that start failing? And doesn’t it remove the benefit of immediate feedback? I’m getting on in years, and sometimes I get to work in the morning not remembering a lot about what I did the night before.

Bob the Builder: I see your point. It would be a lot better if we ran the full set of tests all the time, on every code check-in.

Jeff the Consultant: Great. So you’ll make the necessary changes to the build script?

Bob the Builder: No, I can’t do that.

Jeff the Consultant: Wonderful, thanks… wait – did you just say “No?”

Bob the Builder: Correct. I can’t change the build.

Jeff the Consultant: Okay, I have to ask. Why not?

Bob the Builder: Because we don’t want this build to fail.

Jeff the Consultant: (This is about the point where I had to start physically suppressing laughter) But the build is supposed to fail if there are broken tests. That’s how we know there’s something that needs to be fixed, and who’s responsible – er, best qualified to fix it.

Bob the Builder: Right, I understand that. But the jars we produce in this build get deployed in an overnight process. So we don’t want the build to fail, because if it does, the daily build won’t get deployed.

Jeff the Consultant: Okay, I see that. But why do you want to deploy a build that you know has bugs in it? Or, in this case, that might have bugs in it, since you’re not running all of the tests…

Bob the Builder: Well, the deployment isn’t just for this team. There’s another team on the East Coast that needs that build to be deployed every day.

Jeff the Consultant: Hmm… interesting. Are the tests we’re not running our tests or theirs?

Bob the Builder: Mostly ours, I think. But it doesn’t matter – we can’t let broken tests stop the deployment, because usually only a small number of features are affected by those tests, and lots of people need to see the deployed application the next day in our demo environment.

Jeff the Consultant: Okay, that makes sense – I guess. But isn’t there another way to do this? For example, couldn’t we run the full suite of unit tests at code check-in time, and make the nightly build the one that skips tests and reliably produces the demo jar files? That way at least we’d get back the benefit of immediate feedback and have a better idea whose code is causing a test to start failing.

Bob the Builder: I think we could do that. But I don’t have time to dig into an overhaul of the build process, because I have development responsibilities too, and we’re only 2 days from code freeze.

(Exit cubicle, stage right…)

By this time my head was spinning. I wasn’t sure whether what I’d just heard made sense, or whether I’d just been had. I was pretty sure the whole build arrangement they had settled on was crazy, but I could also see how an organization could have evolved into this arrangement because of conflicting needs.

The root of their problem is the Shimmer conundrum. It’s neither a dessert topping nor a floor wax, but their CI build is certainly trying to be two very different things at the same time. It was probably started as a CI build in the Agile sense, with the primary purpose being immediate feedback in the form of unit test results.

But of course, like any build, it also builds something. Those jar files were ripe for the picking, and some enterprising person had the great idea to deploy them every day so interested parties in the company would be able to see the application “fresh from the oven” – with all of the evolving new features on display.

The problem is that the secondary purpose of this particular build – generation of compiled, deployable software, became more critical to somebody – some person with influence – and the build began to slowly stop meeting its initial purpose.

So, is there a point to this rambling blog post? Indeed there is. It’s a word of caution to those of us in the consulting profession. It’s very easy for those of us who spend a lot of time reading and in other ways learning about “ideal” situations – such as well-run Agile projects – to become easily frustrated when we encounter things that are in some way less than ideal. We have to remember to suspend judgment long enough to understand the big picture – all of the various problems our clients are trying to juggle – and then offer our assistance in helping improve things, at a pace that makes sense in their environment.

I’ll try to remember this myself as I return to work tomorrow…

There is one thing, however, that has bothered me since I started using Hibernate several years ago.  A deep bother.  Not a “this is a nuisance, but I can live with it” bother, but more of a “this is so fundamental an issue that there just has to be a better solution to it than is being offered by Hibernate’s authors” bother.

The problem:  bridging the gap between object identity and database identity.  In most modern database schemas, tables are defined with a numeric primary key column, which is populated through some unique key assignment mechanism (Identity columns in SQL Server or MySQL, sequences in Oracle, etc).

This works great at the database layer.  But problems arise in the JVM (or CLR for you NHibernate folks).  In order for objects representing database entities (those managed by Hibernate) to be handled properly in collections (including Hibernate cache), objects that have the same database identity need to be recognized as equals as well.  In Java, this means overriding equals() and hashCode() such that any two objects that represent the same database entity will be equal, and return the same hash code.

The “obvious” solution, at first blush, would be to base equals() and hashCode() on entity ID values.  This works great for entities created by Hibernate, which represent existing rows in a database, all with unique ID’s.  The problem has to do with new entites that you create programmatically, which haven’t yet been saved.  Those entities will typically have a null ID value – and if you base equality on ID values, any two unsaved instances will be considered equal.

There are actually two problems when dealing with unsaved entity objects.  The first, as stated above, is that they don’t yet have a unique identity – so multiple new instances will collide in collections like Maps and Sets.  The other problem is that the identity of these objects will be set at some point, which violates part of the contract of hashCode() – the part that says its value must not mutate over time.

To avoid all of thes problems, Hibernate’s authors have long recommended a workaround – use “natural keys” instead of identity values for equals() and hashCode(). The problem with this is that many entities simply don’t have a natural key – that is, a set of one or more attributes whose values uniquely identify the entity.  And those that do often fail the hashCode() qualification that the value should not change over time.

There are other possible solutions I’ve seen or come up with myself, but all had at least one glaring weakness.  It seemed that there was no silver bullet for dealing with database-generated identity values in Hibernate.

And therein lies the silver bullet.  Don’t use the database (or Hibernate) to generate identity values.

I stumbled across a wonderful article on O’Reilly’s OnJava site that quite clearly describes this problem (in more detail than I do here), and offers what I believe to be the only real solution.  Check it out at http://www.onjava.com/pub/a/onjava/2006/09/13/dont-let-hibernate-steal-your-identity.html.

And now, as I look out the window at the late February snowstorm, I think maybe it’s time for me to hibernate…

Now that I’ve had a week to let all the new information bounce around in my head for a while and settle into semi-organized piles of data, I have two major takeaways from the conference.

Firstly, Adobe users love their products, and they’re tremendously enthusiastic about the direction Adobe is taking them. In the main sessions, where all 6,000+ attendees were in the same huge room, many new features were met with thunderous applause. You just don’t get that sort of thing at a WebSphere conference.

Secondly, if CS4 is a big success – and I have no doubt but that it will be – we’ll start to see changes in the relationship between designers and developers on teams building RIA’s. CS4 contains a brilliant piece of software, Flex Catalyst, that lets designers using PhotoShop and Flash save content to the same source files as developers using Flex Builder. This provides a seamless transition from initial UI mockup to prototype to working application – and keeps the UI in the hands of the designer a little longer than has traditionally been the case. Designers can now exert great control over not only over the way an applicaiton looks, but also over the way it behaves as a user interacts with the various widgets in the UI. This will bring new challenges to RIA developers who work on teams with designers – developers will have less control, less latitude to say “We can’t do that – it’ll take too long”, or “Sorry, the tool won’t let us do that.” Developers will inherit a prototype that is quite complete in terms of look and feel, and will be focused more on adding functional meat to the UI bones (filling lists with live data, wiring events to real business services, etc.).

Oh, I did forget to mention one thing. This was my first time in San Francisco. What a wonderful city! I’ll definitely be back…

I must admit I’ve been a bit puzzled by the hype – but then I’ve never been one to jump on a bandwagon until I know just where it’s headed. So I started to dig a little deeper into this one, and I’m starting to see how it will be useful. I won’t prognosticate here as to how soon. Rather, I’ll spend a little time explaining what it is and what “problem” it solves, and let you do your own crystal-ball-gazing.

Before I start, a bit about my background. I’m an old-school computer guy. Learned BASIC on a Commodore VIC-20. One of my college profs had kids (nick)named COBOL and FORTRAN. I knew databases before they were relational, and languages before they were object-oriented. So I’ve seen a few major technology shifts (mainframes to PC’s to networked PC’s to the Internet, flat files to B-trees to relational databases to OODB’s, etc.).

I don’t expect the Semantic Web to be as game-changing as any of these. But I do think it’ll be useful and catch fire at some point – and will be of benefit to both consumers of technology and companies that make money through innovative use of technology.

The Semantic Web – What Is It?

I like to explain things through analogy. If I miss your frame of reference here, sorry – I tried. Remember the good old days when lots of data sat around in “flat files?” Maybe they were “positional” in nature, with the first 10 characters representing a social security number, the next 20 a first name, the next 20 a last name, etc. Or maybe they were delimited, with commas, tabs or other characters separating fields. At any rate, they all had one thing in common – a person (let alone a computer program) couldn’t interpret them without the “magic key” – some form of metadata that explained how the contents were arranged. For mainframe files, the “magic key” was often a COBOL file layout. You get the idea.

Fast forward a few years to the advent of relational databases. Suddenly, information was well-organized, and – here’s the kicker – self-explanatory. Not only could you easily run a query and get rows of neatly organized data with explanatory headings, you could also query database metadata – the schema – to inquire about the contents of the database.

Okay. Now, fast forward another decade and a half or so, to the dawn of the Web. Suddenly we have massive amounts of information available for perusal, and it’s (usually) pretty understandable by human readers. But don’t ask software to make heads or tails of an HTML document. A web page might contain information about an individual – name, address, phone number, freckle count, favorite ’80’s hair metal band, etc. – but that information isn’t organized in such a way that a software parser can pick it apart and know what it has (see below).

<html>
<body>
<p>My first name is Jeff.</p>
<p>My last name is Shurts.</p>
<p>My phone number is 555-1212.</p>
</body>
</html>

Enter the Semantic Web. If you were asked to write a piece of software that would dig through a file and display someone’s first name, last name and phone number, wouldn’t you be happier if that file’s content looked like this?

<?xml version="1.0" encoding="utf-16" ?>
<person>
<firstName>Jeff</firstName>
<lastName>Shurts</lastName>
<phoneNumber>555-1212</phoneNumber>
</person>

Ah, much better. Give me a DOM or SAX parser and I’m golden. No ambiguity, no tricky parsing tricks. But wait – XML has been around forever – that can’t be the essence of some Next Big Thing, right?

Right. Plain old XML is great for use in applications where there are a limited number of interested parties, all of whom agree on the format (schema, or DTD) of the data being exchanged. And XML has been used very effectively in this way – expecially in Web Services.

But back to our assignment from a few paragraphs ago. What if we were given a trickier problem… say, to write a web crawler that would scan millions of web pages looking for information about people – specifically first and last names, and phone numbers. We aren’t given a sample XML file or an XML schema. We just know we want information about people.

Now we have a nightmare on our hands. We might be able to look for XML documents (giving up on HTML altogether), and look for tags like <firstName>Jeff</firstName>. But we don’t have any guarantees that the tags were used in a manner consistent with their apparent meaning. Wouldn’t it be great if there were a way that people could (and would) encode data such that it was very clear, in a globally-understood manner, what the data means?

Enter RDF (Resource Description Framework). Based on XML, but forging on to solve bigger and more sophisticated problems, RDF provides a syntactic means of encoding very specific intent along with information. Here is an example:

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:foaf="http://xmlns.com/0.1/foaf/" >
<foaf:firstName>Jeff</foaf:firstName>
<foaf:lastName>Shurts</foaf:lastName>
<foaf:phone>555-1212</foaf:phone>
</rdf:RDF>

This document not only teases out the interesting bits into discrete fields, but also denotes that these fields are very specific attributes that are defined by a particular well-known model (in this case, FOAF, or Friend of a Friend). In a way, RDF does for web-hosted data what WSDL (Web Services Discovery Language) did for web-hosted services – it provides a well-understood description that can be programmatically interrogated and understood.

Okay, I get it. But how is this a game-changer?

Well, so far, it’s not. But it might be, and its chances are getting better. Earlier this year, Yahoo announced plans to include Semantic Web features in its search engine, to provide richer, more intelligent searches, as well as enhanced output from search results.

For those of you who know SQL, wouldn’t it be pretty nice to be able to run the equivalent of this query – not against a local Oracle table, but against the Web?

SELECT firstName, lastName
from People
where lastName = 'Shurts'
and homeCity = 'Saint Charles'

Voila – the Web as a giant database! Starting to see how this might be a game-changer? Finding just the information you are looking for with today’s search engines is an art – you have to find just the right set of keywords, and then hope for the best. In a semantically-rich future, you’ll be able to scour the web with the precision of a database query.

And, to quote the old Ronco commercials I used to watch when I wasn’t writing BASIC code on that VIC 20: “But wait, there’s more!” Not only will information searches be more precise, the results of those searches will be presented in a richer, semantically meaningful way. Instead of the ubiquitous three-liner (page title, URL, and a snippet) for each match, you will one day be presented information that makes sense given the context of your query. You searched for people? You get business-card-like search results. You searched for CD’s? You get track listings and cover art for each CD that matched your query (e.g. select * from albums where backgroundSinger includes ‘Michael McDonald’).

In conclusion…

Well, there it is. That’s as much of the Semantic Web as I’ve been able to wrap my head around so far. So what do you think? Is it a game changer? Or just incremental improvement over current Web technologies? Will it take off, or are there barriers to adoption? (If I’m an online merchandiser, do I really want to help web crawlers snarf my prices so they can include them in comparison-shopping sites?) I’d love to hear from you!

Of course it can; different programming languages have always had different strengths and weaknesses, and RIA languages are no different.  But in what ways?  I’ve been thinking about this quite a bit lately, as Redpoint has been engaged in projects that represent a few variations on the RIA theme.  In particular, I’ve noticed how eerily similar two particular technology stacks are, and yet how different they are in at least one key area.

Consider two applications, both of which use standard Java-based web technology on the application server (say, Tomcat, Spring and Hibernate).  For client-side technology, one uses Java Swing (via Web Start), and the other Adobe Flex.  Both employ middleware (Spring HttpInvoker on the one hand, Adobe Blaze on the other) to accomplish simple and efficient communication between client and server in the form of serialized objects over HTTP (as opposed to a more verbose SOAP or XML-RPC-based solution).

From many angles, these applications are very similar; the app server receives requests containing binary data (serialized from client-side objects), and returns binary data to the client in response.  In neither case do programmers need to pay attention to the object serialization; once it’s configured into the environment early in the project, it’s done, and thereafter it "just works."

But there is one difference, and I think it’s profound.  In the all-Java solution (with the Swing front end), not only does the client receive objects from the server, but – this is key – those objects are instances of the same classes the server uses in its JVM.  Okay, great – but why is this important?

It’s very important if, as an application architect, you’re interested in building a rich set of domain classes that contain all of the important business rules and operations in the application.  (If you’re content with "domain classes" that are little more than containers for data that’s been spewed forth by Hibernate, read no further – the rest won’t interest you.)

In the Tomcat-to-Swing application, we have tremendous flexibility in terms of location of code that some refer to as "transaction script" – code that manipulates a rich set of domain objects in order to produce some meaningful outcome (place an order for the latest Tom Petty CD, transfer funds from your beleaguered mutual fund into bonds, etc.).  Because the rich behavior of the Java domain classes resides on both the server and the client, we have the option of writing our transaction script code on the client as well as the server.

In the Tomcat-to-Flex application, this flexibility, ironically, is gone.  The Flex application sits on the other side of a technology chasm – yes, it is based on objects, and yes, they are returned from the server and magically "appear" in responses received by the client.  But the Flex classes are written in ActionScript, not in Java.  Which means they are not the same behavior-rich set of domain classes that exist in the server JVM.

As an architect of an application with disparate development languages on client and server, you have an important choice to make.  Do you maintain two full-featured sets of domain classes, one in ActionScript and one in Java?  Or do you treat the ActionScript layer as "domain light," eschewing the dual development (and more importantly, dual maintenance), and do all of the meaningful application work on the server (where a lot of traditionalists and web service or ESB lovers will argue that it belongs anyway)?

I’ve built applications both ways.  The traditional approach of pushing all of the meaningful application work onto the server does have its benefits, in terms of simplifying the architecture, and helping to ensure that the client app doesn’t get too "chatty" (communications with a very thin client tend to be less frequent and larger-grained in nature).  But there are benefits to taking advantage of a rich client and doing some of the work there.  Scalability is one obvious benefit; by allowing the client to do some of the work, you by definition relieve some of the load on the server.  All other things being equal, an application that shares processing responsibility between client and server will scale better than one in which the server carries the load alone.  The "thicker client" application can also be more responsive, responding quicker and more meaningfully to the user (this is, after all, why we are building rich applications now instead of relying on creaky old HTML forms, right?)

I’d love to hear from others who have thought about this.  Is having a single set of domain classes, with rich behavior available to both client and server, an important consideration when selecting an architecture?  Is it important enough to sway your choice of client-side development technologies?