Most of these features – closures, ultra-convenient collection-oriented operators like the spread-dot (*.), etc., are well-documented. Some, however, are a bit more arcane, and not realizing they exist can actually cost you a lot of time if you stumble into them inadvertently.

The feature I have in mind at the moment is a very convenient but somewhat surprising bit of collection behavior in Groovy. I’ve been working with the language for quite a while, and have dutifully pored over books and language reference documents to become familiar with all of the groovy goodness at my disposal, but somehow I missed this one until someone on my project team found it – quite by accident.

Here’s the magic find. In Groovy, if you have a list of objects, you can access, on the list itself, properties that belong to objects within the list. The result, somewhat intuitively (but somehow quizzical at the same time), is a list of values of that property across all the members in the list. Here’s an example to illustrate what I mean – just copy this and run it as a Groovy script.

class Player {
def name
def teams

public String toString() {
"${name} - played for ${teams}"
}
}

def players =
[
new Player(name: "Greg Maddux", teams: ["Cubs", "Braves"]),
new Player(name: "Ron Santo", teams: ["Cubs", "White Sox"]),
new Player(name: "Billy Williams", teams: ["Cubs", "Athletics"]),
new Player(name: "Bruce Sutter", teams: ["Cubs", "Cardinals"]),
new Player(name: "Lou Brock", teams: ["Cubs", "Cardinals"]),
new Player(name: "Gary Gaetti", teams: ["Twins", "Cubs"]),
new Player(name: "Ron Cey", teams: ["Dodgers", "Cubs"]),
new Player(name: "Ryne Sandberg", teams: ["Cubs"]),
new Player(name: "Mark Grace", teams: ["Cubs", "Diamondbacks"]),
new Player(name: "Goose Gossage", teams: ["Athletics", "Yankees", "Cubs"]),
new Player(name: "Sammy Sosa", teams: ["Rangers", "White Sox", "Cubs"]),
new Player(name: "Larry Bowa", teams: ["Phillies", "Cubs"]),
new Player(name: "Rick Monday", teams: ["Dodgers", "Cubs"]),
new Player(name: "Dave Kingman", teams: ["Rangers", "Mets", "Cubs"]),
new Player(name: "Joe Girardi", teams: ["Cubs", "Cardinals", "Yankees"])
]

def playerNames = players.name
println "Here, amazingly enough, is a list containing the names of all those players:\n $(playerNames)"

Once you know this behavior exists, it’s pretty easy to find situations where it’s an insanely convenient feature. If you don’t know it exists, you can get some pretty confusing results. In our case, we were running a GORM query to retrieve a domain object using an alternate ID (expecting a single result), and incorrectly used the list() method (we should have used get(), which returns the domain object, whereas list() always returns a list).

In our code, after querying for the domain object, we accessed properties of the object, and were puzzled when instead of getting Strings and Integers and Dates, we were getting single-member lists of Strings and Integers and Dates. Each list contained the value we expected, inexplicably embedded in a List instance. It took a little head-scratching before we figured out that the root cause was that our query was returning a list containing our domain object, and our code that accessed the properties, which would have failed miserably in Java if attempting to treat a List as though it were a domain object, was happily drilling through the wall of that list and getting at its contents.

Syntactic sugar? Definitely. Violation of the principle of least surprise? Maybe. Will I use it, now that I know it’s a tool at my disposal? Absolutely.

Stay Groovy, my friends…

But such peace of mind comes at a cost. I’ve heard estimates from various teams I’ve talked to stating that the cost of writing unit tests for adequate code coverage (usually 50% to 80% on Java projects) can range from equivalent to the production code itself, to five times that. The problem with these estimates, of course, is subjectivity. For example, if you have found a bug using a test and work to fix the test, are you mentally counting that time as test-writing or code-writing?

Estimating factors aside, the plain fact is that testing does amplify the cost of software development – at least in the short term. Which means that a software engineer who really cares about his customer’s bottom line should always be on the lookout for ways to avoid the increased cost of testing.

This line of thinking arose the other day on the project I’m currently working on. It’s a Groovy/Grails web application with some public, customer-facing components and some internal-use components. We’re using Grails scaffolding as much as is practical for the relatively infrequently-used internal-use features to save time and money. In most cases, we have generated and tweaked the UI components (GSP pages) to smooth over some of the uglier bits of scaffolded pages, but we have tried not to generate any of the controller code, instead relying on Grails’ generate-at-runtime approach by including def scaffold = true in these controllers.

Why not generate the source? Because generated code is deceptively expensive! The minute a feature springs into existence in your code base, it needs to be tested – even if that code was generated by a framework like Grails. And Grails controller closures – insert and update, as the case in point here, do a significant amount work for you. Data binding. Optimistic locking exception handling. Routing of validation errors. All things that “just work” in scaffolded code, but which might be broken by someone once the code actually becomes part of your code base.

So when the need arose to customize part of the logic in insert and update closures (to do some custom interpretation of a series of check boxes on a form that couldn’t be made to conform to the default binding mechanism), we began looking for ways to override binding logic without generating and modifying the closures themselves.

As usual, Grails has a solution at the ready – controller interceptors. All we had to do was declare an interceptor that fires before insert and update closures, and in that closure, inspect our list of check boxes and set a params value that would then be assigned to our domain object through the normal binding process.

By using an interceptor, we completely avoided the need to generate (and assume custodianship of) the insert and update closures in this controller – which means we didn’t have to spend valuable time writing unit tests to cover these Grails-supplied features that could be broken once the code actually exists.

I guess I can summarize this little victory this way: Having tested code for an application feature is far better than having untested code. Having no code at all is better than either.

Stay Groovy, my friends.

When I first started test driving the Magic Mouse, I happened to be in iTunes, and happened to need to adjust the volume – which is a horizontal slider in the UI. I thought to myself, “This mouse has the ability to do scrolling in any direction – I wonder what happens if I hover over the volume control and swipe right?” Bam. Of course it worked. It’s Apple, and they think of *everything.*

I was truly impressed. And, being an engineer and naturally inquisitive, I took notice that I was impressed – particularly by such a “little thing.” I then realized that I was impressed not despite the fact that this was a little thing, but because it was a little thing. The sort of thing that other user interface and hardware designers might well have missed.

Which brings me to the moral of this story. I work in a consulting firm, and our goal is not merely to meet our customers’ expectations, but to surpass them. In short, to impress them. I think that sometimes we find this hard to do. We need to remind ourselves that sometimes it’s just the little things we do – things that others might have missed – that will make the biggest impression.

See, technology is my day job.  When I’m not working (or walking the dog, or schlepping kids around town), I usually have some sort of musical instrument in my hands. or in front of me. Guitar, mandolin, bass, keyboards, drums and trumpet, for now.  Hope to add to the collection soon – I think I need a bouzouki.  Or maybe a dobro.  But I digress…

When I have a block of free time and a little peace and quite in the house, I like to break out my recording gear and see what I can throw together.  And, occasionally, share it with a few friends to see what they think.  Or even send a track to a friend who’s a real drummer to lay down a drum track for me to mix in later.

In the past, sharing music online has been inconvenient at best.  Oh, there are music sharing sites like SoundClick, or social sites like MySpace that cater to musicians.  But they all have some pretty serious limitations (like having to actually log into their service to find and listen to tracks).

SoundCloud is different.  It’s a seriously open, cloud-based music storage and publishing service.  I just signed up an hour ago, and within minutes I was uploading my first track.  (I love, by the way, that you specify the file to upload first, and it automatically starts uploading while you fill in the track info form.)

But the really cool part is the openness of the service.  There’s no need for anyone to go to their site to hear my music.  I can email it to them, or drop a widget into Facebook or a blog entry (as I’ve done here), and voila.  Music.  Where I want it.  No strings attached.

And now, if you’ll excuse me, I have a recording project to dive into…

Remember Shimmer on Saturday Night Live? Dan Aykroyd and Gilda Radner? “It’s a dessert topping and a floor wax!” If you’re as old as I am or enough of an SNL fan to have caught that old episode on DVD, you’re probably chuckling a little. Humor has many triggers in the human brain, and absurdity is certainly one of them.

That’s why my initial reaction to something that happened at work the other day was to laugh – a reaction which, thankfully, I was successful in suppressing. Not only because laughing would have been unprofessional, but because, as is often the case, my first impression was superficial and, ultimately, off the mark.

The topic of discussion was the Continuous Integration build at a client. Let’s see if I can recreate the scene. Names, of course, have been changed to protect those who may not want their names to appear here…

Jeff the Consultant: I see that there are 800-odd unit tests in our code base, and I can run them locally, but the CI build only runs 150 or so of them. Is something wrong?

Bob the Builder: No, this is correct.

Jeff the Consultant: Come again?

Bob the Builder: Yes, we do this intentionally.

Jeff the Consultant: Okay, I have to ask. Why? Does it take too long to run all of the tests?

Bob the Builder: No, not really. The build with all the tests runs in 10 minutes or so.

Jeff the Consultant: Then why do we run a just a subset of them on code checkin?

Bob the Builder: Well, we run the full set of tests in a different build that runs nightly.

Jeff the Consultant: Hmm… Doesn’t running the tests at night make it difficult to track down whose code submission is responsible for tests that start failing? And doesn’t it remove the benefit of immediate feedback? I’m getting on in years, and sometimes I get to work in the morning not remembering a lot about what I did the night before.

Bob the Builder: I see your point. It would be a lot better if we ran the full set of tests all the time, on every code check-in.

Jeff the Consultant: Great. So you’ll make the necessary changes to the build script?

Bob the Builder: No, I can’t do that.

Jeff the Consultant: Wonderful, thanks… wait – did you just say “No?”

Bob the Builder: Correct. I can’t change the build.

Jeff the Consultant: Okay, I have to ask. Why not?

Bob the Builder: Because we don’t want this build to fail.

Jeff the Consultant: (This is about the point where I had to start physically suppressing laughter) But the build is supposed to fail if there are broken tests. That’s how we know there’s something that needs to be fixed, and who’s responsible – er, best qualified to fix it.

Bob the Builder: Right, I understand that. But the jars we produce in this build get deployed in an overnight process. So we don’t want the build to fail, because if it does, the daily build won’t get deployed.

Jeff the Consultant: Okay, I see that. But why do you want to deploy a build that you know has bugs in it? Or, in this case, that might have bugs in it, since you’re not running all of the tests…

Bob the Builder: Well, the deployment isn’t just for this team. There’s another team on the East Coast that needs that build to be deployed every day.

Jeff the Consultant: Hmm… interesting. Are the tests we’re not running our tests or theirs?

Bob the Builder: Mostly ours, I think. But it doesn’t matter – we can’t let broken tests stop the deployment, because usually only a small number of features are affected by those tests, and lots of people need to see the deployed application the next day in our demo environment.

Jeff the Consultant: Okay, that makes sense – I guess. But isn’t there another way to do this? For example, couldn’t we run the full suite of unit tests at code check-in time, and make the nightly build the one that skips tests and reliably produces the demo jar files? That way at least we’d get back the benefit of immediate feedback and have a better idea whose code is causing a test to start failing.

Bob the Builder: I think we could do that. But I don’t have time to dig into an overhaul of the build process, because I have development responsibilities too, and we’re only 2 days from code freeze.

(Exit cubicle, stage right…)

By this time my head was spinning. I wasn’t sure whether what I’d just heard made sense, or whether I’d just been had. I was pretty sure the whole build arrangement they had settled on was crazy, but I could also see how an organization could have evolved into this arrangement because of conflicting needs.

The root of their problem is the Shimmer conundrum. It’s neither a dessert topping nor a floor wax, but their CI build is certainly trying to be two very different things at the same time. It was probably started as a CI build in the Agile sense, with the primary purpose being immediate feedback in the form of unit test results.

But of course, like any build, it also builds something. Those jar files were ripe for the picking, and some enterprising person had the great idea to deploy them every day so interested parties in the company would be able to see the application “fresh from the oven” – with all of the evolving new features on display.

The problem is that the secondary purpose of this particular build – generation of compiled, deployable software, became more critical to somebody – some person with influence – and the build began to slowly stop meeting its initial purpose.

So, is there a point to this rambling blog post? Indeed there is. It’s a word of caution to those of us in the consulting profession. It’s very easy for those of us who spend a lot of time reading and in other ways learning about “ideal” situations – such as well-run Agile projects – to become easily frustrated when we encounter things that are in some way less than ideal. We have to remember to suspend judgment long enough to understand the big picture – all of the various problems our clients are trying to juggle – and then offer our assistance in helping improve things, at a pace that makes sense in their environment.

I’ll try to remember this myself as I return to work tomorrow…

There is one thing, however, that has bothered me since I started using Hibernate several years ago.  A deep bother.  Not a “this is a nuisance, but I can live with it” bother, but more of a “this is so fundamental an issue that there just has to be a better solution to it than is being offered by Hibernate’s authors” bother.

The problem:  bridging the gap between object identity and database identity.  In most modern database schemas, tables are defined with a numeric primary key column, which is populated through some unique key assignment mechanism (Identity columns in SQL Server or MySQL, sequences in Oracle, etc).

This works great at the database layer.  But problems arise in the JVM (or CLR for you NHibernate folks).  In order for objects representing database entities (those managed by Hibernate) to be handled properly in collections (including Hibernate cache), objects that have the same database identity need to be recognized as equals as well.  In Java, this means overriding equals() and hashCode() such that any two objects that represent the same database entity will be equal, and return the same hash code.

The “obvious” solution, at first blush, would be to base equals() and hashCode() on entity ID values.  This works great for entities created by Hibernate, which represent existing rows in a database, all with unique ID’s.  The problem has to do with new entites that you create programmatically, which haven’t yet been saved.  Those entities will typically have a null ID value – and if you base equality on ID values, any two unsaved instances will be considered equal.

There are actually two problems when dealing with unsaved entity objects.  The first, as stated above, is that they don’t yet have a unique identity – so multiple new instances will collide in collections like Maps and Sets.  The other problem is that the identity of these objects will be set at some point, which violates part of the contract of hashCode() – the part that says its value must not mutate over time.

To avoid all of thes problems, Hibernate’s authors have long recommended a workaround – use “natural keys” instead of identity values for equals() and hashCode(). The problem with this is that many entities simply don’t have a natural key – that is, a set of one or more attributes whose values uniquely identify the entity.  And those that do often fail the hashCode() qualification that the value should not change over time.

There are other possible solutions I’ve seen or come up with myself, but all had at least one glaring weakness.  It seemed that there was no silver bullet for dealing with database-generated identity values in Hibernate.

And therein lies the silver bullet.  Don’t use the database (or Hibernate) to generate identity values.

I stumbled across a wonderful article on O’Reilly’s OnJava site that quite clearly describes this problem (in more detail than I do here), and offers what I believe to be the only real solution.  Check it out at http://www.onjava.com/pub/a/onjava/2006/09/13/dont-let-hibernate-steal-your-identity.html.

And now, as I look out the window at the late February snowstorm, I think maybe it’s time for me to hibernate…

Now that I’ve had a week to let all the new information bounce around in my head for a while and settle into semi-organized piles of data, I have two major takeaways from the conference.

Firstly, Adobe users love their products, and they’re tremendously enthusiastic about the direction Adobe is taking them. In the main sessions, where all 6,000+ attendees were in the same huge room, many new features were met with thunderous applause. You just don’t get that sort of thing at a WebSphere conference.

Secondly, if CS4 is a big success – and I have no doubt but that it will be – we’ll start to see changes in the relationship between designers and developers on teams building RIA’s. CS4 contains a brilliant piece of software, Flex Catalyst, that lets designers using PhotoShop and Flash save content to the same source files as developers using Flex Builder. This provides a seamless transition from initial UI mockup to prototype to working application – and keeps the UI in the hands of the designer a little longer than has traditionally been the case. Designers can now exert great control over not only over the way an applicaiton looks, but also over the way it behaves as a user interacts with the various widgets in the UI. This will bring new challenges to RIA developers who work on teams with designers – developers will have less control, less latitude to say “We can’t do that – it’ll take too long”, or “Sorry, the tool won’t let us do that.” Developers will inherit a prototype that is quite complete in terms of look and feel, and will be focused more on adding functional meat to the UI bones (filling lists with live data, wiring events to real business services, etc.).

Oh, I did forget to mention one thing. This was my first time in San Francisco. What a wonderful city! I’ll definitely be back…

I must admit I’ve been a bit puzzled by the hype – but then I’ve never been one to jump on a bandwagon until I know just where it’s headed. So I started to dig a little deeper into this one, and I’m starting to see how it will be useful. I won’t prognosticate here as to how soon. Rather, I’ll spend a little time explaining what it is and what “problem” it solves, and let you do your own crystal-ball-gazing.

Before I start, a bit about my background. I’m an old-school computer guy. Learned BASIC on a Commodore VIC-20. One of my college profs had kids (nick)named COBOL and FORTRAN. I knew databases before they were relational, and languages before they were object-oriented. So I’ve seen a few major technology shifts (mainframes to PC’s to networked PC’s to the Internet, flat files to B-trees to relational databases to OODB’s, etc.).

I don’t expect the Semantic Web to be as game-changing as any of these. But I do think it’ll be useful and catch fire at some point – and will be of benefit to both consumers of technology and companies that make money through innovative use of technology.

The Semantic Web – What Is It?

I like to explain things through analogy. If I miss your frame of reference here, sorry – I tried. Remember the good old days when lots of data sat around in “flat files?” Maybe they were “positional” in nature, with the first 10 characters representing a social security number, the next 20 a first name, the next 20 a last name, etc. Or maybe they were delimited, with commas, tabs or other characters separating fields. At any rate, they all had one thing in common – a person (let alone a computer program) couldn’t interpret them without the “magic key” – some form of metadata that explained how the contents were arranged. For mainframe files, the “magic key” was often a COBOL file layout. You get the idea.

Fast forward a few years to the advent of relational databases. Suddenly, information was well-organized, and – here’s the kicker – self-explanatory. Not only could you easily run a query and get rows of neatly organized data with explanatory headings, you could also query database metadata – the schema – to inquire about the contents of the database.

Okay. Now, fast forward another decade and a half or so, to the dawn of the Web. Suddenly we have massive amounts of information available for perusal, and it’s (usually) pretty understandable by human readers. But don’t ask software to make heads or tails of an HTML document. A web page might contain information about an individual – name, address, phone number, freckle count, favorite ’80’s hair metal band, etc. – but that information isn’t organized in such a way that a software parser can pick it apart and know what it has (see below).

<html>
<body>
<p>My first name is Jeff.</p>
<p>My last name is Shurts.</p>
<p>My phone number is 555-1212.</p>
</body>
</html>

Enter the Semantic Web. If you were asked to write a piece of software that would dig through a file and display someone’s first name, last name and phone number, wouldn’t you be happier if that file’s content looked like this?

<?xml version="1.0" encoding="utf-16" ?>
<person>
<firstName>Jeff</firstName>
<lastName>Shurts</lastName>
<phoneNumber>555-1212</phoneNumber>
</person>

Ah, much better. Give me a DOM or SAX parser and I’m golden. No ambiguity, no tricky parsing tricks. But wait – XML has been around forever – that can’t be the essence of some Next Big Thing, right?

Right. Plain old XML is great for use in applications where there are a limited number of interested parties, all of whom agree on the format (schema, or DTD) of the data being exchanged. And XML has been used very effectively in this way – expecially in Web Services.

But back to our assignment from a few paragraphs ago. What if we were given a trickier problem… say, to write a web crawler that would scan millions of web pages looking for information about people – specifically first and last names, and phone numbers. We aren’t given a sample XML file or an XML schema. We just know we want information about people.

Now we have a nightmare on our hands. We might be able to look for XML documents (giving up on HTML altogether), and look for tags like <firstName>Jeff</firstName>. But we don’t have any guarantees that the tags were used in a manner consistent with their apparent meaning. Wouldn’t it be great if there were a way that people could (and would) encode data such that it was very clear, in a globally-understood manner, what the data means?

Enter RDF (Resource Description Framework). Based on XML, but forging on to solve bigger and more sophisticated problems, RDF provides a syntactic means of encoding very specific intent along with information. Here is an example:

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:foaf="http://xmlns.com/0.1/foaf/" >
<foaf:firstName>Jeff</foaf:firstName>
<foaf:lastName>Shurts</foaf:lastName>
<foaf:phone>555-1212</foaf:phone>
</rdf:RDF>

This document not only teases out the interesting bits into discrete fields, but also denotes that these fields are very specific attributes that are defined by a particular well-known model (in this case, FOAF, or Friend of a Friend). In a way, RDF does for web-hosted data what WSDL (Web Services Discovery Language) did for web-hosted services – it provides a well-understood description that can be programmatically interrogated and understood.

Okay, I get it. But how is this a game-changer?

Well, so far, it’s not. But it might be, and its chances are getting better. Earlier this year, Yahoo announced plans to include Semantic Web features in its search engine, to provide richer, more intelligent searches, as well as enhanced output from search results.

For those of you who know SQL, wouldn’t it be pretty nice to be able to run the equivalent of this query – not against a local Oracle table, but against the Web?

SELECT firstName, lastName
from People
where lastName = 'Shurts'
and homeCity = 'Saint Charles'

Voila – the Web as a giant database! Starting to see how this might be a game-changer? Finding just the information you are looking for with today’s search engines is an art – you have to find just the right set of keywords, and then hope for the best. In a semantically-rich future, you’ll be able to scour the web with the precision of a database query.

And, to quote the old Ronco commercials I used to watch when I wasn’t writing BASIC code on that VIC 20: “But wait, there’s more!” Not only will information searches be more precise, the results of those searches will be presented in a richer, semantically meaningful way. Instead of the ubiquitous three-liner (page title, URL, and a snippet) for each match, you will one day be presented information that makes sense given the context of your query. You searched for people? You get business-card-like search results. You searched for CD’s? You get track listings and cover art for each CD that matched your query (e.g. select * from albums where backgroundSinger includes ‘Michael McDonald’).

In conclusion…

Well, there it is. That’s as much of the Semantic Web as I’ve been able to wrap my head around so far. So what do you think? Is it a game changer? Or just incremental improvement over current Web technologies? Will it take off, or are there barriers to adoption? (If I’m an online merchandiser, do I really want to help web crawlers snarf my prices so they can include them in comparison-shopping sites?) I’d love to hear from you!