Most of these features – closures, ultra-convenient collection-oriented operators like the spread-dot (*.), etc., are well-documented. Some, however, are a bit more arcane, and not realizing they exist can actually cost you a lot of time if you stumble into them inadvertently.

The feature I have in mind at the moment is a very convenient but somewhat surprising bit of collection behavior in Groovy. I’ve been working with the language for quite a while, and have dutifully pored over books and language reference documents to become familiar with all of the groovy goodness at my disposal, but somehow I missed this one until someone on my project team found it – quite by accident.

Here’s the magic find. In Groovy, if you have a list of objects, you can access, on the list itself, properties that belong to objects within the list. The result, somewhat intuitively (but somehow quizzical at the same time), is a list of values of that property across all the members in the list. Here’s an example to illustrate what I mean – just copy this and run it as a Groovy script.

class Player {
def name
def teams

public String toString() {
"${name} - played for ${teams}"
}
}

def players =
[
new Player(name: "Greg Maddux", teams: ["Cubs", "Braves"]),
new Player(name: "Ron Santo", teams: ["Cubs", "White Sox"]),
new Player(name: "Billy Williams", teams: ["Cubs", "Athletics"]),
new Player(name: "Bruce Sutter", teams: ["Cubs", "Cardinals"]),
new Player(name: "Lou Brock", teams: ["Cubs", "Cardinals"]),
new Player(name: "Gary Gaetti", teams: ["Twins", "Cubs"]),
new Player(name: "Ron Cey", teams: ["Dodgers", "Cubs"]),
new Player(name: "Ryne Sandberg", teams: ["Cubs"]),
new Player(name: "Mark Grace", teams: ["Cubs", "Diamondbacks"]),
new Player(name: "Goose Gossage", teams: ["Athletics", "Yankees", "Cubs"]),
new Player(name: "Sammy Sosa", teams: ["Rangers", "White Sox", "Cubs"]),
new Player(name: "Larry Bowa", teams: ["Phillies", "Cubs"]),
new Player(name: "Rick Monday", teams: ["Dodgers", "Cubs"]),
new Player(name: "Dave Kingman", teams: ["Rangers", "Mets", "Cubs"]),
new Player(name: "Joe Girardi", teams: ["Cubs", "Cardinals", "Yankees"])
]

def playerNames = players.name
println "Here, amazingly enough, is a list containing the names of all those players:\n $(playerNames)"

Once you know this behavior exists, it’s pretty easy to find situations where it’s an insanely convenient feature. If you don’t know it exists, you can get some pretty confusing results. In our case, we were running a GORM query to retrieve a domain object using an alternate ID (expecting a single result), and incorrectly used the list() method (we should have used get(), which returns the domain object, whereas list() always returns a list).

In our code, after querying for the domain object, we accessed properties of the object, and were puzzled when instead of getting Strings and Integers and Dates, we were getting single-member lists of Strings and Integers and Dates. Each list contained the value we expected, inexplicably embedded in a List instance. It took a little head-scratching before we figured out that the root cause was that our query was returning a list containing our domain object, and our code that accessed the properties, which would have failed miserably in Java if attempting to treat a List as though it were a domain object, was happily drilling through the wall of that list and getting at its contents.

Syntactic sugar? Definitely. Violation of the principle of least surprise? Maybe. Will I use it, now that I know it’s a tool at my disposal? Absolutely.

Stay Groovy, my friends…

But such peace of mind comes at a cost. I’ve heard estimates from various teams I’ve talked to stating that the cost of writing unit tests for adequate code coverage (usually 50% to 80% on Java projects) can range from equivalent to the production code itself, to five times that. The problem with these estimates, of course, is subjectivity. For example, if you have found a bug using a test and work to fix the test, are you mentally counting that time as test-writing or code-writing?

Estimating factors aside, the plain fact is that testing does amplify the cost of software development – at least in the short term. Which means that a software engineer who really cares about his customer’s bottom line should always be on the lookout for ways to avoid the increased cost of testing.

This line of thinking arose the other day on the project I’m currently working on. It’s a Groovy/Grails web application with some public, customer-facing components and some internal-use components. We’re using Grails scaffolding as much as is practical for the relatively infrequently-used internal-use features to save time and money. In most cases, we have generated and tweaked the UI components (GSP pages) to smooth over some of the uglier bits of scaffolded pages, but we have tried not to generate any of the controller code, instead relying on Grails’ generate-at-runtime approach by including def scaffold = true in these controllers.

Why not generate the source? Because generated code is deceptively expensive! The minute a feature springs into existence in your code base, it needs to be tested – even if that code was generated by a framework like Grails. And Grails controller closures – insert and update, as the case in point here, do a significant amount work for you. Data binding. Optimistic locking exception handling. Routing of validation errors. All things that “just work” in scaffolded code, but which might be broken by someone once the code actually becomes part of your code base.

So when the need arose to customize part of the logic in insert and update closures (to do some custom interpretation of a series of check boxes on a form that couldn’t be made to conform to the default binding mechanism), we began looking for ways to override binding logic without generating and modifying the closures themselves.

As usual, Grails has a solution at the ready – controller interceptors. All we had to do was declare an interceptor that fires before insert and update closures, and in that closure, inspect our list of check boxes and set a params value that would then be assigned to our domain object through the normal binding process.

By using an interceptor, we completely avoided the need to generate (and assume custodianship of) the insert and update closures in this controller – which means we didn’t have to spend valuable time writing unit tests to cover these Grails-supplied features that could be broken once the code actually exists.

I guess I can summarize this little victory this way: Having tested code for an application feature is far better than having untested code. Having no code at all is better than either.

Stay Groovy, my friends.

When I first started test driving the Magic Mouse, I happened to be in iTunes, and happened to need to adjust the volume – which is a horizontal slider in the UI. I thought to myself, “This mouse has the ability to do scrolling in any direction – I wonder what happens if I hover over the volume control and swipe right?” Bam. Of course it worked. It’s Apple, and they think of *everything.*

I was truly impressed. And, being an engineer and naturally inquisitive, I took notice that I was impressed – particularly by such a “little thing.” I then realized that I was impressed not despite the fact that this was a little thing, but because it was a little thing. The sort of thing that other user interface and hardware designers might well have missed.

Which brings me to the moral of this story. I work in a consulting firm, and our goal is not merely to meet our customers’ expectations, but to surpass them. In short, to impress them. I think that sometimes we find this hard to do. We need to remind ourselves that sometimes it’s just the little things we do – things that others might have missed – that will make the biggest impression.

See, technology is my day job.  When I’m not working (or walking the dog, or schlepping kids around town), I usually have some sort of musical instrument in my hands. or in front of me. Guitar, mandolin, bass, keyboards, drums and trumpet, for now.  Hope to add to the collection soon – I think I need a bouzouki.  Or maybe a dobro.  But I digress…

When I have a block of free time and a little peace and quite in the house, I like to break out my recording gear and see what I can throw together.  And, occasionally, share it with a few friends to see what they think.  Or even send a track to a friend who’s a real drummer to lay down a drum track for me to mix in later.

In the past, sharing music online has been inconvenient at best.  Oh, there are music sharing sites like SoundClick, or social sites like MySpace that cater to musicians.  But they all have some pretty serious limitations (like having to actually log into their service to find and listen to tracks).

SoundCloud is different.  It’s a seriously open, cloud-based music storage and publishing service.  I just signed up an hour ago, and within minutes I was uploading my first track.  (I love, by the way, that you specify the file to upload first, and it automatically starts uploading while you fill in the track info form.)

But the really cool part is the openness of the service.  There’s no need for anyone to go to their site to hear my music.  I can email it to them, or drop a widget into Facebook or a blog entry (as I’ve done here), and voila.  Music.  Where I want it.  No strings attached.

And now, if you’ll excuse me, I have a recording project to dive into…

Remember Shimmer on Saturday Night Live? Dan Aykroyd and Gilda Radner? “It’s a dessert topping and a floor wax!” If you’re as old as I am or enough of an SNL fan to have caught that old episode on DVD, you’re probably chuckling a little. Humor has many triggers in the human brain, and absurdity is certainly one of them.

That’s why my initial reaction to something that happened at work the other day was to laugh – a reaction which, thankfully, I was successful in suppressing. Not only because laughing would have been unprofessional, but because, as is often the case, my first impression was superficial and, ultimately, off the mark.

The topic of discussion was the Continuous Integration build at a client. Let’s see if I can recreate the scene. Names, of course, have been changed to protect those who may not want their names to appear here…

Jeff the Consultant: I see that there are 800-odd unit tests in our code base, and I can run them locally, but the CI build only runs 150 or so of them. Is something wrong?

Bob the Builder: No, this is correct.

Jeff the Consultant: Come again?

Bob the Builder: Yes, we do this intentionally.

Jeff the Consultant: Okay, I have to ask. Why? Does it take too long to run all of the tests?

Bob the Builder: No, not really. The build with all the tests runs in 10 minutes or so.

Jeff the Consultant: Then why do we run a just a subset of them on code checkin?

Bob the Builder: Well, we run the full set of tests in a different build that runs nightly.

Jeff the Consultant: Hmm… Doesn’t running the tests at night make it difficult to track down whose code submission is responsible for tests that start failing? And doesn’t it remove the benefit of immediate feedback? I’m getting on in years, and sometimes I get to work in the morning not remembering a lot about what I did the night before.

Bob the Builder: I see your point. It would be a lot better if we ran the full set of tests all the time, on every code check-in.

Jeff the Consultant: Great. So you’ll make the necessary changes to the build script?

Bob the Builder: No, I can’t do that.

Jeff the Consultant: Wonderful, thanks… wait – did you just say “No?”

Bob the Builder: Correct. I can’t change the build.

Jeff the Consultant: Okay, I have to ask. Why not?

Bob the Builder: Because we don’t want this build to fail.

Jeff the Consultant: (This is about the point where I had to start physically suppressing laughter) But the build is supposed to fail if there are broken tests. That’s how we know there’s something that needs to be fixed, and who’s responsible – er, best qualified to fix it.

Bob the Builder: Right, I understand that. But the jars we produce in this build get deployed in an overnight process. So we don’t want the build to fail, because if it does, the daily build won’t get deployed.

Jeff the Consultant: Okay, I see that. But why do you want to deploy a build that you know has bugs in it? Or, in this case, that might have bugs in it, since you’re not running all of the tests…

Bob the Builder: Well, the deployment isn’t just for this team. There’s another team on the East Coast that needs that build to be deployed every day.

Jeff the Consultant: Hmm… interesting. Are the tests we’re not running our tests or theirs?

Bob the Builder: Mostly ours, I think. But it doesn’t matter – we can’t let broken tests stop the deployment, because usually only a small number of features are affected by those tests, and lots of people need to see the deployed application the next day in our demo environment.

Jeff the Consultant: Okay, that makes sense – I guess. But isn’t there another way to do this? For example, couldn’t we run the full suite of unit tests at code check-in time, and make the nightly build the one that skips tests and reliably produces the demo jar files? That way at least we’d get back the benefit of immediate feedback and have a better idea whose code is causing a test to start failing.

Bob the Builder: I think we could do that. But I don’t have time to dig into an overhaul of the build process, because I have development responsibilities too, and we’re only 2 days from code freeze.

(Exit cubicle, stage right…)

By this time my head was spinning. I wasn’t sure whether what I’d just heard made sense, or whether I’d just been had. I was pretty sure the whole build arrangement they had settled on was crazy, but I could also see how an organization could have evolved into this arrangement because of conflicting needs.

The root of their problem is the Shimmer conundrum. It’s neither a dessert topping nor a floor wax, but their CI build is certainly trying to be two very different things at the same time. It was probably started as a CI build in the Agile sense, with the primary purpose being immediate feedback in the form of unit test results.

But of course, like any build, it also builds something. Those jar files were ripe for the picking, and some enterprising person had the great idea to deploy them every day so interested parties in the company would be able to see the application “fresh from the oven” – with all of the evolving new features on display.

The problem is that the secondary purpose of this particular build – generation of compiled, deployable software, became more critical to somebody – some person with influence – and the build began to slowly stop meeting its initial purpose.

So, is there a point to this rambling blog post? Indeed there is. It’s a word of caution to those of us in the consulting profession. It’s very easy for those of us who spend a lot of time reading and in other ways learning about “ideal” situations – such as well-run Agile projects – to become easily frustrated when we encounter things that are in some way less than ideal. We have to remember to suspend judgment long enough to understand the big picture – all of the various problems our clients are trying to juggle – and then offer our assistance in helping improve things, at a pace that makes sense in their environment.

I’ll try to remember this myself as I return to work tomorrow…

There is one thing, however, that has bothered me since I started using Hibernate several years ago.  A deep bother.  Not a “this is a nuisance, but I can live with it” bother, but more of a “this is so fundamental an issue that there just has to be a better solution to it than is being offered by Hibernate’s authors” bother.

The problem:  bridging the gap between object identity and database identity.  In most modern database schemas, tables are defined with a numeric primary key column, which is populated through some unique key assignment mechanism (Identity columns in SQL Server or MySQL, sequences in Oracle, etc).

This works great at the database layer.  But problems arise in the JVM (or CLR for you NHibernate folks).  In order for objects representing database entities (those managed by Hibernate) to be handled properly in collections (including Hibernate cache), objects that have the same database identity need to be recognized as equals as well.  In Java, this means overriding equals() and hashCode() such that any two objects that represent the same database entity will be equal, and return the same hash code.

The “obvious” solution, at first blush, would be to base equals() and hashCode() on entity ID values.  This works great for entities created by Hibernate, which represent existing rows in a database, all with unique ID’s.  The problem has to do with new entites that you create programmatically, which haven’t yet been saved.  Those entities will typically have a null ID value – and if you base equality on ID values, any two unsaved instances will be considered equal.

There are actually two problems when dealing with unsaved entity objects.  The first, as stated above, is that they don’t yet have a unique identity – so multiple new instances will collide in collections like Maps and Sets.  The other problem is that the identity of these objects will be set at some point, which violates part of the contract of hashCode() – the part that says its value must not mutate over time.

To avoid all of thes problems, Hibernate’s authors have long recommended a workaround – use “natural keys” instead of identity values for equals() and hashCode(). The problem with this is that many entities simply don’t have a natural key – that is, a set of one or more attributes whose values uniquely identify the entity.  And those that do often fail the hashCode() qualification that the value should not change over time.

There are other possible solutions I’ve seen or come up with myself, but all had at least one glaring weakness.  It seemed that there was no silver bullet for dealing with database-generated identity values in Hibernate.

And therein lies the silver bullet.  Don’t use the database (or Hibernate) to generate identity values.

I stumbled across a wonderful article on O’Reilly’s OnJava site that quite clearly describes this problem (in more detail than I do here), and offers what I believe to be the only real solution.  Check it out at http://www.onjava.com/pub/a/onjava/2006/09/13/dont-let-hibernate-steal-your-identity.html.

And now, as I look out the window at the late February snowstorm, I think maybe it’s time for me to hibernate…