SpringOne 2010: Gradle - A Better Way to Build

I spent the second-last week of October at SpringOne 2GX 2010 in Chicago and I thought some of you might get something useful out of my notes. These aren’t my complete reinterpretations of every slide, but just things I jotted down that I thought were interesting enough to remember or look into further.

Gradle - A Better Way to Build
presented by Hans Docketer

Note that any references to the Gradle User Guide from this blog entry are to version 0.8, as that was the most recent GA version at the time of writing. If you are using a later version of Gradle, you can might want to check the latest User Guide instead.

Hans is the creator and lead developer of the Gradle build system. He’s also the CEO of Gradle Inc., a consulting company specialising in project automation.

First up, Hans explained that Gradle is actually mostly written in Java, with only the DSL being written in Groovy.

Gradle supports building Java, Scala, Groovy, Web projects, etc.

It appeared very easy to add a Maven dependency into the classpath. (Probably easier than in Maven!)

He demoed an example of developing a small, custom build operation that copied an image (of Rod Johnson) into the META-INF/ directory of any output JARs that had 'spring' in the name. While this was possible with only a few lines of code, I have to say that the code wasn’t all that intuitive and I think you would have to know quite a lot about the way Gradle structures things before you can easily write tasks like this yourself.

There’s a very easy method for importing Gradle tasks from external sources by specifying not much more than a URL.

Hans said that, in comparing Gradle to Maven and Ant, he believes the use of Groovy in Gradle instead of XML is not the main differentiator, but its design is where the gains come from.

He gave a pretty impressive demo of how Gradle detects changes in sources, tests, classpaths and even the Gradle script itself, and then only rebuilds those parts of the project that absolutely need to be built (and, of course, the things that depend on them).

While Gradle can be used to build projects that aren’t written in Groovy, it was my observation that you probably need to have a fair level of proficiency in Groovy in order to compose anything other than a basic build script.

It’s pretty easy to separate the declarative parts of a custom task from the imperative parts by defining a class at the bottom of the script that. (I found it interesting that, even though the syntax of a Gradle file is mostly declarative, Hans was still referring to the file as a 'script'.)

Gradle exhibits heavy integration with Ant that is also bi-directional, i.e. Ant tasks can depend on Gradle tasks.

Hans highlighted that Maven’s defaults (which he called a “strong opinion”) for things like project directory structure are one reason that people will avoid migrating existing projects to Maven. While this might be true, I think it’s based on a misconception - in reality, it’s quite trivial to override the defaults of things directory locations in a Maven POM.

Gradle uses coloured output, which I think is pretty cool for a Java-based build tool.

Hans noted that the flexibility of Groovy means that creating custom operations in a build script is pretty easy. Having seen smart people spend a week or two writing a pretty simple custom plugin for Maven, I think making the customisation process easier is definitely a win. (On the other hand, in four years of using Maven, we've only ever created one Mojo. We have, though, often used the Ant plugin to "break out of the box".)

Hans gave a demonstration of dependencies between sub-projects of a larger project which was pretty impressive, showing Gradle building and testing both dependency and dependant projects based on what changes had occurred.

He talked about how performance is a big focus of Gradle, in particular getting the dependency and change behaviour right to ensure that Gradle only ever builds things that have been affected by changes.

Gradle is able to generate a pom.xml file for a project and deploy an artefact to a Maven repository along with the generated POM.

Gradle doesn’t support (out of the box) a ‘release’ action like Maven’s Release Plugin (which creates branches and tags in the code repository, deploys build artefacts to a remote repository and automatically updates version numbers in the build files on head/trunk to the next snapshot). However, Hans said that they eventually want to develop a full deployment pipeline based on Gradle, which will be one of the focus points after version 1.0 has been released.

Want to learn more?

From Amazon...

From Book Depository...

SpringOne 2010: Groovy and Concurrency

I spent the second-last week of October at SpringOne 2GX 2010 in Chicago and I thought some of you might get something useful out of my notes. These aren’t my complete reinterpretations of every slide, but just things I jotted down that I thought were interesting enough to remember or look into further.

Groovy and Concurrency
presented by Paul King

Paul by started by mentioning a library called Functional Java which, to me, looks like an attempt at porting a bunch of ideas present in Scala over to Java, and another one called Kilim, which is an Actors library for Java.

Paul said that his main argument for why you should use Groovy, rather than Scala or Clojure, is that Groovy is closer to the Java syntax and, hence, is more easily integrated with Java. (In my personal experience, I can’t say I’ve ever had any problems integrating Java with Scala. Going the other way (using Scala in Java) has some gotchas, but wouldn't be described as hard.)

Groovy supports a very nifty pipe (‘|’) operator extension to the java.lang.Process class, allowing you to easily pipe stdout to stdin between two or more processes, just like in a shell.

I've now heard Google Collections (now part of the Guava libraries) mentioned for about the 5th time this week. I really should check out what these are because they’re very popular!

Groovy supports adding functions to classes, and even individual objects, at runtime. That is, your code can contain statements that add members to an existing type. This is not creating an inline definition of a new type, but actually changing the type at runtime, as you might do in JavaScript. They call this Dynamic Groovy. I've never really got my head around why meta-programming - programs that write programs (and then run them) - is a good idea, but I've also read Paul Graham saying that this feature gave him a major competitive advantage, so there must be something mind-bending about it. Perhaps I just need to give it a try?

Groovy has an @Immutable annotation that, as well as making the contained fields final (without you declaring them so), is also shorthand for adding getters, toString(), hashCode() and equals() to a class based on just the field names and types. Case classes in Scala provide the same functionality along with the added bonus of making pattern matching simple.

He mentioned two Java concurrency libraries, one called Jetlang for message-based concurrency (the Jetlang site itself refers to Jetlang being similar to Scala's Actors) and JPPF, a grid-computation library. The JPPF intro uses domain language similar to that of Spring Batch, with regards to jobs and tasks.

He talked a bit about GPars (short for Groovy Parallel Systems), a concurrency library specifcally for Groovy.

He also said that shared transactional memory looked interesting but didn’t go into it much beyond mentioning the Multiverse library. I have seen this term bandied around a little, in particular due to a couple of people attempting to implement it in Scala, but I've never looked into it - it's frequency hasn't yet punched through my signal/noise filter.

He gave a little preview of what the GPars guys are trying to achieve with a “Dataflows” library. You use a DSL to describe a dependency graph between steps in your (complex, but parallelisable) algorithm and Dataflows will automatically run in parallel those parts of the algorithm that are not dependent, synchronising at the points where multiple asynchronous outputs are required to enter the next step.

Want to learn more?

From Amazon...

From Book Depository...

SpringOne 2010: Slimmed Down Software: A Lean, Groovy Approach

I spent the second-last week of October at SpringOne 2GX 2010 in Chicago and I thought some of you might get something useful out of my notes. These aren’t my complete reinterpretations of every slide, but just things I jotted down that I thought were interesting enough to remember or look into further.

Slimmed Down Software: A Lean, Groovy Approach
presented by Hamlet D'Arcy

This talk turned out to be much more of a review of Lean principles than about how Groovy supports these principles, but that was fine by me. Obviously the principles are far more important than the language you’re using.

The key principle of Lean Development is to eliminate waste, which in essence means to stop doing things that you don’t need to do.

Interestingly, Hamlet proposed getting more sleep and ingesting less caffeine as good development practices

Meetings are more often than not a form of waste, especially when they seek to produce 100% consensus. Hamlet talked through the four forms of decision arrival, from dictatorship at one end to unanimity at the other, with democracy and something else in the middle. He didn't really make a conclusion out of this, but my guess is that he was warning us to stay away from the extreme ends. Certainly, constantly trying to achieve unanimity would cause a lot of waste.

He highlighted unfinished work as an example of waste. For example, choosing to half-develop 6 features rather than finish 2 or 3 causes waste. He didn’t really go into why this is waste, but my immediate thoughts were:
1) that it requires energy – both from individual developers and from the team – to keep abreast of unclosed loops; and
2) that knowledge learned towards the end of one feature may accelerate the development or prevent a change in another feature if it’s developed later rather than simultaneously.

While talking about unfinished work as a form of waste, he criticised distributed version control (e.g. Git, Mercurial/Hg) based on the fact that local branches, which are essentially code that’s not committed to head, are unfinished work and hence waste.

He recommended Groovy for unit testing, even if the production code isn’t Groovy.

He briefly discussed Value Stream Mapping, which from what I could tell is basically graphing out the flow of a process, including dependencies, time delays and actions, some of which may only be required to be performed every Nth time through the process (e.g. maintenance). My take-away message from the example shown was that you shouldn’t just accept time that is wasted waiting for a process to complete, but should schedule other tasks, that you know will need to be done in the future anyway, to occupy this time. (This is all in reference to analysing one’s process, not the flow of a program.)

While discussing Value Stream Mapping, he mentioned that you really want to be measuring activities and waste in $$$, not something more abstract like time or gut feel. If you’re making business decisions, $$$ is the unit that makes sense.

He referred us to an article on DZone called the “Seven Wastes of Software Development” (Though the top hit for this phrase on Google is a 2002 paper by Mary Poppendieck [PDF])

Hamlet postulated that languages with less syntax, e.g. Groovy (or Scala), allow one to write unit tests that look/read a lot closer to the original requirements specification.

He talked a little bit about EasyB, a Groovy DSL for BDD, and explained how its hierarchical test cases allow you to write multiple tests based on a shared scenario.

He claimed that the Agile idea that “the tests are the documentation” has been over-sold by evangelists and under-delivered by developers.

He showed how the Spock testing framework has you list all your assertions as Boolean expressions in an “expect” block, eliminating the need to write an assertThat(... call on every single line.

He raised the idea that every time we are called upon to make a decision there are three potential outcomes: Yes, No, or Defer.

He talked about something called “Real Options”, which posits that every possible decision outcome has both a cost and a deadline or expiry date. It almost never makes sense to commit to any decision before the expiry of the next option, because that’s when you’ll have the most information. The problem with achieving this is that human brains are wired to eliminate unknowns by locking down decisions as early as possible. The solution to that is to make an action point out of deferred decisions, the required action being to collect more information and reconvene when the decision needs to be made.

All of the above being under the banner of “delayed commitment”, it occurred to me that a good method for getting good at this is to constantly be asking yourself and everyone around you the question: “Is this the best time to be making this decision?”

He mentioned Canoo a couple of times, which was the firm he was working with while experimenting with all this lean and Groovy stuff. (I think they were doing something with App Engine?)

He said that his team stopped using fixed-length iterations because “two weeks is not a unit of value but a unit of time”, i.e. you should be releasing when you have value to deliver, no sooner or later.

He suggested reducing the number of integration tests because having these tests fail due to valid changes to the system is a form of waste. I actually disagree on this one. Around my workplace, the idea that you should delete a test because it keeps failing is an ongoing joke. Obviously you shouldn’t have tests that duplicate each other or if they fail for no reason – both of which result in waste. However, if you’re doing TDD, you probably want to change the test first anyway, so it shouldn’t break when you change the implementation, it should pass! It was very interesting to hear someone working with a dynamic language suggesting having less integration tests. My assumption has always been that, if anything, you would need more tests at this level to prove the correctness of the wiring of your essentially un-typed components than you would with static typing. Now that I think about it a bit deeper, I was probably wrong - you shouldn't need any more tests - you should need exactly the same amount. Full coverage is full coverage. Having more wouldn't prove anything.

I really liked this: He emphasised that when you decide to make a change to your process it is an experiment – you should define a time limit and then assess, as professionally as possible (i.e. without bias), the results of the experiment before deciding whether or not to make a permanent change or to start a different experiment.

He criticised what he called “closed-door architecture”, where a small set of “architects” within an organisation decide what technologies will be used and dictate these to the rest of the developers. He didn’t mention his exact reasoning for talking this down, but the obvious ones I see are the demotivational effect on the non-architect employees and the potential to miss good ideas by not providing everyone with an opportunity to contribute their brainpower and expertise to the problem. I think, in order for this to work well, you need a pretty mature bunch of developers. If you're going to canvas everyone's opinion, then everyone needs to be really good at leaving their ego at the door, otherwise you're going to end up in a six hour meeting about which developer has the best idea rather than which idea is best for the customer.

In the context of introducing Agile practices to an organisation, he discussed an equation from some book that says that the value of a change to an organisation is relative to Why over How, meaning that a big organisational change (large How) has to tackle a big problem or create a big advantage (large Why) in order to provide value. Changes that can provide a large benefit with minimal impact on the work (that's a large Why over a small How) are obviously the sweet spot in terms of increasing value.

Lastly he showed an example of a Groovy script that used a @Grab annotation to download a Maven dependency and bring it into the classpath at runtime. Very cool.

Want to learn more?

From Amazon...

From the Book Depository...