If you've been paying attention to the Scala corner of the web over the last week, you'll no doubt have heard something about Yammer - an enterprise-focussed social network - switching from Scala back to Java.
If you've been on holidays and missed it all, basically what happened was that Coda Hale, the Infrastructure Architect at Yammer, wrote a private email to the Typesafe team giving loads of mostly critical feedback regarding Scala, but the email was subsequently leaked and became the talk of the town, among both Scala supporters, like David Pollak, and critics, such as Stephen Colebourne.
A Painfully Honest Critique
Firstly, let me summarise the main points in Coda's email:
* Yammer have decided to change their "basic infrastructure stack" from Scala to Java.
* The main reason is that Scala has "friction and complexity" that isn't offset by enough productivity gain.
* Some seemingly simple constructs in Scala, e.g. List, actually have very complex interfaces and behaviour that necessitate explaining a lot of concepts to beginners. The belief by some that this complexity can be ignored except by library authors has proven to be false in practice.
* The most vocal members of the Scala online community are generally very academic in their practice and discussion of Scala.
* Existing libraries seem to be being re-written just because people want to practice their use of advanced functional programming concepts, but these re-writes are then being recommended as the library of choice.
* The Yammer team eventually decided the most sensible way to deal with the Scala community was to ignore it.
* Learning Scala properly is important, but doing so is neither quick nor easy.
* Java is not going away, so the choice to use Scala is actually a choice to use Scala and Java, along with the interaction challenges that brings.
* The Yammer team had "endless issues" with SBT, and found themselves writing plugins to replicate Maven functionality.
* Switching to Maven solved most of their problems, but they found it has been marginalised by the Scala community, with poor support and constant encouragement to switch to SBT.
* The backwards-incompatibility of Scala major releases requires library authors to cross-compile, which has resulted in library re-invention when authors are no longer committed to the project.
* The backwards-incompatibility also causes headaches for developers, which has resulted in many production Scala deployments not being upgraded to 2.9.
* Yammer found idiomatic Scala code performed very poorly. By eliminating for-comprehensions, avoiding Scala's collection library, avoiding closures and marking everything with private[this], they were able to achieve speed ups of 100x in some components.
So that was that. For what it's worth, Coda's email is also quite complimentary of Scala. I think he gives the impression that he likes it and would really like to keep using it, but the realities of making it work in a day-to-day production-deploying team have weighed them down so much that the benefits have been eclipsed.
A Frustratingly Measured Official Response
Not long after Coda's email became public, Typesafe published on their blog what is quite obviously a public response to the email, and it's that blog I'd like to take a closer look at today.
Here are the main points from the Typesafe blog:
* Typesafe are investing in Scala IDE support by contributing to the Scala IDE for Eclipse
* They are addressing the learning curve by developing training courses, but also provided references to some of the free resources on the web.
* 'scalac' is slow because it has more work to do than 'javac', but they are investigating using parallel and incremental techniques to improve speed, with a focus on improving sbt.
* If you don't want to (or can't) upgrade your Scala version, you can purchase commercial support from Typesafe to get access to long-term maintenance releases of previous versions.
* In order to reduce people's major-release upgrade issues, they are planning to fold a bunch of existing libraries into the Scala release .
* They take performance seriously, but are wary of engaging in premature optimisation.
* Typesafe are not just focused on the Scala language and core libraries but also on frameworks to help build applications more easily.
* They describe the Scala community using a couple of niceties as well as the words "opinionated" and "quirky".
* They say there's many places where Scala could be improved and they're hard at work doing so.
* They mention Yammer amongst a group of other companies with successful production deployments of Scala.
What Aren't They Telling Us?
What I think is most interesting about this response is the things it doesn't mention and the places where it's specific about what they're doing to solve issues.
Firstly, there's no mention of Maven. There is some commitment to improving compile times, but they seem to suggest that this will be for sbt users only. It has to be considered odd that Maven has been given so little attention by the chief proponents of Scala. One of the main advantages often cited for adopting Scala is its easy integration with Java, "so you can continue using all your favourite libraries". But the commitment to prolonging the life of people's Java API knowledge doesn't extend to the Java build chain. Should it? Why is Maven important? It's important because it works, and it's almost ubiquitous. It has had years and years of work poured into it and has been tested and improved by feedback from hundreds of thousands of users and plugin developers. It has a global ecosystem that is well understood. It has become a lingua franca; many people would know how to build their project with Maven but couldn't tell you how to build it with javac and jar. If you want to distribute your project to the world through Maven Central via Sonatype's Nexus repository, they've got fantastically detailed instructions for how to do so... with Maven. The Java-fluent world - which Typesafe want to convert into the Scala world - don't need another build tool. We need relatively minor improvements to Maven and its plugins to ensure we can keeping working without having to learn something completely new.
Secondly, Typesafe give no commitment to eliminating the backwards compatibility problems in future major releases. In fact, one of the parts of their workaround - the offering of paid support for people who want to continue using older releases - actually makes it in their interest to not solve the problem. The other half of their workaround - bringing certain libraries into the Scala release - will mean that, as a developer, your backwards compatibility issues will only be solved if your taste in libraries is similar to those of the Typesafe team. If you like a library that hasn't been deemed worthy of inclusion, you'll be left with all the same cross-compiling and "version-x-for-scala-y" dependency issues that you have now.
Thirdly, they seem to fob off Yammer's performance problems as an issue specific to their environment, suggesting that fixing such things for everybody would be "premature optimisation". I've heard this argument before: that it only makes sense for companies like Yammer and Twitter to spend time optimising tiny bits of code because it can save them millions across all their computers. But the same argument applies to small businesses just as well. If my business is growing rapidly, but I have limited cash flow, having a 100x performance increase is going to make a significant contribution to my bottom line if I can serve all of my customers on one machine instead of 100 machines. "Premature optimisation" is a phrase that should be reserved for application developers, not framework developers. Just optimise the hell out of it, please.
Finally: the community. While Typesafe dropped a small comment about the community in its response, its really not an issue that they can solve, though I do think it will greatly effect the measure of their success in the long term. Really, the comments in Coda's email should be a wake up call to the Scala community at large: a sizable Scala software development team, full of very smart people who've written some very scalable and successful software, found the Scala community to be so fractured and unhelpful that they eventually decided to ignore it and, I would assume, not be a part of it. That's not just a loss for the community, but also a warning, because for every developer that finds the community unhelpful and so decides to program Scala in his own microcosm, there will be numerous others who decide to not bother with Scala at all for the same reason. This alone could be a weighty enough issue to relegate Scala to a hobby language in the long term rather than seeing wide-spread adoption in the enterprise community.
Hope and Trepidation
When I originally read through Coda's email I was nodding my head in agreement and hoping that Typesafe would be listening carefully and would take it as an opportunity to shift their focus, maybe just temporarily, but long enough to knock over these major objections. Unfortunately, when I read through Typesafe's response, I didn't see any real hint that they plan to fix the problems that I find most annoying. In some cases I even saw excuses about why they won't be doing anything. I don't think "We're focusing on making sbt better" is a response to "Why isn't there decent Maven support?". I know they've been meeting with influential Scala teams in the days since this response, I just hope they listen to what they've got to say.
So now I'm left feeling a little unsure about Scala. Will it continue to improve for me and people like me? People who like to use IDEA + Maven, not Eclipse + sbt, who are educated in creating applications rather than composing mathematical proofs? Or is the current state of the art in Scala, along with its broad range of minor but resonating annoyances, what I should expect to still be grappling with in five years time? Only time will tell whether Typesafe can navigate this wave of criticism and find a promised land of general popularity, but I think it is an indication that Scala's window of opportunity has entered the waning phase, and now is the critical time at which to capitalise on the extant interest.
What do YOU think?
Do you sympathise with the Yammer team's frustrations?
Are you confident that Typesafe will listen and respond to current criticisms with relevant and prompt action?
Thursday, December 8, 2011
Wednesday, October 26, 2011
Top 10 Reasons Java Programs Envy Scala
I presented a talk at the 2nd meeting of the new ScalaSyd Meetup last night. I talked through the "Top 10 Reasons Java Programs Envy Scala" in an attempt to give Java developers a taste of some little things that could make them much more productive if they switch to Scala.
If you'd like to watch the Prezi, or listen to the talk (or both!), here are the links:
Presentation: Top 10 Reasons Java Programs Envy Scala (Graham Lea)
Audio Simulcast: Top 10 Reasons Java Programs Envy Scala (Graham Lea)
At the end of the talk, I mention that people who are tempted to learn Scala might want to look at my blog post, Graham's guide to learning Scala.
If you'd like to have a look at the code examples used in the talk you can view them on or clone them from GitHub.
Tuesday, October 11, 2011
Presenting at ScalaSyd, October 26
In two weeks' time I'll be presenting a talk at the recently-formed ScalaSyd Meetup. The title of the talk is "Top 10 Reasons that Java programs envy Scala"; the synopsis: "A whirlwind tour through 10 productivity-increasing features of Scala that aims to inspire Java developers to jump on the Scala bandwagon for their own coding pleasure."
There will be some other talks, Atlassian are providing the venue, pizza and drinks and JetBrains have donated some swag to dish out. It should be a good night so if you'll be in Sydney on Wednesday October 26, make a plan to drop in and make sure you RSVP on Meetup.com. Hope to see you there!
There will be some other talks, Atlassian are providing the venue, pizza and drinks and JetBrains have donated some swag to dish out. It should be a good night so if you'll be in Sydney on Wednesday October 26, make a plan to drop in and make sure you RSVP on Meetup.com. Hope to see you there!
Friday, September 23, 2011
Code Writers vs Code Creators
It's occurred to me recently that one of the key turning points for a professional software developer is the realisation that you don't have to make do with the tools you've got - you can create your own tools. This is why 'The Pragmatic Programmer' recommends activities like learning scripting languages and using code generation.
What is your Goal?
As a software engineer, my key goal is to CREATE software. A lot of the time, creating will mean WRITING software (i.e. typing out a program), but that isn't always the case. Many pieces of code have large amounts of boilerplate, or contain elements that are repeated again and again across the code base. Usually this means that the code could have been derived from some description of the end result that was simpler than the code itself. And wherever that is the case, the code is a candidate for being tool-aided or tool-generated.
What is a Tool?
Now, tools can take many forms. Sometimes you just need to abstract a super class from some existing code, rather than copying code, and that base class becomes a tool for solving any similar problem. If you have a similar problem across a number of different applications, you might pull code out into a library that can be reused across those code bases.
I'm using a pretty loose definition of 'tool' here. Basically I consider anything a tool if it has no function by itself, but can be used to create things which can perform a function. Base classes and libraries definitely fit this definition, but I hope that for the majority of developers this kind of reuse is bread and butter.
The kind of tool-creation I think we should aspire to is a little more high-level than this. At regular points in the creation of software, every programmer, and sometimes even whole teams, will come across problems where they feel awkward with the solution and with the process for creating it. So what I'm advocating is that developers and teams need to have the courage in these situations to put down the normal tools and say, "This isn't working. We need better tools for this task." Sometimes there will be existing tools for solving your problem, and sometimes you'll have to create the right tool yourself. But the key quality is to be on the lookout for situations where the productivity of your code creation is being hampered by something, and to be ready to say out loud, "let's take the time to fix this". The problem might be irrelevant boilerplate, repetetive details, or it may even be something not directly related to coding, like a randomly failing build or an automated process that sends you too many emails. Anything that's stealing time away from value creation and could potentially be semi-automated can become the target of tooling.
What's the Catch?
Of course, there's common sense to be used in deciding when to employ a tool or create one. If you're going to write your own tool, you need to make a really honest (i.e. pessimistic) estimation of the ROI (return on investment): Could the time it will take you to write the tool end up being longer than the time it will save you? Is this problem (and hence the tool to solve it) happening again and again or is it just a one off? Will the tool require ongoing maintenance that will significantly offset the gain it provides? You also need to have the discipline to create just the tool that YOU need, and to not try to solve the larger, generic set of problems that are like yours, but which you don't have. Remember: the purpose of creating a tool is to get you to the finish line faster, not to divert you from the race entirely because you're finding it uninteresting.
Are Your Eyes Open to Opportunities for Greater Productivity?
At the end of the day, though, I believe there will often come times for every developer where they find a problem that justifies the creation of a custom tool. If this hasn't been your experience, I'd like to challenge you to think about this: is the charge of your occupation to write lines of code as fast as possible, or to deliver working solutions as fast as possible? Assuming you're not paid by the line, you're really being paid to be a Code Creator, not just a Code Writer. That means you have a responsibility to use the right tools for the job, and hence a mandate to create them if they don't exist.
Want to learn more?
What is your Goal?
As a software engineer, my key goal is to CREATE software. A lot of the time, creating will mean WRITING software (i.e. typing out a program), but that isn't always the case. Many pieces of code have large amounts of boilerplate, or contain elements that are repeated again and again across the code base. Usually this means that the code could have been derived from some description of the end result that was simpler than the code itself. And wherever that is the case, the code is a candidate for being tool-aided or tool-generated.
What is a Tool?
Now, tools can take many forms. Sometimes you just need to abstract a super class from some existing code, rather than copying code, and that base class becomes a tool for solving any similar problem. If you have a similar problem across a number of different applications, you might pull code out into a library that can be reused across those code bases.
I'm using a pretty loose definition of 'tool' here. Basically I consider anything a tool if it has no function by itself, but can be used to create things which can perform a function. Base classes and libraries definitely fit this definition, but I hope that for the majority of developers this kind of reuse is bread and butter.
The kind of tool-creation I think we should aspire to is a little more high-level than this. At regular points in the creation of software, every programmer, and sometimes even whole teams, will come across problems where they feel awkward with the solution and with the process for creating it. So what I'm advocating is that developers and teams need to have the courage in these situations to put down the normal tools and say, "This isn't working. We need better tools for this task." Sometimes there will be existing tools for solving your problem, and sometimes you'll have to create the right tool yourself. But the key quality is to be on the lookout for situations where the productivity of your code creation is being hampered by something, and to be ready to say out loud, "let's take the time to fix this". The problem might be irrelevant boilerplate, repetetive details, or it may even be something not directly related to coding, like a randomly failing build or an automated process that sends you too many emails. Anything that's stealing time away from value creation and could potentially be semi-automated can become the target of tooling.
What's the Catch?
Of course, there's common sense to be used in deciding when to employ a tool or create one. If you're going to write your own tool, you need to make a really honest (i.e. pessimistic) estimation of the ROI (return on investment): Could the time it will take you to write the tool end up being longer than the time it will save you? Is this problem (and hence the tool to solve it) happening again and again or is it just a one off? Will the tool require ongoing maintenance that will significantly offset the gain it provides? You also need to have the discipline to create just the tool that YOU need, and to not try to solve the larger, generic set of problems that are like yours, but which you don't have. Remember: the purpose of creating a tool is to get you to the finish line faster, not to divert you from the race entirely because you're finding it uninteresting.
Are Your Eyes Open to Opportunities for Greater Productivity?
At the end of the day, though, I believe there will often come times for every developer where they find a problem that justifies the creation of a custom tool. If this hasn't been your experience, I'd like to challenge you to think about this: is the charge of your occupation to write lines of code as fast as possible, or to deliver working solutions as fast as possible? Assuming you're not paid by the line, you're really being paid to be a Code Creator, not just a Code Writer. That means you have a responsibility to use the right tools for the job, and hence a mandate to create them if they don't exist.
Want to learn more?
| From Amazon... | From Book Depository... |
Thursday, August 4, 2011
How to Create a Webapp with Scala, Spring, Hibernate and Maven in Late 2011
Way back in the ancient history of January 2010, I wrote a walk-through of how to create a webapp with Scala, Spring Hibernate and Maven.
As with all things, technologies have progressed, as have other things, including the way code is shared online. So I decided it was time to update the code form that post and share it somewhere more modern.
You can now find my code for starting a Scala + Spring + Hibernate + Maven Webapp on GitHub.
At the time of writing, this project uses Scala 2.9.0-1, Spring 3.0.5.RELEASE and Hibernate 3.6.5.Final. I hope to keep it updated as new versions of these libraries are released, and also to add some more flesh to the example as time permits. (Ha!)
In you're looking more for a tutorial rather than just code, the original post still serves as a great walk-though of all the steps that went into creating the sample webapp, as well as showing lots of mistakes that I made that are probably quite common.
Want to learn more?
Here's some books you might find useful if you plan to go further with Spring, Hibernate, Scala or Maven:
Spring in Action - Craig Walls & Ryan Breidenbach (Manning)
Hibernate in Action - Gavin King & Christian Bauer (Manning)
Spring Recipes - Gary Mak (Apress)
Programming in Scala - Martin Odersky, Lex Spoon & Bill Venners (Artima)
Maven - A Developer's Notebook - Vincent Massol & Timothy M. O'Brien (O'Reilly)
As with all things, technologies have progressed, as have other things, including the way code is shared online. So I decided it was time to update the code form that post and share it somewhere more modern.
You can now find my code for starting a Scala + Spring + Hibernate + Maven Webapp on GitHub.
At the time of writing, this project uses Scala 2.9.0-1, Spring 3.0.5.RELEASE and Hibernate 3.6.5.Final. I hope to keep it updated as new versions of these libraries are released, and also to add some more flesh to the example as time permits. (Ha!)
In you're looking more for a tutorial rather than just code, the original post still serves as a great walk-though of all the steps that went into creating the sample webapp, as well as showing lots of mistakes that I made that are probably quite common.
Want to learn more?
Here's some books you might find useful if you plan to go further with Spring, Hibernate, Scala or Maven:
From Amazon
From The Book Depository
Spring in Action - Craig Walls & Ryan Breidenbach (Manning)
Hibernate in Action - Gavin King & Christian Bauer (Manning)
Spring Recipes - Gary Mak (Apress)
Programming in Scala - Martin Odersky, Lex Spoon & Bill Venners (Artima)
Maven - A Developer's Notebook - Vincent Massol & Timothy M. O'Brien (O'Reilly)
Subscribe to:
Posts (Atom)
