If you've been paying attention to the Scala corner of the web over the last week, you'll no doubt have heard something about
Yammer - an enterprise-focussed social network - switching from Scala back to Java.
If you've been on holidays and missed it all, basically what happened was that
Coda Hale, the Infrastructure Architect at Yammer, wrote
a private email to the
Typesafe team giving loads of mostly critical feedback regarding
Scala, but
the email was subsequently leaked and became the talk of the town, among both Scala supporters, like
David Pollak, and critics, such as
Stephen Colebourne.
A Painfully Honest Critique
Firstly, let me summarise the main points in Coda's email:
* Yammer have decided to change their "basic infrastructure stack" from Scala to Java.
* The main reason is that Scala has "friction and complexity" that isn't offset by enough productivity gain.
* Some seemingly simple constructs in Scala, e.g. List, actually have very complex interfaces and behaviour that necessitate explaining a lot of concepts to beginners. The belief by some that this complexity can be ignored except by library authors has proven to be false in practice.
* The most vocal members of the Scala online community are generally very academic in their practice and discussion of Scala.
* Existing libraries seem to be being re-written just because people want to practice their use of advanced functional programming concepts, but these re-writes are then being recommended as the library of choice.
* The Yammer team eventually decided the most sensible way to deal with the Scala community was to ignore it.
* Learning Scala properly is important, but doing so is neither quick nor easy.
* Java is not going away, so the choice to use Scala is actually a choice to use Scala and Java, along with the interaction challenges that brings.
* The Yammer team had "endless issues" with SBT, and found themselves writing plugins to replicate Maven functionality.
* Switching to Maven solved most of their problems, but they found it has been marginalised by the Scala community, with poor support and constant encouragement to switch to SBT.
* The backwards-incompatibility of Scala major releases requires library authors to cross-compile, which has resulted in library re-invention when authors are no longer committed to the project.
* The backwards-incompatibility also causes headaches for developers, which has resulted in many production Scala deployments not being upgraded to 2.9.
* Yammer found idiomatic Scala code performed very poorly. By eliminating for-comprehensions, avoiding Scala's collection library, avoiding closures and marking everything with private[this], they were able to achieve speed ups of 100x in some components.
So that was that. For what it's worth, Coda's email is also quite complimentary of Scala. I think he gives the impression that he likes it and would really like to keep using it, but the realities of making it work in a day-to-day production-deploying team have weighed them down so much that the benefits have been eclipsed.
A Frustratingly Measured Official Response
Not long after Coda's email became public, Typesafe
published on their blog what is quite obviously a public response to the email, and it's that blog I'd like to take a closer look at today.
Here are the main points from the Typesafe blog:
* Typesafe are investing in Scala IDE support by contributing to the Scala IDE for Eclipse
* They are addressing the learning curve by developing training courses, but also provided references to some of the free resources on the web.
* 'scalac' is slow because it has more work to do than 'javac', but they are investigating using parallel and incremental techniques to improve speed, with a focus on improving sbt.
* If you don't want to (or can't) upgrade your Scala version, you can purchase commercial support from Typesafe to get access to long-term maintenance releases of previous versions.
* In order to reduce people's major-release upgrade issues, they are planning to fold a bunch of existing libraries into the Scala release .
* They take performance seriously, but are wary of engaging in premature optimisation.
* Typesafe are not just focused on the Scala language and core libraries but also on frameworks to help build applications more easily.
* They describe the Scala community using a couple of niceties as well as the words "opinionated" and "quirky".
* They say there's many places where Scala could be improved and they're hard at work doing so.
* They mention Yammer amongst a group of other companies with successful production deployments of Scala.
What Aren't They Telling Us?
What I think is most interesting about this response is the things it doesn't mention and the places where it's specific about what they're doing to solve issues.
Firstly, there's no mention of
Maven. There is some commitment to improving compile times, but they seem to suggest that this will be for sbt users only. It has to be considered odd that Maven has been given so little attention by the chief proponents of Scala. One of the main advantages often cited for adopting Scala is its easy integration with Java, "so you can continue using all your favourite libraries". But the commitment to prolonging the life of people's Java API knowledge doesn't extend to the Java build chain. Should it? Why is Maven important? It's important because it works, and it's almost ubiquitous. It has had years and years of work poured into it and has been tested and improved by feedback from hundreds of thousands of users and plugin developers. It has a global ecosystem that is well understood. It has become a lingua franca; many people would know how to build their project with Maven but couldn't tell you how to build it with javac and jar. If you want to distribute your project to the world through Maven Central via Sonatype's Nexus repository, they've got fantastically detailed instructions for how to do so... with Maven. The Java-fluent world - which Typesafe want to convert into the Scala world - don't need another build tool. We need relatively minor improvements to Maven and its plugins to ensure we can keeping working without having to learn something completely new.
Secondly, Typesafe give no commitment to eliminating the
backwards compatibility problems in future major releases. In fact, one of the parts of their workaround - the offering of paid support for people who want to continue using older releases - actually makes it in their interest to not solve the problem. The other half of their workaround - bringing certain libraries into the Scala release - will mean that, as a developer, your backwards compatibility issues will only be solved if your taste in libraries is similar to those of the Typesafe team. If you like a library that hasn't been deemed worthy of inclusion, you'll be left with all the same cross-compiling and "version-x-for-scala-y" dependency issues that you have now.
Thirdly, they seem to fob off Yammer's
performance problems as an issue specific to their environment, suggesting that fixing such things for everybody would be "premature optimisation". I've heard this argument before: that it only makes sense for companies like Yammer and Twitter to spend time optimising tiny bits of code because it can save them millions across all their computers. But the same argument applies to small businesses just as well. If my business is growing rapidly, but I have limited cash flow, having a 100x performance increase is going to make a significant contribution to my bottom line if I can serve all of my customers on one machine instead of 100 machines. "Premature optimisation" is a phrase that should be reserved for application developers, not framework developers. Just optimise the hell out of it, please.
Finally: the
community. While Typesafe dropped a small comment about the community in its response, its really not an issue that they can solve, though I do think it will greatly effect the measure of their success in the long term. Really, the comments in Coda's email should be a wake up call to the Scala community at large: a sizable Scala software development team, full of very smart people who've written some very scalable and successful software, found the Scala community to be so fractured and unhelpful that they eventually decided to ignore it and, I would assume, not be a part of it. That's not just a loss for the community, but also a warning, because for every developer that finds the community unhelpful and so decides to program Scala in his own microcosm, there will be numerous others who decide to not bother with Scala at all for the same reason. This alone could be a weighty enough issue to relegate Scala to a hobby language in the long term rather than seeing wide-spread adoption in the enterprise community.
Hope and Trepidation
When I originally read through Coda's email I was nodding my head in agreement and hoping that Typesafe would be listening carefully and would take it as an opportunity to shift their focus, maybe just temporarily, but long enough to knock over these major objections. Unfortunately, when I read through Typesafe's response, I didn't see any real hint that they plan to fix the problems that I find most annoying. In some cases I even saw excuses about why they won't be doing anything. I don't think "We're focusing on making sbt better" is a response to "Why isn't there decent Maven support?". I know they've been
meeting with influential Scala teams in the days since this response, I just hope they listen to what they've got to say.
So now I'm left feeling a little unsure about Scala. Will it continue to improve for me and people like me? People who like to use IDEA + Maven, not Eclipse + sbt, who are educated in creating applications rather than composing mathematical proofs? Or is the current state of the art in Scala, along with its broad range of minor but resonating annoyances, what I should expect to still be grappling with in five years time? Only time will tell whether Typesafe can navigate this wave of criticism and find a promised land of general popularity, but I think it is an indication that Scala's window of opportunity has entered the waning phase, and now is the critical time at which to capitalise on the extant interest.
What do
YOU think?
Do you sympathise with the Yammer team's frustrations?
Are you confident that Typesafe will listen and respond to current criticisms with relevant and prompt action?
