September 23, 2014

Clojure in the Enterprise?

This was originally posted on corfield.org back in April 2013 and I noticed it was recently referenced by Eric Normand in his recent blog post Convince your boss to use Clojure so I figured it was time to update the article and bring it onto my new blog.

A question was asked in early 2013 on a Clojure group on LinkedIn about reasons to migrate to Clojure for enterprise applications in a Java shop. It's a fairly typical question from people in the Java world when they hear the buzz about Clojure, and of course asking the question on a Clojure group garnered a lot of positive responses about why Clojure is a good choice. I didn't feel anyone had really addressed a core aspect of the original question which was, essentially, "Why should I, as a Java web developer, using JPA, JSF etc, choose Clojure instead for an enterprise application?". The key considerations here are "enterprise application" and "Java web devloper, using JPA, JSF etc". Clojure is rightly praised for big data projects, simplified concurrency due to immutable data, and the conciseness of its solutions. The general advice when introducing Clojure to an organization is to take a grass roots approach: use it for some tooling first, or a small low-risk (but perhaps high-profile) project and show how well it works in a Java-dominated world. Then you get more and more developers trying it out and gradually the organization adopts it for more and more projects. It's good advice, and it's often how Clojure has crept into Java shops so far (as opposed to those fast-moving small shops that already have a tendency toward polyglot development).

I didn't feel anyone had really talked about how radical Clojure seems to a conservative "enterprise" company that's already bought into the Java way of doing things from top-to-bottom (as indicated by the original poster's references to JPA and JSF). I had a couple of people ask me to turn my (fairly length) response on the group into a blog post, so I did, and here's the updated version for 2014...

The "enterprise" in general is extremely conservative about change. Those companies use Java because it's "safe": it's pretty much ubiquitous, it's been around for decades, it's easy to hire low-to-mid-range developers, you can outsource it relatively easily. If you have a company that's bought into the whole Java stack and uses Java for "everything", you have a deeply entrenched company and it may be extremely difficult to bring Clojure in. You might have some success bringing Scala in - as a "better Java" - because Scala is less alien to Java developers and the marketing behind Scala is focused on serving the "enterprise" already.

Having talked to a number of Java developers who work in big "enterprise" shops, one of the biggest problems is simply that the average skill level in such shops is often pretty mediocre, especially if some of the work is outsourced (and considered acceptable quality). The processes tend to be old-fashioned, the tooling tends to be years behind the bleeding edge. Even bringing Scala in can be very disruptive and the worst part is you might just end up with Java-in-Scala, without a lot of the benefits of moving to a highly expressive, (partly) functional language.

Clojure is very alien to the average Java developer. If you have a shop full of "average" Java developers who have grown up only knowing "OOP in Java", they will struggle really badly with the functional concepts in Clojure (and Scala - which is why you get Java-in-Scala instead of idiomatic Scala). Unlearning all those OOP habits to write effective functional code is really, really hard for a lot of people. Writing high performance idiomatic functional code can be even harder. Don't underestimate that difficulty! Take away objects, mutable state, variables, loops... a lot of Java developers are immediately all at sea and have no idea how to solve even basic problems. They'll try to bend Clojure to their OOP way of thinking and they'll most likely fail.

Clojure is great for bleeding edge developers and it is also great for new developers who have not yet learned the bad habits of OOP (conflating state and identity, mutability, loss of data history / time concepts).

Everything that everyone said in the original comment thread on LinkedIn is true: Clojure has many wonderful benefits, you can start with a small, eager team and a high profile but relatively low-risk project, and so on. But don't underestimate the difficulty of getting a conservative organization to switch from the COBOL of the modern era (Java) to something as radical as Clojure currently is, nor the difficulty of retraining your army of low-to-med-range Java-only programmers.

At my company, we first took Clojure to production in mid-2011 and cross-training the team - a small group of developers eager to learn new things - is an ongoing process, slow and steady. We're at 22KLOC in Clojure with over 5KLOC in our test suite, and about 1KLOC of ClojureScript now. That's probably the equivalent of at least 80KLOC of Java (and quite likely north of 150KLOC). We're a small company with a very supportive management team. At this point, almost all new development is in Clojure with maintenance still continuing on the "legacy" code base (another factor to consider, by the way: what about your existing Java code base?). We love Clojure. It's made development a lot more fun. We're able to solve harder problems, make changes faster, leverage multi-core concurrency more effectively, and we have a much smaller code base to maintain. But we are about as far from "enterprise" as you can imagine, while not actually being a start-up (World Singles has been around for over a decade).

The half dozen paragraphs above are an edited version of what I posted on LinkedIn (and slightly updated from my original blog post) but there's something else I wanted to cover in this post: the mindset of Clojure.

A common question amongst folks new to Clojure, especially those who want to do web development, is "Where are the 'batteries included' frameworks?". If you come to Clojure from Ruby on Rails, you'll probably be surprised to see no full-stack web framework. If you come to Clojure from Java, you might be looking for something like Spring (or Hibernate). The Clojure community favors small, composable libraries over opinionated frameworks. If you want to do web development in Clojure, you can pick a library for routing, a library for basic HTTP request/response plumbing, a library for persistence, and so on, and you'll compose those together with your application code to build the solution you like best. For almost every task, Clojure has several libraries, each focused on doing one thing well and an expectation to "play nicely with others", mostly through the idiom of building on Clojure's core data structures (mostly maps and vectors, but also sets and the ubiquitous sequence abstraction). For some, this is seen as a lack of structure or a lack of maturity but it's a very deliberate decision by the core of the Clojure community. Instead of just falling into a particular way of doing something, Clojure encourages you to pick'n'mix, focusing on the problem at hand and solving it in the most elegant way, rather than trying to shoehorn your solution into someone else's way of thinking.

A couple of examples from World Singles might illustrate this. We use two different libraries for generating HTML/XML, which have very different approaches. We use Selmer to generate HTML emails. We use Hiccup to generate XML documents that we use with 3rd party web services. Selmer takes mustache-style templates with substitutions for data structures and some embedded control structures. Hiccup takes Clojure data structures and renders them to HTML/XML. For our web service interaction, we tend to have data structures that we want to render to XML so it is more natural to annotate the data with more data and let Hiccup take care of the rendering. With our HTML emails, we want something close to HTML with a controlled set of substitutions, for member profile data etc. Similarly we use two different testing libraries for different situations. We use Expectations for most of our testing because we like the language it uses: "Expect {some conditions} to be true of {our test expression}". It lets us write very natural, expressive tests. However, for our WebDriver based tests (using clj-webdriver), we prefer the more assert-based approach of Clojure's built-in testing library (clojure.test). These end-to-end tests are organized in a much more procedural way, asserting certain things about the state of the UI at various points in each scenario we are testing. Finally, we use two different date/time libraries: date-clj and clj-time. The former is great for interacting with data stores since it uses java.util.Date and supports basic date manipulation (adding / subtracting a number of time units). The latter is great for more complex date/time manipulation (midnight-based instants, periods, intervals, range checking) but is based on Joda Time and needs to be explicitly converted to/from java.util.Date at the boundaries.

Clojure's wealth of choices can be overwhelming but it's also very freeing. The "enterprise" often wants a more proscribed way to do things, with more consistent solutions across different teams and different abilities. For more ideas on getting Clojure into your "enterprise" shop, take a look at Neal Ford's Master Plan for Clojure Enterprise Mindshare Domination presentation from Clojure/conj 2011.

Tags: clojure expectations