Development Costs of User Interface Development (and mitigating them with Clojure/Script)
Alternative titles: ("Enjoy some more razor-sharp carbon-fiber scaffoldings of paradigm abstractions on which to cut yourself!" "Clojure – these asbestos-gloves-for-web-development didn't kill me immediately!" "Tooling is a great way to spend time and feel good about yourself without making any real progress at all." "One neat trick for driving UI developers insane with the inadequacy of their toolchain!")
This first section is the philosophy behind this particular insane quest of mine. If you're looking for help with your Clojure environment, grep downwards.
The most painful and frictive step in most user interface developers' workflows is the cycle between writing some code, and observing the changes so wrought on the final product (be that a label's new value, the size of some new elements, or sub-view scaking and positioning). Over the past two or three years I've learned of a few older paradigms for building UIs where the person building the interface in question can write the code, and reload it into the running application for immediate feedback.
At this point most Lispers are probably laughing. Recompiling code and incrementally loading it into a running program has been a basic feature of Lisp environments since literally before I was born. Despite that, the vast majority of modern UI development workflows look like:
- wrangle app into state dev is hacking on today
- decide what needs changing
- write that code
- restart the application under development with the new binaries
Of course, swap details for your local development platform.
My (proposed) metrics for evaluating UI development environments:
- minimize duration between writing code and discovering errors in that code (e.g., compiler feedback, runtime exceptions)
- minimize repeated work and wasted time (clicking through UI's, manually setting up application state, recompiling the whole application, reloading the web page)
- minimize duration between written code and visibly different UI elements
- minimize context switches
- minimize enforced idle periods
Backend developers (in particular the Python and Ruby groups, Java to a lesser extent, and in all cases highly organization-dependent. Haskaliens, I've noted, rarely write tests, preferring instead to lean on the type system and their knowledge of how constrain program behavior with it. If you figure out how QuickCheck came out of the Haskell group, let me know) attempt to attend to the minimization of time between writing of code and testing its behavior by doing something on the spectrum of unit or integration tests.
Unit and integration tests are well, good, and in some situations downright imperative for these dynamically-typed languages where runtime behavior is nigh-unknowable thanks to the dynamically-typed nature of the programming environments. Many testing programs for software written in languages like this hypothetical that I've seen wrap a net of tests around the (for example) user-input-processing functions, attempting to make and enforce guarantees about the shape of data streaming from those functions into others, for the systems under construction rapidly grow so large, complex, and complected with themselves that formally reasoning about the whole thing (as a team at least, thinly-spread across the application surface area) is not even worth attempting. Hence, tests.
Tests, though, "can only show the presence of bugs, never their absence".
Developers are generally on their own when it comes to minimizing repeated work and wasted time, and let me tell you – are hilariously bad at it. Not only do they love to grow their own lovely little bonsai trees of complexity, they're also more than happy to run to the store twice a day for fertilizer that starts expiring as soon as they've paid, and whose half-life is measured in jokes about your mother. Some are hell-bent on wasting your time, and will go to such lengths as restarting the server, reloading the web page, and clicking through the whole application on every single code iteration – "just in case. I've been burned by state before". Others, great minds some, in an effort to speed up their development loops in the face of hostile and antagonistically-designed tooling, will rip the application apart and cobble together a development harness for whatever particular view they're working on at the time, so they don't have to go through all of the clicking through the damn thing. This is a laudable project, but hurts later in re-integration time when they want to stuff that selfsame view back down into the application view tree where it belongs – "what precisely state does this thing rely on again?".
Minimizing context switches (between JS in the browser and Python in the server, for instance), are really important. So important, in fact, that my friends in UI who make the jump from "web technologies" to the holistic environment of iOS or Android development frequently claim they'll never go back, because their flow and productivity are simply so much higher on the cohesive platform.
That's why everything's so miserable, in somewhat less than a thousand words. Developers love complexity, our UI toolchains are downright hostile, and nobody has the intellectual fortitude to imagine a better world.
Except for weirdos like me, who listen in bars to hoary old explorers about the glories of the Symbolics temples, and the costs of a world that runs on C and CSS. I have gone to truly absurd lengths in order to drive my personal productivity up with a ridiculously arcane toolchain.
The toolchain consists of Clojure server-side, ClojureScript in the browser, and it's all tied together with Emacs. It boasts minimal time between code-writing and effect-reviewing, provides excellent tooling to minimize clicking-through-of-UIs, is readily munged into ongoing automated testabity thanks to the Lispy ease of refactoring, boasts next to no context switching overhead (beyond differences between JS and the JVM), thanks to an integrated development environment and using the same wrapper language around the host languages in the browser and server, and almost entirely boils off enforced idle periods.
Before I (briefly) document (some) of the quirks of this setup, keep in mind that whatever I do to support my own insane version of interactive development, I've also set myself the constraint of configuring the whole toolshed in such a way that someone who doesn't use Emacs can dive into a project shaped like this and (from a tooling perspective) have a pretty similar experience.
I shan't provide much by way of code or configuration in this post. Other people do that better. What I shall do below, is lay out precisely which tools I use in which scenarios, and how I've gotten them to behave together.
To that end, the tools:
- server-side project structure
The project is structured in such a way that calling `lein repl` at the terminal launches a developer into a console from which the system can be piloted. It has functions like `start`, `reset`, `seed-db`, and other useful tools like `compile-cljs`. This is a user/developer's interface to the server-side code under development, and is capable of clearing out its own code and recompiling the entire application codebase from disk in under 400ms.
- on-demand recompilation of arbitrary code segments from Emacs
This actually works spectacularly (with some quirks I'll reveal later). I can write code, evaluate it, and instruct the frontend to redisplay its entire self without altering the application state. Idem for the backend, and entirely transparently at that. I don't have to worry about where the code is going, I say "emacs compile and ship this to the relevant process" and it does so. It's goddamned amazing.
This part entailed using Weasel, configuring the Piggieback middleware, and setting `cider-cljs-repl` in Emacs to the string:
(do (require 'weasel.repl.websocket)
(weasel.repl.websocket/repl-env :ip \"127.0.0.1\" :port 9001))
Which does the dirty work of starting a REPL server to which the browser can connect, and then switches itself into a namespace I've chosen with some useful functions for hacking on the UI. The code to connect from the browser can be found in the Weasel documentation. One hack that I've implemented in this project is that the call to the frontend's main function actually lives in the application's root HTML page, immediately after it's pulled in with the `goog.require` machinery.
Once you get all of the above working, you can launch a sibling repl (grep around in the Cider sources for `sibling`) for the ClojureScript work that hooks into the JVM process running the server-side code to avoid the time and resource costs of booting a whole new Clojure-hosting JVM from scratch.
- one-button, from-CLI code-reloading witchcraft
It's called 'figwheel', and it's awesome if you're not trying to do something insane like talk to the browser from directly in Emacs. User/developers start the interaction loop with code running in the browser by calling `lein figwheel` at the terminal. The Clojure project's `project.clj` contains a `:figwheel` key specifing output directories and asset paths and the like. Figwheel goes so far as to reload all compiled frontend (as we call the browser) resources and load them into the browser every time that any of the ClojureScript files change. This strikes me as a bit heavy-handed (and it can in fact be tuned), but I never ran into the dreaded change-pileup, where the compiler never digs out from underneath its queue of requested recompiles.
Also, figwheel's configured with `:on-jsload` pointed at the project main function. NB: this will not get called initially, only when Figwheel determines that the JS needs reloading. To that end, I've configured the HTML page which loads the compiled JS to then call the exported `main` function.
Believe it or not, casual reader, this is all pretty vanilla ClojureScript shit. I hope you begin to grok the kinds of complexity even I can't keep from running afoul of.
Quirks (or, wherein I reveal some chairs in the epic tower I'm climbing to the moon atop):
Because I've also chosen to take a dependency on Reagent, which itself has a depdendency on Facebook's React library, when I do want to refresh the browser state, I have to make a manual call to `reagent/render`. In the 'control room' (as I refer to the namespaces I drop user/developers into for controlling the application), I abstract that away as `reset`, in the same vein as the backend. The reasons for this aren't entirely clear to me, but I made it a huge way down the road with this Dick-van-Dyke agglomeration of web technologies before Spolsky bit me particularly hard – and it's not even a hard bite, per-se, so much as it is one I just don't understand particularly well.
The reason (I think) that one must call `render` manually when working with these tools, is because the code is already loaded into the React component lifecycle, which instantiated the component and is now maintaining its rendering, and doing so from the instantiated object, not the code from which the object was originally instantiated.
And, how is everything?
Quite peachy, thank you very much. I have hot-reloading of UI code without having to trash application state, direct connections between running server code, running browser code, and my editor (implying all the niceties of inline test feedback and generally speaking a programmable editor designed to work with running Lisp systems).
The cognitive overhead is incredibly slim, and the speed at which I can move using a single language in these disparate environments from the comfort of my editor is a breath of fresh air.
Clojure, while no Common Lisp, has enough Lisp power that I'm using it to great effect to strangle web-application development challenges. The language in question has wrapped itself around the complexity and sheer size of the JVM, and does an admirable job of doing the same for the historically miserable webapp development workflow.
I didn't even take an inordinate amount of time to pull all of this together. I estimate a total of 10 hours over this project lifetime fixing small tooling gripes one-by-one as I evolved this setup from the novice ClojureScript project/interaction setup folks recommend. Trickiest part by far (but also the most rewarding, and the one I left the longest so as to learn the most about the systems under hack before biting it off) was the "really seriously totally interactive development workflow from Emacs".