CH

December 26, 2015

[UPDATE] WOTPASTE changes

Filed under: Uncategorized — @ 12:00 a.m.
[UPDATE] WOTPASTE changes

From the last post:

In the absence of feedback leading to material design and implementation changes…

Stan pointed out the pure braindamage of asking for "signed farts", which is to say, "unsavory or pointless bits of text that nobody would ever care to hang on to", Mircea argued for "going big". Also nixed: registration or authentication of any sort.

There was also an epic newline related ratfuck, as well as many suggestions to simply make the thing accept whatever trash the wind blows in. It's all, as always, in the logs.

So:

Changes:

  • Anyone may post
  • Heathen newlines are coerced to Unix newlines

Under consideration:

  • Not mutilating newlines
  • Not operating a public urinal

More to come later, I'm sure.

December 25, 2015

[ANN] WOTPASTE Alpha (oh and I found another fucked up corner of GPG...)

Filed under: Uncategorized — @ 12:00 a.m.
[ANN] WOTPASTE Alpha (oh and I found another fucked up corner of GPG...)

I present the entirely unimaginatively named WOTPASTE!

In a fit of pique at Windows-style CRLFs complicating Bitcoin Foundation work yet a-motherfucking-gain, I hauled off and implemented a classic "three curls and a grep" solution of my own to the lack of an accountable pastebin operator in the WoT (because clearly, I don't have enough to do what with the muscle car, new mother and child. Oh, and mod6 asked for one.). I'm damned confident that it won't shit CRLFs1 (or anything other than regular old unix newlines) into the output, but also look forward to being horribly surprised at how some slip through anyways.

Expect brief no-notice outages over the next few days while I flatten rough spots and tend to operational tasks. In the absence of feedback leading to material design and implementation changes, I expect it to be stably up, robustly backed up and reliable by January 1st.

Have at! Holler in #bitcoin-assets if things don't seem right to you.

Footnotes:

1

As an aside, did you catch my discovery-of-old-news-new-only-to-me in the logs the other day? It turns out that one can freely convert clearsigned GPG documents forth and back between Unix newline and Windows "carriage-return linefeed" newline semantics without invalidating the signatures. This means one can produce 2 files that are identical to a casual inspection, but whose SHA's are entirely different, that both validate.

December 21, 2015

Development Costs of User Interface Development (and mitigating them with Clojure/Script)

Filed under: Uncategorized — @ 12:00 a.m.
Development Costs of User Interface Development (and mitigating them with Clojure/Script)

Alternative titles: ("Enjoy some more razor-sharp carbon-fiber scaffoldings of paradigm abstractions on which to cut yourself!" "Clojure – these asbestos-gloves-for-web-development didn't kill me immediately!" "Tooling is a great way to spend time and feel good about yourself without making any real progress at all." "One neat trick for driving UI developers insane with the inadequacy of their toolchain!")

This first section is the philosophy behind this particular insane quest of mine. If you're looking for help with your Clojure environment, grep downwards.

The most painful and frictive step in most user1 interface developers' workflows is the cycle between writing some code, and observing the changes so wrought on the final product (be that a label's new value, the size of some new elements, or sub-view scaking and positioning). Over the past two or three years I've learned of a few older paradigms for building UIs2 where the person building the interface in question can write the code, and reload it into the running application for immediate feedback.

At this point most Lispers are probably laughing. Recompiling code and incrementally loading it into a running program has been a basic feature of Lisp environments since literally before I was born. Despite that, the vast majority of modern UI development workflows look like:

  1. wrangle app into state dev is hacking on today
  2. decide what needs changing
  3. write that code
  4. "compile"3
  5. restart the application under development with the new binaries

Of course, swap details for your local development platform.

My (proposed) metrics for evaluating UI development environments:

  • minimize duration between writing code and discovering errors in that code (e.g., compiler feedback, runtime exceptions)
  • minimize repeated work and wasted time (clicking through UI's, manually setting up application state, recompiling the whole application, reloading the web page)
  • minimize duration between written code and visibly different UI elements
  • minimize context switches
  • minimize enforced idle periods

Backend developers (in particular the Python and Ruby groups, Java to a lesser extent, and in all cases highly organization-dependent. Haskaliens, I've noted, rarely write tests, preferring instead to lean on the type system and their knowledge of how constrain program behavior with it. If you figure out how QuickCheck came out of the Haskell group, let me know) attempt to attend to the minimization of time between writing of code and testing its behavior by doing something on the spectrum of unit or integration tests.

Unit and integration tests are well, good, and in some situations downright imperative for these dynamically-typed languages where runtime behavior is nigh-unknowable thanks to the dynamically-typed nature of the programming environments. Many testing programs for software written in languages like this hypothetical that I've seen wrap a net of tests around the (for example) user-input-processing functions, attempting to make and enforce guarantees about the shape of data streaming from those functions into others, for the systems under construction rapidly grow so large, complex, and complected with themselves that formally reasoning about the whole thing (as a team at least, thinly-spread across the application surface area) is not even worth attempting4. Hence, tests.

Tests, though, "can only show the presence of bugs, never their absence"5.

Developers are generally on their own when it comes to minimizing repeated work and wasted time, and let me tell you – are hilariously bad at it. Not only do they love to grow their own lovely little bonsai trees of complexity, they're also more than happy to run to the store twice a day for fertilizer that starts expiring as soon as they've paid, and whose half-life is measured in jokes about your mother. Some are hell-bent on wasting your time, and will go to such lengths as restarting the server, reloading the web page, and clicking through the whole application on every single code iteration – "just in case. I've been burned by state before". Others, great minds some, in an effort to speed up their development loops in the face of hostile and antagonistically-designed tooling, will rip the application apart and cobble together a development harness for whatever particular view they're working on at the time, so they don't have to go through all of the clicking through the damn thing. This is a laudable project, but hurts later in re-integration time when they want to stuff that selfsame view back down into the application view tree where it belongs – "what precisely state does this thing rely on again?".

Minimizing context switches (between JS in the browser and Python in the server, for instance), are really important. So important, in fact, that my friends in UI who make the jump from "web technologies" to the holistic environment of iOS or Android development frequently claim they'll never go back, because their flow and productivity are simply so much higher6 on the cohesive platform.

That's why everything's so miserable, in somewhat less than a thousand words. Developers love complexity, our UI toolchains are downright hostile, and nobody has the intellectual fortitude to imagine a better world.

Except for weirdos like me, who listen in bars to hoary old explorers about the glories of the Symbolics temples, and the costs of a world that runs on C and CSS. I have gone to truly absurd lengths in order to drive my personal productivity up7 with a ridiculously arcane toolchain.

The toolchain consists of Clojure server-side, ClojureScript in the browser, and it's all tied together with Emacs. It boasts minimal time between code-writing and effect-reviewing, provides excellent tooling to minimize clicking-through-of-UIs, is readily munged into ongoing automated testabity thanks to the Lispy ease of refactoring, boasts next to no context switching overhead (beyond differences between JS and the JVM), thanks to an integrated development environment and using the same wrapper language around the host languages in the browser and server, and almost entirely boils off enforced idle periods.

Before I (briefly) document (some) of the quirks of this setup, keep in mind that whatever I do to support my own insane version of interactive development, I've also set myself the constraint of configuring the whole toolshed in such a way that someone who doesn't use Emacs can dive into a project shaped like this and (from a tooling perspective) have a pretty similar experience.

I shan't provide much by way of code or configuration in this post. Other people do that better. What I shall do below, is lay out precisely which tools I use in which scenarios, and how I've gotten them to behave together.

To that end, the tools:

  • server-side project structure

    The project is structured in such a way that calling `lein repl` at the terminal launches a developer into a console from which the system can be piloted. It has functions like `start`, `reset`, `seed-db`, and other useful tools like `compile-cljs`. This is a user/developer's interface to the server-side code under development, and is capable of clearing out its own code and recompiling the entire application codebase from disk in under 400ms.

  • on-demand recompilation of arbitrary code segments from Emacs

    This actually works spectacularly (with some quirks I'll reveal later). I can write code, evaluate it, and instruct the frontend to redisplay its entire self without altering the application state. Idem for the backend, and entirely transparently at that. I don't have to worry about where the code is going, I say "emacs compile and ship this to the relevant process" and it does so. It's goddamned amazing.

    This part entailed using Weasel, configuring the Piggieback middleware, and setting `cider-cljs-repl` in Emacs to the string:

(do (require 'weasel.repl.websocket)
    (cemerick.piggieback/cljs-repl
      (weasel.repl.websocket/repl-env :ip \"127.0.0.1\" :port 9001))
    (in-ns 'frontend.foo.bar))

Which does the dirty work of starting a REPL server to which the browser can connect, and then switches itself into a namespace I've chosen with some useful functions for hacking on the UI. The code to connect from the browser can be found in the Weasel documentation. One hack that I've implemented in this project is that the call to the frontend's main function actually lives in the application's root HTML page, immediately after it's pulled in with the `goog.require` machinery.

Once you get all of the above working, you can launch a sibling repl (grep around in the Cider sources for `sibling`) for the ClojureScript work that hooks into the JVM process running the server-side code to avoid the time and resource costs of booting a whole new Clojure-hosting JVM from scratch.

  • one-button, from-CLI code-reloading witchcraft

    It's called 'figwheel', and it's awesome if you're not trying to do something insane like talk to the browser from directly in Emacs. User/developers start the interaction loop with code running in the browser by calling `lein figwheel` at the terminal. The Clojure project's `project.clj` contains a `:figwheel` key specifing output directories and asset paths and the like. Figwheel goes so far as to reload all compiled frontend (as we call the browser) resources and load them into the browser every time that any of the ClojureScript files change. This strikes me as a bit heavy-handed (and it can in fact be tuned), but I never ran into the dreaded change-pileup, where the compiler never digs out from underneath its queue of requested recompiles8.

    Also, figwheel's configured with `:on-jsload` pointed at the project main function. NB: this will not get called initially, only when Figwheel determines that the JS needs reloading. To that end, I've configured the HTML page which loads the compiled JS to then call the exported `main` function.

    Believe it or not, casual reader, this is all pretty vanilla ClojureScript shit. I hope you begin to grok the kinds of complexity even I can't keep from running afoul of.

Quirks (or, wherein I reveal some chairs in the epic tower I'm climbing to the moon atop):

Because I've also chosen to take a dependency on Reagent, which itself has a depdendency on Facebook's React library9, when I do want to refresh the browser state, I have to make a manual call to `reagent/render`. In the 'control room' (as I refer to the namespaces I drop user/developers into for controlling the application), I abstract that away as `reset`, in the same vein as the backend10. The reasons for this aren't entirely clear to me, but I made it a huge way down the road with this Dick-van-Dyke agglomeration of web technologies before Spolsky bit me particularly hard – and it's not even a hard bite, per-se, so much as it is one I just don't understand particularly well.

The reason (I think) that one must call `render` manually when working with these tools, is because the code is already loaded into the React component lifecycle, which instantiated the component and is now maintaining its rendering, and doing so from the instantiated object, not the code from which the object was originally instantiated.11

And, how is everything?

Quite peachy, thank you very much. I have hot-reloading of UI code without having to trash application state, direct connections between running server code, running browser code, and my editor (implying all the niceties of inline test feedback and generally speaking a programmable editor designed to work with running Lisp systems).

The cognitive overhead is incredibly slim, and the speed at which I can move using a single language in these disparate environments from the comfort of my editor is a breath of fresh air.

Clojure, while no Common Lisp, has enough Lisp power that I'm using it to great effect to strangle web-application development challenges. The language in question has wrapped itself around the complexity and sheer size of the JVM, and does an admirable job of doing the same for the historically miserable webapp development workflow.

I didn't even take an inordinate amount of time to pull all of this together. I estimate a total of 10 hours over this project lifetime fixing small tooling gripes one-by-one as I evolved this setup from the novice ClojureScript project/interaction setup folks recommend. Trickiest part by far (but also the most rewarding, and the one I left the longest so as to learn the most about the systems under hack before biting it off) was the "really seriously totally interactive development workflow from Emacs".

Footnotes:

1

There's this weird parallel between dope dealers and their users and software dealers and their users. "The first one's free!" both dealers say, before webbing their unsuspecting prey in an only-very-expensively-maintained complex hairball of C++, CSS, or heroin-seeking as the case may be.

2

As near as I can tell, McCLIM is a "modern implementation" of the long-lived Lisp UI models of presentation methods, objects, frames and playback buffers, that evolved out of the Symbolics machines and the Metaobject Protocol. (The previous sentence is a miserable undereducated absurd abstraction of a lot of complicated history). Symbolics' Concordia document editing and typesetting software would have been conceived of in this vein – written in the same language as the operating system, it had access to objects being edited by other programs in the Lisp image, could size and…let's just say that it was very Smalltalk-y, to the extent that I understand that storied environment…

3

You would not believe the eldritch horrors wrought upon poor JavaScript files these days.

4

Not worth attempting for normals, that is to say. There are definitely people who are more than capable of both writing these edifices of complexity and also navigating their shoals and undertoes with grace and aplomb.

Now, do you think managers want 1/1,000,000 type folks capapble of marshalling this scale of complexity for their custom Customer Relationship Management applications on the field representatives' mobile devices? Or do you think they'd prefer teams of entirely-average folks whose output can be managed predictably, if slowly?

Especially given that there's no hard upper bound on how much money the truly determined can blow on software, BigCorp management is nearly always going to prefer tools that lend themselves to use by the kind of people who…work at BigCorp.

5

Poetically relicensed from the great Dijkstra.

6

Why must they build UIs? I hear the peanut gallery asking. Because that is the entire market right now. Nobody said we enjoyed it.

7

Or, you know, get off. Knowing when you're actually honing the sword of your development toolchain and when you're just masturbating is a critical point of self-awareness.

8

The Google Closure compiler (related, but way down there in the giant pile of inscrutable dependencies) does an excellent job of cacheing its output, driving builds of JS from CLJS from ~1025ms (cold) to ~416ms (with cache).

9

Which, per past experiences with libraries provided by other people, I have vendored right the hell into this project thank you very much. I don't like to wake up with brand new JS running in my face because I took a dependency on some CDN.

Also, take deps like that and then try to work on the plane. I dare you.

10

This pattern of "reloading" code is vitally important to the Clojure community, because restarting the JVM via Leiningen is so damn painful. This can be gotten around, and nominally boot times of less than a second achieved by precompiling the base image. I have yet to actually play with this, but an acquaintance claims to have gotten the whole Clojure boot time to under a second by doing just that.

11

This, by the way, would be an excellent location to use some of Common Lisp's features around mutating object definitions at runtime when the class from which they were instantiated changes.

---