Mike Brinker

Spec-Driven Development: Skeleton Coding

Wed, 11 Nov 2026 18:59:15 GMT

Source: Skeleton Coding — Aditya Golatkar et al. · arXiv

At the philosophical level, I would say this paper explores the inextricable symbiosis between form and function. The way software’s behavior is verified depends on the structure of the software. Cynically, one may conclude “oh look, when humans write the code, the quality is better!”. I myself have desired a better abstraction of defining and discussing code with agents. Ultimately, this paper may just be showing or rediscovering the truth encoded in this comic:

In my experience, I spend the most time correcting the output of agents when it comes to the structure of the program. For whatever reason, they seldom build the structure the way that I would. The structures they generate often do not follow trajectory I have in my head for the ideal overall structure of the system but rather adhere to “local” examples in the current codebase which are desirable generally or in this particular case. Or they misuse an existing pattern. Or they employ unnecessary artifices.

This paper implicitly identifies an ability gap in current models which could be improved in the future. Models today can create programs that behave as expected (generally so, to some degree) but do not create programs with structure that skilled people would consider to be “good”. This matches my experience interacting with agents every day.

The paper concludes that structure specifications improve correctness more than behavioral specifications alone. Improvement in correctness is higher when the structural specifications are written in a code-like format compared to prose.

Favorite Quotes

Ideally, this separation of concerns between the what and the how promises to free developers from the laborious task of writing code. However, we show that it can actually exponentially increase the effort required.

Developers rarely begin with a complete specifications; they adapt it to correct issues and steer the agent toward the goal. However, since prompting a model does not perform Bayesian conditioning [8], there is no guarantee that providing additional specifications will eventually converge to the goal. However, adding formally verifiable contracts [9, 10, 11] to the skeletons circumvent the issue (Theorem 3.1). Even when informal contracts are not used, an imperfect verifier can still ensure faster convergence to a valid implementation (Theorem 3.2).

Structural specifications produce less eroded code than behavioral specifications: Across all benchmarks and models, structural specifications reduce code erosion by up to 24.7%, compared to the task-only prompt. Meanwhile, behavioral specifications increase it. Verbosity varies little across conditions.

Originally encountered on X.

Components of Agent Memory

Tue, 10 Nov 2026 17:00:58 GMT

Source: The components of agent memory — Lance Martin · Anthropic

This article explores the design of memory systems for agents. A pertinent question alluded to is: how can an actor decide what to remember now when it doesn’t know what’s next? The ideas contain much anthropoligizing of Claude (comparing memory systems to the human brain). While I think there are problems with constantly seeking to understand agents through an anthropological lens, many of the connections are admittedly interesting. I can sympathize with the stated tradeoff where taking good notes while working on something can be distracting, but perfecting the notes later is resource-intensive. I appreciated the writing style that includes references to the humanities. A bit of Kierkegaard and Borges enriches almost any experience!

The memory system described contains a hierarchy (from top to bottom):

Injected context (smallest)
Memory representations (e.g. wikis)
Memory Notes
Transcripts (event logs)

“Dreaming” is then a process that moves information up the hierarchy while verifying against the layers below.

Observations

Opus 5.5 and Sonnet 5.5 had very similar scores on CL-Bench.
There is explicit mention of Claude being evaluated on its ability to play Pokemon Red. This connects to rumors that the advances gained in current frontier models largely come from reinforcement learning via playing games.
The better the model, the more selective it was in recording memories.
Giving explicit instructions for how to use memory helped older models much more than newer ones.
Providing a search tool in addition to raw file access to the memory wiki reduced token usage and improved eval scores.

Sources referenced:

Continual Learning Bench (CL-Bench) 1.0 (eval)
LLM Wiki
Dreaming with a dynamic workflow (cookbook)
Knowledge wiki with a dynamic workflow (quickstart)

The October 2026 Inflection

Fri, 23 Oct 2026 14:12:00 GMT

In 2025, there was a widely regarded inflection point in the software engineering professional community. Opus 4.5 was released on November 24. I remember being thrilled that when I asked it to implement something on my personal projects, the result was something that generally worked. Up to this point, it would take multiple iterations of prompting to get a working result (if one wasn’t writing any code themselves). Then, from December 25 - 31 in 2025, Anthropic gave Pro and Max users double the usage. I remember coming back to work in January feeling different about the role that AI would have in my daily work as a software engineer. Much of the commentary I saw online matched this sentiment. Over time Simon Willison, Martin Fowler, and Andrej Karpathy (among others) acknowledged the impact of this inflection point.

In October 2026, I believe we’re at another inflection point. If the aforementioned one convinced people that agents were capable of producing the work, this new point that we find ourselves at has more people than ever convinced that agents can or will have the ability to replace human software engineers. I’ve been conflicted and skeptical about this at least since GitHub Copilot was launched in 2022. While I don’t think that we can be sure about the future holds for profession and our employment in it, it sure seems like we’ve crossed a line with no chance of turning back.

I have been experimenting with large language models since 2022. For what felt like a long period of time, the people who had influenced my most in my career were silent or distant about the prospect of using AI to write code. Then, in the last 12 months, all of them have seemingly been converted to this new way of working.

DHH, the creator of Ruby on Rails, who, in my opinion, spoke most elegantly about the utility of aesthetic beauty of code acknowledged the November Inflection:

Then we get to November 24th, 2025. Opus 4.5, to me, was the dividing line, where suddenly… I didn’t even try it on the 24th. I think I tried it on the 26th. I gave it a couple of tasks, and I realize that the quality of the output is uncannily close to what I would’ve written. And I remember just leaning back and thinking, “What just happened?”

How did we go from this autocomplete mess that I was talking to you about in the summer to this just a few short months later? How did we get both the increase in intelligence and then also the increase in usability?” This agent harness question where I don’t know if Opus 4.5 was that much smarter than Opus 4, which is what we had in the summer, but its ability to instrument your computer, to use tools, to check its own work, to apply its intelligence in such a way that you could get real meaningful work out of it, was completely different. And I think this is then the big change that happens for almost anyone who paid attention and started playing with it over the Christmas break.

On September 23, In a sermonic talk emblematic of the entire point I’m making, DHH publicly declared his retirement from handwritten code.

In June 2025, “Uncle” Bob Martin, whose Clean Code left an indelible mark on my coding style , emphasized caution with AI tools. He began experimenting public with AI in January 2026. By April he wasn’t reviewing the code produced by his agents. He often tweeted about building custom harnesses for his agents to ensure acceptable code quality, but by September he abandoned that too.

Kent Beck, the creator of XP and popularizer of Test-Driven Development was relatively ahead of the curve. In April 2025, he spoke of “coding at 3AM for the first time in decades” due to his use AI.

Justin Searls, who I fondly remember on TDD, not to mention his personality test for programmers stated:

Over a year ago, I stopped reading code and started automating adversarial agents to perform recursive code reviews instead.

The result: low-defect systems that do what I want. Can’t believe anyone engaging seriously with these tools still believes human code review is superior

DHH, Kent Beck, Uncle Bob, Martin Fowler, and Justin Searls. In terms of people who write books and speak on software engineering, this is who I learned from. 3 of them were architects of the Agile Manifesto. They are all-in on AI now. While I’ve been experimenting with LLMs and code generation since GPT 3.5 and haven’t written basically any code by hand for the last 1.5+ years (!!), I think I now have to admit to myself that it’s time to be all-in.

Defining a FontFamily for Greek

Thu, 09 Jan 2025 16:00:00 GMT

I enjoy a beautiful typeface for Greek (or any language for that matter). One of my favorites is the unnamed Greek font used as the default in Logos Bible Software. That aside, SIL has a free font which looks great, especially in Greek: Gentium Book Plus.

I decided it would be nice to use this font whenever Greek Unicode characters are present on this site. I wasn’t sure how to accomplish this at first, but it turns out that CSS allows you to specify Unicode ranges when you are defining a font. Along with properly managing the cascade of styles on the site, font definitions like this did the trick:

@font-face {
  font-family: "SortsMillGoudy";
  src: url("../fonts/sorts-mill-goudy/SortsMillGoudy-Regular.ttf")
    format("truetype");
  font-weight: normal;
  font-style: normal;
  unicode-range: U+0000-024F;
}

@font-face {
  font-family: "GentiumGreek";
  src: url("../fonts/gentium/GentiumBookPlus-Regular.ttf") format("truetype");
  font-weight: 400;
  font-style: normal;
  unicode-range: U+0370-03FF, U+1F00-1FFF; /* Greek ranges */
  font-display: swap;
}

For English and Code, unicode-range: U+0000-024F seems to do the trick. And for Greek, I’m using unicode-range: U+0370-03FF, U+1F00-1FFF;.

And now, some Greek:

καὶ τοῦτο προσεύχομαι ἵνα ἡ ἀγάπη ὑμῶν ἔτι μᾶλλον καὶ μᾶλλον περισσεύῃ ἐν ἐπιγνώσει καὶ πάσῃ αἰσθήσει…

The Dative of Possession

Mon, 30 Dec 2024 16:00:00 GMT

Was your name given to you or created by you?

I was confused during a recent Greek lesson in Scripturial. Phrases like “my name” and “your name” were being taught as ὄνομα σοι and ὄνομα μοι. To me, σοι and μοι looked like the dative forms when I was expecting the genitive.

After a quick consultation with ChatGPT (GPT-4o), I was under the impression that for this phrase in particular, Greek speakers would say something like “the name for me” rather than “the name of me” (the latter of which I was expecting based upon my current mental map between Greek and English). In trying to confirm ChatGPT’s answer, I stumbled across this interesting B-Greek post.

The folks discussing this very question on the forum confirmed GPT-4o’s suggestion that the dative is used in Greek to express this idea. They link to a grammar that calls this The Dative of Possession:

The dative of possession is closely related to the dative of advantage. Simple possession is expressed by the genitive (§888.1); the one for whom something exists (i.e. in his or her interest) is put in the dative…

In the forum, the initial answers sounded very similar to what GPT-4o had told me, but later on, some contributors started posting statistics to compare the frequency of constructions like ὅνομά μοι and ὄνομά μου. It ends up looking like both constructions are used somewhat widely and there is variance across time and also whether 1st, 2nd, or 3rd person is being used.

When I think about it, this construction makes sense. One’s name is given (a name “for” you) rather than generated from them. I suspect that in Greek there is a bit more psychological separation between one’s identity/being and one’s name than there is in English? In English, I am Mike, my name is Mike, etc… “Mike” is not a name for me, it’s my name.

I continue to find Scripturial a very useful way to improve my Greek, while working at a pace that fits with everything else I have going on in life. Like this one though, there are instances where a construction that is taught leaves me with questions that I need to resolve on my own. I suspect that’s a worthwhile tradeoff to make (letting my curiosity dictate further research) in order to keep the curriculum compact.

This is enough for me to know that I need to recognize ὅνομά μοι and ὄνομά μου as serving a similar purpose without assuming they are identical. I wonder if B-Greek content is part of OpenAI training data? I would guess so. In any case, I’m a bit late to the party, but B-Greek looks like great place to discuss and learn about these sorts of things online.

Ματαιότης in Romans 8 and Ecclesiastes

Tue, 24 Dec 2024 16:00:00 GMT

Does Romans 8 suggest a Pauline interpretation of Qohelet?

I was recently reading through Romans in my morning study time. I noticed that with each successive chapter, I was reading slower and slower. But the time I hit Romans 6, I was getting through less than one chapter a day. At the moment, I’ve been in Romans 8 for several days, and I’ve hit something interesting that I wanted to write down.

I paused on Romans 8:20 -21:

For the creation has been subjected to futility, not willingly, but because of the one who subjected it, in hope that the creation itself also will be set free from its servility to decay, into the glorious freedom of the children of God

W. Hall Harris III et al., eds., The Lexham English Bible (Bellingham, WA: Lexham Press, 2012), Ro 8:20–21.

τῇ γὰρ ματαιότητι ἡ κτίσις ὑπετάγη, οὐχ ἑκοῦσα ἀλλὰ διὰ τὸν ὑποτάξαντα, ἐφʼ ἑλπίδι 21 ὅτι καὶ αὐτὴ ἡ κτίσις ἐλευθερωθήσεται ἀπὸ τῆς δουλείας τῆς φθορᾶς εἰς τὴν ἐλευθερίαν τῆς δόξης τῶν τέκνων τοῦ θεοῦ

Michael W. Holmes, The Greek New Testament: SBL Edition (Lexham Press; Society of Biblical Literature, 2011–2013), Ro 8:20–21.

Surely I’ve had more thoughts about this passage over the years than I can remember. This time though, I was trying to ask some basic questions about the idea of creation being subjected: who subjected it? For what purpose was it subjected? I found my answers and an unexpected (and delightful!) rabbit trail in the commentary of James D.G. Dunn (which I had never read before):

The point Paul is presumably making, through somewhat obscure language, is that God followed the logic of his purposed subjecting of creation to man by subjecting it yet further in consequence of man’s fall, so that it might serve as an appropriate context for fallen man: a futile world to engage the futile mind of man. By describing creation’s subjection as “unwilling” Paul maintains the personification of the previous verse. There is an out-of-sortness, a disjointedness about the created order which makes it a suitable habitation for man at odds with his creator.

By “futility” Paul probably has in mind the same sense of futility of life which found expression in Jewish thought most clearly in Ecclesiastes—that weariness and despair of spirit which cannot see beyond the stultifying repetitiveness of life, the endless cycle of decay and corruption, the worthlessness of a lifelong effort which may be swept away overnight by a storm or be parched to nothingness in a drought, the complete insignificance of the individual in the tides of time and the currents of human affairs—all indeed that, had man but realized it, was going to make it impossible for him to be “as god,” made it inevitable that he would become subservient to useless idols and mere things. Yet for Paul, of course, that is not, could not be the last word.

James D. G. Dunn, Romans 1–8, vol. 38A, Word Biblical Commentary (Dallas: Word, Incorporated, 1988), 487–488.

With my basic questions of comprehension satisfied in the first paragraph, I turned with interest to this idea of “futility”. I had never considered a connection to Ecclesiastes here before, despite my long-term interest in it. As I thought about it, I became excited because I began to wonder if we have a latent New Testament interpretation of Ecclesiastes here. There has been much argument and little consensus in the Christian tradition about the use or application of Ecclesiastes. Is it an accurate view of how things are, with a conclusion that is so orthodox it’s nearly boring ()? Or is it a cautionary tale of what life is like in the world without God (i.e. “under the Sun”)?

Well, before getting to far ahead of myself, I was desiring a concrete linkage between this passage and Ecclesiastes that was perhaps more solid than interpretation of Dunn and my penchant to see an allusion.

ματαιότης is the word for “futility” in verse 20. I hadn’t seen this one before, so I took a quick look at the word family in NIDNTTE:

μάταιος G3469 (mataios), vain, empty, worthless, futile, deceptive; ματαιότης G3470 (mataiotēs), emptiness, futility, worthlessness; ματαιόω G3471 (mataioō), to render futile or worthless; μάτην G3472 (matēn), in vain, to no end; ματαιολογία G3467 (mataiologia), useless talk, empty prattle; ματαιολόγος G3468 (mataiologos), babbling, subst. idle talker

Pretty consistent! Now, in the past, I had become convinced of a translation like “vapor” being best for hebel in the Hebrew text of Ecclesiastes, but something made me wonder how the LXX translated it. So I took a look at the LXX and what did I find?

Ματαιότης ματαιοτήτων, εἶπεν ὁ ἐκκλησιαστής, ματαιότης ματαιοτήτων, τὰ πάντα ματαιότης.

Henry Barclay Swete, The Old Testament in Greek: According to the Septuagint (Cambridge, UK: Cambridge University Press, 1909), Ec 1:2.

The opening speech of Ecclesiastes (not to mention the rest of it) is chock full of ματαιότης! So I felt better about a connection between the outlook of Ecclesiastes and Paul’s use of ματαιότης in Romans 8. So what does it all mean?

The big thing that came to mind was that in these comments in Romans 8, Paul is talking about an “emplotted” state of affairs, that is the act of creation being subject to futility is not how things originally were or will finally be, but instead it is how they are now at this point in the “story”.

This has potentially big implications for me in my understanding of Ecclesiastes – the very real and resonant way the Qohelet speaks about world is both factual (truthful about how things are) and yet not-as-they-should-be. So, potentially, from Paul’s allusion to it here, we can see that he understood Ecclesiastes to be accurately depicting the world and our experience of it while proclaiming that things will not always be this way. I had never noticed this before, but it fits well with what I understand Paul to be saying up to this point in Romans.

Conclusions? Well, I’m interested to find out what might be in the tradition about the connection here. This has also made me a bit curious about what the concrete criteria might be to declare an allusion. I would guess that use of a word isn’t enough on its own, but in this case there is the lexical link and a thematic “kinship”.

βίβλος and βιβλίον

Mon, 23 Dec 2024 16:00:00 GMT

Earlier this year I started incorporating Scripturial into my Greek learning. Early on, Scripturial introduced βιβλίον as “book”. So for the last few months, thinking about “book” I reached for βιβλίον. I recently encountered Βίβλος and was a bit confused. I thought βιβλίον was a neuter noun referring to “book”. Βίβλος appeared to be a masculine noun that also referred to book or scroll.

The Lexham Theological Wordbook confirmed what I thought I knew about βιβλίον:

βιβλίον (biblion) n. neut. scroll, document. A written document. This is most common term for a scroll in the NT.

The entry for βίβλος surprised me though:

βίβλος (biblos). n. fem. book. Refers to any written composition, including sacred books.

βίβλος is a feminine noun that refers to any written composition?

As I often do with questions that I guess have rudimentary answers, I quickly asked ChatGPT (to the extent I can verify its answers; they seem fairly trustworthy with lexical questions in Ancient Greek). ChatGPT told me that βιβλίον is “diminutive” (originally referred to a little Βίβλος) and eventually became the most common usage. NIDNTTE confirmed and expanded this suggestion:

Presumably, the word came to mean “papyrus” because this writing material was exported to Greece through Byblos, where the plant was prepared. The diminutive form βυβλίον/βιβλίον referred to a strip of βίβλος, thus a piece of writing material, and then by extension it came to mean “document, letter” (Hdt. 3.128.2–3); eventually it too could refer to a scroll or book and thus could be used synonymously with βίβλος (a “doubly diminutive” form βιβλίδιον is attested as early as Demosth.). From the pl. τὰ βιβλία, “the books,” derives Lat. biblia and thence Eng. Bible… In the LXX the term βίβλος occurs almost 30× (βύβλος only in 2 Chr 17:9; Dan 9:2 [Th.]; 1 Esd 1:31), but βιβλίον c. 185×.

Moisés Silva, ed., New International Dictionary of New Testament Theology and Exegesis. (Grand Rapids, MI: Zondervan, 2014), 511.

Already by the time of the Septuagint, it looks like βιβλίον was far more common. Even in the New Testament, I found 10 instances of βίβλος and 34 for βιβλίον.

Even after multiple years of personal study in Ancient Greek, the mere gender of a word can get the best of me! There’s much to be curious about here. What came first, the origin of papyrus (The port city Byblos) or the word for papyrus? What characteristic of language do we see in the way the diminutive form became most common? Learning never ends and the language is more than my understanding of it.

Working with AI, State of the Union June 2024

Fri, 21 Jun 2024 16:00:00 GMT

This post explores what it is like to work with AI in a software development context. With our current rate of growth, this experience changes significantly every 3 - 6 months. In this installment, I chronicle an experiment of exploring a new concept by building a PoC using Claude Sonnet 3.5 and Anthropic’s new Artifacts feature.

This week, Anthropic released Claude Sonnet 3.5 along with a new feature they call Artifacts. According to Anthropic’s numbers, this new model outperforms even the recently released (and poorly named) GPT-4o. Buried in that same article, you will find a description of Anthropic’s new Artifacts feature. As far as I can tell, Artifacts allow you to focus on making and improving a particular thing (program, image, etc.) throughout the larger context of a conversation with the model. Many artifacts can exist per conversation. Programmatic ones allow you to see code with a running preview and are versioned!

Below, I’ll briefly describe my experience with this new release. For additional context, you may want to try the result yourself. As a test case, I wanted to explore the process of creating alignment data for an audio Bible. Why? Much of my day job revolves around developing an alignment tool for text-based Bibles. If you’re not familiar, “alignment data” describes the relationships between texts at the level of words, groups of words, or parts of words. At Biblica, we’re focused on aligning translations to Hebrew and Greek sources so that we can project source annotations through translations that are written in modern languages. Anyway, I wanted to explore a user experience focused on aligning an audio bible.

So what was it like collaborating with AI in June 2024?

The new Artifacts feature is a big step in the right direction. It just removes all kinds of annoyances that have come from coding with GPT-4 in the past. It keeps the thing you are working on front and center, the artifact in context properly, and allows you to easily switch between versions without scrolling through a long conversation.

The coding abilities of Claude Sonnet 3.5 are also excellent. It used a variety of libraries during our prototyping when appropriate. It also had good first-draft styling and design ideas. The most impressive part was that, for a fairly complex domain (aligning an audio bible, dealing with multiple languages, etc.), it rarely took more than a sentence or two to describe my intent. This last part is worth emphasizing: this model understood my ideas as I produced them. Never once did it seem to misunderstand what I wanted to accomplish.

There is still a lot of work to be done, though.

Most of the logic we built for this little concept demo was written in a single React component. In my local code repository, I had a few other boiler files for loading and displaying the component. Claude was aware of those but was mostly just writing code for my main react component. If I wanted to do some refactoring and split this component up, there did not seem to be a good or easy way to accomplish that with what is available today. Working in a single component isn’t the end of the world though, especially if Claude is writign all the code for me. Unfortunately, though, things got sketchy when the component grew to ~200 LOC. The model’s code output began to have rudimentary syntax errors like missing opening braces for an if statement. Beyond that, it didn’t want to reproduce the entire component during each iteration, so it would print fragments of the component that I would have to carefully replace in my local code repository. Beyond the syntax errors, the fragments were often functional, but where to put them wasn’t always obvious.

Overall, the prototyping experience was speedy and productive in the beginning – magically so. But when we hit that threshold, a lot of the magic disappeared.

I had some more UI/UX ideas, but I decided to stop when the syntax errors stopped. Then, it was time to deploy to Netflify so I could share the working PoC. At this point, I ran into issues with hosting the mp3 file for the demo in Netlify. The prototype used audio ranges, which the Netlify HTTP server did not seem to support. I was asking Claude for advice on this when I hit my Claude request limit. I’m a paying customer for Claude Pro, mind you, and I still hit the limit before I was able to take this small PoC across the finish line.

I ended up finishing the deployment without Claude’s help. One of the features that we had programmed for the PoC was left broken (the audio snippet for particular tokens should play when you click on them).

All in all, the early prototyping experience was honestly magicial. Claude was bringing my ideas to life, typing all the code for me. But even with my subscription, Claude couldn’t take me from idea to deployment. This was for a PoC, the core logic of which was ~250 LOC. I don’t think I could depend on Claude for larger projects, not even for a whole half-day work session.

When it comes to Generative AI, there is so much to be excited about. We also have a long way to go. I’m looking forward to what comes next.

Try the result on Netlify
Look at the code on Github

A Canticle for Leibowitz

Sun, 07 Jan 2024 16:00:00 GMT

Walter M. Miller Jr. · 1959 · 334 pages

I first heard about this book from my custom GPT, Reading Companion (a mentor for book discovery). Try it yourself and you might discover an amazing book!

This fine specimen obtained at my local Barnes & Noble.

A Canticle for Leibowitz was a Hugo Award winner in 1961. The book is set on post-apocalyptic earth. Over a series of centuries, the plot follows a series of monks whose use scribal practices to preserve the last known written remnants from the previous age. The book has an incredibly good combination of positive features — unique humor, characters that engender empathy, meditations on perennial wisdom topics, and geeky linguistic quirks to boot.

One character that I could feel for is a monk named Brother Francis. We are introduced to him as he is fasting in the desert as part of a process from being promoted from novice to take lifelong vows. It’s made clear that he feels suicide to be his only other viable option if he can’t make it into the order. He’s not a good fighter and doesn’t have the stomach to live out life in what seems like a very Mad Max type of situation. The life of a monk in the Albertian Order of Leibowitz can be austerely severe in its own ways though. A successful life is spent developing myopia as one copies often unintelligible texts by candlelight.

There are several scenes where the order of monks exhibit a combination of ignorance and reverence for the texts that I can only describe as charming. I won’t ruin all the fun, but imagine a dark age monk preserving circuit diagrams with no understanding of electricity and being so enamoured with the symmetry and intricacy of the design that he cannot help but illuminate his copy of the diagram with swords, crosses, flowers, and the like.

Perhaps the centerpiece of the book is when scientific inclination begins to return to humanity. There is now a leading scholar who is pushing to achieve the understanding and command of the world that was lost after the “deluge” (though his pride is doomed to merely rediscover what came before). This leading scholar makes a journey to the abbey where the monks preserve texts. With a skeptical eye akin to 19th century German Biblical Criticism, the scholar studies the faithfully preserved texts with plenty of theories about how their transmission could have gone wrong. Several conversations the monks have with the scholar give shape to a dynamic where I’ve felt tension myself. That is, on the hand, the religious inclination to preserve ancient knowledge (perhaps not fully understood) and on the other, the scientific proclivity for certainty that we see even today. The way that Miller uses the future setting to shed light on discussions like these that we have in our own time is even-handed, artful, and thought-provoking.

As time goes on, humanity gets ever closer to achieving the power and command they once had. This phase of the novel gives the author opportunities to explore theodicy in a way that feels honest and fresh to this reader. I could go on — there is a memorable visit with the Pope of New Rome that takes place, a prophetic Hebrew character that is interwoven in the plot and many more interesting and enjoyable moments. If you have any interest in Science Fiction, Christianity, scribal practices, linguistics, and the like, I think A Canticle for Leibowitz is likely to be worth your time.

Rebooting My Writing Practice

Mon, 01 Jan 2024 16:00:00 GMT

This year, one intention I have is to reboot my public writing practice.

In 2023, I learned the same lesson over and over again: writing improves my ability to think.

Early in 2023, I had the privilege of attending the Linking Your Thinking workshop. I went into the experience hoping to learn the one true way to organize my second brain. While that initial hope was severely deflated, I did learn the value of production as it pertains to thinking. Through a series of exercises at the workshop, I saw that the act of encoding ideas/thinking in writing leads to new ideas and new connections between them. If I spend even 30 minutes a day on some sort of writing practice, I find myself spontaneously having more ideas – generally, life gets more interesting. I used to think of my notes as some sacred cache of knowledge I was creating (that no one actually cared about), but now I see my notes as a vehicle for thought.

Additionally, through continuing with my Bullet Journaling practice, I saw that logging my activities, thoughts, questions, and gratitude helped me gain a better perspective on my circumstances and responses. Processing thoughts with logging didn’t make all my problems go away, but it did help me be a bit more present amidst the constant churn of distractions that surround me in daily life.

Finally, as I’ve embraced AI (read: LLMs with a chat interface), I’ve seen that often the largest barrier to getting them to help me in the way that I want is encoding sufficient context regarding my task at hand. I think that for knowledge workers to benefit from generative AI tools, it’s going to mean more writing, not less. I expect that figuring out the right context to encode in writing will be a determining factor for how helpful AI will be for the average knowledge worker as it inevitably continues its explosion onto the market.

I don’t have grand plans for this iteration of my website, but I do intend for it be be an outlet for writing and learning in public.

Here’s to 2024!

On 10 Years of Making Software

Wed, 30 Nov 2022 16:00:00 GMT

This year I realized I am now 10 years into a career in making software. As I reflected on my experience thus far I began to see certain themes that continue to resonate with me in my work. The writing below is an attempt to explore those themes in the hope of better understanding my motivations and what I should seek in the next phase of the journey.

As the point in time that I began Making Software moves farther and farther away, I oscillate between feeling as if it began a long time ago and as if it was yesterday. Surely though, much has changed in that time — not only about me but about the world we all live and work in. When I started in 2012, open offices were just washing ashore at the conservatively-cultured bank that I worked at. I was put on an agile scrum team of five developers along with a QA and a Scrum Master. We were literally making a spreadsheet for the web. While learning to sling code in Java 6 I was fed a robust diet of methodological guidance both in theory and in practice. TDD, Story Points, Sprints, Clean Code. There were just as many new buzzwords as experiences in those early months. I watched mustachioed, grown men passionately yell at groups of other humans about ESBs (Enterprise Service Bus… don’t look it up). We strove to automate our deployments, bring agility to company culture, and iteratively deliver value to customers. The best days were a surreal mixture of science fiction, fantasy, and late 90s office decor. Sometimes when the developers would get together to solve a problem, I felt like an X-Wing pilot listening to General Dodonna’s briefing as I was about to assault the Death Star.

On the worst days, I often felt as though I had tasted the future (surely an AGILE mindset could solve the problems besetting IT projects and society at large) but was held back by the red tape and stodgy outlooks of middle-management at the medium-sized regional bank where I held my employ.

Much has changed since those days for me and others — some of which might even be worth discussing. My goal here is to reflect on what it is about making software that I continue to find so enriching. Most of this is inextricably tied up in my journey: the extremely helpful mentor I had early on, the insane situations I happened upon, the friends I made, the mistakes I committed, and the internal commitments, failures, fears, and strivings of my own fickle heart. Perhaps though, the themes here are deep enough that they are helpful to others for reflection as well.

Identity

I’m able to trace my interest in computers and technology as far back as the 3rd grade. Sure, there was the glory of Sega Genesis that came before that, but what kid isn’t in awe of video games? No, it was that time in the 3rd grade when every student was assigned a Year-long project. We were to select any subject we could think of. Then, periodically throughout the year, we were to do a variety of research and projects related to that subject. At the end of the year, there was a fair where all of the students displayed their work. I had no idea what subject to pick. My father, in one of his famously creative and wise moments, suggested — with a smile on his face — that I should study The Internet. I knew this had something to do with computers and that was enough; I said yes.

Of the things I remember in the project a few are worth mentioning. It was 1998. We had 56k dial-up. My Dad took me to visit an ISP. At that time, our area had not yet succumbed to relying on giant corporations for internet access. The ISP we visited was operated out of an old house by what appeared to be a single person. The only thing that I remember about the visit is that the guy at the ISP asked me “Would you like to see The Internet?” His question implied that if I could see it then I might possibly be able to understand what it was. My heart leaped and I was able to mumble something that sounded affirmative, so the man opened a closet in the house to reveal a rack of network switches with cables strewn about, all awash in the glow and flicker of LED lights. I likely just stared and drooled as he said things I didn’t understand before closing the door. The encounter was characteristic of what would fuel my ongoing interest — a combination of Sci-Fi, awe, wonder, and not having any idea how the things that I was looking at were able to do what they did.

For the finale of my Year-long project, I provided input to my Dad as he built a website for me complete with a page that described the things that I liked to do, a guestbook, and animated GIFs. It was hosted on Angelfire and created with NetObjects Fusion. The small glimpses of HTML that I got would never fully leave my consciousness.

Despite some very geeky moments early on though, I never thought I could be a programmer. There was too much mystique around it and I didn’t have enough confidence in my aptitude to think that I might be able to ever do that myself. For many years I got my kicks from reading PC Gaming magazine, spending whole summers downloading shareware games, and making my own software distribution that I gave to classmates. Of course, things changed significantly when I was put on a software development team for an 8-month rotation at one of my first jobs after undergrad.

I was working at that conservative regional bank that I mentioned earlier. They had a program where they hired new grads and put them through three 8-month rotations in different departments of the IT organization before placing them somewhere permanently. As I said before, I never thought I could be a programmer, but the kind HR rep told me on my first day that I was being placed on an Agile software development team for my first 8 months and my only goal was to learn. The open-ended goal, my 20 years of accumulated curiosity, and the mentor I was assigned were enough to get over the confidence hump and find joy in making software.

I quickly fell in love with being on a team of geeks, trying to figure out what to build as we built it. There was a sort of geek religion that we all observed in that open office. The great Sci-Fi films were our Old Testament and Clean Code was our New Testament. Like any good religious community, we debated solutions to the problem of evil and the ever-present issue of how to practice our daily liturgies (let the reader understand: Vim vs. Emacs). The deed was done and I was now to do this work professionally.

The Nature of the Work

There are two sides to the coin of Making Software that acted in tandem to bring me to full devotion. First, there are worlds upon worlds of abstractions, architecture, tools, methods, and patterns that one can get lost in. I call this part of the work Engineering. Second, there is the harder-to-describe work of first, understanding your users and their world, and second, bringing your skills to bear upon their problems. I use the term Development for this part of the work. At their best, these seemingly disparate disciplines interplay healthily and produce an emergent effect that I think of as Craftsmanship. The first two are what one spends their time on. The latter becomes The Infinite Game.

Engineering

Early on in my 8 months with the Agile software development team, someone walked me through the codebase of an app that an adjacent team had been working on. It was built with Spring MVC. At that point, I had never heard of MVC before, so it was an exercise in explaining the difference between the front-end, the back-end, how the controllers interacted with models, etc. I remember the feeling of excitement just from being shown how the different layers of abstraction handled different concerns. There is a thrill in seeing how something works. I think for people like me whose formative years were less connected with nature and using one’s hands to make something, there is a way in which this type of engineering is fulfilling. It’s also an escape. There are entire technical worlds to get lost in. One’s present social issues can fade into the background when you’re comparing architectural approaches or ways to represent your data.

Some people probably live more fully in the world of Engineering than I do. For me, it’s an (interesting) means to the end of providing value in the real world.

Development

I think of a software project as an endeavor to create a product. We start with a diverse set of problems, ideas, research, analyses, and hunches. Over time the team processes those raw materials into a new creation — a semi-tangible system or product is brought into existence. And now, this new thing invites users into an experience or encounter that is perhaps somewhat unique from all that existed before it. It may solve a problem or only exist as a means of enrichment. Discovering what to make is the journey and the moments of joy or adventure along the way usually make it worth the price of admission.

The goal of Development is of course to produce something valuable. In my experience though, we tend to underestimate the value that is inherent in the journey itself. In some cases, the value provided to the business is just as much in the understanding that comes from trying to figure out what to build as it is in the result itself. Software often has to codify processes and practices that may have been implicit until now. Making Software requires us to understand the process and transform it. Beyond merely identifying a process, digitizing a process forces us to change how it works. Richard Hamming summarizes this well. He’s comparing aspects of the industrial revolution to the digital revolution:

Indeed, one of the major items in the conversion from hand to machine production is the imaginative redesign of the equivalent product. Thus in thinking of mechanizing a large organization, it won’t work if you try to keep things in detail exactly the same, rather there must be a larger give and take if there is to be a significant success. You must get the essentials of the job in mind and then design the mechanization to do that job rather than trying to mechanize the current version — if you want a significant success in the long run.¹

Making software is a process that forces people to learn about the nature of a thing in the real world to the point where they can create a digital experience that interfaces with reality. Learning about the nature of a thing and representing it — even improving it — are skills in their own right. Making Software as a means of learning and discovery is renewable motivational fuel for me. I suppose that’s why I keep moving towards working in domains that I’m personally interested in.

Craft

I then think of Craft as a property that emerges from the Engineering and Development disciplines. There is a thrill in the Engineering and a joy in the Development. When rightly held together I find that care, pride, enjoyment, and beauty begin to inform the way that I work.

Make it work, make it right, make it beautiful

Due to the influence of those I was working with, I learned early on that Making it work is only the first step of an engineering process. After it works it is just as important to Make it right. In those early days, I remember spending whole afternoons finding the right way to implement something long after we had it working. This was likely when I first caught a whiff of Craft. The programmer who is finished when it works has stopped short. There are limitless ways to implement something but some implementations are objectively worse than others once you start thinking about scaling, maintainability, and (gasp!) beauty. When I started caring about being able to do the work rather than progressing my career — when I started finding systems, code, and my tools beautiful — that’s when Craft emerged. Now admittedly, my sense of aesthetics related to being a programmer is somewhat intense compared to that of others. I remember a season when I was almost exclusively writing CoffeeScript in Vim — unicode that rolled off my fingers into my modal editor was so terse and expressive that it seemed to be valuable in and of itself. More than once I felt more like a poet than a programmer in that season.

Craft is about more than aesthetics though. When it takes hold you find a strange commitment within yourself to the work. The concept of The Long Road characterizes this well:

People aspiring to become masters of software craftsmanship need to plan for the long term. This long (yet bright) journey will bring you a rich set of abilities. You will become skilled at learning, problem-solving, and developing strong relationships with your customers. You will come to wield knowledge and technology as the samurai uses his short and long swords. You will come to comprehend and appreciate the deeper truths of software development. But all this will take time.

You should be prepared for the length of this journey… you should keep in mind the expectation that you will be a working software developer even when you are middle-aged. Let this influence the jobs you take and the scope of your ambitions. If you’re still going to be working in 20 years, then you can do anything. No one is so far ahead that you can’t match their skill level given the decades you will have to hone your craft. No business or technical domain is closed to you… rather than counting the days to retirement, the craftsman will willingly and joyfully work into her final decades.²

Creating

From a philosophical point of view, I believe “there’s nothing new, only fresh ways of combining old stuff.”³ As creatures ourselves everything we can do is derivative of the creation itself in some way that is impossible to disentangle. When Making Software though, there is sometimes an undeniable sensation that not only is something being created or born in front of our eyes, but that we are a part of that process. Whether derivative or not, that sensation can be exhilarating.

At its best, this amounts to a sense of mystery and wonder while one does the work — watching something be created from (seemingly) nothing all while not knowing how the experience will unfold. In what some considered a stuffy office job — surrounded by geeks, ugly office decor, and sub-par coffee — there is to be found creativity, joy, hope, expectation, uncertainty, and delight. And what makes Making Software the best is that when wielded carefully, these delights can be accessed almost daily. Early on I was struck by an excerpt from Frederick Brooks (emphasis mine) that captures what it can be like:

As the child delights in his mud pie, so the adult enjoys building things, especially things of his own design. I think this delight must be an image of God’s delight in making things, a delight shown in the distinctiveness of each leaf and each snowflake…

The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures… Yet the program construct, unlike the poet’s words, is real in the sense that it moves and works, producing visible outputs separately from the construct itself. It prints results, draws pictures, produces sounds, moves arms. The magic of myth and legend has come true in our time. One types the correct incantation on a keyboard, and a display screen comes to life, showing things that never were nor could be.⁴

Fred Brooks, The Mythical Man-Month: Essays on Software Engineering

In the early months of my first gig, I watched as our team discovered what we should build and how we should build it as we were building it. My mentor showed me both in microcosm and macrocosm that one often has to make something to figure out what should be made. This discovery by way of making was fractal — that is, it appeared at every granularity of the project. Out of thin air, I watched the castle be designed and built simultaneously and I was one of the wizards doing it, all while surrounded by post-it notes, aging Herman-Miller chairs, and bad coffee (have I mentioned the coffee was really bad?). Working like a wizard while surrounded by people who thought it was just as much fun as I did? Surely, I had stumbled upon some sort of secret life cheat code that enabled work to be fun and delightful.

Craving

Outside of Identity, The Nature of the Work and Creating, there is at least one more category of experience that is worth talking about. When one is engrossed in the work, there is another plane of delight that intersects with these other categories. This final experience is very circular. It’s something akin to a machine whose purpose is to create fuel for itself. It’s a regenerative motivation that creates… motivation. As we create we are enmeshed in rhythms of tension and release. We create tension when we write some code not knowing if it will do what we want it to. The release comes when it works. Sometimes we stack tension upon tension to get a hit of combined releases. As GeePawHill explains,⁵ humans exploit this sort of rhythm in music, TV show plotlines, and more.

From my earliest moments as a software developer, I remember experiencing these concentric cycles of tension and release. Take for instance the one actual programming class that I had in school, Java Software Development for Business Majors. Even in taking the class, there was an immediate onset of tension. Externally, there were the swirling rumors that this was a class that many before had failed. The professor was teaching remotely that semester and would mostly just give assignments on a glorified piece of forum software. I was but one of 30 or so non-programmer guppies taking the class; surely I wouldn’t get much help from my classmates. Internally, I had a lifelong interest in computers and the internet, but never felt as though I was smart enough to be a programmer. In light of all this, would I be able to teach myself to program to avoid failing the course, my schooling, and surely life? The tension was palpable. Just as palpable as the tension though, were those moments of resolution — getting a bit of code to compile and consequently not feeling like failure in the moments before the flash of joy subsided. I often wondered if programming worked like an addiction for me. Each cycle of tension and resolution only led to a greater anticipation of the next.

I learned that going through the gauntlet of getting something to work involved a loop of tension and release that seemed to engender the desire for another, perhaps larger loop. Making Software was a chase of some sort filled with miniature instances of these tension and release loops. I find that when I can get properly plugged into a task or solving a problem, Space-Time itself melts away as I focus on reaching a solution. As I write code, loops of tension are opened and it’s my job to close them. When I do get there, it all just makes me want to do it again. Shepherding these cycles is likely very important for sustained motivation and productivity but it’s also just really enjoyable. Some of the moments that remain most fond to me are when the time available in an afternoon seemingly evaporates as I enter flow — letting go of almost all my other cares — to load up the problem and the system into mental RAM and cravingly chase the solution. In all my feeble encounters with the humanities, one that has stuck with me is T. S. Eliot’s reflections on Time in Burnt Norton:

Time past and time future
Allow but a little consciousness.
To be conscious is not to be in time⁶

The craving pursuit of Making it work is often a journey into the act of creating so deep that one becomes conscious and therefore exits time, for a time.

Hamming, Richard. The Art of Doing Science and Engineering: Learning to Learn. Stripe Press, p. 19 ↩
Hoover, Dave and Oshineye, Adewale. Apprenticeship Patterns: Guidance for the Aspiring Software Craftsman. O’Reilly, 2010. p. 39 ↩
“I think a lot of liberal arts education (and not just in music) gets it wrong by telling to the students to “be themselves.” There’s nothing new, only fresh ways of combining old stuff. If you are in an academic environment, students should just go to the library a lot” Ethan Iverson, Twitter, 3 December, 2022 Link ↩
Brooks, Frederick P., Jr., The Mythical Man-Month: Essays on Software Engineering. Addison-Wesley, 1982. ↩
See RAMPS - Rhythm is Tension and Release. Rhythm is one of the five “drives” that he identifies as essential for motivation. ↩
Eliot, T. S. Four Quartets. New York, Harcourt, Brace and Co, 1943. ↩

Heroes and Housekeeping

Thu, 20 Feb 2020 16:00:00 GMT

I came across an interesting article over at the Doist blog. It seems their front-end team highly values Deep Work, but was having trouble balancing this with production support demands that would arise. They arrived at a solution that looks a lot like my One-Hand Approach to Production Support where each month, a “hero” is set aside from the team to focus on handling urgent issues. Here’s a bit from the article:

Each month one person on each product team becomes the “Hero”. Their main responsibilities are to communicate with the support team, triage bugs, file them in our bug tracker, and fix the most urgent ones. On top of those tasks, the Hero takes care of smaller platform parity improvements, to ensure that our various apps don’t diverge too much, as well as refactorings, to improve the general health of our code base.

That’s a lot of work for one person. It’s a very reactive, communication-heavy role that can get stressful, particularly after a new feature is released. That’s why we rotate the role on a monthly basis.

While in my experience one week was plenty of time playing the hero, the general purpose of the role is similar to what I have seen work. One person is sacrificed to deal with distractions for a time in order to protect the productivity of the rest of the team.

Flatten the Cost-of-Change Curve, you must.

Thu, 30 Jan 2020 16:00:00 GMT

In software development, there is a key success factor that often goes unnamed. It lies below many debates that software teams have and is at the root of many failure and success stories. Kent Beck aptly described this factor in the early chapters of the first edition of his Extreme Programming Explained book. In my experience developers and managers are unlikely to have read this book so I’m here doing my best to accurately convey them. This thinking is not original to me.

First, let me attempt to express the essence of the idea:

A software system’s cost of change is inversely proportional to its potential ongoing economic advantage.

In argument form, it might look like this:

Software is a highly malleable medium
Software’s ability to change is a primary competitive advantage
Therefore, consistently low cost of change is essential to maximizing its intrinsic benefits

Let’s take a look at each point of the argument.

Software is a highly malleable medium

Software has induced a revolution into our world that is on the order of the printing press. We now interface with all the domains of our lives with the help of software. Nearly everything we create and consume is aided or mediated by software.

Why?

Software itself is highly malleable which allows its output to likewise be very pliable.

There are many angles to this (sociological, economical, cultural, etc…) but it seems to me that a key factor is how easy it can be to change software and thus modify its output. Costs associated with changing software (including time) differ from other media by orders of magnitude.

Software’s ability to change is a primary competitive advantage

Software has taken the world by storm precisely because it can be so malleable at such a low cost.

Double-entry accounting done on paper includes an enormous amount of work if something changes once calculations are made. With software, the change is easy and thus the recalculations are automatic.

Yes, submitting information digitally rather than physically reduces errors (by validating input) and slashes delivery costs, but at the end of the day the value of a digital submission process is in how easy it is to change. Adding a new validation rule, requiring a different type of information or adding a review process is relatively inexpensive to implement once a system is up and running.

Publishing is another industry revolutionized by software. Just think about the difference between publishing updates to an ebook compared to printing the second edition of a physical book.

It is economically “better” to employ software for a given task precisely because it can be so easy to modify and update.

Therefore, consistently low cost of change is essential to maximizing its intrinsic benefits

So, how is this discussion relevant to an individual software developer or a software team?

To achieve these economic advantages, software must remain inexpensive to change.

We all know that possibly the most common problem to crop up in software teams is that as a codebase or system ages and gains more functionality it becomes very difficult to change. If code isn’t clean and readable and if relevant tasks aren’t documented and automated, making small changes to the system can take inordinate amounts of time and effort. This can be magnified by having a multiplicity of contributors who may come and go, each knowing a part of the system better than the whole. As functions get longer, if statements are added, comments become stale, and files get huge, a system loses the economic benefits that made it profitable in the first place.

In light of this, software teams must focus on keeping their products’ cost of change consistently low.

There is an ever growing pool of tools and methods to achieve this. Test-driven development, automated checks, automatic code styling, lint rules, CI jobs, keeping your dependencies up to date, and code review processes are among some of the most prominent.

If a software team fails to keep the cost of change low (“not enough time!”, “I can do that later”, “We don’t really need that yet”) they are really just biding their time until the next expensive rewrite.

If a team cannot exercise discipline to manage the cost of change, as a wise man once said, “then the Emperor has already won”.

Healthy Codebase Habit: Regular Dependency Updates

Fri, 13 Dec 2019 16:00:00 GMT

Do you have a habit of updating your team’s dependencies? Congratulations, you’ve got fresh dependencies.

Do you lack a habit of upgrading your projects dependencies? Oh man, you’re dependencies are probably stale.

I’ve learned that it’s better in almost every way to regularly update your dependencies. Read on for more.

Stale Dependencies

Stale dependencies lead to suffering.

This is your team on stale dependencies

Everyone can guess what happens when you don’t keep your dependencies up to date on a project. Dependencies get out of date, then become harder and harder to upgrade. This is true, but there are actually deeper, darker problems to projects with stale dependencies.

The first couple of projects that I worked on as a software developer did not value regular dependency updates. They were java projects in a large-ish IT organization. The progression went something like this:

First, an approved version of a large framework was picked. This was usually at least the n - 1 version. From here the project began and dependencies would begin to accumulate. There was very little discussion over the fact that as time passed the project dependencies would become more and more out date, but this was the case.

Next, a developer faced a problem and wished to solve it. A developer faces a problem and realizes that one of the project’s dependencies could easily solve the problem. They go to upgrade a dependency to use it’s new feature and spend a few days in dependency resolution sheol until they give up.

Then, the team accumulates an identity that is disconnected from its technology community. The team is stuck on old technology so why read about what is up and coming? Why contribute back to the open source community? Why spend time thinking about ways we can improve and new patterns we can adopt?

Finally, much toil is had and the cycle repeats. Silly workarounds solve problems rather than new patterns available latest dependencies. One broken window turns into 50. Eventually a new project needs to be funded to bring the app into a usable state again.

All in all, stale dependencies cut a team off from its community and the valuable feedback loops contained therein. They incur large costs down the line and make daily life on a software development team less enjoyable.

Some problems with stale dependencies

Major frameworks lock you into a specific version
Difficulties resolving dependencies that have desired features
Team becomes a disconnected user of the dependencies they use
- unable to contribute back to the community
- unable to take advantage
Cost of modifying product becomes so high, a new project must be funded to upgrade the app once it becomes a security concern or liability
All of these negative side effects increase cost in sundry ways

Fresh dependencies

Fresh dependencies breed new feedback loops.

This is your team on fresh dependencies

If you haven’t worked on a project that prioritized fresh dependencies, you may not be aware of the advantages that spring up. I myself was surprised.

After spending a while on Java projects with stale dependencies, I joined a team in Javascriptland. In Javascriptland, ‘new’ things were happening so often that the team’s evergreen app would become unworkable from an engineering point of view if they didn’t regularly keep dependencies fresh. When I first joined the team I was surprised at how much the codebase was constantly improve. I eventually identified fresh dependencies as a core practice that enabled these feedback loops.

Dependency updates signal the misuse of frameworks, libraries, and APIs as breaking changes were encountered. Dependency maintainers regularly find ways that the APIs of their projects are miused or underused. To counter this, they make deprecations and modify the way the dependency should be consumed. Watching these changes happen makes a team aware of things maintainers are doing to improve their product. Often, seeing this happen is infectious and spurs the improvement of internal APIs or even the overall adoption of say functional programming principles over the imperative. Immutability can be contagious.

Constantly improving lint rule sets help developers improve their codebases. If you work in Javascriptland you should definitely be taking advantage of establish lint rulesets like the famous Airbnb repo. Upgrading these rulesets along with your dependencies will often introduce new rules that break your build. Taking time to explore why these rules were introduced and how you can follow them will lead you down new avenues of improving your codebase and programming practices. As the community learns, you learn.

Updating your language/runtime helps you to take advantage of new language features. Typescript is a great example of this. With typescript, if your regularly update to the latest stable version and read the blog post they produce explaining new features, you are incrementally learning about what will become part of the javascript spec in the next 12 months. Your codebase is able to take advantage of these features right away and you are using them as soon as they come out.

Proactively upgrading dependencies preemptively removes potential blockers. Rather than finding out you can’t use some feature that you need or that you will have to migrate a bunch of old code to maintain platform support, you will be ahead of the curve on the platforms, frameworks and libs that you use. No more worrying about running out of time to upgrade before something breaks in production.

Some advantages to fresh dependencies.

Dependency updates signal the misuse of frameworks, libraries, and APIs as breaking changes were encountered
Constantly improving lint rule sets help developers improve their codebases
Updating your language/runtime helps you to take advantage of new language features
Proactively upgrading dependencies preemptively removes potential blockers

In Sum

Keep your team, project, and personal development healthy by updating your codebase’s dependencies at a regular interval. Like our health or our relationships, doing good things often yields unpredictable benefits, while neglecting them will create emergency situations and make you feel like a lame human being.

Please share in the comments: advantages you have seen from keeping dependencies up to date, or stale dependency horror stories you’ve lived through.

'Who You Are' is 'What You Do'

Sat, 27 Jul 2019 16:00:00 GMT

In the the time that I’ve spent developing software so far in my life, I’ve seen a tendency for folks to create a false dichotomy – a dualism – between accomplishing goals (“pragmatism”) and desiring to accomplish them in the theoretical ‘best way’ (“idealism”). Alas, logic abhors a dualism and it’s my experience that as soon as you begin creating such a dichotomy, you’ve fallen into a trap. In fact, I’m about to tell you that who you are (the way you do things and your motivation) is inextricably connected to the things you produce (goals you achieve).

Let’s look at this first through the lens of organizations and the products they produce. I’ve often seen that when a cultural problem exists in a company (selling things before they exist, not allowing employees to balance their work and life appropriately, favoring results above all else, etc) there is a desire to deliver and fix the cultural problems later. What many do not realize though is that the products produced by a culture are born in that context and bear the mark of their origin. In the same way that Conway taught us that the shape of an organization impacts the architecture of a system, we can learn that an organization’s motives shape its methods, and those methods produce a product that resembles its maker.

Step back for a moment and consider this at the personal level. We often are driven to set goals for ourselves with the hope that once we accomplish said goal, we will have proof that we have become the person that we want to be (a marathon runner, someone with impecible hygiene, etc). James Clear, in his recent book Atomic Habits, goes to great lengths to show that it is the journey towards a goal that shapes us, not the completion of a goal. One golden quote from the book is:

Every action you take is a vote for the type of person you wish to become.

The way you do things shapes who are you, and your identity – your heart – is what drives your actions.

Now consider this from the software development point of view. Those who buy into the false dichotomy between ideals and results will say things like “we don’t need TDD, we need to ship software” or more perversely “Let’s wait to start using TDD until we get this first release out.” (You can exchange “TDD” in those phrases for other methods such as Continuous Delivery or Evolutionary Database Design) Those who understand that picking only ideals or results is a false choice will realize that “we can only deliver our first release and respond to change if our software can be safely modified”. In fact, favoring results over ideals is itself an ideal. There is no escaping the fact that your value system breeds a culture which births your products. Methods will always come out of ideals or vision, it’s just a matter of what ideal you choose to embrace.

Once an individual developer, team, or organization understands this, there is a freedom that will arise. They will realize that they can take time to think about how they do things, in fact, they must think about the how. There are times when it is profitable to slow down and reason about something out of your organization vision rather than a desire to achieve a goal by a certain date.

Don’t agree? Let me know in the comments.

The Software Developer's Path to Wisdom

Thu, 09 May 2019 16:00:00 GMT

Have you ever worked with someone who was seemed able to know the right thing to do in almost all situations? This person wasn’t only very good at solving technical problems but knew how to handle people, systems, and situations too. This was a person that possesed wisdom. In this blog post I will set out to define wisdom and then present the way that I think about how someone becomes wise in a given domain.

Wisdom?

What is wisdom? There are many alternative defitions that you will encounter. First, let’s begin with what it isn’t:

What wisdom is not:

Experience
Old age
Intellectual prowess
Common sense

Any of the above factors may correlate to a wise person, but wisdom itself is something more. My definition:

Wisdom is the ability to understand reality and act accordingly.

Knowledge, experience, and emotional intelligence are all crucial to being able to apply wisdom, but on the whole wisdom comes across as that someone develops over time.

The Model (Brinker’s Model of Ideological Adoption?)

As I have thought about the acquisition of wisdom in my life and reflected on various environments in which I have sought it, I began to see a pattern. I noticed that whenever someone is presented with something new (an idea, tool, method, or system) there are stages of adoption which the person undergoes as they try that new thing out. For the sake of example, we’ll just call the thing in question “the new thing”. Here is what I’m calling the stages for now:

Ignorance
Awareness
Zeal
Wisdom

Ignorance

The first stage, ignorance, is when you are blissfully unaware of the new thing. You haven’t heard of it. You may have experienced the problem that the new things solves but that’s about it.

Awareness

The second stage, awareness, is when you become aware of the new thing, but for whatever reason don’t attempt to apply it. Maybe you think it won’t be useful, or maybe you think it will be useful but that you don’t have time to try it out. Finally, you may be aware of the new thing but don’t fully understand it.

Zeal

The third stage, zeal, is a dangerous one. It could fairly be called the ‘cage stage’ because you should be locked in a cage until you are through it. In this stage you understand the new thing to a degree and seek to apply it to every thing and every one, often yielding unfortunate results. You haven’t yet learned when the new thing isn’t applicable, so you are an evangelist for it, blind to any situation where it isn’t useful and self-wise regarding its application. You know you are in this stage when you annoy friends/family/colleagues by blathering on and on about the glory of the new thing.

Wisdom

The fourth stage, wisdom, is the stage you want to be in. In this stage, the shininess has worn off a bit, and you are on a journey of knowing when not to use the new thing. This stage asymptotically approaches perfection. You know you are getting here when you can find some aspect of the new thing that you disagree with or could be a problem in some contexts. After the cage stage, this is where you become sane and tolerable regarding the new thing.

Wisdom in Software Development

I have seen myself move through these stages at a macro level with regards to software development (and many other things) and also at a micro-level with certain tools, methods, and technologies. For a case study, let’s take a look at the way I adopted Test-Driven Development.

A Case Study: Test-Driven Development

Ignorance

In college I had a programming class or two. No one ever mentioned TDD. I had no idea what it was. I did notice that the one programming project I had seemed to break often as I was changing it.

Awareness

In my first programming job I joined a team that lived and taught TDD. As soon as I began pairing with my team members I was introduced to the concept. It seemed like it might be helpful, but writing a program that tests the program which I’m writing felt very odd to think about. I always began coding a new feature and then heard one of my coworkers say “did you write a test for that?”. As I learned how to write tests and was introduced to mocks, stubs, and spies, I became proficient but still didn’t consistently write tests first.

Zeal

On my first software project as a full-fledged junior, I engaged in TDD with much zeal.

Was I about to define a class? Write a test first.
Extracting something into a private method? Write a test first.

Not only did I behave this way myself but I demanded that the team maintain a near 100% coverage rate.

The result?

The tests were brittle. The slightest refactor caused a lot of rework in the tests. We went to great lengths to test our getters and setters. Some teammates found me annoying.

Wisdom

I eventually realized that some of my tests provided no value. Testing getters and setters didn’t buy us much. And writing a test before defining a class? That was the essence of an overzealous junior developer. I scaled back a bit and focused on writing tests that the project needed. Eventually I was even able to grok the difference between London and Detroit style TDD and understand where I fit in the scheme of things. At this point people could bear to hear me talk about TDD again. I was even able to mentor and help a few people figure it out along the way. I’m not a zen master by any means, but the benefits that TDD provides as a tool are clear to me and I also understand when it isn’t that helpful. Now I remain on the long road of wisdom.

In Sum

Hopefully you see some concordance with reality in this model. It’s been a helpful tool for me to understand people and team dynamics in some cases.

Was this helpful in any way? Did I miss a step? Do you have a better name for this thing? Let me know in the comments.

A Software Engineering Manifesto

Sat, 13 Apr 2019 16:00:00 GMT

Disclaimer: “Best practices” are less like objective truths and more like pragmatic successes in the software engineering field. This is not to say that the correct way or even best way of doing something doesn’t exist, but rather given our finiteness and the external forces of the worlds we inhabit, one should expect some variability here. While I state these in the indicative sense (“X shall be Y”) at the end of the day, the best way to write and ship software will differ from context to context and will (hopefully!) look different in coming years.

Deployments Shall Be

Automated
Scheduled
Repeatable
Often
While the system is in use

Databases Shall Be

Migrated automatically during deployments
Locally accessible for developers
Optimized just in time

Code Shall Be

Version controlled
Style-enforced at build time by automated tooling
As simple as could possibly work
Nearly comment free
Deleted if unused
Test-driven
Treated as a liability
Paired on as needed
Deleted with joy

Project Management Shall Be

Pull-based
Optimized for end-results, not implementation details

Developers Shall be

Courageous to simplify the system in almost all circumstances
Teliocentric in their efforts
Mildly over-affectionate towards their text editors
Allowed to work uninterrupted for generous amounts of time

System Design Shall Be

Evolving as necessary with every feature
Changing as the team’s understanding of the problem domain changes
Ultimately enforced at code review time
Consistent whenever possible
Delegated to widely-used open source implementations when possible

Did I miss something? Was I completely wrong in some way? Was there something I got right? Let me know in the comments.

An Atlas for Learning React

Fri, 29 Mar 2019 16:00:00 GMT

Learning something like React takes time. For every excellent react resource that exists on the internet, there are several poorly written tutorials showing you how to make a todo list app. This ‘atlas’ serves as a guide for myself and others. It will probably be updated over time. Information found in the glossary of terms is largely indebted to the various resources mentioned here.

Conceptual Overviews

The official React guide to main concepts
React as a UI Runtime (Learning react from first principles)

Tools

Create React App, A CLI and set of sensible defaults for react projects
React devtools, Chrome and Firefox extension fo view react stuff along with dev tools

Ecosystem

Next.js, Server-side rendering
Redux

Tutorials

People

Jordan Walke, creator of React
Dan Abramov, creator of Redux and Create React App

Glossary of Terms

Elements

React elements are plain javascript objects that represent something that react will render in the DOM.

Treated as immutable by react
Can reference valid DOM tags or user-created components

Components

React components functions or instances of classes that receive props and return elements.

Can be as simple as a js function that receives an object and returns and element
Can also be a class that extends React.component
Components start with an uppercase char
Must not mutate props

State

State is essentially a property on component classes that is accessible at this.state.

Only mutate state using setState
setState is asynchronous
setState merges properties shallowly
state is local to a given component and uses a top-down data flow

Mounting

Mouting is when an element is rendered to the DOM for the first time
Unmounting is when an element is removed from the DOM

Lifecycle Methods

componentDidMount
componentWillUnmount

Did you come across some excellent resources when learning React? Did you have a turning point where react suddenly ‘made sense’ to you? Let me know in the comments.

Linux, Entropy, Random Number Generation, and you

Sat, 09 Mar 2019 16:00:00 GMT

Recently while debugging a performance issue with a JRuby application, I learned that the JVM defaults to /dev/random as its randomness source. This default resulted in a performance problem in the application. This post will cover the purpose of /dev/random and /dev/urandom in general, the cause of the performance problem that I experienced, and the solution that I landed on.

Why Random Number Generation (RNG) is important in computing

In most unix-like operating systems, the files located at the path /dev/random and /dev/urandom server as random number generators. True randomness is a concept that is very important in computing, especially with regards to cryptography. Cryptography often depends on a key or set of keys that are secret. If an actor was able to identify a pattern used to generate secret keys, they could likely find a way to guess what a secret key is. Inserting randomness into the process of generating a secret key makes it harder for an actor to find a pattern in key generation.

Think for a moment about how you might write a program that would generate random values. Perhaps the program could do some math on a seed value that results in integers produced that don’t appear to follow any particular pattern. But how does one select a seed? If there is a pattern in the seed used to generate random values then you haven’t accomplished much. One might resort to using a date or timestamp and then performing math to generate a seed but in that case the seed could be guessed easily enough.

A source of true randomness would be quite valuable and quite difficult to construe.

Linux and RNG

To accomplish sufficiently random number generation, the linux operating system keeps an ‘entropy pool’ which is a collection of random values that one hopes to be unknown by some malevolent actor. The entropy pool then acts as a seed to a process that tries to ‘randomly’ generate numbers. This results in a stream of random output that is available at /dev/random on the filesystem. The entropy pool is populated from a variety of sources like user input activity and other environmental noise that device drivers may observe.

Things get a little complicated when the entropy pool runs low.

When entropy runs low, /dev/random stops or blocks. This causes any process requiring data from the random stream to wait until the entropy pool is full enough to continue generating random numbers. There is another randomness stream on most linux system at /dev/urandom that does not block when the entropy pool runs low. Choosing not to block when the entropy pool runs low can be considered ‘less secure’ but one must think about the situations that can cause this to happen and judge accordingly whether or not /dev/urandom is safe to use.

The JVM and /dev/random

By default, many versions of the JVM will default to using /dev/random. Depending on the size of your entropy pool and your application’s need for random numbers, this can put you in a situation where you are blocked until the entropy pool reaches its required size.

I noticed the performance problem when I modified a JRuby application to use stronger ciphers for TLS. Presumably the strong ciphers required more random input at times and the entropy pool on my servers wasn’t always up to the task. After doing some research, (I found this article to be helpful) I decided to use /dev/urandom/ for the needs of our application.

Addendum: Useful commands and config

Here are a few helpful commands and settings related to entropy pools in linux and configuring the JVM accordingly. The commands assume RHEL or another system like it.

Available entropy: cat /proc/sys/kernel/random/entropy_avail
Does a CPU have a hardware RNG? cat /proc/cpuinfo | grep -i rdrand | echo $?
JVM flag for default randomness source: -Djava.security.egd=file:/dev/urandom

Have you dealt with problems related to /dev/random’s blocking behavior? Did I get something wrong here? Let me know in the comments.

Refactoring Doesn't Require Domain Knowledge

Sun, 24 Feb 2019 16:00:00 GMT

I came across this great GeePaw Hill thread on twitter about refactoring and domain knowledge. The final tweet in the thread sums it up well:

When I’m refactoring, I do my best work when I am entirely domain-blind, and am working with and thinking of all and only what the code that is actually there actually does. Prioritize reading code over studying the application domain.

Have a pleasantly odd Sunday!

Over the years I have learned to trust the fact that the code never lies. In addition to possibly saying some interesting things about epistemology, this aphorism has proven trustful to me time and time again. While devs (and others) ~~argue~~ discuss what some code does, how it’s working, etc… in almost every case the wise thing to do is say very little and begin reading the code.

There are lessons in this thread about the value of readable code as well. In the thread, there seems to be a correlation between the refactoring effort (making the code easier to understand) and the discovery of the performance problem. The easier code is to read, the easier it is to reason about, which enables developers to make improvements to it, or at very least decreases their chance of screwing it up as they change it. An explanation of the value of clean (read: readable) code is worthy of its own post.

Alas! The whole thread is a great read. Read it unrolled on ThreadReader.

IDEs are dead. Long Live Development Environments. (Part 3)

Sun, 10 Feb 2019 16:00:00 GMT

I’ve already written about why you should step away from your IDE and how I’ve benefited from doing so. Now let’s take a look at how this can be accomplished. As always these steps are based on my experience as a finite human being and thus your mileage may vary. The five steps are:

1. Begin using a single tool directly, outside of your IDE.

The first step is begin interacting directly with one of the tools that your IDE abstracts. Choose a tool, and make it a goal to only use it directly (rather than from inside of your IDE) from now on. Use the interface that that the tool defaults to. For example, if you choose to use Git directly, use the command line interface rather than another third party client. If nothing else you will gain the advantage of learning a particular tools in more depth. Some ideas for which tool to pick:

Dependency Management
Build System
Version Control
Test Execution / Coverage
Code Quality / Linting Tools
Application Servers

2. Use all tools directly. Eventually only rely on the text editor in your IDE.

Repeat step one until you can use all or most of your development tools directly rather than through your IDE. The goal here is for the IDE to basically just become a text editor. At this point your development experience might feel slower than it was before, but what you lose in daily productivity will be rewarded in the long run with a better knowledge and command of your tools.

This isn’t the place that you want to be long term, so work to get a good understanding of your tools quickly so that you can move onto steps 3 and 4.

3. Switch text editors.

Once you reach step two, you will most likely realize that your IDE is not a great text editor. It’s big, slow, and is most likely lacking advanced features that specialized text editors have. Now that you only really have your IDE because you need a text editor, it’s time to find a your text editor. Finding the right text editor could be a blog series of its own, but here are a few things to think about.

Finding your text editor is what some will call a journey. Ideally you want something that has:

cross platform support
large online community
customization capabilities
staying power

Text editors that have these qualities will like have a larger learning curve, but that’s ok! You spend hours of your life editing text every day. Why not dig in and get good at it? I’m not the first one to say this:

Use a Single Editor Well

The editor should be an extension of your hand; make sure your editor is configurable, extensible, and programmable.

(From The Pragmatic Programmer)

4. Slowly bring IDE functionality into your text editor.

This step always in progress for most developers. Once you are on the path to mastering your text editor, you will find it beneficial to begin adding little shortcuts to your editor to increase your productivity. For example, rather than switching to new terminal window to run your unit tests, write a macro (or something equivalent for your editor… perhaps a plugin?) that runs all tests in the visible file based upon a keystroke of your choice. Some ideas to enhance productivity in your editor:

A fuzzy finder for file contents or names
Ability to see the VCS history for a given file
A shortcut to automatically format markup (chunks of XML, JSON, etc…)

At this point I anticipate some objections. Why leave your IDE if the goal is just to rebuild it eventually? There are two main reasons why bringing functionality back into your text editor is not just an insane re-doing of what you already had with your IDE.

First, recognize that by this point you have gained an enhanced knowledge of all the tools in your toolkit and become a text editing wizard. Any shortcut you add back is added from a position of knowledge (of your tools) rather than dependence (on your IDE).

Second, with this approach you only bring what modifications you need. You will end up with something much lighter than an IDE overall.

5. Build your environment.

Now that you are living and working with a set of development tools that you have glued together through your expert knowledge and experience, you can take it one step farther by noticing patterns in your daily development activities and automating those. For example, if you find yourself opening the same three terminal emulator tabs (and starting various tools in them) write a script that does that!

Terminal Emulator: Customized Windows, Tabs, and Panes
A Tiling Window Manager: i3, xmonad, kwm, Amethyst
Window Manager with the Terminal: Tmux, Tmuxinator, Screen
Make your environment portable: Store your configurations on GitHub or create a docker image that houses all of your tools and their dependencies

This is how I broke my dependence on an IDE. Have you? Let’s hear how you did it in the comments.

All Web Abstractions Leak

Sat, 02 Feb 2019 16:00:00 GMT

Gary Bernhardt had a great thread on twitter recently.

There’s no such thing as abstraction in web tools and never has been. Every framework claims that it will hide the complexity from you, but it always leaks through. You end up having to understand the framework, its dependencies, and the web primitives they’re built on.

When I first joined a software development team, I remember having my mind blown as someone on the team walked me through the layers that made up the application. It was a SpringMVC web app that consisted of a view described the user interface, a model that held the data to be plugged into the view, and a controller that tied the model and view together and served requests. Hibernate was also involved in retrieving the data that populated the model. It was like staring under the hood of a car for the first time, seeing how carefully chosen components worked together to create this thing called a ‘web app’. All of those layers! Not to mentioned future joys I would experience like debugging an ORM, tuning a database, becoming fluent in the unspeakable incomprehensibilites of WebSphere, learning languages that compile into other languages, and all of the other things I’ve buried in my subconscious. Early on though it was fascinating to see those layers and exciting to think I was going to be able to contribute to the creation of an application that someone would be using in real life.

Now days though, when I consider all of the things someone might have to learn in order to be a productive web developer, it really is daunting. The markup languages, stylesheets, a javascript framework, a language that compiled to JS, a build tool and dependency management, all just for the client side of the application. Imagining a world where the web (good luck well-defining that btw) is as abstracted to a developer and my operating system is to me… well it seems to good to be true.

The thread as some great examples and comments from others. Read it unrolled on ThreadReader.

The One-Hand Approach to Production Support

Sat, 26 Jan 2019 16:00:00 GMT

Across the industry, as the teams trend away from being project-focused toward being product-focused, an increasing number of software developers find themselves being responsible for ‘support duties’. These responsibilities may include answering user questions, attempting to recreate reported bugs, and trying to recover from a system outage in production. There are reasons why it is good (this is the spirit of DevOps in my opinion) that developers will increasingly participate in these duties, but for teams that are taking these responsibilities on for the first time, confusion abounds.

Assumptions

Production incidents will happen
It is in the best interest of the team and product for developers to become skilled at handing production incidents
We should aspire to Antifragility in our systems, our teams, and as individuals.

The All-Hands by Default Approach

To my surprise I have seen managers and developers who prefer an unorganized, all-hands approach to production support. Typically, a scenario will play out like this:

Production incident is reported to some sort of channel that is ‘public’ to all developers
Since there is not a dedicated person to handle all issues for this given time frame, a manager type will soon be contacting developers to find out who is working on the issue.
Meanwhile, a developer notices the problem and immediately notifies his/her peers
Physical or virtual huddle happens and ‘all hands’ are on deck for fixing the issue
Typically developers that have more experience with the system (or perhaps have more mature skill sets) lead the troubleshooting until a solution is found

I suppose developers and managers might prefer this approach for the following reasons:

Having a huddle creates an atmosphere or appearance of productivity and progress
When all developers are in the huddle, each can suggest solutions or hypotheses that are unique to their skillset and experience (seemingly shortening the time that it takes to solve the problem)
From a developer’s point of view, this approach may provide more psychological safety since they aren’t the only one working on an issue

Problems of the All-Hands Approach

There are hidden costs to the all-hands approach.

First, it may seem like having a support incident channel that is public to all developers is a good idea, but I think what most managers don’t realize is the amount of distraction that channels such as these create. If a developer is told that part of their job responsibility is to monitor logged incidents, their concentration is going to be almost permanently compromised to some degree. One extreme example I recently heard was a developer, who faced with the responsibilities of supporting an application, opened up the application logs every morning and had them visible on her screen for the entire day. If something went wrong with the application, she claimed to be able to detect a change in the shape of the text that was flowing on the screen. Imagine if this person’s manager understood that this person was compromising their concentration nearly 100% of the time to visually monitor application logs!

Less extreme situations still are quite a drain on a developer’s ability to focus. If you are not convinced of this I suggest taking a look at Deep Work by Cal Newport (decent summary here).

In addition to a near constant compromising of developer focus, consider the sheer cost of involving everyone on the team in each production incident. As I mentioned in point number five of the All Hands approach, it has often been my observation that despite all hands being present, a single developer usually ends up leading the charge at finding a solution and applying it. Given this, it is especially egregious to think that all of the members are pulled into a huddle for the incident when a subset of them will actually be contributing meaningfully to the solution.

The One-Hand (by Default) Approach

What if I could suggest an approach to handling production support that decreased distraction to the team and fostered improvement of skills and self-confidence for individual developers? That’s exactly what the One-Hand approach is. It goes something like this:

Notifications of production incidents are fed to a developer who is ‘on call’ for the given time period (let’s say they are on call for the current week). Developers who are not on call are expected to continue with their normal work.
The developer works with the user (and outside teams if necessary) to resolve the issue.
The on-call developer knows that as a case of last-resort, they can involve their mentor (a developer peer) or lead to get the issue resolved.

With such an approach you decrease developer distraction and increase individual problem solving skills over time.

There are some cultural presuppositions that need to be installed for this to work properly. Primarily, the developer on call needs to see a production incident as an individually assigned challenge — from which they can learn and be gratified for solving. Additionally, the developer on call needs to understand that they are providing value as they protect the team from distraction, meanwhile the team as a whole needs to cherish distraction-free focus time (aka “Deep Work”) as a means to applying their unique skillsets so that they can deliver value.

Finally, the decision to bring another developer into a huddle needs to be understood as something undesirable but potentially necessary in a given instance. The developer on-call should set out thinking of pulling someone else in as a last resort, but the rest of the team should not think of this as an ostrasizable offense. If the team is mature enough to work under these assumptions, the One-Hand approach yields the following benefits:

Protects the majority of the team from distraction
Removes the latent stress of the entire team watching a public channel, waiting for something bad to happen
Forces all team members to ‘level up’ the various skills needed for production support
Each developer who handles an incident will have increased confidence and know-how
Over time you decrease the amount of single-person dependencies the team has

This approach goes hand-in-hand with the idea of Antifragility. A team needs to be strengthened by various stressors in its environment. A system needs ‘adaptive fault tolerance’ (See the Antifragile Software Manifesto). An individual needs to be tested and strengthened to a degree in order to become better (interestingly, Jonathan Haidt even applies Antifragility to childhood development).

In Sum

Yes, there might be some situations where the All-Hands approach is needed. If these instances are not rare on your team, something else is probably gravely wrong.

Yes, the lead of the team or mentor will still be generally distracted as junior developers come to them with production incidents. This is the sad, lonely road walked by the developer who chooses to mentor or take the lead on a team.

Ultimately, the One-hand (by default) approach ends up delivering a more focused and capable team and a generally saner work environment.

Do you violently disagree me? Do you have a legendary story from that night you were on-call? Let me know in the comments.

IDEs are dead. Long Live Development Environments. (Part 2)

Sat, 12 Jan 2019 16:00:00 GMT

If you haven’t yet, you may want to refer to part 1 of this series where I present a reasoned argument against using an IDE. Now let me try to explain why it was so beneficial for me.

I attribute major positive impacts to my professional competency as a developer to pulling away from my IDE and creating a development environment for myself. Below, I attempt categorize the positive impacts into 4 major groups.

1. Increased competency with individual tools

I first noticed that by using some tools directly, I began to understand them better than I did when I was using them through the abstraction of my IDE. Interacting with a tool on its terms (the interface that it provides) forces you to learn the concepts inherent with the tool rather than the way they have been abstracted away by your IDE.

For example,

Git. Git’s CLI is well known as being confusing and inconsistent. Many choose to use Git through their IDE of with applications like Tig, SourceTree, Kraken, etc… While these tools can be beneficial in specific use cases (I find no shame in relying on SourceTree or BitBucket for complicated diffs) they generally obfuscate the underlying concepts of Git. Learning to use Git’s CLI has allowed me to extract more value from this notoriously complicated tool.
Dependency Management (NPM, Yarn, Bundler, Gradle) Using the command line interface for these tools rather than the menus in an IDE forced me to better understand the role that they play in the software development lifecycle. When I was a pure IDE user, I knew that in some cases I had to run a gradle refresh or gradle build but these were actually more like debugging steps when something went wrong in my local environment. After learning the tools individually and directly I better understood when I needed to use them.

2. Improved debugging skills

I next noticed that when a production issue arose I had become much more comfortable with jumping on servers and getting to the bottom of an issue. When I was purely an IDE user, the thought of understanding how an application was running on in our target environment seemed much more foreign than the comfort of my Eclipse’s GUI. Working primarily in a terminal and interacting with tools via their CLI led me to be much more comfortable with ops-related tasks. It also helped me come to understand the OS I was using more thoroughly.

3. Becoming a technical resource

After I noticed my increased ability with individual tools and improved debugging skills in my day to day software development activities, a third interesting benefit began to manifest — people began to come to me when they had problems with an individial tool. This may seem like an obvious progression as you read this blog post, but as it was happening to me, I was not planning to become ‘the Git guy’ when I started using Git directly via its CLI but nonetheless from time to time coworkers would come to me with a problem and much to my surprise I was often able to help them.

4. Adaptability

Finally, I noticed that when I switched development stacks, I was able to adapt easily to a new set of tools. Once I became somewhat ‘intimate’ with the tools in the first stack that I learned (Java based webapps) I then moved to a fullstack Javascript environment. I was a little nervous moving from Java to fullstack JS, but I found myself asking diagnostic questions based on tooling to get me through it. Rather than making a list of all the new tools I needed to learn to be a successful fullstack Javascript developer (and then researching them one by one), I was able to use the mental models I had of the tooling in my previous stack to gradually come to understand the new one. I would speak to myself like this:

Alright, I need to pull down the dependencies for my project. In the past I used Gradle for that. What do I have now? NPM. Great. Let’s see what the equivalent of gradle build is for NPM.

In Sum

In the end I was often pleasantly surprised to find that from stack to stack, smart people have developed tools that share similar roles and very often these tools even use similar terminology. (Moving from Yarn to Bundler for example required very little research)

So, are you sold on the benefits of ditching your IDE yet? Read my step-by-step guide to getting rid of your IDE. Otherwise, let me know in the comments if you have found similar benefits.

IDEs are dead. Long Live Development Environments. (Part 1)

Sat, 05 Jan 2019 16:00:00 GMT

Preface

Generally, I don’t think there are one-size-fits-all solutions in software development and engineering. I’m going to argue that it will be good for you to step away from your IDE because it was good for me to do so. If you totally disagree with me, that’s cool! Tell me your experience in the comments section.

The Sacrifice You Didn’t Know You Were Making

When it comes to being productive as a software developer there are many categories of personality that manifest regarding productivity. I speak from experience here, I’ve dabbled in nearly all of these. There are coffee addicts, nootropic pill-poppers, noise-cancelling headphone donners, binaural beat dweebs, flow-state seekers, text-editor worshippers, point-and-click mouse nuts, keyboard-only wingdings, the list goes on. I would like to take this moment to introduce what I consider a dangerous software developer personality, the IDE indentured servant.

The IDE indentured servant is the person who relies on their trusty IDE as a prized productivity tool but doesn’t realize that it’s actually holding them back. Whatever gains they may have gotten in short-term productivity (which I believe is a false productivity), they completely sacrifice when it comes to long-term productivity.

I’m here to tell you today that IDEs are a drain on your productivity.

Use an IDE if you want to stunt your software developer development.

Use an IDE if you want to be a lesser developer.

Do I have your attention? Cool.

Let’s go back to productivity for a second. Everyone knows a developer who’s measure of productivity is getting a feature deployed to a test environment so that they can start the next feature. These are the people who feel good about getting something ‘done’. These are also the people who usually forget to finish things in my experience. I have trouble with this myself actually. The gratification I get from ‘finishing’ something can lead me to prematurely move onto something else if I’m not careful. Let’s call this short term productivity.

Conversely we have long term productivity; finishing things that provide value and getting them right.

Slow is smooth and smooth is fast.

Measure twice and cut once.

There is truth to these aphorisms! (not least because Phil Dunphy champions the former). It’s the developer who desires long term productivity that needs to ditch her IDE.

Why the IDE hate?

Think about what an IDE actually does. The most charitable description would be that it ties together many tools and utilizes them to give instant feedback to the developer. This actually might be true, but there is one terrible side effect that most don’t consider: IDEs abstract the developer away from the very tools that they are using. This abstraction generates a distance between the developer and their tools. This distance may be helpful in the short term (imagine training wheels on a bike) but will only hurt in the long term (try riding your bike off road being dependent on training wheels).

Don’t believe me?

Consider this for a moment: becoming competently productive with a particular IDE is orthogonal to being competently productive in a given language technology stack.

Learning how to use an IDE doesn’t actually teach you how to use the underlying tools that it ‘integrates’. Please join me in a short though experiment.

The Junior Developer + Git

Take version control for example. Imagine a junior developer who joins a team and begins their first project, which happens to be written in Ruby. For the sake of the experiment, imagine that they know nothing about Git. They reach for the RubyMine IDE from JetBrains. RubyMine has a fair amount of VCS tooling off the shelf that helps you do things with Git. There is some overlap between the terminology used by RubyMine’s VCS features and the Git CLI, but it’s not 100%.

So the junior dev works on this project for 6 months. They’ve been using the RubyMine VCS tools to commit and push their changes. At the end of 6 months, how much have they learned about Git?

I think we all know the answer. They have learned very little about Git. They have probably memorized the menu items or keyboard shortcuts that are needed to commit and push to a branch, but they probably aren’t aware of the differences between the Git index, staged files, and committed files for example.

I’m not saying it’s logically necessary for a developer to know the internals of Git to be successful. Being a software developer is about providing value to the people that we are making the software for. You don’t have to be a Git ninja to provide value. However, There will be a point though when the developer who doesn’t know Git very well will likely lose some of their work, blow away a remote branch accidentally, or get very confused when someone asks them to “Cherry pick” a commit. In this moment who is the person that every one turns to? It’s the guy who always uses the Git CLI.

Some Personal Experience

My first project was in Java and we used CVS (shudder) for version control. I used the CVS tooling built into Eclipse every time I pushed, pulled, compared, etc… just like the rest of my team. There was one time where something went horribly wrong with the CVS repo on our CVS server. You know who my team needed help from to get it fixed? The token Emacs user that sat near us. He had been using the CVS CLI all along. He made quick work of the problem that we had. It was trivial enough for him to fix that none of us ever understood what the problem was.

I humbly suggest that this will happen not only with VCS tools but dependency management tools, interpreters, compilers, application servers, linters, code coverage tools, and build tooling.

In Sum

In order to integrate developer tools into a single environment, IDEs abstract tooling away which has the unfortunate side effect of obscuring the developer from their tooling.

It is my contention that this may be good for short-term productivity, but is almost certainly bad for long-term productivity.

Take a look at part 2 of this series to understand why stepping way from an IDE was so beneficial to me.

Do you completely disagree with me? Excellent! Please let me know in the comments.

How does DNS work?

Thu, 27 Dec 2018 16:00:00 GMT

Whether you realize it or not, DNS is a workhorse of the internet ecosystem. It provides the glue that allows someone like your grandma, (who knows that you want a pair of Adidas Ultraboost from footlocker.com for Christmas) to talk TCP with a server on the other side of the world.

It’s amazing when you think about it…

Your grandma really is quite kind.

So, let’s start from first principles and dig in to understand DNS a bit more.

What exactly is DNS?

DNS, or the Domain Name System is an abstraction layer that maps the human-readable web addresses that we know and love to IP addresses.

For a long time, that was about the extent of my knowledge of DNS. I knew that there were servers out in the ether of the internet that contained those mappings. Later on, I learned that one could use nslookup to find existing mappings. Beyond that, I knew very little. The majority of the time this was enough, but every now and then I would hear someone mention an A record or a CNAME record and I couldn’t contribute much to the conversation other than “Ah yes, that has something to do with DNS”. If you are like me, the things that you know that you don’t know serve as bread crumbs for the things that you want to learn. Let’s follow those breadcrumbs a bit.

The lifecycle of a DNS query

Interview questions like “Tell me everything that happens when I navigate to google.com” are quite fun to answer for me. This is mainly because there are endless ways to tell the story of all the things that go on when a user does x. At some point though, we have to limit the detail of our explanations in order to actually provide value. Be forewarned: the explanatory lifecycle of a DNS query below is not exhaustive! Exhaustive references can be useful but not so much in didactic contexts.

So what happens when a domain name query is issued to DNS?

Someone named Jeffrey asks their computer to go to google.com
The operating system of Jeffrey’s computer consults its own cache to see if the IP address for the requested name is already known. If it is, life is good! If not…
Jeffrey’s operating system reaches out to a DNS cacher/recursor which we will refer to as the local DNS server from here onward. The local DNS server is very often provided by your ISP, but sufficiently advanced LANs will often have their own local DNS server.
The local DNS server will check its cache. If it has an answer to the query, life is good! If not…
The local DNS server will recursively contact other DNS servers until it can provide an authoritative answer. The recursive DNS request looks something like:
- Local DNS server asks a root DNS server where to find the .com TLD (top level domain) server. The root DNS server provides the answer.
- Local DNS server then asks the .com TLD DNS server for the “google” name server. The .com TLD DNS server provides the answer.
- Local DNS server asks the “google” DNS server for the google.com IP. The “google” name server provides an answer.
The local DNS server returns the IP address provided by the “google” name server to Jeffrey’s computer. (and hopefully caches the answer!)

What is a DNS record?

The response provided by a DNS server represents a record that exists in the Domain Name System. One might find that the responses representing records are referred to as a record themselves. YMMV.

Below I explain a few of the common DNS record types. There are many more. For more details you may want to consult this helpful and complete list of DNS record types.

`CNAME` Records

A CNAME record is said to be “the canonical name for an alias” in RFC 1035. CNAME records map an alias to another name. Upon receiving a CNAME response, the computer issuing a DNS query will restart the query using the value that the CNAME record points too. Note: While DNS is actually resolving the IP of the canonical name, the original name request (the alias) is what users will see in their browser.

CNAME records are intended to point to a canonical name, but there is nothing that prevents one from creating a CNAME record that point to another alias. One could even create an infinite loop if they so desired.

`DNAME` Records

A DNAME record “provides redirection for a subtree of the domain” accord to its RFC (RFC 6672). For this reason is it known as a delegation record.

`A` Records

A records are described as providing “a host address” in RFC 1035. A records are the meat and potatoes(?) of DNS. The provide a simple mapping from a name to an IP address.

Did I get something wrong? Is there another subset of DNS functionality that needs to be explained? Let me know in the comments.

The Bad Plus

Sat, 08 Dec 2018 16:00:00 GMT

I’ve found over the years that there is some music that is well-suited for listening while doing software-development related work and some music is not. Rather than try to derive a special formula for “coding music” I’m going to tell you about some music that works for me. One group I’ve been listening to for awhile that works particularly well is The Bad Plus.

The Bad Plus is a jazz piano trio that might not sound like a jazz group to you at first blush. They employ jazz idioms and instrumentation on popular cover tunes and their own originals as well. I find a wide range of mood and tone in their music: calm and nearly ambient, cacophonic excitement, and even straight ahead tunes with lots of energy. Despite the diversity it all works nicely when I’m clacking away on the keyboard.

I highly recommend catching them live if they are ever playing in your town. The occasions I’m which I had the priviledge of hearing The Bad Plus did not disappoint – YouTube has a nice selection of live recordings available. One can only hope they release more live music in the future.

Song Selection for Work

Anthem for the Earnest
Flim
As the Moment Slips Away
Beauty Has It Hard
We are the Champions
Forces
Prehensile Dream
Silence is the Question
1983 Regional All-Star
Newport Jazz Festival, 2006
Moers Festival, 2017

The Serverless Sea Change

Fri, 07 Dec 2018 16:00:00 GMT

I read a great article at InfoQ this week called The Serverless Sea Change. Generally, the argument made is that code leads to technical debt, therefore the solution with the least amount of code wins. For example:

If you are building a Serviceful Serverless application, you will need to spend significantly more time on research than you would with an application where you will write everything. This is because you will be implementing much more functionality of the application on services, and you need to verify that you are selecting the correct service. You also need to figure out the right way to integrate the service with your application. So instead of spending an hour or two looking for packages you might leverage, you should think about spending days and even weeks writing proof-of-concept code and testing different options.

Another way to say this is in two equations:

2 Weeks Research + 1 Day Development → N Lines of Code to Maintain

1 Day Research + 2 Weeks Development → 10 ⋅ N Lines of Code to Maintain

On average, ten times more lines of code is ten times more technical debt, which means increasingly slower and less predictable future development velocity, and systems that cannot be well maintained by the average developer.

I’ve generally remained skeptical as to whether the advantages of serverless architectures outweigh the benefits, or to put it another way, if there are any advantages when all is sasaid and done. For me, the looming issues are:

Vendor Lock-in
Conversely, the portability of any artifact targeting a FaaS platform
The amount of glue needed to make serverless work (any AWS Lambda demo I’ve seen spends more time gluing AWS services together than it does writing code)
In the case of AWS, many of the glue services are still beta products
Ease of local development
The Cold Starts problem

I remain skeptical, but this article does a great job of explaining and arguing for the benefits. Also, perhaps I am wrong (!) and we are a few years away from a true “Serverless Sea Change”.

Be sure to check out the whole article over at InfoQ.

Mike Brinker

Spec-Driven Development: Skeleton Coding

Favorite Quotes

Components of Agent Memory

Observations

Sources referenced:

The October 2026 Inflection

Defining a FontFamily for Greek

The Dative of Possession

Ματαιότης in Romans 8 and Ecclesiastes

βίβλος and βιβλίον

Working with AI, State of the Union June 2024

A Canticle for Leibowitz

Rebooting My Writing Practice

On 10 Years of Making Software

Identity

The Nature of the Work

Engineering

Development

Craft

Make it work, make it right, make it beautiful

Creating

Craving

Footnotes

Heroes and Housekeeping

Flatten the Cost-of-Change Curve, you must.

Software is a highly malleable medium

Software’s ability to change is a primary competitive advantage

Therefore, consistently low cost of change is essential to maximizing its intrinsic benefits

If a team cannot exercise discipline to manage the cost of change, as a wise man once said, “then the Emperor has already won”.

Healthy Codebase Habit: Regular Dependency Updates

Stale Dependencies

This is your team on stale dependencies

Some problems with stale dependencies

Fresh dependencies

This is your team on fresh dependencies

Some advantages to fresh dependencies.

In Sum

'Who You Are' is 'What You Do'

The Software Developer's Path to Wisdom

Wisdom?

The Model (Brinker’s Model of Ideological Adoption?)

Ignorance

Awareness

Zeal

Wisdom

Wisdom in Software Development

A Case Study: Test-Driven Development

Ignorance

Awareness

Zeal

Wisdom

In Sum

A Software Engineering Manifesto

Deployments Shall Be

Databases Shall Be

Code Shall Be

Project Management Shall Be

Developers Shall be

System Design Shall Be

An Atlas for Learning React

Conceptual Overviews

Tools

Ecosystem

Tutorials

People

Glossary of Terms

Elements

Components

State

Mounting

Lifecycle Methods

Linux, Entropy, Random Number Generation, and you

Why Random Number Generation (RNG) is important in computing

Linux and RNG

The JVM and /dev/random

Addendum: Useful commands and config

Refactoring Doesn't Require Domain Knowledge

IDEs are dead. Long Live Development Environments. (Part 3)

1. Begin using a single tool directly, outside of your IDE.

`CNAME` Records

`DNAME` Records

`A` Records