I have a Toshiba Libretto 70CT which is still running, but with 120 Mhz CPU and 16 Mb of RAM I don’t use it much, and would love a modern version - the form factor was small, but the keyboard was still big enough to touch type on.
I similarly like small devices. A lot of people use the GPD devices for retro gaming and there’s a lot of competition in the handheld retro gaming consoles. I recently got a Anbernic RG552 because it’s based on a Rockchip RK3399 chipset which I have some experience with. I got a mainline Linux kernel running on it and have full NixOS:
The OP mentions how the Micro reminds him of his once daily driver the EeePC 1000HE. Now, I recently became obsessed with the EeePC line because it’s small, can easily run Alpine Linux and X11, and just looks cool. Parts are easy to come by and it’s hackable beyond the standard upgrades for the true diehards. But OP hits it on the head when he says
Finally, this little thing has a soul. Like the Eee
I love my little Eee PC 900 (Pepsi, I call it) and I’m glad OP recaptured that feeling with hardware with his Micro too.
Similarly on the underground: rather than mindlessly reaching for the phone and scrolling through news, I choose to pull out the Micro and read some code.
I found this too when traveling my EeePC - forces me to do what I think I should be doing more of: writing!
Nice to meet a fellow enthusiast. I have a white one that I use for writing, making notes and light coding on the couch.
I found it to be too slow and too small for X with the distros I tried and I settled for text only. tmux is your friend. I even have a half finished blog post lying somewhere about FDD: framebuffer driven development. It really does wonders for you concentration!
Indeed, I don’t find myself within X often but netsurf works well and giving it a bit more RAM helped. Otherwise, I’m just using screen and jumping between virtual consoles (if need be) - also lynx (once you have the settings to your liking) is great on these EeePCs
What a great writeup. Here’s why it hits the nail on the head. TDD on its own is great. It gets people talking about the design of their code. It gets people caring about correctness. But the “TDD maximalist” position is unhelpful, arrogant, annoying, and flat out incorrect.
Uncle Bob’s “TDD is double entry bookkeeping” doesn’t make any sense. The reason double entry bookkeeping works is because accounting is zero-sum. Money must move between accounts, because it’s a finite resource. If you write a test, code can be written to pass it in an infinite number of ways. Computation is not a finite resource. The analogy is not good.
The TDD maximalist position that “TDD improves design because the design ends up more testable that way” is the definition of circular reasoning. I loved the point about long functions sometimes being the better abstraction - this is absolutely true in my experience. Artificially separating a concept that’s just inherently complex doesn’t improve anything objectively, it just gives someone with an obsession over small functions an endorphin hit. The same is true of dependency injection and other TDD-inspired design patterns.
I know tons of people who have done TDD for years and years and years. As mentioned in the article, it has a lot of great benefits, and I’ll always use it to some degree. But in those years working with these other people, we have still always had bug reports. The TDD maximalist position says we were just doing it wrong.
Well, if a methodology requires walking a tightrope for 5 years to get the return on investment, maybe it’s not a perfect methodology?
I don’t think this is right: while Freud has largely been discarded in the details, his analytical approach was utterly new and created much of modernity. Uncle Bob … well, the less said, the better.
While I generally agree with your take, I’m going to quibble with part of it:
The TDD maximalist position that “TDD improves design because the design ends up more testable that way” is the definition of circular reasoning.
This is not circular reasoning, though the inexact phrasing probably contributes to that perception. A more exact phrasing may help make it clear that this isn’t circular:
All else equal (and I’m not sure what that might mean when discussing design), a more testable design can be considered a better design.
A way to ensure that the design is more testable is to require comprehensive tests, and reject designs where comprehensive tests are hard to achieve.
TDD is a practice which generates comprehensive tests, and which rejects designs that cannot be comprehensively tested.
TDD therefore improves this dimension of software design.
Put this way, this may be less controversial: it allows room for both those who subscribe to TDD, and for those who object to it. (Possible objections include “all else can’t be equal” – I think DHH’s comments on Test-induced Design Damage fall in this category – or that other practices can be as good or better at generating comprehensive tests, like TFA’s discussion of property-based testing.)
All else equal (and I’m not sure what that might mean when discussing design), a more testable design can be considered a better design.
This is why the reasoning is circular. This point right here can be debated, but it’s presented as an axiom.
TDD is a practice which generates comprehensive tests, and which rejects designs that cannot be comprehensively tested.
TDD does nothing inherently, this is another false claim. It does not reject any design. In practice, programmers just continuously copy the same test setup code over and over and do not care about the signal that the tests are giving them. Because TDD does not force them to. It happily admits terrible designs.
It does reject designs: designs that cannot be tested cannot be generated when following strict TDD, and designs which are difficult to test are discouraged by the practice. (Edit to add: In practice, this requires developers to listen to their tests, to make sure they’re “natural” to work with. When this doesn’t happen, sure, blindly following TDD probably does more harm than good.)
Sure, there’s some design out there that won’t even support an end to end test. The “naturalness” of a test though can’t be measured, so the path of least resistance is to just make your tests closer and closer to end to end tests. TDD itself will not prevent that natural slippage, it’s only prevented by developer intervention.
That’s the TDD maximalist point of view is a nutshell - if you don’t get the alleged benefits of it, you did it wrong.
For what it’s worth, I’m not a TDD maximalist, I try to be more pragmatic. I’m trying to have this come across in some of the language I’m using: non-absolute terms, like “encourage” or “discourage”, instead of absolutes like “allow” or “deny”. If you’re working with a code base that purports to follow TDD, and you’re letting your unit tests drift into end-to-ends by another name, then I’d argue that you aren’t listening to your tests (this, by the way, is the sense in which I think “naturalness” can be measured – informally, as a feeling about the code), and that (I’m sorry to say, but this does sometimes happen) you’re doing it wrong.
Doing it right doesn’t necessarily mean using more TDD (though it might!), it’s just as easy to imagine that you’re working in a domain where property-based tests are a better fit (“more natural”) than the style of tests generated via TDD, or that there are contextual constraints (poorly-fitting application framework, or project management that won’t allow time to refactor) that prevent TDD from working well.
I’m mostly with you. I’m definitely what Hillel refers to in this article as a “test-first” person, meaning I use tests to drive my work, but I don’t do strict TDD. The way that people shut their brains off when talking about TDD kills me though.
It does reject designs: designs that cannot be tested cannot be generated when following strict TDD, and designs which are difficult to test are discouraged by the practice.
Well, yeah, that’s why “maximalist TDD” has such a mixed reputation…
First, there are plenty of cases where you’re stuck with various aspects of a design. For example, if you’re writing a driver for a device connected over a shared bus, you can’t really wish the bus out of existence, so you’re stuck with timing bugs, race conditions and whatever. In these cases, as is often the case with metrics, the circular thing happens in reverse: you don’t get a substantially more testable design (you literally can’t – the design is fixed), developers just write tests for the parts that are easily tested.
Second, there are many other things that determine whether a design is appropriate for a problem or not, besides how easy it is to implement it by incrementally passing automatic unit tests written before the code itself. Many of them outweigh this metric, too – a program’s usefulness is rarely correlated in any way to how easy it is to test it. If you stick to things that The Practice considers appropriate, you miss out on writing a lot of quality software that’s neither bad nor useless, just philosophically unfit.
For example, if you’re writing a driver for a device connected over a shared bus, you can’t really wish the bus out of existence, so you’re stuck with timing bugs, race conditions and whatever. In these cases, as is often the case with metrics, the circular thing happens in reverse: you don’t get a substantially more testable design (you literally can’t – the design is fixed), developers just write tests for the parts that are easily tested.
I’ve written device drivers and firmware code that involved shared buses, and part of the approach involved creating a HAL for which I could sub out a simulated implementation, allowing our team to validate a great deal of the logic using tests. I used an actor-model approach for as much of the code as possible, to make race conditions easier to characterize and test explicitly (“what happens if we get a shutdown request while a transaction is in progress?”). Some things can’t be tested perfectly – for example we had hardware bugs that we couldn’t easily mimic in the simulated environment – but the proportion of what we could test was kept high, and the result was one of the most reliable and maintainable systems I’ve ever worked on.
I’m not trying to advocate against DFT, that’d be ridiculous. What I’m pointing out is that, unless DFT is specifically a goal at every level of the design process – in these cases, hardware and software – mandating TDD in software without any deliberation higher up the design chain tends to skew the testability metric. Instead of coming up with a testable design from the ground up, developers come up with one that satisfies other constraints, and then just write the tests that are straightforward to write.
Avoiding that is one of the reasons why there’s so much money and research being poured into (hardware) simulators. Basic hiccups can be reproduced with pretty generic RTL models (oftentimes you don’t even need device-specific logic at the other end), or even just in software. But the kind of scenarios that you can’t even contemplate during reviews – the ones that depend on timing constraints, or specific behavior from other devices on the bus – need a more detailed model. Without one, you wind up not writing tests for the parts that would benefit the most from testing. And it’s what you often end up with, because it’s hard to come up with reliable test models for COTS parts (some manufacturers do publish these, but most don’t). That’s not to say the tests you still get to write are useless, but it doesn’t have much of an impact over how testable the design is, nor that there aren’t cheaper and more efficient ways to attain the same kind of correctness.
My experience with TDD development in this field has been mixed. Design teams that emphasize testability throughout their process, in all departments, not just software, tend to benefit from it, especially as it fits right in with how some of the hardware is made. But I’ve also seen TDD-ed designs with supposedly comprehensive tests that crashed if you so much as stared at the test board menacingly.
Therefore, designs that lend to better test coverage are more likely to be correct than designs that don’t.
End to end testing has more coverage per test case though. So, by this logic, you would just write only end to end tests, which is in contrast to the isolated unit testing philosophy.
So, by this logic, you would just write only end to end tests, which is in contrast to the isolated unit testing philosophy.
I’m not sure that actually is in opposition to TDD philosophy, though. My understanding of TDD is that it’s a pretty “top-down” approach, so I imagine you would start with a test that is akin to e2e/integration style before starting work on a feature. A “fundamentalist” would leave just about all of the implementation code “private” and not unit test those individual private functions unless they became part of the exposed public API of the software. A more pragmatic practitioner might still write unit tests for functions that are non-trivial even if they are not public, but I still think that TDD would encourage us to shy away from lots of unit tests and the “traditional” test pyramid.
I could be totally off base with my understanding of TDD, but that’s what I’ve come away with from reading blogs and essays from TDD advocates.
End to end tests can have difficulty with “testability”, IMO:
When an end-to-end test fails, it can be very difficult to reason backwards from the failure to the fault. This is a byproduct of the fact that end-to-ends cover more code per test case: The coverage may be high, but the link between the code being tested and the test can be tenuous.
Similarly, it’s very difficult to create an end-to-end test that targets a specific part of the code.
Finally, one can’t write an end-to-end test until a new facility is complete enough to be integrated. I’d rather establish confidence about the code I’m writing as early as possible.
Edit to add: This is probably giving the wrong impression. I like E2E tests, whole system behavior is important to preserve, or to understand when it changes, and E2E’s are a good way of getting to that outcome. But more narrowly targeted tests have their own benefits, which follow from the fact that they’re narrowly targeted.
I was going to object along similar lines - but I agree with the sibling - your 1) does not make a good argument.
I’d say more something like: A testable design is better, because it allows for writing tests for bugs, helping document fixes and prevent regressions. Tests can also help document the business logic, and can improve the overall system that way too.
Just saying tests are good; let’s have more tests! - Doesn’t say why tests are good - and we do indeed get circular reasoning.
I’ve inherited a number of legacy systems with no tests - and in the few I’ve shoe-horned in tests with new features and bug fixes - those have always cought some regressions later. And even with the “wasted” hours wrangling tests to work at all in an old code base; I’d say tests have “paid their way”.
If those code bases had had tests to begin with, I think a) the overall designs would probably have been better, but perhaps more importantly b) writing tests while fixing bugs would have been easier.
I also think that for many systems, just having unit tests will help a great deal with adding integration tests - because some of the tooling and requirements for passing in state/environment (as function arguments, mocks, stubs etc) is similar.
I do consider point 1 nearly axiomatic. What does “testable” mean? Trivially, it means something like “easy to test”, which suggests properties like:
Easy to set up preconditions and provide inputs;
Easy to verify postconditions and validate outputs;
Easy to address interesting points in the module’s semantic domain.
Achieving these requires that the interface under test be pretty easy to understand. All else equal (whatever that might mean when talking about design), having these qualities is better than not having them. (Note, also, that I talk more about “testability”, not “tests”: I value testability more than I value tests, though TDD does strongly encourage testability.)
Still, I don’t believe this is circular. Point 1 is presented axiomatically, but the remaining points that attempt to characterize the meaning only depend on earlier points, without cycles.
Struggling with work. I’m not problem solving fast enough. I’ve done a bunch of practice and will do more so “practice more” doesn’t seem to be helping.
I may need to do a career pivot but I’ve had a chunk of my identity wrapped up in “I’m a guy who writes code” so I’m not quite ready to give up yet.
I’m very good at simple process oriented scripting, which is how I spent most of my career up to now. I thought if I had the chance to do more complex software engineering I’d rise to the occasion but I’m not there yet :)
What kind of problems aren’t being solved “fast enough”? I’m curious because I am (again) deciding if pretending to enjoy $WORK is worth the effort. Sometimes I think of Lesane and hear them:
“I made a G today,” but you made it in a sleazy way
In my case it’s problem solving around debugging complex code problems, and also being able to pick up new technologies and glue them together rapidly enough.
If I may… maybe try on “I’m a guy who solves problems and/or builds things” or something like that, and see how that fits? Writing code is just one way of building things or solving problems.
Thank you very much that perspective is very helpful.
The next bit, once I have my own head on straight, is figuring out how I should be piloting my own career in a way which will both allow me to thrive and continue to put bread on the table for my family :)
Still drinks 8 or 9 Diet Cokes a day?! I guess it’s true, that stuff does make one goes crazy later in life. I will say, insightful to hear his thoughts on debuggers and interesting cultural thing about “big companies” disdaining them and their Linux bias.
Steele suggests the trick in balancing size against utility is that a language must empower users to extend (if not change) the language by adding words and maybe by adding new rules of meaning.
I don’t buy this. Authors generally don’t extend e.g. English with new words or grammar in order to write a novel. Programmers generally don’t need to extend a programming language with new words or rules in order to write a program.
A programming language, like a spoken/written language, establishes a shared lexicon in which ideas can be expressed by authors and widely understood by consumers. It’s an abstraction boundary. If you allow authors to mutate the rules of a language as they use it, then you break that abstraction. The language itself is no longer a shared lexicon, it’s just a set of rules for arbitrarily many possible lexicons. That kind of defeats the purpose of the thing! It’s IMO very rare that a given work, a given program, benefits from the value this provides to its authors, compared to the costs it incurs on its consumers.
Programmers generally don’t need to extend a programming language with new words or rules in order to write a program.
Maybe you would disagree, but I would argue that functions are essentially new words that are added to your program. New rules do seem to be a bit less common though.
This would be my interpretation as well. Steele defines a language to be approximately “a vocabulary and rules of meaning”, and clearly treats defining types and functions as to be adding to that vocabulary throughout his talk. My (broad) interpretation of a generalized language is based on that idea that the “language” itself is actually just the rules of meaning and all “vocabulary” or libraries are equal whether they be “standard” or “prelude” or defined by the user.
I understand the perspective that functions (or classes, or APIs, or etc.) define new words, or collectively a new grammar, or perhaps a DSL. But functions (or classes, or APIs, or etc.) are obliged to follow the rules of the underlying language(s). So I don’t think they’re new words, I think they’re more like sentences, or paragraphs.
Authors generally don’t extend e.g. English with new words or grammar in order to
FWIW a thing that shitposters on Tumblr have in common with William Shakespeare is the act of coining new vocabulary and novel sentence structure all the time. :)
The author is saying that simple systems need this extensibility to be useful. English is far from small. Even most conventional languages are larger than the kernel languages under discussion, but chief argument for kernel languages is their smallness and simplicity. And those languages are usually extended in various ways to make them useful.
I would say, and I think that the author might go there too, that certain libraries that rely heavily on “magic” (typically ORMs) also count to some degree as language extensions. ActiveRecord and Hibernate, for instance, use functionality that is uncommonly used even by other practitioners in their respective languages.
The article is about languages, right? Languages are a subset of systems. The abstraction established by a language is by definition a shared context. A language that needs to be extended in order to provide value doesn’t establish a shared context and so isn’t really a language, it’s a, like, meta-language.
It’s about using languages as a case study in how systems can be of adequate or inadequate complexity to the tasks they enable their users to address, and how if a tool is “simple” or perhaps inadequately complex that the net result is not that the resulting system is “simple” but as @dkl speculates above that the complexity which a tool or system failed to address has to live somewhere else and becomes user friction or – unaddressed – lurking unsuitability.
This creates a nice concept of “leverage” being how a language or system allows users to address an adequate level of complexity (or fails to do so), and begs how you can measure and compare complexity in more meaningful and practical terms than making aesthetic assessments both of which I want to say more about later.
I suppose I see languages as systems that need to have well-defined and immutable “expressibility” in order to satisfy their fundamental purpose, which isn’t measured in terms of expressive power for authors, but rather in terms of general comprehension by consumers.
And, consequently, that if a language doesn’t provide enough expressivity for you to express your higher-order system effectively, the solution should be to use a different language.
If you consider that each project (or team) has slightly different needs, but there aren’t that many slightly different language dialects out there that add just one or two little features (thankfully!) that these projects happen to need. Sometimes people build preprocessors to work around one particular lack of expressibility in a language (for one well-known example that’s not an in-house only development, see yacc/bison). That’s a lot of effort and produces its own headaches.
Isn’t it better to take a good language that doesn’t need all that many extensions, but allows enough metaprogramming to allow what your team needs for their particular project? This allows one to re-use the 99% existing knowledge about the language and the 1% that’s different can be taught in a good onboarding process.
Nobody in their right mind is suggesting that projects would be 50% stock language, 50% custom extensions. That way lies madness. And indeed, teams with mostly juniors working in extensible languages like CL will almost inevitably veer towards metaprogramming abuse. But an experienced team with a good grasp of architecture that’s eloquent in the language certainly benefits from metaprogrammability, if even it’s just to reduce the amount of boilerplate code. In some inherently complex projects it might even be the difference between succeeding and failing.
I do think the industry tends to be dominated by junior-friendly programming languages, as if the main concern was more about increasing headcount than it was about expressing computation succinctly and clearly.
Nobody in their right mind is suggesting that projects would be 50% stock language, 50% custom extensions. That way lies madness.
Paul Graham made roughly that claim when he wrote about why viaweb could only have been written in common lisp, but I think his numbers were more like 70/30. But you did qualify it with only people in their right mind, so maybe that doesn’t count.
Isn’t it better to take a good language that doesn’t need all that many extensions, but allows enough metaprogramming to allow what your team needs for their particular project?
YMMV, but I don’t think so. My experience with metaprogramming in industry has been negative. I find it usually adds more complexity than it removes. And I don’t really agree with the framing that a language should be extensible, at the ·language· level, by its users. I see that as an abstraction leak.
My experience with metaprogramming in industry has been negative. I find it usually adds more complexity than it removes.
I’d put that in the “hell is other people” basket - indeed, very often metaprogramming gets abused in ways that make code more complicated than it has to be. So you definitely have a point there. But then I’ve seen horror shows of many other kinds in “industrial” codebases in languages without metaprogramming. In the wrong hands, metaprogramming can certainly wreak more havoc, though!
Still, I wouldn’t want to go back to languages without metaprogramming features, as I feel they allow me to express myself more eloquently; I prefer to say things more precisely than in a roundabout way.
And I don’t really agree with the framing that a language should be extensible, at the ·language· level, by its users. I see that as an abstraction leak.
I never considered that perspective, but it makes sense to view it that way. In my neck of the woods (I’m a Schemer), having an extensible language is typically considered empowering - it’s considered elitist to assume that the developer of the language knows everything perfectly and is the only one who should be allowed to dictate in which direction the language grows. After all, the language developer is just another fallible programmer, just like the language’s users are.
As a counterpoint, look at the featuritis that languages like Python and even Java have spawned in recent years. They grew lots of features that don’t even really fit the language’s style. Just because something’s gotten popular IMHO doesn’t necessarily mean it ought to be put in a language. It burdens everyone with these extra features. Think about C++ (or even Common Lisp), where everyone programs in a different subset of the language because the full language is just too large to comprehend. But when you want to leverage a useful library, it might make use of features that you’ve “outlawed” in your codebase, forcing you to use them too.
There’s also several examples of the Ruby and Python standard libraries having features which have been surpassed by community libraries, making them effectively deprecated. Sometimes they are indeed removed from the standard library, which generates extra churn for projects that were using them.
I’d rather have a modestly-sized language which can be extended with libraries. But I readily admit that this has drawbacks too: let’s say you choose one of many object/class libraries in an extensible language which doesn’t supply its own. Then if you want to use another library which depends on a different class library, you end up with a strange mismatch where you have entirely different class types, depending on what part of the system you’re talking to.
I’ve seen horror shows of many other kinds in “industrial” codebases in languages without metaprogramming. In the wrong hands, metaprogramming can certainly wreak more havoc, though!
Sure! In pure quantitative terms, I agree: I’ve seen way more awful Java (or whatever) than I have awful metaprogramming stuff. But the bad Java was a small subset of the overall Java I’ve witnessed, whereas the bad metaprogramming was practically 100% of the overall metaprogramming I’ve witnessed.
I fully acknowledge that I’m biased by my experience, but with that caveat, I just haven’t seen metaprogramming used effectively in industry.
This is an important point. You are allowing authors to add new syntax, not requiring it. You can do most of the same tricks in Common Lisp or Dylan that you can do in Javascript or Kotlin, by passing funargs etc., it’s just that you also have the ability to introduce new syntax, within certain clearly defined bounds that the language sets.
Just as you have to learn the order of arguments or the available keywords to a function, Lisp/Dylan programmers are aware that they have to learn the order of arguments and which arguments are evaluated for any given macro call. Many of them look exactly like functions so there’s no difference. (I like that in Julia, as oppose to Common Lisp and Dylan, macro calls must start with the special character “@”, since it makes a clear distinction between function calls and macro calls. But I don’t know much about Julia macros.)
A programming language, like a spoken/written language, establishes a shared lexicon…
Yes, and I don’t believe macros change this significantly, although this depends to a certain extent on the macro author have a modicum of good taste. The bulk of Common Lisp and Dylan macros are one of two flavors:
“defining macros” – In Common Lisp these are usually named “defsomething” and in Dylan “define [adjectives] something”. When you see define frame <chess-board> (<frame>) ... end you know you will encounter special syntax because define frame isn’t part of the core language. So you go look it up just as you would lookup a function for which you don’t know the arguments.
“with…” or “…ing” macros like “with-open-file(…)” or “timing(…)”
These don’t change the complexity of the language appreciably in my experience and they do make it much more expressive. (Think 1/10th to 1/15th the LOC of Java here.)
Where I believe there is an issue is with tooling. Macros create problems for tooling, such as including the right line number in error messages (since macros can expand to arbitrarily many lines of code), stepping or tracing through macro calls, recording cross references correctly for code browsing tools, etc.
This seems founded in the idea that if you just know the language, you can look at code and understand what it does. But this is always convention. For instance, consider the classic C issue of “who owns the pointer passed to a function?” Even with an incredibly simple language, there’s ambiguity about what conventions surround a library call - do you need to call free()? Will the function call free()? Can you pass a stack pointer, or does it have to be heap allocated? And so on. More powerful type systems can move more information into the type itself, but more powerful types tend to be included in more powerful languages; for instance, in D, if you pass an expression to a function, if the parameter is marked as lazy, the expression may actually be evaluated any number of times, though it’s usually zero or one, and you have no idea when the evaluation takes place. So just from looking at a function call, foo(bar), it may be that bar is evaluated before foo, or during foo, or multiple times during foo, or never.
Now a macro could do worse, sure, but to me it’s a difference of degree, not kind. There’s always a spectrum, and there’s always ambiguity, and you always need to know the conventions. Every library is a grammar.
This seems founded in the idea that if you just know the language, you can look at code and understand what it does. But this is always convention. For instance, consider the classic C issue of “who owns the pointer passed to a function?” Even with an incredibly simple language, there’s ambiguity about what conventions surround a library call - do you need to call free()? Will the function call free()? Can you pass a stack pointer, or does it have to be heap allocated? And so on.
The devil’s in the details. But when I’m answering questions like “who calls free()?” I don’t need to re-evaluate my understanding of language keywords or sigils or operator semantics. I see this as a categorical difference. I suppose other people may not.
I think it’s worth addressing the elephant in the room: there’s language as in formal language (mathematics) and then there’s language as reasoned by linguistics, which also acknowledges and studies the sociopolitical and anthropological aspects of language. It may be useful to analyze PLs through the lens of the former, but programming languages also do an awful lot of things that human languages do: give rise to dialects, have governing bodies, have speakers that defy governing bodies, borrow from each other, develop conventions, nurture communities, play host to ecosystems, and so forth.
We now arrive at the core of your assertion, which is that there is a hard line between syntax that comes for free and syntax that is defined by the userland programmer. This is part of the very same line that the author is asking us to imagine blurring:
Building on Steele’s deliberate blurring of the line between a “language” and a “library”, I suggest this train of thought applies to libraries as well. Libraries – language extensions really – help us say and do more but can fail to help us manage complexity or impose costs on their users in exactly the same ways as language features.
It is furthermore dangerous to equate the malleability of syntax with the extensibility of a language. Macros are a form of extensibility, yet conversely, extensibility does not require a support for macros. Take Python, a language that does not support macros, for example. Python allows you to engage in so much operator overloading that you could read the expression (a + b) << c and still have no clue what it actually does without looking up how the types of a, b, and c define such operations. Ultimately, it is conventions — be they implicit or documented, be they cultural or tool-enforced — that dictate the readability of a language, just as @FeepingCreature has demonstrated with multiple examples.
The author covers a great deal of the triumphs and tragedies of language extensibility in the “Convenience? Or simplicity through design?” section. The monstrosity of the Common Lisp LOOP macro, as well as userland async / await engines, are brought up because they brutally defeat static analysis, whereas the .use method in Kotlin is shown as an example that manages to accomplish all of the needs of WITH-FOO macros with none of the ill effects of macros. In fact, there is another commonly used language that manages to achieve this the same way Kotlin does: Ruby. In Ruby, any method call can take an additional block of code, which gets materialized into a Proc object, which is a first-class function value. This means in Ruby it is routine to write code that looks like File.open("~/scratch.txt") { |f| f.write("hello, world") }. This is an example straight out of the Ruby standard library, and it is the convention for resource management across the entire Ruby ecosystem.
Now, even though Ruby — like Python — does not support macros, and even though it got resource management “right”, just like Kotlin did, it has nevertheless managed to dole out some of the most debilitating effects of metaprogramming. Ask anyone who’s worked on a Rails project, myself included, and they will recount how they were victimized by runtime-defined methods whose names were generated by concatenating strings and therefore remain nigh unsearchable and ungreppable. A great deal of research and literature have been dedicated to making Ruby more statically analyzable, and they all converge towards the uncomfortable conclusion that its capability of evaluating strings as code at runtime and its Smalltalk-like object system both make it incredibly difficult for machines to reason with Ruby code without actually running it.
there’s language as in formal language (mathematics) and then there’s language as reasoned by linguistics, which also acknowledges and studies the sociopolitical and anthropological aspects of language.
Programming languages can be analyzed in the formal/mathematical sense, but when they are used, they are used by humans, and humans necessarily don’t understand languages through this lens. I’m not saying a programming language is strictly e.g. linguistic, but I am saying that it cannot be effectively described/modeled purely via formal or mathematical means. A language isn’t an equation, it’s an interface between humanity and logic.
Authors generally don’t extend e.g. English with new words or grammar in order to write a novel.
Well, no, but it helps that English language already has innumerable – legion, one might say – words, an English dictionary is a bountiful, abundant, plentiful universe teeming with sundry words which others have already invented, not always specifically for writing a whole novel, often just in order to get around IRL. It’s less obvious with English because it’s not very agglutinative but it has been considerably extended in time.
Language changes not only through the addition of new words and rules but also through other words and rules falling out of use (e.g. in English, the plural present form of verbs lost its inflection by the 17th century or so), so it’s not quite fair to say that modern English is “bigger” than some of its older forms. But extension is a process that happens with spoken languages as well.
It’s also super frequent in languages that aren’t used for novels and everyday communication, too. At some point I got to work on some EM problems with a colleague from a Physics department and found that a major obstacle was that us engineers and them physicists use a lot of different notations, conventions, and sometimes even words, for exactly the same phenomenon. Authors of more theoretical works regularly developed their own (very APL-like…) languages that required quite some translation effort in order to render them into a comprehensible (and extremely verbose) math that us simpletons could speak.
(Edit: this, BTW, is in addition to all that stuff below, which @agent281 is pointing out, I think the context of the original talk is relevant here).
If I had to guess I would think that this was a nudge to Scheme programming language from Steel’s point of view (or Common Lisp, but this one is quite large in comparison). Quite easy to extend Scheme that way and he does have a history with lisp-family languages. But that’s just a guess.
Thanks for this. The video (2012) linked at the end was also super helpful. As someone that’s been working on an older go project for a short time, I do feel there’s mocks/fakes/whathaveyou that maybe shouldn’t have been written and went looking for some content akin to this but wasn’t coming up with much.Thanks again.
I have a ipxe script burned onto most of the Intel nics in my house that boot directly to netboot.xyz for the last two or three years. I use it off and on when I’m to lazy to find my ventoy based thumb drive. It’s great in a pinch. This is the very basic script I have running on my cards.
Look, PKI/the CA cabal clearly isn’t perfect. In fact it’s extremely flawed, especially if you aren’t enforcing Certificate Transparency which presumably iPXE isn’t. But I think it’s really silly to imply that HTTPS doesn’t raise the bar for the attacker? At minimum they now have to actually compromise a CA instead of being able to just set up an ordinary HTTP proxy and serve malicious images.
PKI is scary enough that nobody dares to even touch it, but one
person at each department/company/whatever who Is The PKI Guy who
everybody expects to know everything and fix everything remotely
related to PKI. (at least from my experience)
I think many, many other security measures will fail before lack
of TLS will be an issue. But it is still very cool that it’s
even possible!
The entire process was very painful, lots of money and time were wasted for trial and error. As the CTO of our company (SudoMaker), I did this sorely as my own personal interest and didn’t spent much time caring for the company over the 2 years. And our company isn’t in a good condition now.
However, I finally made it. This is at least a meaningful thing. Hope you can understand my feelings, enjoy this project, share it to others, and maybe support it. Let’s get over this hard time together.
trying to get nix going at current $JOB. Going to try to introduce it to speed up dev between various services that run various databases. I’ve been learning more and more about dockerTools and baking my own images with nix and it’s been both insightful and fun. I hope I can spread the joy of nix since at a previous job it didn’t get a great reception but I think it was a “pearls before swine” type thing…or more so my lack of imparting information clearly shrug
One caveat is that this causes what is called “import-from-derivation” (IFD) which muddles the clean split between evaluation and building. While it is evaluating it is forced to stop evaluating to start building, so that it can complete evaluating. This causes a lot of performance problems, especially since the amount of things you need to build to finish evaluating may be quite significant.
In general, the Nix community tries to recommend against using IFD when possible because of these problems. It is often described as a very nice footgun.
One thing to note is that builtins.fetchGit returns some revision data that might be useful:
There are whole households out there that don’t have a single graduate degree in them. Amazing, I know!
That said, I didn’t actually know LL was the one behind TLA+, so it was a useful read for me too. (Also, it turns out he actually does look somewhat like the fluffy lion on the cover of the LaTeX book!)
This is a hypertext book that I intended to be the way to learn TLA+ and PlusCal, until I realized that people don’t read anymore and I made the video course.
I only learned about his writing LaTeX after reading his TLA+ book. Dude doesn’t have much ego, and he only made his name better known when he started actively advocating for formal methods.
without googling, can you name one thing Leslie Lamport is known for?
yes: 70.1%
no: 29.9%
Not as good as I’d like, but not as bad as the article makes it sound. Granted, this is a heavily skewed demographic. I ran the same poll at work and got 0% yes.
I can kind of see why that bothered him. Many people viewed TeX as the real, manly typesetting system, and LaTeX was for weaklings. Lamport received the impression of a hippie simplifying Knuth - more an enthusiastic college lecturer than a “real” computer scientist.
OTOH LaTeX made TeX usable, and facilitated the distribution of a lot of science. That has to count for something.
Yeah, the best debugger is a good walk to think it over and maybe a bunch of print statements and then a debugger if “stuff” gets to that point.
I have a Toshiba Libretto 70CT which is still running, but with 120 Mhz CPU and 16 Mb of RAM I don’t use it much, and would love a modern version - the form factor was small, but the keyboard was still big enough to touch type on.
Oh, nice! Yeah would be great if you could get on the net with that bad boy. So slick.
I similarly like small devices. A lot of people use the GPD devices for retro gaming and there’s a lot of competition in the handheld retro gaming consoles. I recently got a Anbernic RG552 because it’s based on a Rockchip RK3399 chipset which I have some experience with. I got a mainline Linux kernel running on it and have full NixOS:
Photo
https://github.com/puffnfresh/nix-files/blob/master/machines/rg552/configuration.nix
Now I need to figure out find a window manager which works with joysticks OR I’ll have to write my own :)
How do you control it – connect peripherals?
The window manager I’m currently running is Phosh, which is designed for phones and tablets. The RG552 has a touchscreen so it works fairly well.
It has a USB-C port on top so I can connect a keyboard.
It has internal WiFi and is running SSH. The WiFi driver for the chip is awful but it’s just connected via USB so can be swapped:
https://www.patreon.com/posts/63637882
I’ll soon swap it for a chip which also has Bluetooth so I can more easily connect a keyboard.
Nice! Hell yeah. Nix forever!
The OP mentions how the Micro reminds him of his once daily driver the EeePC 1000HE. Now, I recently became obsessed with the EeePC line because it’s small, can easily run Alpine Linux and X11, and just looks cool. Parts are easy to come by and it’s hackable beyond the standard upgrades for the true diehards. But OP hits it on the head when he says
I love my little Eee PC 900 (Pepsi, I call it) and I’m glad OP recaptured that feeling with hardware with his Micro too.
I found this too when traveling my EeePC - forces me to do what I think I should be doing more of: writing!
Nice to meet a fellow enthusiast. I have a white one that I use for writing, making notes and light coding on the couch.
I found it to be too slow and too small for X with the distros I tried and I settled for text only.
tmuxis your friend. I even have a half finished blog post lying somewhere about FDD: framebuffer driven development. It really does wonders for you concentration!Indeed, I don’t find myself within X often but netsurf works well and giving it a bit more RAM helped. Otherwise, I’m just using
screenand jumping between virtual consoles (if need be) - also lynx (once you have the settings to your liking) is great on these EeePCsWhat a great writeup. Here’s why it hits the nail on the head. TDD on its own is great. It gets people talking about the design of their code. It gets people caring about correctness. But the “TDD maximalist” position is unhelpful, arrogant, annoying, and flat out incorrect.
Uncle Bob’s “TDD is double entry bookkeeping” doesn’t make any sense. The reason double entry bookkeeping works is because accounting is zero-sum. Money must move between accounts, because it’s a finite resource. If you write a test, code can be written to pass it in an infinite number of ways. Computation is not a finite resource. The analogy is not good.
The TDD maximalist position that “TDD improves design because the design ends up more testable that way” is the definition of circular reasoning. I loved the point about long functions sometimes being the better abstraction - this is absolutely true in my experience. Artificially separating a concept that’s just inherently complex doesn’t improve anything objectively, it just gives someone with an obsession over small functions an endorphin hit. The same is true of dependency injection and other TDD-inspired design patterns.
I know tons of people who have done TDD for years and years and years. As mentioned in the article, it has a lot of great benefits, and I’ll always use it to some degree. But in those years working with these other people, we have still always had bug reports. The TDD maximalist position says we were just doing it wrong.
Well, if a methodology requires walking a tightrope for 5 years to get the return on investment, maybe it’s not a perfect methodology?
Uncle Bob is to software what Sigmund Freud is to psychology
I don’t think this is right: while Freud has largely been discarded in the details, his analytical approach was utterly new and created much of modernity. Uncle Bob … well, the less said, the better.
Just read Otto Rank.
While I generally agree with your take, I’m going to quibble with part of it:
This is not circular reasoning, though the inexact phrasing probably contributes to that perception. A more exact phrasing may help make it clear that this isn’t circular:
Put this way, this may be less controversial: it allows room for both those who subscribe to TDD, and for those who object to it. (Possible objections include “all else can’t be equal” – I think DHH’s comments on Test-induced Design Damage fall in this category – or that other practices can be as good or better at generating comprehensive tests, like TFA’s discussion of property-based testing.)
This is why the reasoning is circular. This point right here can be debated, but it’s presented as an axiom.
TDD does nothing inherently, this is another false claim. It does not reject any design. In practice, programmers just continuously copy the same test setup code over and over and do not care about the signal that the tests are giving them. Because TDD does not force them to. It happily admits terrible designs.
It does reject designs: designs that cannot be tested cannot be generated when following strict TDD, and designs which are difficult to test are discouraged by the practice. (Edit to add: In practice, this requires developers to listen to their tests, to make sure they’re “natural” to work with. When this doesn’t happen, sure, blindly following TDD probably does more harm than good.)
Sure, there’s some design out there that won’t even support an end to end test. The “naturalness” of a test though can’t be measured, so the path of least resistance is to just make your tests closer and closer to end to end tests. TDD itself will not prevent that natural slippage, it’s only prevented by developer intervention.
That’s the TDD maximalist point of view is a nutshell - if you don’t get the alleged benefits of it, you did it wrong.
For what it’s worth, I’m not a TDD maximalist, I try to be more pragmatic. I’m trying to have this come across in some of the language I’m using: non-absolute terms, like “encourage” or “discourage”, instead of absolutes like “allow” or “deny”. If you’re working with a code base that purports to follow TDD, and you’re letting your unit tests drift into end-to-ends by another name, then I’d argue that you aren’t listening to your tests (this, by the way, is the sense in which I think “naturalness” can be measured – informally, as a feeling about the code), and that (I’m sorry to say, but this does sometimes happen) you’re doing it wrong.
Doing it right doesn’t necessarily mean using more TDD (though it might!), it’s just as easy to imagine that you’re working in a domain where property-based tests are a better fit (“more natural”) than the style of tests generated via TDD, or that there are contextual constraints (poorly-fitting application framework, or project management that won’t allow time to refactor) that prevent TDD from working well.
I’m mostly with you. I’m definitely what Hillel refers to in this article as a “test-first” person, meaning I use tests to drive my work, but I don’t do strict TDD. The way that people shut their brains off when talking about TDD kills me though.
Well, yeah, that’s why “maximalist TDD” has such a mixed reputation…
First, there are plenty of cases where you’re stuck with various aspects of a design. For example, if you’re writing a driver for a device connected over a shared bus, you can’t really wish the bus out of existence, so you’re stuck with timing bugs, race conditions and whatever. In these cases, as is often the case with metrics, the circular thing happens in reverse: you don’t get a substantially more testable design (you literally can’t – the design is fixed), developers just write tests for the parts that are easily tested.
Second, there are many other things that determine whether a design is appropriate for a problem or not, besides how easy it is to implement it by incrementally passing automatic unit tests written before the code itself. Many of them outweigh this metric, too – a program’s usefulness is rarely correlated in any way to how easy it is to test it. If you stick to things that The Practice considers appropriate, you miss out on writing a lot of quality software that’s neither bad nor useless, just philosophically unfit.
I’ve written device drivers and firmware code that involved shared buses, and part of the approach involved creating a HAL for which I could sub out a simulated implementation, allowing our team to validate a great deal of the logic using tests. I used an actor-model approach for as much of the code as possible, to make race conditions easier to characterize and test explicitly (“what happens if we get a shutdown request while a transaction is in progress?”). Some things can’t be tested perfectly – for example we had hardware bugs that we couldn’t easily mimic in the simulated environment – but the proportion of what we could test was kept high, and the result was one of the most reliable and maintainable systems I’ve ever worked on.
I’m not trying to advocate against DFT, that’d be ridiculous. What I’m pointing out is that, unless DFT is specifically a goal at every level of the design process – in these cases, hardware and software – mandating TDD in software without any deliberation higher up the design chain tends to skew the testability metric. Instead of coming up with a testable design from the ground up, developers come up with one that satisfies other constraints, and then just write the tests that are straightforward to write.
Avoiding that is one of the reasons why there’s so much money and research being poured into (hardware) simulators. Basic hiccups can be reproduced with pretty generic RTL models (oftentimes you don’t even need device-specific logic at the other end), or even just in software. But the kind of scenarios that you can’t even contemplate during reviews – the ones that depend on timing constraints, or specific behavior from other devices on the bus – need a more detailed model. Without one, you wind up not writing tests for the parts that would benefit the most from testing. And it’s what you often end up with, because it’s hard to come up with reliable test models for COTS parts (some manufacturers do publish these, but most don’t). That’s not to say the tests you still get to write are useless, but it doesn’t have much of an impact over how testable the design is, nor that there aren’t cheaper and more efficient ways to attain the same kind of correctness.
My experience with TDD development in this field has been mixed. Design teams that emphasize testability throughout their process, in all departments, not just software, tend to benefit from it, especially as it fits right in with how some of the hardware is made. But I’ve also seen TDD-ed designs with supposedly comprehensive tests that crashed if you so much as stared at the test board menacingly.
The way they phrased that does make it kind of circular, but we can phrase it differently to avoid being circular:
End to end testing has more coverage per test case though. So, by this logic, you would just write only end to end tests, which is in contrast to the isolated unit testing philosophy.
I’m not sure that actually is in opposition to TDD philosophy, though. My understanding of TDD is that it’s a pretty “top-down” approach, so I imagine you would start with a test that is akin to e2e/integration style before starting work on a feature. A “fundamentalist” would leave just about all of the implementation code “private” and not unit test those individual private functions unless they became part of the exposed public API of the software. A more pragmatic practitioner might still write unit tests for functions that are non-trivial even if they are not public, but I still think that TDD would encourage us to shy away from lots of unit tests and the “traditional” test pyramid.
I could be totally off base with my understanding of TDD, but that’s what I’ve come away with from reading blogs and essays from TDD advocates.
End to end tests can have difficulty with “testability”, IMO:
Edit to add: This is probably giving the wrong impression. I like E2E tests, whole system behavior is important to preserve, or to understand when it changes, and E2E’s are a good way of getting to that outcome. But more narrowly targeted tests have their own benefits, which follow from the fact that they’re narrowly targeted.
I was going to object along similar lines - but I agree with the sibling - your 1) does not make a good argument.
I’d say more something like: A testable design is better, because it allows for writing tests for bugs, helping document fixes and prevent regressions. Tests can also help document the business logic, and can improve the overall system that way too.
Just saying tests are good; let’s have more tests! - Doesn’t say why tests are good - and we do indeed get circular reasoning.
I’ve inherited a number of legacy systems with no tests - and in the few I’ve shoe-horned in tests with new features and bug fixes - those have always cought some regressions later. And even with the “wasted” hours wrangling tests to work at all in an old code base; I’d say tests have “paid their way”.
If those code bases had had tests to begin with, I think a) the overall designs would probably have been better, but perhaps more importantly b) writing tests while fixing bugs would have been easier.
I also think that for many systems, just having unit tests will help a great deal with adding integration tests - because some of the tooling and requirements for passing in state/environment (as function arguments, mocks, stubs etc) is similar.
I do consider point 1 nearly axiomatic. What does “testable” mean? Trivially, it means something like “easy to test”, which suggests properties like:
Achieving these requires that the interface under test be pretty easy to understand. All else equal (whatever that might mean when talking about design), having these qualities is better than not having them. (Note, also, that I talk more about “testability”, not “tests”: I value testability more than I value tests, though TDD does strongly encourage testability.)
Still, I don’t believe this is circular. Point 1 is presented axiomatically, but the remaining points that attempt to characterize the meaning only depend on earlier points, without cycles.
Struggling with work. I’m not problem solving fast enough. I’ve done a bunch of practice and will do more so “practice more” doesn’t seem to be helping.
I may need to do a career pivot but I’ve had a chunk of my identity wrapped up in “I’m a guy who writes code” so I’m not quite ready to give up yet.
I’m very good at simple process oriented scripting, which is how I spent most of my career up to now. I thought if I had the chance to do more complex software engineering I’d rise to the occasion but I’m not there yet :)
What kind of problems aren’t being solved “fast enough”? I’m curious because I am (again) deciding if pretending to enjoy $WORK is worth the effort. Sometimes I think of Lesane and hear them:
In my case it’s problem solving around debugging complex code problems, and also being able to pick up new technologies and glue them together rapidly enough.
Sorry you’re not happy with your current gig.
Ha, I think I’m unhappy at any gig at this point. But, yeah, I think this sounds like your cup of tea! Don’t doubt yourself!
If I may… maybe try on “I’m a guy who solves problems and/or builds things” or something like that, and see how that fits? Writing code is just one way of building things or solving problems.
Thank you very much that perspective is very helpful.
The next bit, once I have my own head on straight, is figuring out how I should be piloting my own career in a way which will both allow me to thrive and continue to put bread on the table for my family :)
Still drinks 8 or 9 Diet Cokes a day?! I guess it’s true, that stuff does make one goes crazy later in life. I will say, insightful to hear his thoughts on debuggers and interesting cultural thing about “big companies” disdaining them and their Linux bias.
Thank you! This is awesome and will make it (one hopes) easier to introduce nix in an enterprise setting / work with nodejs projects.
I don’t buy this. Authors generally don’t extend e.g. English with new words or grammar in order to write a novel. Programmers generally don’t need to extend a programming language with new words or rules in order to write a program.
A programming language, like a spoken/written language, establishes a shared lexicon in which ideas can be expressed by authors and widely understood by consumers. It’s an abstraction boundary. If you allow authors to mutate the rules of a language as they use it, then you break that abstraction. The language itself is no longer a shared lexicon, it’s just a set of rules for arbitrarily many possible lexicons. That kind of defeats the purpose of the thing! It’s IMO very rare that a given work, a given program, benefits from the value this provides to its authors, compared to the costs it incurs on its consumers.
Maybe you would disagree, but I would argue that functions are essentially new words that are added to your program. New rules do seem to be a bit less common though.
This would be my interpretation as well. Steele defines a language to be approximately “a vocabulary and rules of meaning”, and clearly treats defining types and functions as to be adding to that vocabulary throughout his talk. My (broad) interpretation of a generalized language is based on that idea that the “language” itself is actually just the rules of meaning and all “vocabulary” or libraries are equal whether they be “standard” or “prelude” or defined by the user.
I understand the perspective that functions (or classes, or APIs, or etc.) define new words, or collectively a new grammar, or perhaps a DSL. But functions (or classes, or APIs, or etc.) are obliged to follow the rules of the underlying language(s). So I don’t think they’re new words, I think they’re more like sentences, or paragraphs.
FWIW a thing that shitposters on Tumblr have in common with William Shakespeare is the act of coining new vocabulary and novel sentence structure all the time. :)
See: 1984, full of this
Also, Ulysses no?
The author is saying that simple systems need this extensibility to be useful. English is far from small. Even most conventional languages are larger than the kernel languages under discussion, but chief argument for kernel languages is their smallness and simplicity. And those languages are usually extended in various ways to make them useful.
I would say, and I think that the author might go there too, that certain libraries that rely heavily on “magic” (typically ORMs) also count to some degree as language extensions. ActiveRecord and Hibernate, for instance, use functionality that is uncommonly used even by other practitioners in their respective languages.
The article is about languages, right? Languages are a subset of systems. The abstraction established by a language is by definition a shared context. A language that needs to be extended in order to provide value doesn’t establish a shared context and so isn’t really a language, it’s a, like, meta-language.
Not really.
It’s about using languages as a case study in how systems can be of adequate or inadequate complexity to the tasks they enable their users to address, and how if a tool is “simple” or perhaps inadequately complex that the net result is not that the resulting system is “simple” but as @dkl speculates above that the complexity which a tool or system failed to address has to live somewhere else and becomes user friction or – unaddressed – lurking unsuitability.
This creates a nice concept of “leverage” being how a language or system allows users to address an adequate level of complexity (or fails to do so), and begs how you can measure and compare complexity in more meaningful and practical terms than making aesthetic assessments both of which I want to say more about later.
Right.
I suppose I see languages as systems that need to have well-defined and immutable “expressibility” in order to satisfy their fundamental purpose, which isn’t measured in terms of expressive power for authors, but rather in terms of general comprehension by consumers.
And, consequently, that if a language doesn’t provide enough expressivity for you to express your higher-order system effectively, the solution should be to use a different language.
Reasonable people may disagree.
If you consider that each project (or team) has slightly different needs, but there aren’t that many slightly different language dialects out there that add just one or two little features (thankfully!) that these projects happen to need. Sometimes people build preprocessors to work around one particular lack of expressibility in a language (for one well-known example that’s not an in-house only development, see yacc/bison). That’s a lot of effort and produces its own headaches.
Isn’t it better to take a good language that doesn’t need all that many extensions, but allows enough metaprogramming to allow what your team needs for their particular project? This allows one to re-use the 99% existing knowledge about the language and the 1% that’s different can be taught in a good onboarding process.
Nobody in their right mind is suggesting that projects would be 50% stock language, 50% custom extensions. That way lies madness. And indeed, teams with mostly juniors working in extensible languages like CL will almost inevitably veer towards metaprogramming abuse. But an experienced team with a good grasp of architecture that’s eloquent in the language certainly benefits from metaprogrammability, if even it’s just to reduce the amount of boilerplate code. In some inherently complex projects it might even be the difference between succeeding and failing.
I do think the industry tends to be dominated by junior-friendly programming languages, as if the main concern was more about increasing headcount than it was about expressing computation succinctly and clearly.
Paul Graham made roughly that claim when he wrote about why viaweb could only have been written in common lisp, but I think his numbers were more like 70/30. But you did qualify it with only people in their right mind, so maybe that doesn’t count.
YMMV, but I don’t think so. My experience with metaprogramming in industry has been negative. I find it usually adds more complexity than it removes. And I don’t really agree with the framing that a language should be extensible, at the ·language· level, by its users. I see that as an abstraction leak.
I’d put that in the “hell is other people” basket - indeed, very often metaprogramming gets abused in ways that make code more complicated than it has to be. So you definitely have a point there. But then I’ve seen horror shows of many other kinds in “industrial” codebases in languages without metaprogramming. In the wrong hands, metaprogramming can certainly wreak more havoc, though!
Still, I wouldn’t want to go back to languages without metaprogramming features, as I feel they allow me to express myself more eloquently; I prefer to say things more precisely than in a roundabout way.
I never considered that perspective, but it makes sense to view it that way. In my neck of the woods (I’m a Schemer), having an extensible language is typically considered empowering - it’s considered elitist to assume that the developer of the language knows everything perfectly and is the only one who should be allowed to dictate in which direction the language grows. After all, the language developer is just another fallible programmer, just like the language’s users are.
As a counterpoint, look at the featuritis that languages like Python and even Java have spawned in recent years. They grew lots of features that don’t even really fit the language’s style. Just because something’s gotten popular IMHO doesn’t necessarily mean it ought to be put in a language. It burdens everyone with these extra features. Think about C++ (or even Common Lisp), where everyone programs in a different subset of the language because the full language is just too large to comprehend. But when you want to leverage a useful library, it might make use of features that you’ve “outlawed” in your codebase, forcing you to use them too.
There’s also several examples of the Ruby and Python standard libraries having features which have been surpassed by community libraries, making them effectively deprecated. Sometimes they are indeed removed from the standard library, which generates extra churn for projects that were using them.
I’d rather have a modestly-sized language which can be extended with libraries. But I readily admit that this has drawbacks too: let’s say you choose one of many object/class libraries in an extensible language which doesn’t supply its own. Then if you want to use another library which depends on a different class library, you end up with a strange mismatch where you have entirely different class types, depending on what part of the system you’re talking to.
Sure! In pure quantitative terms, I agree: I’ve seen way more awful Java (or whatever) than I have awful metaprogramming stuff. But the bad Java was a small subset of the overall Java I’ve witnessed, whereas the bad metaprogramming was practically 100% of the overall metaprogramming I’ve witnessed.
I fully acknowledge that I’m biased by my experience, but with that caveat, I just haven’t seen metaprogramming used effectively in industry.
This is an important point. You are allowing authors to add new syntax, not requiring it. You can do most of the same tricks in Common Lisp or Dylan that you can do in Javascript or Kotlin, by passing funargs etc., it’s just that you also have the ability to introduce new syntax, within certain clearly defined bounds that the language sets.
Just as you have to learn the order of arguments or the available keywords to a function, Lisp/Dylan programmers are aware that they have to learn the order of arguments and which arguments are evaluated for any given macro call. Many of them look exactly like functions so there’s no difference. (I like that in Julia, as oppose to Common Lisp and Dylan, macro calls must start with the special character “@”, since it makes a clear distinction between function calls and macro calls. But I don’t know much about Julia macros.)
Yes, and I don’t believe macros change this significantly, although this depends to a certain extent on the macro author have a modicum of good taste. The bulk of Common Lisp and Dylan macros are one of two flavors:
“defining macros” – In Common Lisp these are usually named “defsomething” and in Dylan “define [adjectives] something”. When you see
define frame <chess-board> (<frame>) ... endyou know you will encounter special syntax becausedefine frameisn’t part of the core language. So you go look it up just as you would lookup a function for which you don’t know the arguments.“with…” or “…ing” macros like “with-open-file(…)” or “timing(…)”
These don’t change the complexity of the language appreciably in my experience and they do make it much more expressive. (Think 1/10th to 1/15th the LOC of Java here.)
Where I believe there is an issue is with tooling. Macros create problems for tooling, such as including the right line number in error messages (since macros can expand to arbitrarily many lines of code), stepping or tracing through macro calls, recording cross references correctly for code browsing tools, etc.
Do you not see code that defines new syntax as categorically different than code which uses defined syntax?
What is a language if not a well-defined grammar and syntax?
I don’t see how something which permits user modification of its grammar/syntax can be called a language. It’s a language construction set, maybe?
This seems founded in the idea that if you just know the language, you can look at code and understand what it does. But this is always convention. For instance, consider the classic C issue of “who owns the pointer passed to a function?” Even with an incredibly simple language, there’s ambiguity about what conventions surround a library call - do you need to call free()? Will the function call free()? Can you pass a stack pointer, or does it have to be heap allocated? And so on. More powerful type systems can move more information into the type itself, but more powerful types tend to be included in more powerful languages; for instance, in D, if you pass an expression to a function, if the parameter is marked as lazy, the expression may actually be evaluated any number of times, though it’s usually zero or one, and you have no idea when the evaluation takes place. So just from looking at a function call,
foo(bar), it may be that bar is evaluated before foo, or during foo, or multiple times during foo, or never.Now a macro could do worse, sure, but to me it’s a difference of degree, not kind. There’s always a spectrum, and there’s always ambiguity, and you always need to know the conventions. Every library is a grammar.
The devil’s in the details. But when I’m answering questions like “who calls
free()?” I don’t need to re-evaluate my understanding of language keywords or sigils or operator semantics. I see this as a categorical difference. I suppose other people may not.I think it’s worth addressing the elephant in the room: there’s language as in formal language (mathematics) and then there’s language as reasoned by linguistics, which also acknowledges and studies the sociopolitical and anthropological aspects of language. It may be useful to analyze PLs through the lens of the former, but programming languages also do an awful lot of things that human languages do: give rise to dialects, have governing bodies, have speakers that defy governing bodies, borrow from each other, develop conventions, nurture communities, play host to ecosystems, and so forth.
We now arrive at the core of your assertion, which is that there is a hard line between syntax that comes for free and syntax that is defined by the userland programmer. This is part of the very same line that the author is asking us to imagine blurring:
It is furthermore dangerous to equate the malleability of syntax with the extensibility of a language. Macros are a form of extensibility, yet conversely, extensibility does not require a support for macros. Take Python, a language that does not support macros, for example. Python allows you to engage in so much operator overloading that you could read the expression
(a + b) << cand still have no clue what it actually does without looking up how the types ofa,b, andcdefine such operations. Ultimately, it is conventions — be they implicit or documented, be they cultural or tool-enforced — that dictate the readability of a language, just as @FeepingCreature has demonstrated with multiple examples.The author covers a great deal of the triumphs and tragedies of language extensibility in the “Convenience? Or simplicity through design?” section. The monstrosity of the Common Lisp
LOOPmacro, as well as userlandasync/awaitengines, are brought up because they brutally defeat static analysis, whereas the.usemethod in Kotlin is shown as an example that manages to accomplish all of the needs ofWITH-FOOmacros with none of the ill effects of macros. In fact, there is another commonly used language that manages to achieve this the same way Kotlin does: Ruby. In Ruby, any method call can take an additional block of code, which gets materialized into aProcobject, which is a first-class function value. This means in Ruby it is routine to write code that looks likeFile.open("~/scratch.txt") { |f| f.write("hello, world") }. This is an example straight out of the Ruby standard library, and it is the convention for resource management across the entire Ruby ecosystem.Now, even though Ruby — like Python — does not support macros, and even though it got resource management “right”, just like Kotlin did, it has nevertheless managed to dole out some of the most debilitating effects of metaprogramming. Ask anyone who’s worked on a Rails project, myself included, and they will recount how they were victimized by runtime-defined methods whose names were generated by concatenating strings and therefore remain nigh unsearchable and ungreppable. A great deal of research and literature have been dedicated to making Ruby more statically analyzable, and they all converge towards the uncomfortable conclusion that its capability of evaluating strings as code at runtime and its Smalltalk-like object system both make it incredibly difficult for machines to reason with Ruby code without actually running it.
Programming languages can be analyzed in the formal/mathematical sense, but when they are used, they are used by humans, and humans necessarily don’t understand languages through this lens. I’m not saying a programming language is strictly e.g. linguistic, but I am saying that it cannot be effectively described/modeled purely via formal or mathematical means. A language isn’t an equation, it’s an interface between humanity and logic.
I would accept that Common Lisp can be called a language construction set but I don’t see how it’s useful or accurate to say it’s not a language.
Well, no, but it helps that English language already has innumerable – legion, one might say – words, an English dictionary is a bountiful, abundant, plentiful universe teeming with sundry words which others have already invented, not always specifically for writing a whole novel, often just in order to get around IRL. It’s less obvious with English because it’s not very agglutinative but it has been considerably extended in time.
Language changes not only through the addition of new words and rules but also through other words and rules falling out of use (e.g. in English, the plural present form of verbs lost its inflection by the 17th century or so), so it’s not quite fair to say that modern English is “bigger” than some of its older forms. But extension is a process that happens with spoken languages as well.
It’s also super frequent in languages that aren’t used for novels and everyday communication, too. At some point I got to work on some EM problems with a colleague from a Physics department and found that a major obstacle was that us engineers and them physicists use a lot of different notations, conventions, and sometimes even words, for exactly the same phenomenon. Authors of more theoretical works regularly developed their own (very APL-like…) languages that required quite some translation effort in order to render them into a comprehensible (and extremely verbose) math that us simpletons could speak.
(Edit: this, BTW, is in addition to all that stuff below, which @agent281 is pointing out, I think the context of the original talk is relevant here).
If I had to guess I would think that this was a nudge to Scheme programming language from Steel’s point of view (or Common Lisp, but this one is quite large in comparison). Quite easy to extend Scheme that way and he does have a history with lisp-family languages. But that’s just a guess.
Thanks for this. The video (2012) linked at the end was also super helpful. As someone that’s been working on an older go project for a short time, I do feel there’s mocks/fakes/whathaveyou that maybe shouldn’t have been written and went looking for some content akin to this but wasn’t coming up with much.Thanks again.
I have a ipxe script burned onto most of the Intel nics in my house that boot directly to netboot.xyz for the last two or three years. I use it off and on when I’m to lazy to find my ventoy based thumb drive. It’s great in a pinch. This is the very basic script I have running on my cards.
This script loads code over plaintext HTTP and then uses that untrusted code to boot. That seems… very sketchy.
Why not at least use HTTPS? (Is this an iPXE limitation?)
Which CAs would you trust in that case? A local, or ALL of the public CAs?
Look, PKI/the CA cabal clearly isn’t perfect. In fact it’s extremely flawed, especially if you aren’t enforcing Certificate Transparency which presumably iPXE isn’t. But I think it’s really silly to imply that HTTPS doesn’t raise the bar for the attacker? At minimum they now have to actually compromise a CA instead of being able to just set up an ordinary HTTP proxy and serve malicious images.
PKI is scary enough that nobody dares to even touch it, but one person at each department/company/whatever who Is The PKI Guy who everybody expects to know everything and fix everything remotely related to PKI. (at least from my experience)
I think many, many other security measures will fail before lack of TLS will be an issue. But it is still very cool that it’s even possible!
Yes, HTTPS is supported
A bunch of other documentation was recently uploaded as well.
https://metacpan.org/pod/Time::Piece
Yes, indeed. I ended up there as well. Nevertheless, was fun to see this little tidbit and see why perl is, for lack of a better word, strange.
Strange? More like it’s POSIX that’s strange.
PHP’s
localtimehas the same “issue”.Glad it was all worth it.
I didn’t happen to see the last bit:
trying to get nix going at current $JOB. Going to try to introduce it to speed up dev between various services that run various databases. I’ve been learning more and more about dockerTools and baking my own images with nix and it’s been both insightful and fun. I hope I can spread the joy of nix since at a previous job it didn’t get a great reception but I think it was a “pearls before swine” type thing…or more so my lack of imparting information clearly shrug
One caveat is that this causes what is called “import-from-derivation” (IFD) which muddles the clean split between evaluation and building. While it is evaluating it is forced to stop evaluating to start building, so that it can complete evaluating. This causes a lot of performance problems, especially since the amount of things you need to build to finish evaluating may be quite significant.
In general, the Nix community tries to recommend against using IFD when possible because of these problems. It is often described as a very nice footgun.
One thing to note is that builtins.fetchGit returns some revision data that might be useful:
Thanks, that makes sense and good to know re: fetchGit
Them: “Leslie Lamport may not be a household name,[…]”
Me: “The hell he isn’t.” [rage close]
(I opened it back up and read it anyway. It was actually really interesting. But my rage close was real.)
There are whole households out there that don’t have a single graduate degree in them. Amazing, I know!
That said, I didn’t actually know LL was the one behind TLA+, so it was a useful read for me too. (Also, it turns out he actually does look somewhat like the fluffy lion on the cover of the LaTeX book!)
Yeah, I knew he did Lamport clocks but didn’t know he was also the guy who did LaTeX and TLA.
Inverse for me, I never made the connection between LaTeX Lamport and clock Lamport.
One did happen before the other.
Can we really be sure about that.
I only learned about his writing LaTeX after reading his TLA+ book. Dude doesn’t have much ego, and he only made his name better known when he started actively advocating for formal methods.
I saw that and ran a Twitter poll
Not as good as I’d like, but not as bad as the article makes it sound. Granted, this is a heavily skewed demographic. I ran the same poll at work and got 0% yes.
And even then, people tend to know him more for LaTeX than for his much more important work on distributed computing. Which I’ve heard bothered him.
I imagine that there are more people writing professional documents than working on distributed computers.
On the other hand, I imagine that there are more people using distributed computers than people writing documents. Probably.
I can kind of see why that bothered him. Many people viewed TeX as the real, manly typesetting system, and LaTeX was for weaklings. Lamport received the impression of a hippie simplifying Knuth - more an enthusiastic college lecturer than a “real” computer scientist.
OTOH LaTeX made TeX usable, and facilitated the distribution of a lot of science. That has to count for something.
I know him for Lamport clocks and not much else :)
Heheh. I saw the Twitter poll and thought of this story. I follow you on Twitter now.