This post is largely a response to The myth of the Lisp genius and the ensuing conversation on reddit. In particular, I want to respond to the apparent confusion about where the claimed increased productivity could come from in a language like Lisp versus a more traditional language like C, Java or C#.
For me, the core of the productivity gain comes from this statement:
Lisp is a programmable programming language.
- John Foderaro, CACM, September 1991
Consider how natural languages affect your communication with others and your own thought processes. Without a proper shared vocabulary and cultural idioms, it can be very a cumbersome process to express complex and nuanced ideas. It's generally a trial and error process of statement->paraphrase->restatement until both parties feel they've come to a shared understanding. Our mind can even have trouble holding on to concepts that we can't adequately express in language.
Programming a system can feel much the same, when you first start out. Code, test, recode until you feel the code is doing what you intend. As a programmer works on a system, he or she will be developing a cognitive model of the concepts involved. This will in turn lead to a whole vocabulary and language that programmers can use when discussing the system with one another. But, in non-programmable programming languages, they're still required to dumb-it-down when actually writing out the code. This constant re-specification of higher-level concepts in the lower-level language can sometimes lead to bugs in the writing and can inhibit quick recognition of the higher-level intent when reading code. It can also tether the entire thought process to the level of the language.
On the other hand, Lisp allows the programmer to raise the level of abstraction in the programming language itself, so that you can more directly express in code the cognitive model that's been developed, which makes the entire programming process much quicker. This ability largely stems from two facts of Lisp. First, it's syntax is extremely simple. When writing Lisp code, you're largely writing abstract syntax trees serialized in nested lists, rather than expressions that have to be parsed into ASTs. Second, Lisp allows you to write code at several layers of the language. All programming languages allow you to specify code to run at execution time. Lisp also allows you to write code that will be executed by the compiler to generate the expression which should be compiled. Since the generated Lisp code is just a bunch of ASTs serialized to nested lists, and Lisp is very good at LISt Processing, it's relatively easy to write these code generators.
When you combine these two things, a very regular syntax with seamless usage of code generators, then it becomes much easier to add vocabulary and idioms that blend right in with the language, making the code much more straightforward to read, write and reason about.
