Re: The myth of the Lisp Genius

This post is largely a response to The myth of the Lisp genius and the ensuing conversation on reddit. In particular, I want to respond to the apparent confusion about where the claimed increased productivity could come from in a language like Lisp versus a more traditional language like C, Java or C#.

For me, the core of the productivity gain comes from this statement:

Lisp is a programmable programming language.

- John Foderaro, CACM, September 1991

Consider how natural languages affect your communication with others and your own thought processes. Without a proper shared vocabulary and cultural idioms, it can be very a cumbersome process to express complex and nuanced ideas. It's generally a trial and error process of statement->paraphrase->restatement until both parties feel they've come to a shared understanding. Our mind can even have trouble holding on to concepts that we can't adequately express in language.

Programming a system can feel much the same, when you first start out. Code, test, recode until you feel the code is doing what you intend. As a programmer works on a system, he or she will be developing a cognitive model of the concepts involved. This will in turn lead to a whole vocabulary and language that programmers can use when discussing the system with one another. But, in non-programmable programming languages, they're still required to dumb-it-down when actually writing out the code. This constant re-specification of higher-level concepts in the lower-level language can sometimes lead to bugs in the writing and can inhibit quick recognition of the higher-level intent when reading code. It can also tether the entire thought process to the level of the language.

On the other hand, Lisp allows the programmer to raise the level of abstraction in the programming language itself, so that you can more directly express in code the cognitive model that's been developed, which makes the entire programming process much quicker. This ability largely stems from two facts of Lisp. First, it's syntax is extremely simple. When writing Lisp code, you're largely writing abstract syntax trees serialized in nested lists, rather than expressions that have to be parsed into ASTs. Second, Lisp allows you to write code at several layers of the language. All programming languages allow you to specify code to run at execution time. Lisp also allows you to write code that will be executed by the compiler to generate the expression which should be compiled. Since the generated Lisp code is just a bunch of ASTs serialized to nested lists, and Lisp is very good at LISt Processing, it's relatively easy to write these code generators.

When you combine these two things, a very regular syntax with seamless usage of code generators, then it becomes much easier to add vocabulary and idioms that blend right in with the language, making the code much more straightforward to read, write and reason about.

EclipseCon 2011 and High Performance Tabular Databinding

This past November, I was honored to have my talk, High Performance Tabular Databinding, be selected from amongst the first batch of EclipseCon 2011 Audition Sessions. Now, the conference proper is fast approaching, so I'm working to finalize the materials for the full talk. I've already watched the recommended video series, and am taking Scott's suggestions to heart in order to eliminate all suck from my talk. As I progress in the prep, I'm also noticing that I had originally shot for far more information than I feel could be properly covered in 20 minutes, so I'm likely to remove the NatTable/XViewer comparison and focus simply upon databinding with SWT Tables. That'll give me the opportunity to dive more deeply into my methods for gathering performance data, and what I found in that data. I'd be interested in hearing from anyone considering attending the talk though. What would make the 20 minutes most worthwhile for you? All in all, I'm very excited to see you all in San Jose in a couple weeks.

Android vs iOS: A(nother) Developer’s Perspective

While reading Android vs iOS: A Developer’s Perspective from the Whereoscope Blog, I realized a few things that I'd like to share about my own perspective on Android vs iOS development.

For me, the single biggest win of the Android platform is its openness. I don't just mean the freely available documentation (and source) for the platform, though that is a big part of it. What I mean more is the ability for a hobbyist like myself to get software running on the device itself. When I got my iPhone, the first thing I wanted to do with it was write an app for it. I was a bit late to the iPhone party, so the SDK was already out (in fact, the availability of the SDK was one of the final straws that convinced me to buy one), but I was extremely disappointed to find you had to sign up for a paid developer account before Apple would allow you to test software you were writing, even on a device you (ostensibly) own. I can see charging a fee before allowing developers to upload their app to Apple's servers for dissemination through the App Store, but I'd like the ability to fully test software I'm writing on the target device during the development process. And I don't want to start paying the $99/year tithe to Apple until I'm actually ready to start marketing the app.

When I bought my Android phone, I literally had a hello world app of my own running on the device in 15 minutes. Sure, there's a fee to Google in order to host your apps on the Marketplace just as there is to host apps on the App Store, but that fee can be paid by the developer when they're ready to market the application, after they're confident it works to their satisfaction when physically installed on a device. Also, if a developer wants to have friends beta test their application, it's as simple as walking them through a configuration setting change and pointing them at a URL. Finally, Android developers have the ability completely opt-out of the Marketplace and market/sell their applications completely independently, if they so choose. That's the sort of openness (and freedom) that I care about as a developer.

Regarding the observation that "the software is of lower quality, on average," on the Marketplace, I must say that I side with Google's approach to this as well. I'd rather have a wide open Marketplace with crowd-sourced quality control than have a single gatekeeper which is ostensibly in place purely for quality control reasons, but is free to reject or delay apps indefinitely with little to no reason given. And then there's patentgate... I'm just not comfortable entrusting Apple Inc. with ultimate authority over my work.

Eclipse Databinding Aggregation

In RCP apps I'm working on, especially when displaying numeric data in a Table, I often find myself wanting to have some aggregation support built into Eclipse's databinding framework. As an example, if I have a list of Students, and each Student has a list of ExamResults, it would be nice to easily bind a TableColumn to the average of each student's exam results. To that end, I've created a new project, net.juckel.rcp.databinding.aggregate. I'm relatively new to extending the databinding framework, so I'd greatly appreciate comments on the the API design, as well as suggestions for how to extend the functionality. As a quick example (pulled from the tests on github), here's how you'd create an observable map for the example given above (average score for each student in a list).

IObservableList observableBeanList = new WritableList(studentList,
    Student.class);
IObservableSet observableBeanSet = new ListToSetAdapter(
    observableBeanList);
IObservableMap map = AggregateProperties.average(
    EMFProperties.list(ClassroomPackage.Literals.STUDENT__TEST_RESULTS)
        .values(ClassroomPackage.Literals.EXAM_RESULT__SCORE))
    .observeDetail(observableBeanSet);

C'mon, Apple!

As noted at Daring Fireball and others, Apple has made the following change in their iPhone Developer Program License Agreement.

Old:

3.3.1 — Applications may only use Documented APIs in the manner prescribed by Apple and must not use or call any private APIs.

New:

3.3.1 — Applications may only use Documented APIs in the manner prescribed by Apple and must not use or call any private APIs. Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited).

The fact that Apple refuses to allow me to install software I write onto the iPhone 3G that I "own," has already convinced me to not purchase another iPhone (or iPad), but this change makes me want to immediately blend my current iPhone just so I don't continue to add to the statistics Jobs uses in his keynotes to demonstrate the iPhone's popularity.

I'm a large fan of the hardware Apple puts out. The iPhone, iPad and Macbooks are all fantastic from a hardware viewpoint. But as AAPL has exploded over the past decade, Apple has been getting more and more brazen in their lock-in policies. Now, not only does Apple want to control the hardware platform and software platform, they also want to dictate the tools that developers use to perform their work. That's just unconscionable.

I really wanted my next laptop to be a nice, new unibody Macbook Pro (to upgrade my early 2008 model), but unless Apple makes a pretty hard about-face on this, I'll have to vote with my wallet and take my business elsewhere.

More praise for git at Eclipse

Yesterday, @iamkevb tweeted: #SWT needs help with a carbon bug on 10.6. If you have time and hardware, attach patches to http://bugs.eclipse.org/294929. I happened to have the hardware to test it, so I downloaded a few carbon builds, and succeeded in reproducing the error using the steps posted to bugzilla. Luckily, I found that the latest 3.6 integration build didn't exhibit the same error. Now, I'm no carbon programmer, so I just posted my results as a comment, hoping that someone who knew more about SWT would be able to deduce what changed between 3.5.2RC3 and the latest in the 3.6 integration build. Today, Silenio Quarti posted that he believed the error was fixed as a result of bug #275686. Now, this is where it git comes in. I only had about 30 minutes before I had to be out the door, but I figured that would be plenty of time to whether Silenio was correct. I headed over to http://dev.eclipse.org/git/ to find out the URL for the jface.text project (turned out bug 275686 was a jface change, and didn't even touch the Carbon bindings specifically).

$ git clone git://dev.eclipse.org/org.eclipse.jface/org.eclipse.jface.text.git
 [.. snip ..]
$ cd org.eclipse.jface.text
$ git log --grep='275686'
commit a7d0c902cae6382be0b0812a773f651f75a801d8
Author: mkeller 
Date:   Thu Oct 8 19:22:34 2009 +0000

    Bug 275686: [painting] Rulers should not draw outside of SWT.Paint
$ git checkout -b b294929 origin/R3_5_maintenance
$ git cherry-pick a7d0c902c
Finished one cherry-pick.
# On branch b294929
# Your branch is ahead of 'origin/R3_5_maintenance' by 1 commit.

Then I opened my 3.5.2RC3 carbon build of eclipse, imported the org.eclipse.jface.text project, and File->Export->Deployable plug-ins and fragments. Choose Install into host, and restart the workbench. Voila, Silenio was correct and that commit solved the issue. All told, the above took about 10 minutes, and left enough time for me to sing the praises of git usage at Eclipse in this blog post before I had to be out the door.

Using git to submit a patch to Eclipse

I've been getting more and more comfortable with git of late, so when it came time for me to poke around at an Eclipse bug (#130854), I decided to work off the git mirrors rather than CVS. I must say, although the Eclipse CVS tools making working with CVS relatively tolerable, using git is a huge improvement for external contributors like myself. Since I was working on org.eclipse.jface, I started with:

git clone git://dev.eclipse.org/org.eclipse.jface/org.eclipse.jface.git
git clone git://dev.eclipse.org/org.eclipse.jface/org.eclipse.jface.snippets.git

After importing those projects into my workspace, I noticed some compilation errors due to recent changes in SWT. Now, the bug I was working on was an old bug, so I didn't have to chase the latest development, so I just switched to the R3_5_maintenance branch instead.

git checkout -b b130854 origin/R3_5_maintenance

Now the project compiled just fine, so I could start poking around at the issue at hand. All told, this was probably just 3-5 minutes, as opposed to drilling through a CVS tree in Eclipse for 3-5 minutes, followed by fetching a coffee while waiting for the branch to check out (another 3-5 minutes), then another 3-5 after realizing I really wanted a different branch. At this point, it was back to a normal git workflow. I mucked around with the code, had about 3 separate commits, all broken, before I realized where I wanted to actually make the change. Just reset HEAD on my branch to ignore those commits, committed the change that actually fixed my bug, then formatted a patch file to send to bugzilla.

git add ...
git commit ...
git reset ...
# repeat until satisfied
git format-patch origin/R3_5_maintenance

All in all, a very quick and painless experience, though this would be relatively easy even using CVS. The real benefit of git would come into play if I had a longer-lived local branch (trying to add a new feature). I'd have full version control available locally to maintain my work as a patchset against the latest upstream work. A hearty thanks to everyone at Eclipse that's been pushing forward on making git repositories available. Even if projects don't migrate to using git as their canonical repo, just having a git mirror available makes working from the outside significantly easier than it has been in the past.