Tuesday, February 26, 2008

Ooh! Ooh! My turn! Why Lisp?

Brace yourself. The first reason you should be using Lisp is a non-reason, an answer to an objection, a negation, a let-down: Lisp is just an ordinary programming language.

I'll wait while you pick yourself off the floor. What a terrible place to start! But it must be said. It is the first hurdle on the road to Lisp. Xenophobia never feels like fear of the strange, it feels like dislike. But one of the best young Lispers to show up at a Lisp-NYC meeting sat down, ordered a beer, and was smart enough to make it his first question: Can you use Lisp for conventional programming? Can it read a database? Can it run a GUI?

Yep.

Lisp at one level of understanding is just a normal high level programming language. Those parentheses look quite different but constitute only a superficial difference from the conventional chicken scratch syntax of semi-colons, braces, full stops, and brackets. When a conventional programmer sits down to program Lisp, they feel quickly at home: Lisp is just another 3GL.

I feel terrible about this. I am supposed to be up on a soapbox preaching eternal salvation and ecstasies glorious and unknown and we will touch on that below but in the end your correspondent is just a simple application programmer and if you check my Road to Lisp you will discover that I (re)discovered Lisp just to find an easier way to create educational math software.

Lispers are accused of religous fanaticism and zealotry -- nope, our enthusiasm is all about getting applications built faster with less pain. Which brings me to the bad news: the next reason I have to offer for using Lisp is almost as boring as the first: Lisp is mature. Turns fifty or so in 2008. In 1994 when it was old enough to be President of the United States it got an ANSI Standard. How stable can you get?

So far we have ordinary, old, and codified and I am losing my audience. Send in the clowns!

Fine, a little pizazz: it ain't just old. The old translates to polish. Lisp did not begin with a standard and it did not begin as a single-implementation BDFL strangulation such as Python or Perl or Ruby. It began as...an experiment?

Two score and ten years ago our foregeeks started from the simple idea that code and data were one and began evolving a huge tool chest of HLL functionality spread across several competing dialects. For some reason they took the engineering so seriously that whenever a nooby picks up one of these tools surprise! surprise! it turns out to work precisely the way they need it to. It seems like magic but it is not. The magic is just a long history of use and careful refinement by very good engineers and was possible precisely because there was no standard and there was no BDFL.

Back to the yawns: most Common Lisp implementations are native compiled. Yeah, yeah, I know, computers are fast and we are all doing Web apps where network latency decides all but I work on a desktop interactive expert system that needs all the horsepower it can get and to tell you the truth when building something like that more horsepower makes more better features possible so isn't it nice to compile down to the metal?

More yawns: ephemeral GC, the best OO model you can imagine, multiple implementations to choose from including free ones.... hey, even I am getting bored. But my Road also reveals that I stumbled onto this staid dependable workhorse only after squandering mucho dollars and hours on half-baked gleams-in-the-eye programming tools that promised a world of productivity and delivered only reboots. At which point boring looks pretty damn exciting.

Left as an exercise is connecting the stable/standard/refined dot with other dots of dynamic interactive languages still emerging from sea and periodically burying their installed bases in volcanic ash. Can you scream Perl 6? Sher ya can. Has Guido finally axed first-class functions yet? It is fun watching languages evolve, but if you will excuse me I really do have an application to get out.

With the dull stuff out of the way we can indulge at long last in some Why Lisp? fireworks.

We'll build slowly, starting with Lisp being interactive, dynamic, reflective, functional, compiled, mature, standardized, has a vast array of library functions and data structures (including the wonderful and ubiquitous list), multiple value return, special variables, CLOS (including multiple inheritance and multi-methods), the MOP, first-class functions and omigod! closures, destructuring-bind, LOOP, lexical scope, untyped variables, typed data, object identity for everything but numbers and characters and now repeat a few of those things a few times to make clear how helpful they are to getting applications built, such as the vast toolchest. Gasp.

I call that starting slowly because everything above can be found in some language. It is wonderful that only Common Lisp has them all, but is there anything that really sets Lisp off? Is there some protein that cannot be mimicked by other invading dynamic language organisms, some shibboleth their tongues cannot form?

More prosaically, is there some quality those languages cannot copy without becoming Lisp?

At ILC 2002 former Lisp giant now Python advocate Peter Norvig was for some reason allowed to give the keynote address like Martin Luther leading Easter Sunday mass at the Vatican and pitching Protestantism because in his talk Peter bravely repeated his claim that Python is a Lisp.

When he finished Peter took questions and to my surprise called first on the rumpled old guy who had wandered in just before the talk began and eased himself into a chair just across the aisle from me and a few rows up.

This guy had wild white hair and a scraggly white beard and looked hopelessly lost as if he had gotten separated from the tour group and wandered in mostly to rest his feet and just a little to see what we were all up to. My first thought was that he would be terribly disappointed by our bizarre topic and my second thought was that he would be about the right age, Stanford is just down the road, I think he is still at Stanford -- could it be?

"Yes, John?" Peter said.

I won't pretend to remember Lisp inventor John McCarthy's exact words which is odd because there were only about ten but he simply asked if Python could gracefully manipulate Python code as data.

"No, John, it can't," said Peter and nothing more, graciously assenting to the professor's critique, and McCarthy said no more though Peter waited a moment to see if he would and in the silence a thousand words were said.

Come to think of it...

46 comments:

  1. On InfoQ McCarthy said the same about Ruby:
    http://www.defmacro.de/?p=13. And it's always the same answer :)

    ReplyDelete
  2. Lisp has two fundamental features.
    1) Programs and data are the same.
    2) nil eq () eq false

    Almost all other programming
    languages miss the first features.
    Scheme screwed up when the broke
    the second feature.

    Without these features, it is
    not lisp.

    Python is just one of thousands
    of languages that pretend to be
    lisp "without the parens". Python
    is a lost cause, its just too young
    to bother dying but it will.

    ReplyDelete
  3. Michael, my post was a bit more work than I would have liked but the link to McCarthy's InfoQ interview made it worthwhile a dozen times over. Thx.

    The bad news is that you made me a liar: McCarthy's hair and beard are neater than mine. And I am in deep trouble: he loves Linux, I am always ragging on it, now I have to take the pill and install Debian. :(

    There was some great stuff in there, the standout line (well, it was a little precious, but who am I to be complaining about that) being his concern that software be usable by children and generals. And his self-deprecating comment (also in violent disagreement with my Grand Unification Theory of Program Language Convergence) that the only reason new languages are copying Lisp is to be different.

    But there was one astonishing quiet little thing he did during the interview that simply leaves me in awe, and it happened in and around the bit you cited. Can anyone guess?

    Thx again, Miguel

    ReplyDelete
  4. Tim, Word up, but how about every form returning a value? Valueless statements compel the imperative. This along with the code/data thing really has me stumped about anyone as smart as Norvig even comparing Python and Lisp. Python has all sorts of wicked cool powerful stupid pet tricks and I totally get that as, yes, making apps easier to build. But to call it a Lisp? I dunno...

    ReplyDelete
  5. hmm in CL form can return no values ;)
    CL-USER> (defun foo () nil)
    FOO
    CL-USER> (length (multiple-value-list (foo)))
    1
    CL-USER> (defun foo () (values))
    STYLE-WARNING: redefining FOO in DEFUN
    FOO
    CL-USER> (length (multiple-value-list (foo)))
    0

    ReplyDelete
  6. btw, is (values) ==> "no values" a value ? ;)

    ReplyDelete
  7. It's been around fifty years and yet there are really are not that many libraries for it, particularly GUI and graphic libraries. Sure there are a few, but not as many as other languages have access to or as well designed. And what I've seen many of those libraries are not maintained.

    ReplyDelete
  8. I'm curious: what's the largest lisp program you ever wrote, how large is it, and what does it do?

    ReplyDelete
  9. Honestly. A few lines. And even then just working through examples from tutorials. I also wrote a small game using .NET and Flash , but even with that lisp was only a small part of it. But I don't see how that invalidates my point concerning GUI and graphical libraries. I can guarantee you that if LISP had better library support I would have written more than just a few lines in it.
    I'm in no way knocking lisp I think it's a great language if for no other reason than its homoiconic, I just feel that LISP would be more accepted if it had better library support. Contradicting myself a bit I've messed with New LISP and it looks promising

    ReplyDelete
  10. stever: But it seems that things are slowly getting better.
    Basic OpenGL support is there but admittedly situation with GUI toolkits could be better (though I guess Kenny has his own opinion about that). It is kinda funny that CL has some half dozen different, more or less maintained GTK bindings. I guess it is partly because most serious users aren't that much into desktop apps.
    Luckily using C libraries from CL is very easy and the results port well across implentations and operating systems.

    ReplyDelete
  11. stever: I have not hear much good of newLisp. If you want a batteries included lispy language I'd rather suggest PLT Scheme.
    But for most bang-per-buck you should really consider SBCL or one of the commercial CLs.
    (:shameless-plug "If you are interested in '(2d) games in lisp I occasionally write about the topic at www.yellow-hut.com/blog")

    ReplyDelete
  12. Flatland I'll definitely checkout yellow-hut. And I too think things are slowly getting better just wish they would hurry up a bit ;)
    I've also heard not so good things about New Lisp, but the truth is my understanding of lisp is not that in depth to understand the faults.
    Someday I hope someone develops a graphical IDE that has lisp for the backend and a flash/squeak like interface for the front end. There are some IDE out there that have different aspects of what I want like VistaSmalltalk, Factor,Flash, or squeak.

    ReplyDelete
  13. "Python is a lost cause, its just too young to bother dying but it will."

    It's so lost that it is kicking Lisp's ass in the number of useful applications created with it. And, get this--people *love* programming in it--how dare those ignorant fools!

    Which is more important? Building something interesting or navel gazing over some properties of a language?

    ReplyDelete
  14. (values) might be the perfect example of the verb "to quibble", just perfect for Usenet, not so good for talking to me. :) Or maybe you are unaware that (when (values) (print "I love quibbles!)) works like a charm? I think I have heard of something else that sets (values) off from nil, but the only one I am sure of is that it keeps you from getting the extra nil printed at the REPL. :) Come to think of it, if the REPL can tell the difference there must be some way. Anyone know? But the reason every form returning a value matters is so I can write non-imperative code meaning an if statement can be an argument to a function, and (values) works fine there so it really is a FOQ (frequently offered quibble). :)

    ReplyDelete
  15. Libraries? A couple of answers here. First of all, CFFI rocks, Lisp can use any C library. I use a very tight interface to Tcl/Tk, and others use GTk, tho I have no argument with assertions they could be better maintained. Commercial Lisps offer nice GUIs and database wrappers and sometimes great native DBs such as AlllegroCache and AllegroGraph, it's just the "free software" geniuses scraping by with weak open source libraries. :) But my main answer is that in these intial smug Lisp posts I am talking about the language itself, and that a great language will (and in this case is) eventually attract the users needed to produce even solid open source stuff, and then they will be as better as is Lisp itself because Lisp macros allow more than just bindings, they allow a higher-level API. This should probably be its own blogpost....

    ReplyDelete
  16. what's the largest lisp program you ever wrote, how large is it, and what does it do?

    Approximately 80KLOC and it managed clinical trials and you just gave me an idea for a great post to the blog. :)

    This http://www.theoryyalgebra.com/ weighs in at 36LOC with the OpenGL-based custom GUI I wrote. Navigate to the "demo" link to see what it does.

    ReplyDelete
  17. Ooops, right, (length (multiple-values-list...)).

    ReplyDelete
  18. [Python is] so lost that it is kicking Lisp's ass in the number of useful applications created with it.

    And Java and maybe C++ apps outnumber Python apps even though Python is much better for programming than both. Wrong metric. The question is where will we be in five years, the answer is using Common Lisp, because of language features.

    Which is more important? Building something interesting or navel gazing over some properties of a language?

    I tried to point out in the post to which you have responded that I am nothing but an application programmer. When I rave on a language feature such as every form returning a value, it is because it translates into programmer productivity.

    I decided to break off my Why Lisp? post to emphasize the code/data point of McCarthy, but in an upcoming continuation I will likewise get specific about exactly how the code/data equivalence of CL translates into very practical advantages to the working programmer.

    ReplyDelete
  19. But there was one astonishing quiet little thing he did during the interview that simply leaves me in awe, and it happened in and around the bit you cited. Can anyone guess?

    What left you in awe????!

    ReplyDelete
  20. What left you in awe????!

    I just re-posted the challenge over on comp.lang.lisp, let's give them six hours to find it.

    To me it is wonderful because in two words (hint) McCarthy gave me a glimpse of genius, of a heightened awareness so clearly beyond me that I could not even formulate envy, like watching Michael Jordan play basketball, that moment where competiveness with others falls away and only naked admiration remains and joy at the other's sublime ability.*

    Be careful, it is a sneaky little thing.

    * Of course the others he landed with are ragging on him for so clumsily giving away his alien status.

    ReplyDelete
  21. Kenny, great post. I agree on all of your points.
    Specifically, the maturity of CL as a language is apparent every time I code. It's like having the giants of ages past standing behind you with their hands on your shoulder.
    The amount of thought and care put into it by smart people is amazing.

    ReplyDelete
  22. eh, that stuff about (values) was mine ;) I just wrote reply (repl), but after a while I realized that you got it (oopsed). I'm posting my example anyway because I spend a couple minutes writing it ;)
    CL-USER> (let (list)
    (loop (format t "~& feed-me > ")
    (if (setq list (multiple-value-list (eval (read))))
    (dolist (el list) (print el))
    (format t "~&; No value"))))

    feed-me > (values 1 2 3)

    1
    2
    3
    feed-me > (values nil)

    NIL
    feed-me > nil

    NIL
    feed-me > (values)

    ; No value
    feed-me > (defun foo () (values))

    FOO
    feed-me > (foo)

    ; No value

    ReplyDelete
  23. "And Java and maybe C++ apps outnumber Python apps even though Python is much better for programming than both. Wrong metric."

    Java and C++ are/were for people who follow the job trends: people *love* Python and Common Lisp.

    "When I rave on a language feature such as every form returning a value, it is because it translates into programmer productivity."

    I'll take "batteries included", a BDFL who aides in keeping the language modern, and terse readable syntax productivity over multiple implentations of abandoned libraries and scattered efforts any day.

    5 Michael Jordans using Python will best 5 Michael Jordans using Common Lisp on a project built from scratch in just about any problem domain. People have voted with their feet and Python has reaped the rewards.

    ReplyDelete
  24. "I tried to point out in the post to which you have responded that I am nothing but an application programmer."

    How's that Algebra software coming? Lisp attracts people that like to talk about telescopes rather than look through them. There's no shortage of people that would rather tell you how they feel than actually accomplish something. Those types of folks are right at home with Lisp.

    ReplyDelete
  25. Java and C++ are/were for people who follow the job trends: people *love* Python and Common Lisp.

    OK, fair distinction, but then you have to retract the "app count" metric as an issue. I mean, it was your metric.

    I'll take "batteries included", a BDFL who aides in keeping the language modern, and terse readable syntax productivity over multiple implentations of abandoned libraries and scattered efforts any day.

    Admit it, you have no idea how macros change the programming process.

    5 Michael Jordans...

    I think you are getting a little too worked up, the concepts "five" and "Michael Jordan" cannot be combined.

    People have voted with their feet and Python has reaped the rewards.

    Are you referring to all the space freed up by the exodus to Ruby?

    I think Rails started the exodus, but the superiority of Ruby blocks over the joke Python calls lambda sealed the deal.

    See? Language features do matter.

    ReplyDelete
  26. How's that Algebra software coming?

    Looks like I should make it a blog post of its own, definitely entitled "A Cautionary Whale" with full cred to Diablo. But it is going amazingly well, I am just not working on it though I keep promising myself to resume work tomorrow.

    I am currently allowing myself to wallow in the perfect solution to creating random Algebra problems. I might have to punt on that to get it out the door. If only I had a deadline to impose some discipline!

    Hell, today I start work on Eek, an implementation of McCarthy's Elephant 2000. As my one of my fortune cookies says, find another job, program for fun again.

    The Algebra software is fun, but it has been for a long time and the idea of working on E2K is fresh and fresh always distracts me from boring things like not going broke.

    ReplyDelete
  27. "OK, fair distinction, but then you have to retract the "app count" metric as an issue. I mean, it was your metric."

    People who work in languages that they love languages can't write apps?

    "Admit it, you have no idea how macros change the programming process."

    I have seen plenty of people get religious about them yet fail to show people the actual value by using rediculous examples that are solved in an easy manner in other languages. Maybe you could change that?

    "the concepts "five" and "Michael Jordan" cannot be combined."

    Sorry--it's an obscure basketball reference. I figured you might know it since you brought His Airness up.

    "Are you referring to all the space freed up by the exodus to Ruby?"

    The only exodus I've seen regarding Ruby is from Java and PHP web programmers to Rails. Nobody likes those divas anyway.

    You must not be paying very close attention: Ruby and Rails has been getting slammed left and right lately.

    On the contrary, Python usage is on the rise, particularly with the web frameworks.

    "See? Language features do matter."

    I agree that they do to an extent, but I don't think the code as data argument is as important as Lispers like to believe.

    Python as a language and community is very welcoming to newcomers: it's incredibly easy to find, install and get started with. You can create all kinds of applications with it right out of the box. There is no shortage of good documentation, help and community resources.

    Advanced language features like code as data are nice, but they appeal to a subset of potential users. Most people just want to get moving on making something.

    ReplyDelete
  28. >>(setf open-source 1)

    1
    >>(eql (not open-source) nil)

    T
    >>

    ReplyDelete
  29. What left you in awe????!

    "He nodded"

    ReplyDelete
  30. What left you in awe????!

    "He nodded"


    Bingo!

    ReplyDelete
  31. Python as a language and community is very welcoming to newcomers: it's incredibly easy to find, install and get started with. You can create all kinds of applications with it right out of the box. There is no shortage of good documentation, help and community resources.

    You are saying nothing about the language itself, and nothing in principle impossible for another language (like Lisp) as its community grows.

    In brief, those are commodity advantages. Please note that I am not saying they are not advantages, I am saying they are commodity advantages. Hell, COBOL used to have them.

    Advanced language features like code as data are nice, but they appeal to a subset of potential users.

    There you go again, dissing advantages you do not even understand. See below for my proposed cease fire pending rectification of this gap.

    Meanwhile, I have an objection: I doubt we can brush "nice advanced features" under the "select few" rug like that. There is no such animal, I submit, no nice feature that only a few will appreciate.

    There may be heights of abstraction lesser lights may not be able to reach (I once found someone cutting and pasting code because they did not know how to make a function in VAX Basic (an industrial-strength Basic)) but I hope those programmers do not count. :)

    And there may be disputed nice features such as strong static typing of variables, but where we agree on a feature, a language without must have a deficit compared to one that does, and a deficit for any user.

    Most people just want to get moving on making something.

    And I made clear in my post that getting things made is my bottom line, too.

    Let's agree to continue after my post on "The Joy of Macros" so we are not talking past each other so badly.

    ps. As for Ruby vs. Python, fine, but why did Ruby get all those Java/PHP developers when Python if anything was the more established alternative?

    pps. Yes, I have heard the backlash on Rails, it kinda makes my point that that got Ruby in the mindshare door but something more fundamental kept it there. Why do you think so many of them bypassed Python's more established camp?

    ReplyDelete
  32. Ok, so maybe I'm an idiot. "He nodded"? Could someone explain why that is so monumental? The interviewer nodded and JM noticed it. What's the big deal?

    ReplyDelete
  33. The interviewer nodded and JM noticed it.

    You parsed it wrong. :) That is not what happened. Think about trials, where witnesses sometimes do what comes naturally and respond by nodding or shaking their heads, then reparse.

    ReplyDelete
  34. btw, while we are waiting for my Ode to Macros one could search comp.lang.python for a long thread on "why macros?" several years back involving yours truly and others. Lotsa good groundwork in there.

    ReplyDelete
  35. i'm not familiar with ruby but i know it doesn't differentiate between statements and expressions (syntactically at least) which i'm certain is lisp-inspired. no idea about python

    re: niftyness of macros, this article tries to convey their power: http://www.defmacro.org/ramblings/lisp.html

    does it succeed? i dunno. it might for some people

    ReplyDelete
  36. Can you use Lisp for conventional programming? Can it read a database? Can it run a GUI?

    Ok. Show me? Well, actually I don't care about the GUI. But I do care about web apps and I do care about databases.

    So where can I find out how to build a really simple database backed Lisp web application?

    I hear a lot about language syntax, clever algorithms, macros, and the like. But I almost never hear someone say how you connect a Lisp app to SQL Server or PostgreSQL and the like.

    I take you at your word, but I'd love to see some short and sweet explanations of how to achieve this sort of stuff.

    ReplyDelete
  37. So where can I find out how to build a really simple database backed Lisp web application?

    I hear a lot about language syntax, clever algorithms, macros, and the like. But I almost never hear someone say how you connect a Lisp app to SQL Server or PostgreSQL and the like.

    I take you at your word, but I'd love to see some short and sweet explanations of how to achieve this sort of stuff.


    http://www.paragent.com

    ReplyDelete
  38. re: niftyness of macros, this article tries to convey their power: http://www.defmacro.org/ramblings/lisp.html


    Ha-ha, I started my post on macros (which I probably will not finish cuz it will take like three days to write and everyone wants me back at work on the Algebra software) I began with some C preprocessor macros, the point being, "Why do I even have to explain the need for macros?!!!". :)

    It also occurred to me that it would be a lot faster to just link everyone to Graham's On Lisp (which I do in just about every post) or other such sites so thanks for the link. But it is a little scary to see someone so enamored of XML. :)

    ReplyDelete
  39. Ok. Show me? Well, actually I don't care about the GUI. But I do care about web apps and I do care about databases.

    I will refer you to comp.lang.lisp since I am a dinosaur (and I prefer OODBs and now RDF to relational). The commercial apps all have interfaces to SQL, and CL-SQL is a popular open version. I am actually using Amazon S3 as my database, and if I was not using AllegroCL with its ODB AllegroCache and RDF AllegroGraph I would use CFFI to get to Redland RDF.

    AllegroCL also has a Webserver (AllegroServe, not sure how they thought of the name) and they made it open source so there is a Portable Aserve, but something code named Hunchentoot (open source) seems to be the open way to go: http://www.weitz.de/hunchentoot/

    ReplyDelete
  40. MythMoth makes a good point. If the Lisp advocates want to convince non-Lispers that Lisp is practical, they should offer more practical examples and tutorials.

    Do you think you can squeeze in a few snippets of your application code? I'd be very interested to see what real-life Lisp looks like.

    Thanks!

    ReplyDelete
  41. T, I do not mean to be unresponsive, but MythMoth asked about practical stuff like libraries for Web apps and SQL access and I answered him.

    Er, come again?

    Had you not confused me thus, I would have said great minds think alike because I gots to get back to work on my Algebra software and it occurred to me I could keep this blog going by chatting about issues (code) as they came up.

    Just remember, you still have to deconfuse me. :)

    ReplyDelete
  42. Did my last replay go through?

    Anyway, I'm sorry, I didn't want to appear hostile. I'm in the process of learning Lisp and MythMoth's post clicked with how I'm currently feeling: I wish discussions on practicality of Lisp came with more practical examples and tutorials. And since we have a Lisp application programmer, I just had to bring it up :)

    ReplyDelete
  43. I wish discussions on practicality of Lisp came with more practical examples and tutorials. And since we have a Lisp application programmer...

    Right, but the post to which you are responding says the first thing to understand is that Lisp works like any other language, so I have a couple hundred KLOC to offer.

    You may need to be more specific. :)

    Or check out this on-line book (you might like the title):

    http://www.gigamonkeys.com/book/

    ReplyDelete
  44. Using Eric Marsden's (link: http://www.chez.com/emarsden/downloads/) postgres library, connecting to a database looks like this:

    (with-pg-connection (conn "testdb" "login" :host "dbhost" :password "secret")
    (with-pg-transaction conn
    (when (member "test_date" (pg-tables conn) :test #'string=)
    (pg-exec "DROP TABLE test_date"))
    (pg-exec conn "CREATE TABLE test_date(a timestamp, b abstime, c time, d date)")
    (pg-exec conn "INSERT INTO test_date VALUES "
    "(current_timestamp, 'now', 'now', 'now')")))

    ReplyDelete
  45. I'm writing this comment a bit late, but there is a language, started around ~2000, and revisited recently, which operates it's own code as data (called `block`) and vice versa. It's called REBOL, and modern reincarnation is Red.

    ReplyDelete