The Road to qooxdoo Part III: Why It Rocks

In the episodes one and two we learned all about how a Lisp God and Interwebby know-nothing found a way to support a client needing an enterprise Ajax Web application, said way turning out to be qooxdoo after solid efforts to make jQuery, Dojo, and YUI fill the same bill. But we did not learn very much detail about qooxdoo, we just learned that it comes with lots of doc, lots of examples, a solid community, and that the engineering is first rate. I may not have mentioned this, but it also was dramatically faster/smoother than the others on my acid test: scrolling a datagrid. 

Okay, that is a lot as a general recommendation. I mean we did not learn much detail. Until now. Here are four Big Things that really stand out:

1. HTML? CSS? Never heard of 'em.
First comes a shocker. The qooxdoo user codes no HTML or CSS and needs know nothing about those two disappointments. Just as my Lisp compiler reads my Lisp code and spits out optimized native machine code, qooxdoo takes your qooxdoo/Javascript and produces the HTML and CSS to drive the browser. Upside: qooxdoo developers write less code and never have to worry about browser variability again.

To a Web noob like me there is a second big win here: I do not have to learn HTML, CSS, or the vagaries of the different Web browsers my client might want supported. I am a bit of a rarity in my ignorance of those things but soon I should have a lot of company now that qooxdoo has set the bar as high as it has on insulating the web app developer from lower level technology.

By the way, in case you are wondering: yes, there is a widget (two, actually) that will accept classic HTML.

2. I do declare. Not!
The second thing that hits us is a consequence of the first: we are going to be writing a lot more Javascript than we would be with libraries such as Dojo where authors still write HTML/CSS. With declarative tools we sit in Lisp or Ruby or PHP on the server generating HTML. Not qooxdoo (but see next)  and not YUI.

I am not at all happy about this but I will live and if it ever bothers me enough I will spend a weekend exploring the qooxdoo markup contrib called QxTransformer. Right now I need to catch up on a backlog of functionality the client has had to do without because of the six weeks it took me to identify qooxdoo as The One.

3. OO is not the Grail, but OO rocks
Stendhal said it best in "On Love" in the chapter on Infatuation: (paraphrasing poorly) we do the object of our desire two injustices, first setting them on a pedestal unreasonably high and then, when they inevitably disappoint, setting them too low. OO is not the Grail, it is just a profound advance in serious application development. All by way of introduction of another stunner: qooxdoo's slightly more substantial object model layered atop the core Javascript OO capabilities. 

To be honest this struck me as a bit of a yawn until I decided to start carving up my existing code into distinct qooxdoo classes. This is a very big application and I needed code re-use almost immediately and doing it in HTML with Javascript glue was...well, before qooxdoo I was pretty much starting to build my own abstraction layer atop the core tools--there was no way around it.

But now that I have used qooxdoo's OO to partition my rapidly growing code base I have more than once had that wonderful sense of getting something for nothing after doing a major refactoring (you know, taking all the pieces and throwing them up in the air) and having it Just Work after two or three tweaks. Furthermore, I will not bore you with the details but I have not even started leveraging the full capabilities of qooxdoo's object layer: even more benefit will be reaped moving forward because of some nifty reactive capabilities.

This came as a pleasant surprise. I do more OO in more ways than you can imagine in part because I normally program with Lisp's CLOS which is to normal OO what normal OO is to binary arithmetic but by the time I stumbled onto qooxdoo I had been doing Javascript, HTML, and CSS for a couple of months under a lot of pressure so I never had the time to build a sensible OO abstraction atop Javascripts primitive little OO pretensions. Thus Javascript had endowed me with a bit of child mind and I was able to experience the mundane code structuring win of OO as if for the first time. It rocks.

3. Layout the Wazoo
I may have lied. This may have been the first stunner I felt in my gut. The others I knew would be huge if they panned out, but this was the first one I saw in action working as beautifully as had the other frameworks left me hanging. With other frameworks I could find no way to get built-in layout schemes to use the full window real estate available. That cuts two ways: I either ended up with unused whitespace (and with our app we need all we can get) or my application panels grew so large that the browser itself put up scroll bars, meaning I had to window-scroll my scroll panes around. No, that was not a candidate for production release. Note that to a large degree this is the same as point #1: a framework is not hiding HTML/CSS from me if I am still victim to surprising browser decisions, such as oh gosh that is big let's use a scroll bar here.

How good is qooxdoo at layout? When I popped open the Firebug debugger in FireFox, where by default it begins by seizing application window real estate, the qooxdoo layout engine handled the downsizing impeccably, right down to adjusting scroll bars to accurately reflect the new dimensions.

4. Message In A Bottle

This feature is huge in a small way, by which I mean I could probably roll my own version of qooxdoo's Message mechanism in a an hour but--wait for it--I did not have to. They did. That is huge, the huge thing being that the qooxdoo team really has set out to create a compleat Web application development environment and with such a beast comes pleasant surprises such as their Message mechanism which allows a nice decoupling of widgets: one widget can dispatch "Make it so" and anyone responsible for "it" knows to roll up their sleeves simply by having subscribed to "Make it so".

Beyond the Message mechanism we have the Event mechanism. The documentation of every widget lists the events it might generate and what data the event will carry, and in toto we have an engine with an event model dozens of times richer than pure DOM.

5. I know, I said "4"

Number five is an umbrella win, too detailed to list: as I whined above, this Lisp God is now writing an awful lot of Javascript leveraging more and more qooxdoo all the time. Again and again your highly critical correspondent is finding nothing but good news in qooxdoo. 

Yes, I could make a bug report every other day if I had the time, but a long time ago I figured out that I am not looking for a perfect library, I am looking for a powerful library that is fundamentally sound (so anything broken can be readily fixed) and actively maintained (so things actually get fixed).

Summary

qooxdoo: fast, powerful, complete, rich, well-engineered, and it hides browsers, HTML, and CSS. Great doc, loads of examples, fine community. It's all good.

The Road to qooxdoo - Part II

Last time, we learned how one self-appointed Lisp God with no knowledge of web programming ended up programming the web and then soon enough embarked on a six week grand tour of Javascript frameworks in search of a platform suitable for an enterprise web application and where his tour ended: qooxdoo. That episode ended with a promise of more information on qooxdoo, but first a note on the methodology: there was none.

My first experiment with web programming had thanks to the marvels of Lisp in general and AllegroCL's web tools in particular mushroomed into a full-blown web application aided here and there by jQuery widgets before we slowed down enough to think, Damn, we could benefit from a full-blown Web application framework.

 So I eyeballed the usual suspects and ran them through googlefight and picked the most promising one and used that to redo what I had accomplished to date. Which was: tab controls, select boxes, data grids as views into server-side data stores, tree views, text input, tool tips, pushbuttons, pop-up windows, and probably one or two things I am forgetting.

How did I assess the libraries? We needed a professional look, good performance, and maximum productivity. Sorry if that seems blindingly obvious. The productivity assessment has some beef: I looked for lots of documentation and tons of examples describing a rich set of widgets at once high-level, feature-rich, and yet highly authorable which may not be a word but it means I get to override stuff without resorting to back doors. 

I considered also two metaqualities: on-line support and internals code quality. Having the Lisp HyperSpec an F1 away is just a little faster than asking for the same information on comp.lang.lisp. ie, Community matters. As for code quality, the good thing about open source is not that it is free, it is that I can use the source to understand things at 3am when the community might be quiet; I can add print statements to debug my code; and maybe I can find and fix a bug in the library. But if the code is of low quality good luck with anything more than the print statement insertion.

It was a rough six weeks. Three times in six weeks I logged onto support groups and said, "Hi, this is my first day..." as I rebuilt a relatively feature-rich web app three times while staggering up to speed three times on three environments. All while the client is getting absolutely no new functionality. I feel bad all the while because they need this stuff now. I am letting them down and there is the strong possibility they will be letting me go: I had been brought on to do the Web front-end. I like learning, but not so much with a gun to my head and a time bomb ticking.

Are we having fun yet? Well, yes. The last framework we tried was qooxdoo and it has worked out great. I can now give the client pretty much what they want fast and looking good once they pony up for a graphic artist because I just saw this impressive example of what can be done with their "appearance" framework. But I ain't no designer.

Funny thing, though: qooxdoo  like YUI and Dojo came up short during the evaluation. It had no example of the widget I needed most, a fairly standard widget at that: a datagrid backed by a remote store. Bad sign. Second, qooxdoo did no better on googlefight than does Lisp, and that is pretty bad. Finally, qooxdoo like YUI lacks a declarative programming model [though the declarative QxTransformer has been revived since my original investigation] which means I cannot just sit in my Lisp IDE catching XHRs and tossing back HTML/CSS. This would put a non-trivial dent in my productivity.

So how did qooxdoo prevail? What can I tell you? Bullet charts are useless, expert systems work great as long as expert humans interpret their output, and resumes tell us nothing about the work we will get out of people. 

qooxdoo did have a contributed example of a remote data store which pretty much worked and when I looked at the qooxdoo internals to learn how to make it work better I found very good code and rather quickly had the datagrid I needed. Otherwise qooxdoo did have loads of examples and documentation and decent community (I say "decent", you'll say "great"--my standard is comp.lang.lisp) and then we get to the bottom line.

While resurrecting my app for the third time with qooxdoo, yes, I worked just as hard as I did using Dojo and YUI. But the effort was different. With qooxdoo I was struggling to learn a very nice framework that needs an index: I was slaving for six hours to find the simple six lines of code I had to write. With YUI I was struggling with a framework that did not work all that well and scared me more and more the more I looked inside it. With Dojo the struggle was with a good framework but the struggle was such that I suspected the struggle would never diminish and probably increase exponentially as the application grew in complexity.

As for the declarative thing, hey, I am a programmer, I can write code. And Javascript has a lot of Lisp soul, I am not stuck with something daft like PHP or Python. Sure, having to spend more time in a JS IDE means a ten percent productivity hit but ten percent we can do. The terror we developers dread is the tool nightmare death by a thousand cuts two steps forward two steps back rope drag death. 

I will blog on rope drag sometime soon, but first let's get specific about qooxdoo. In Part III of my Road to qooxdoo.

The Road to qooxdoo

[The title of this rather boring entry derives from a fun little survey I kicked off years ago.]

OK, why should you listen to a Lisper on Javascript frameworks? Because we use Lisp so obviously we have better instincts when it comes to developing applications. In fact, this road would not have ended so well had not Lars, a fellow Lisper developing a killer new Lisp Web programming framework called Symbolic Web (SW) brushed aside my enthusiasm for Dojo and mentioned he found qooxdoo interesting, although for now SW uses a little jQuery. "A little" because the whole idea of Symbolic Web is to do most of your programming in Lisp which of course is the only way to go but SW is still emerging from the sea and I had a requirement now (well, six weeks ago, but there is major new release of SW just out I need to look into but as this Road will reveal I am not likely to change horses now because of what I found in qooxdoo (not that SW will not eventually catch up)). Gasp.

The second question might be how a Lisp god and Interwebby know-nothing ended up doing heads-down intense Interweb development. Well, it was the Lisp credentials. A former client using mostly Lisp was having great luck with a Ruby/Rails front end but was looking for more speed and hoped having a Lisp image serve the pages directly would make for better response.

What happened next was pretty neat, and again Lisp conquers all. I told the client I would look at what they had and what they needed for a few days to decide if even I could help them, because I can do a lot of things but one thing I cannot do is exaggerate how little I knew about Web programming and even static HTML eight weeks ago.

A glance at the existing screens and Rails code had me thinking, "No way", but I was curious anyway about even getting an application to drive a web site so I fired up a tutorial on Web app development using tools from my Lisp vendor of choice, Franz.

How good are Lisp, Franz, the tutorial, and the WebActions tool? In the three days I was supposed to be deciding whether I could help the client I managed to execute a respectable fraction of the first component they needed. Sorry, no screenshots, NDA and all that.

So now there I am with a contract and a browser toolbar sagging from the weight of bookmarks to sites on HTML, CSS, and for the hell of it jQuery though I was not sure why but everyone was jumping up and down about it. And then I needed it.

Well, not jQuery. I needed a razzle-dazzle spreadsheet-like grid presenting data from an arbitrarily large table of data held on the server. Now at this point I had already rolled my own Ajax paging mechanism, but something superficial but nonetheless important for that this being an important application was holding me back: my HTML looked like crap. Meta-holding me back was my sense from surfing the Web that getting good results out of HTML/CSS was a black art requiring years of practice which certainly leaves me out. Enter FlexiGrid. A URL which as I write responds "Please contact the billing/support department as soon as possible."

Which brings me to why I dumped jQuery. What I found was small, yes, and add-ons, yes, but a hodge-podge of add-ons does not a professional development environment make. By the time I punted I had contribs from three people and while each was lovely alone together they looked like, well, a great decorator on a multiple-personality binge. And beyond appearance there came a surprising result: FlexiGrid did not offer a "row selected" event. Meta-worse was the author's response. "Right, no row-selected event." Ooops. Next up was jqGrid and that was fine but I still had Crazed Decorator Syndrome going and now I am looking for the next widget I need and a light went on: we are building an application here, not a Web page/site. It is wonderful that jQuery is small, but small is exactly the wrong thing for developing a sophisticated application with a coherent look and feel never mind coherent engineering underneath so widgets can play well together. Enter Dojo. But first...

In case you are wondering where the client went, they are encouraging my search for a pre-fab solution. Probably seeing the gorgeous FlexiGrid widget compared to my ugly efforts did not hurt, but the client has good instincts anyway (they use Lisp, right?) so they are leaning towards my original assessment that I am overmatched if I am going to try to turn out a hot application on short experience with HTML and CSS even though at this point I have wondered aloud to them how much HTML/CSS I might have mastered in the same time I was spending beating my head against pre-fab, this being the classic failed deal we make with the devil when we reach for 4GL: yeah, wow, a complete browser screen in five minutes. Gee, could we get the first sub-item appearing on the same row with the item? No. The client wants it. It cannot be done. That was Datatrieve on VAX/VMS twenty-five years ago, but nothing has changed. 4GLs get their apparent power by making decisions for me which works only if I (and my client) are not making any decisions.

But no, the client says "find a framework" and I decide to try again with Dojo. Again?

I had looked at Dojo before. The word on Dojo was incredibly good. Of course that word was from their Web site. When I tried to confirm by eyeballing the doc I could not find it. I mean, there were some great links, but all the pages they led to were blank. Next! But now I was back, encouraged by the news that OpenLaszlo had just placed a bet on Dojo. IBM as well, but OpenLaszlo is more important because I know they are geniuses, they have a dataflow hack like mine.

Oooh-ooh! Another reason Dojo looked good was a benchmark I had found showing the thing was fast. I have no idea if the benchmark is valid, but the pictures are gorgeous so I trust it. So why did I dump Dojo?

Well, OK, I did not really. It came in second. The question is why did I keep looking such that I found the eventual winner. Easy: too frickin hard to get to work. The documentation is awful. They know this and are addressing it but did I mention we need this now? Overall Dojo fit the bill of being a Compleat Solution with solid backing and a great future, but after banging my head against the wall over something (ah, it was the six hours I spent trying to figure out the "build" mechanism for a compressed production release of my JS culminating in watching a blurry video that finally showed how to do it) I went for a better way surf (as in there's gotta be) and saw YUI touted as a serious solution.

A quick glance at the enormous volume of well-organized documentation and exhaustive examples and the client and I agreed at once to give it a try. And I will tell you right now YUI has the best graphic designers, because their stuff looks hot. Dojo is a close second there, too. YUI is a complete solution with a big backer, gorgeous style, and more doc than you can imagine. How did it get dumped?

Stuff was not working well. Even with all the doc and examples it was hard to get working. When I dug into the source to see why things were not working, I did not like what I saw. And the death knell: YUI abandons the declarative model. Why is this so bad? Hellooooo, Lisp? Remember?

Up thru Dojo I was programming in Lisp 90% of the time, doing just enough JS to get information back to the Lisp server application where HTML could be generated and sent back, even that Lispily thanks to HTML generating macros. And YUI was going to make me give that up and be as hard to figure out as Dojo and I suspect worse under the hood. So regretfully I informed the client my battles with Dojo would be resuming.

So how does this road end up at qooxdoo?

The weekend was coming up. Weekends are mine, so I decide to follow-up on Lars's hint about qooxdoo. qooxdoo is the ultimate threat to YUI because qooxdoo also says Just Code Javascript, and if I am forced to do that by YUI I may as well do it in qooxdoo where the code is as good as YUI's is bad. (I looked at about fifty lines of it for five minutes, I should know.) Anyway....

qooxdoo is amazing. They need help on the graphic design (never look directly at their tab control, use a mirror) but other than that the engineering is terrific. [My bad: they simply leave anything more than the utilitarian look up to the developer, witness this before/after case study.] The documentation and examples are also abundant here, and the support is great. They even do dataflow! Yes, I wish I could spend all my time in Lisp sending over HTML, but two things on that.

One, there is a side-project that brings a declarative model back into qooxdoo. QxTransformer. I have no idea where that stands, but I did see a note on-line where a developer spoke of it in the past tense. [Now resurrected.]

Two, hey, I can write some Javascript. I did ten years of C before I did fifteen years of Lisp, Javascript is not a problem. And it even has lambda, something this guy still does not understand. Lisp/markup would be better, but when dealing with mission-critical stuff (the client is doing valuable data crunching but does not win until they make it accessible) any team can roll up its sleeves and crank out the work even if they have to do without the brilliance of Lisp.

This entry is getting long and I am getting tired and my bartenders are starting to worry about me so I will leave a detailed praise singing of qooxdoo to my next entry, but allow me a bit of a rant. What a mission-critical effort cannot stand is fundamentally flawed solutions that seem to offer a fast track but in fact bleed developers to death. This can be jQuery with an incomplete solution forcing me to try to build an application out of mom and pop contribs or YUI with fundamentally bad code or some other framework that might be just plain slow.

As a developer I am not looking not to work, I am looking to have my work yield quality results predictably. I might spend four hours trying to figure out how to do something in qooxdoo but once I do it turns out to be insanely simple and work beautifully. Moral: their documentation needs an index! But I am keeping my big yap shut because documentation is a bitch and they have done a ton of it and I do not document my stuff so...I keep my big yap shut. But the point is that my work now yields results reliably and predictably and this will accelerate as I learn the framework and even learn better how to navigate the documentation.

qooxdoo not only fits the bill of being a complete solution well documented with many examples but it also has high-quality code under the hood, which both gives me confidence that I have found the right tool and not incidentally serves as useful further documentation. More next time.

Why Lisp Packages are so Easy (Hard)

Slobodan Blazeski wrote:
> May somebody please explain me with simple words why cl packages are
> hard to use, 'couse I really don't get it?
> The best is probably to hear from somebody who just learned lisp or
> somebody who teaches lisp (do such people exist anymore?) and his/her
> students has problems with packages.

...or a good teacher, defined as someone who remembers what it was like before they knew what they were teaching.

The first problem is one of those immune system deals. Packages have a lot of the same proteins as, say, C dot aitch definition files so the recognition systems of programmers coming to Lisp from other languages (you know, everyone) does its damndest to parse them as dot aitches.

In fact, packages have nothing to with anything but symbols, neither the functions nor global values they might name. More expansively: packages are about mapping string names to first class Lisp symbol objects and dividing this up into discrete mappings called packages so the same string "Hi, Mom!" can map to multiple |Hi, Mom!| symbols in multiple packages.

The second problem is that symbols take us straight to black-belt Lisp in which one must understand the Lisp reader and fancy things like read-time and compile-time and all that. Example:

I remember getting messed over because some symbol was mysteriously appearing the base CL-USER package. I could not for the life of me see where it was being introduced. I turned the problem over to a few trees worth of monkeys and one of them came up with this rule:

"Never use a normal symbol like |Hi, Mom!| in a defpackage form coded in a source file with (in-package :cl-user) at the top which would likely be the default even without that."

Instead, use a string like "Hi, Mom!" or an uninterned symbol like #:|Hi, Mom!|.

Strings are dicey because you might want to export 'banana and be silly enough to code "banana". Lisp will take you seriously and create |banana| instead of BANANA, case sensitivity being what it is except when it is not when Lisp folds 'banana to 'BANANA which is what it does unless you are in Modern (case sensitive) mode.

Did I mention "black-belt"? The fundamental idea is simple, but noobs get thrown off mostly by the non-simular simularity with C and then in the debugging stage by case sensitivity seeming to come and go at random and the whole read-time compile-time thang.

hth, kxo

A Beginners Guide to ASDF (Ha!)

OK, there is at least one thing about which Lispers are not so smug: ASDF (Another System Definition Facility), the de factor standard for the Lisp library equivalent of Makefiles, and my post today was prompted by ANFOBA (Another Noob Effed Over By ASDF) washing ashore on comp.lang.lisp, and how I almost responded there:

Francogrex wrote:

Hi, I am struggling to understand how to work with asdf.

I have been using it for ten years and I am about 50-50 now on getting it to work. I am assured it does a great job in working with a multitude of Lisps/OSes, and I can assure you it is a disaster at working with programmers.

Come back here and post console output freely, you will need help, especially because one of its great triumphs is useless error messages.

Hints:

(1) It will work. You will have to promise it your first-born, but it will work.

(2) No, PB was not joking: you have to separately push locations onto some magical special variable. Leave off a slash (or add a slash, I forget) and ASDF will take your head off, so don't do that.

(3) No, PB was not joking, the syntax is:

(asdf:operate 'asdf:load-op :cl-pdf)

Why is it not (asdf:load-system :cl-pdf)? In my experience, extremely smart programmers think it astoundingly brilliant when "one function does everything!!!!", remembering the ridiculous op-code first parameter left as an exercise.

Since you asked, this is because extremely smart programmers are tone deaf to programmer productivity; they are so smart they do not even notice when they have created something unuseable, they just brain through it.

(4) ASDF has a featurebug: it loads in the wrong order in order to make McCLIM work. Don't ask. If the author of the .ASD file retreated to the desert for forty days to slave over the dependencies this will not be a problem. Otherwise do what I do: take a sledgehammer to the beast:

(asdf:operate 'asdf:load-op :cl-pdf :force t)

That abandons the whole point of having a build system, but it gets ASDF to work. When for the love of god I am just trying to build this frickin thing!!! that is a nice tradeoff of compile-time vs tracking down and short-sheeting the author of ASDF.

(5) On success ASDF emits more messages than an IBM OS/360 core dump. This is so any errors will scroll well out of view and anyway stand out like a healthy thumb from the surrounding...um, other healthy thumbs. Anyway, do what I do: ignore all messages other than a backtrace and Just Try the Hello World(tm).

If you get a backtrace, Just Post It to c.l.lisp.

(6) If hello world fails, f*ck! But at least now you are a Real Lisp Programmer(tm), this is what we live with.

hth, kenny