I Feel A Naggum (RIP) Coming On: Quads

I sometimes begin c.l.l rants with "I feel a naggum coming on...". What is a naggum? Normally:

naggum (n): A rant along one of Erik Naggum(1965-2009)'s themes.

That might be self-referentially hopeless which is fine because that is not what I am talking about, I just thought that would be a clever title. In this case a "naggum" is a nugget of Erikian technology. First, his specification of what he called quads (see below), and my poor implementation (even further below and good luck even figuring out how to test it) .

kt

#|

From: Erik Naggum (erik@naggum.no)

Subject: Re: XML->sexpr ideas

Newsgroups: comp.lang.lisp

Date: 2004-01-19 04:24:43 PST

* Kenny Tilton

| Of course it is easy enough for me to come up with a sexpr format off

| the top of my head, but I seem to recall someone (Erik? Tim? Other?)

| saying they had done some work on a formal approach to an alternative

| to XML/HTML/whatever.

|

| True that? If so, I am all ears.

Really? You are? Maybe I didn't survive 2003 and this is some Hell

where people have to do eternal penance, and now I get to do SGML all

over again.

Much processing of SGML-like data appears to be stream-like and will

therefore appear to be equivalent to an in-order traversal of a tree,

which can therefore be represented with cons cells while the traverser

maintains its own backward links elsewhere, but this is misleading.

The amount of work and memory required to maintain the proper backward

links and to make the right decisions is found in real applications to

balloon and to cause random hacks; the query languages reflect this

complexity. Ease of access to the parent element is crucial to the

decision-making process, so if one wants to use a simple list to keep

track of this, the most natural thing is to create a list of the

element type, the parent, and the contents, such that each element has

the form (type parent . contents), but this has the annoying property

that moving from a particular element to the next can only be done by

remembering the position of the current element in a list, just as one

cannot move to the next element in a list unless you keep the cons

cell around. However, the whole point of this exercise is to be able

to keep only one pointer around. So the contents of an element must

have the form (type parent contents . tail) if it has element contents

or simply a list of objects, or just the object if simple enough.

Example: 123 would thus be represented by (foo nil "123"),

123456 by (foo nil "123" bar nil "456"), and

123456 by #1=(zot nil (foo #1# "123"

bar #1# "456")).

Navigation inside this kind of structure is easy: When the contents in

CADDR is exhausted, the CDDDR is the next element, or if NIL, we have

exhausted the contents of the parent and move up to the CADR and look

for its next element, etc. All the important edges of the containers

that make up the *ML document are easily detectible and the operations

that are usually found at the edges are normally tied to the element

type (or as modified by its parents), are easily computable. However,

using a list for this is cumbersome, so I cooked up the «quad». The

«quad» is devoid of any intrinsic meaning because it is intended to be

a general data structure, so I looked for the best meaningless names

for the slots/accessors, and decided on QAR, QBR, QCR, and QDR. The

quad points to the element type (like the operator in a sexpr) in the

QAR, the parent (or back) quad in the QBR, the contents of the element

in the QCR, and the usual pointer to the next quad in the QDR.

Since the intent with this model is to «load» SGML/XML/SALT documents

into memory, one important issue is how to represent long stretches of

character content or binary content. The quad can easily be used to

represent a (sequence of) entity fragments, with the source in QAR,

the start position in QBR, and the end position in QCR, thereby using

a minimum of memory for the contents. Since very large documents are

intended to be loaded into memory, this property is central to the

ability to search only selected elements for their contents -- most

searching processors today parse the entire entity structure and do

very little to maintain the parsed element structure.

Speaking of memory, one simple and efficient way to implement the quad

on systems that lack the ability to add native types without overhead,

is to use a two-dimensional array with a second dimension of 4 and let

quad pointers be integers, which is friendly to garbage collection and

is unambiguous when the quad is used in the way explained above.

Maybe I'll talk about SALT some other day.

--

Erik Naggum | Oslo, Norway

Act from reason, and failure makes you rethink and study harder.

Act from faith, and failure makes you blame someone and push harder.

|#

(in-package :ukt)

;;;(defstruct (juad jar jbr jcr jdr)

(defun qar (q) (car q))

(defun (setf qar) (v q) (setf (car q) v))

(defun qbr (q) (cadr q))

(defun (setf qbr) (v q) (setf (cadr q) v))

(defun qcr (q) (caddr q))

(defun (setf qcr) (v q) (setf (caddr q) v))

(defun qdr (q) (cdddr q))

(defun (setf qdr) (v q) (setf (cdddr q) v))

(defun sub-quads (q)

(loop for childq on (qcr q) by #'qdr

collecting childq))

(defun sub-quads-do (q fn)

(loop for childq on (qcr q) by #'qdr

do (funcall fn childq)))

(defun quad-traverse (q fn &optional (depth 0))

(funcall fn q depth)

(sub-quads-do q

(lambda (subq)

(quad-traverse subq fn (1+ depth)))))

(defun quad (operator parent contents next)

(list operator parent contents next))

(defun quad* (operator parent contents next)

(list operator parent contents next))

(defun qups (q)

(loop for up = (qbr q) then (qbr up)

unless up do (loop-finish)

collecting up))

(defun quad-tree (q)

(list* (qar q)

(loop for childq on (qcr q) by #'qdr

while childq

collecting (quad-tree childq))))

(defun tree-quad (tree &optional parent)

(let* ((q (quad (car tree) parent nil nil))

(kids (loop for k in (cdr tree)

collecting (tree-quad k q))))

(loop for (k n) on kids

do (setf (qdr k) n))

(setf (qcr q) (car kids))

q))

#+test

(test-qt)

(defun test-qt ()

(print (quad-tree #1='(zot nil (foo #1# ("123" "abc")

. #2=(bar #1# (ding #2# "456"

dong #2# "789")))))))

(print #1='(zot nil (foo #1# ("123" "abc")

. #2=(bar #1# (ding #2# "456"

dong #2# "789")))))

#+xxxx

(test-tq)

(defun test-tq ()

(let ((*print-circle* t)

(tree '(zot (foo ("123")) (bar (ding) (dong)))))

(assert (equal tree (quad-tree (tree-quad tree))))))

(defun testq ()

(let ((*print-circle* t))

(let ((q #1='(zot nil (foo #1# ("123" "abc")

. #2=(bar #1# (ding #2# "456"

dong #2# "789"))))))

(print '(traverse showing each type and data preceded by its depth))

(quad-traverse q (lambda (q depth)

(print (list depth (qar q)(qcr q)))))

(print `(listify same ,(quad-tree q))))

(let ((q #2='(zot nil (ding #2# "456"

dong #2# "789"))))

(print '(traverse showing each "car" and itd parentage preceded by its depth))

(print '(of data (zot (ding (dong)))))

(quad-traverse q (lambda (q depth)

(print (list depth (qar q)

(mapcar 'qar (qups q)))))))))

;;;(defun tree-quad (tree)

(defun testq2 ()

(let ((*print-circle* t))

(let ((q #2='(zot nil (ding #2# "456"

dong #2# "789"))))

(print '(traverse showing each "car" and itd parentage preceded by its depth))

(print '(of data (zot (ding (dong)))))

(quad-traverse q (lambda (q depth)

(print (list depth (qar q)

(mapcar 'qar (qups q)))))))))

American & Iran: Separated at Birth?

Am I the only one grooving specifically on the fact that Iranians are telling their authority figures to go f*ck themselves? Here is the country we thought we hated but it turns out they are as kick-ass as us when it comes to political freedom, and we utterly respect them for their strength. Omigod, Americans and Iranians are going to get along great!

How to teach math

> On Sat, 09 May 2009 15:30:52 -0400, Kenneth Tilton wrote:

Well, i was not really trolling, I was forking the thread to make fun of  the New Math that tried to get the numeral/number distinction across to five year olds.

Someone responded:

You find it better to start with medieval concepts working gradually on to the mathematics of XIX century, while explaining each next year what was wrong with the things they learnt a year ago?

I said all that? Ma's gonna be right proud. But... 

Funny you should ask. Yes, I suspect the path society took to get to what it knows now about math is the path an individual neuronal mass should follow. ie, kids should encounter zero and roman numerals and place value and algebraic variables in the same order society developed those ideas.  The history of math is your math curriculum guide.

The New Math erred by selecting the logical organization of mathematical concepts as its curricular pole star. Next came Constructivism, which wanted kids to reinvent math. From scratch. Cool idea, but too slow.

Instead, let the history of mathematics dictate the order in which things are directly taught. Maybe go further and teach math as history with less emphasis on computation. Math often advanced when needed to solve real problems. Maybe we can shut up the little devils asking why they need to learn this stuff.

As for explaining all along the way what was wrong with the ideas taught the day before, hey, ever read a book on programming? They typically develop a chunk of code iteratively, presenting ever more improved variations on a primitive original. Come to think of it, ever develop some software? Same thing.

Here's the deal: most folks do not even know zero had to be invented. One understands zero better if one has done without it and then the teacher invents it for you. Something like that.

Cells: The Secret Transcript

The Boss asked me to give the group fifteen minutes on Cells because I have been talking about it for a while as a future better mousetrap for us and then suddenly last week threatened actually to apply it to qooxdoo and the front end.

I forwarded to everyone a link to a reasonably complete yet relatively brief write-up which tells you everything you need to know about Cells. I know that if I were in your shoes I would not have read it so I presume no one has. But I would like to determine how many folks I will be boring to tears if I review said document, so I will first cut to the chase and ask if anyone has any questions based on what they read.

........silence..............

OK. Cells is at once the simplest and hardest thing in the world for programmers to understand. Simple because the idea is just to have slot values of objects work like cells in a spreadsheet, and everyone knows how spreadsheets work. What is hard is understanding that one can program computers this way.

I only have fifteen minutes so: Yes, you can. Program inputs are assigned by good old imperative code to input cells the same way a user types values into a spreadsheet when they are doing what-if analysis or recording, say, actual monthly expenditure into a budget spreadsheet. Intermediate, derived, and aggregate cells compute new values based on those new inputs from predefined rules just as user changes to a spreadsheet propagate to other spreadsheet cells. Observers on cells let the emergent working model manifest its decisions with more good old fashioned imperative code, usually by simply updating the screen or playing a sound or controlling some external device over a serial port or updating a database.

Boom, we're done. Why is it so great? What part of the superiority of functional and declarative paradigms should I explain first? As I outlined in the material you did not read, a lot of things have to happen when a program receives an input. The programmer coding the event handler has to look at the event and decide all the things that have to happen in light of that event, and any things that follow from those first things. Not only must they reliably see to all those things, but they must do them in the right order. The analogy to a real spreadsheet is quite strong, if you imagine hand-implementing a spreadsheet with old-fashioned pencil and paper.

So the first things Cells does is eliminate a lot of work and thus a lot of bugs. Because the work eliminated is tedious, Cells also makes programming a lot more fun. But there is more.

The declarative paradigm means I always know why a slot has a certain value, because all the logic appears in one place, in the rule assigned to that slot. Without Cells any number of lines of code may have assigned a value to a particular slot and a unified deriving rule certainly cannot be divined even if one were to track them all down.

There is more. Most people hate OO because it never quite panned out. Objects turned out not to be reusable. One of the nicest features of Cells is that two different instances can have different rules for the same slot. That makes objects reusable. Yayyyyyyy.

I did not use Cells in the Kleaner because that was more of a straight calculation running from start to finish. I like to decribe the role for Cells being in any situation where one has an unpredictable stream of data and one is keeping a model with a sufficiently large amount of internal state consistent with that stream of inputs. Two examples being a GUI and a RoboCup client. The Kleaner worked by compiling statistics from a fixed store and then translating exactly once dirty data into corrected data.

Where Cells would have been useful would have been in implementing Phil's ideas about an ongoing stream of data leading to rediscernment of things previously discerned. Cell rules would take new raw inputs and propagate them over to tables of probabilities which would then reach out to existing cleaned data and possibly redecide from the original raw state a new cleaned state.

So no, I do not use Cells for everything, but in this case it was only because the full functionality had not been addressed.

A fun note is that I have in the past applied Cells to a database, specifically the old AllegroStore persistent CLOS database. This works two ways. One is that a user can be looking at a screen and as the underlying data changes the screen changes. That may sound like old news but with Cells one doe not have to write any code to make it happen. One just says "this view shows this users overdue books" and when the date changes the overdue status on every book gets updated and a new book appears in the list on the screen if someone happens to be looking, simply by someone having written code to list overdue books on the screen as if it were an unchanging value.

The other thing that happens is hinted at above. Things like overdue books and amount of fines owed and paid can be calculated from scratch by reading a users entire history of checkouts and returns, but sometimes it is useful to record such derived values in the database and update them incrementally as books are checked out and returned. We can have code in programs do it and hope they run at the right time, or we can have the code in the database (as datapoints mediated by Cells) and be sure they run and run immediately. We get timeliness of data, efficiency, and we still get consistency even as we introduce redundancy.

I left something out. That is bad because the thing I left out is the thing I am planning to do with Cells on my own anyway and then apply to the FE. Cells makes it dead easy to drive a separate framework from Lisp. In my note I mentioned tcl/Tk and Gtk. These are two killer C GUI frameworks with their own homebrewed little object models. We want to program in Lisp, and we want our models driven by Cells for all the reasons above. No problem. We build a model out of instances of CLOS classes mapping isomorphically onto Tk or GTk classes and use Cell observers to pipe information (thru an FFI or even literally a pipe) to the C library or runtime to drive there the creation and animation of C instances.

Works great, and one amazing programmer Peter Hildebrandt pulled off a trifecta in which he had Cells driving and driven by both GTk and a C physics engine, name forgotten.

For a while I kinda marvelled at how Cells could be so useful for such disparate activities, and do so in the same application, the two activities being building an application model and having some other programming framework dance to that model's tune.

I figured it out in time for ECLM 2008, not they were able to understand me. I opened by telling them that Cells was the single most powerful library they could use, because Cells is about change and nothing is more fundamental than change.

Programming is hard because like someone doing a spreadsheet on paper we programmers end up with the burden of propagating change thoughout our models. It is tedious work, it must be done reliably, and there is a lot of it as internal program state multiplies, exponentially a lot. This exponential growth in interdependence of program state is what led Brooks to declare that a silver bullet was not only unlikely to be found but that it would be impossible to find; he felt the complexity was ineluctable because as states multiply there is nothing that can be done to avoid the explosion of interdependence.

I wrote to Dr Brooks recently and asked him if he had ever looked at dataflow. He said he was familiar with the concept, but no. Oops.

Tinkering

Spring cannot come soon enough for Bobi, who is losing it badly these days on comp.lang.lisp:

Slobodan Blazeski wrote:

Dear board members

I'm baseball player for a several time periods (days,

moths ,years,decades) I've noticed that interest in baseball is

dwindling, and baseball is becoming less and less relevant and will

soon become extinct with only baby boomers supporting it, and even

those are either going to die or switch to golf. In order to save our

favorite sport I propose we make drastic changes and adapt more modern

things like:

a. Playing on the beach sand wearing swimwear like in beach

volleyball, very modern sport. Check Thiobe for growth rate

b. Replacing bats  with  hockey sticks. Note that hockey is popular in

many world countries and we should think international

c. Including  24-Second Shot Clock like in NBA that will make our

sport more lively and fast paced

d. Square playing fields should be replaced with the more common

rectangular one like found in many popular sports : soccer, football,

tennis etc

Including this will make baseball prosper.

 

very truly yours

 

Concerned Semi-Ex Baseball Player

Avenue of delusional weirdos Number 23

Bobi may not be as crazy as he thinks he is. Baseball suffered extreme popularity anxiety in the late Sixties and did indeed tinker with the game. Thinking more offense would attract more fans the pitching mound was lowered so pitchers did not get extra energy into the ball from falling into a pitch. The American League adopted the designated hitter to eliminate the 11% nil pitcher from batting lineups (eliminating as well an awful lot of interesting strategy). They avoided the salary caps of the NBA and instituted free agency (well, no, they lost a lawsuit) which allowed bigger markets like NYC, Boston, and LA to buy better teams, and bigger markets are always good for ratings. Minnesota fans will follow the Dodgers, Los Angeles fans will not follow the Twins. 

The changes went beyond the playing field. Ballparks added mascots and a disgusting cacophony of party music between innings so loud you can barely talk, and limited alcohol sales late in games to make the experience more family-friendly cuz you know how the losing fans get in their third hour of drinking.

Now baseball is hugely popular again so tinkering with grand institutions can work. Right?

Wrong. In the end, baseball is just a great game: multi-dimensional and deep. Quality tells, and which quality one emphasizes matters. Hockey and basketball have non-stop action and are fading in popularity, while baseball and football like great music have a variety, a rhythm, a balancing of quiet against intense. Baseball has the pitch, football has the snap. All scales from small to large from inning or drive to the game or season always and invariably end up condensed into one point of explosive tension when the pitcher releases or the center snaps the ball.

Intense without quiet merely exhausts. A boxing match with two brawlers spurning defense landing bombs back and forth brings the crowd to its feet but those who love the sport do so for its nickname, The Sweet Science. They still talk about one genius of defense who won a round without throwing a punch. Between evenly matched fighters one solid punch (forget the knockout, the cartoon haymakers of Rocky n) brings the crowd screaming to its feet, the culmination of rounds of careful, tentative, mutual exploration. A single knockdown becomes a cause for pandemonium and one punch knockouts almost do not happen between the best and when they do they are talked about for a long time. I digress.

Tinkering. Basketball has all the action in the world and now faces its own popularity crisis. Racism is one factor, another is probably the salary cap that has San Antonio in the championship series instead of New York. Another problem: poor defense, and a twenty-point lead does not mean anything. 

But worst of all is the lack of dimensionality. There just is not that much to these games to argue about over the water cooler. Baseball? Boston still talks about the time Grady Little [thx, Xach. ed.] left Pedro Martinez in one inning too long against the Yankees in game seven of the ALCS. Come on, he had thrown a hundred pitches! Everyone knows Pedro is useless after a hundred pitches! You just never hear anything like that about hockey or basketball, which both boil down to great athletes pretty much just playing run and gun.

Baseball never needed tinkering, though tinker they did. The fundamental quality of the game first ensured its survival throught the hard times when fans strayed for the quick fix of non-stop hockey and basketball action. Now the richness, subtlety, and sophistication of the game has some stadiums selling out most games of the year of a very long season.

Moral for Lisp left as an exercise.

Tilton's Law: Solve the Failure First

The team was at my throat.

"Just use the new search!," they bellowed.

The mission critical, project saving, do or die demo to upper management was eight hours away and we had not even begun the always dicey process of moving the software from the development system to one within reach of the Demomeister, and I was trying to find out why the old search was so slow.

"Soon," I replied.

We had a new search I was told was a screamer but I continued poking around putting in metrics trying to figure out why the old search was so slow. Had we not been a virtual remote telecommuting team I would not have lived to tell this tale but we were so they had no choice and I reassured then that "soon" meant ten minutes and they shut up.

Why was I still trying to understand the perplexing sloth of the old when a whole new replacement module was available and working fine and pretty much the demo on which all our jobs and a cool project depended was coming on like freight train?

Tilton's Law: Solve the failure first.

Early on we learned the other side of that coin: Solve the first problem. The commonality is...no, let's do the war story first, war stories are more fun than preaching.

Back we go a quarter of a century to my first contract with a client who would become my sole recurring client for the next decade. I was being hired to take over maintenance of an application whose author had been one of the first to die of AIDS. I was reminded of the whole business by a conversation with another developer recently about the nature of working on OPC. Other People's Code.

In my IT career I have worked always at the poles of software development, either writing new code or performing massive overhauls of OPC, never that relaxed zone between in which one simply maintains and extends in small ways a long-lived system. The second pole (OPC overhauls) always seemed to me an intimate one-way encounter with some anonymous predecessor, an encounter usually involving me roundly and steadily cursing them out. You can imagine then how eerie it was working on this system from this predecessor who was not so anonymous this time, especially when I learned that the poor guy was in bad shape during one stint but needed the money and so worked on the code I was now working on even as his fate rose up to meet him (this well before the days of the cocktails of today that make ones fate less certain). This guy I do not remember cursing out so much.

But I digress. Our lesson today is how to piss off your coworkers by insisting on solving a failure first, by which I mean even if you do decide to punt on X make sure you understand how X failed. I am not alone in this. In 2001 the movie when the crew determines that the unit Hal said was no good was fine he says fine let's put it back in and let it fail. Sure, he was really looking for a way to kill the crew but we learned in 2010 that Hal was just a computer system and I think the bit about putting the supposedly OK/not OK system back in to see if it failed was one of Hal's systems working nominally in accordance with Tilton's Law: we need to understand broken things.

And now at long last, my unsolved failure. My predecessor's, actually. The application was a securities database with a nightly feed of data applied to the cumulative DB by a batch program. This is late 80s, primitive stuff. A security could have three IDs because three groups were tracking securities and each had their own ID system. We had tens of thousands of records in our VAX/VMS RMS file, and a separate RMS key for each of the three possible IDs. So far so yawn. Here comes the fun part.

Two of the IDs were populated all the time. The other one was populated five percent of the time. Big deal, right? Right, very big deal, the poster boy for Solve the Failure First. What happened was this quesswork reconstruction:

My predecessor Paul (I picked "Paul" because it easier to type than predecessor) had a problem. His program ran an initial test load of a hundred securities in a few seconds. Fine. Everything looked good. So then he ran it against a full daily feed, which would include news of every security traded that day so it would be -- OK, I confess I completely forget even the order of magnitude, let's say tens of thousands and declare up front that that is idiotic and I am sorry, but here is what happened: the damn thing ran forever. There probably was no immediate specific great mystery because Paul probably had the program printing something (a count, the last ID recorded, something) right to his VT-100 console as it went and he could see that the program had started out zooming along but then gradually got slower and slower until just adding one security to the database (and this is just good old ISAM, mind you) took... wait for it... twenty seconds. Oh. My. God. What on earth is happening?

Paul got a clue. Every once in a while two records were written out bang-bang, as fast as at the start. Dig dig dig puzzle puzzle...ah, there it is. Any record for which we have all three IDs is written out in nothing flat. Any record (you know, the ninety-five percent) with just two will (by the end of the run) be written out three per minute, 180/hour, or 1000/fuggedaboutit.

Paul realized what was going on. The ISAM file system had no problem storing data with duplicate keys, which was a good thing because Paul was storing a whole lot of data with one key 95% the same: spaces. Poor ISAM it seemed was chugging thru all the duplicates looking for the last one after which it would record the latest duplicate. And apparently it took twenty seconds back then to walk (effectively) the entire index of a hundred-thousand record file.

Now the good news is that we would never need to look something up using spaces as the key value sought, so....what can we do? Paul was no slouch. He popped open the RMS reference manual and to his delight discovered he was not the first to pass this way and gleefully added the option "NULL_VALUE=SPACES" (translated: "if the value is spaces, Just Don't Index this record on this key") to the key definitions in the file definition script he was using to initialize the file and recreated the file and re-ran the program from scratch.

The change did not help. At all. I think we all know that feeling, as visceral as a dropping elevator.

There it was, the option explicitly intended to solve the problem he had explicitly encountered, and it did not change a thing. Impossible. But this happens to us programmers all the time. We know what to do. Compile the damn code, because we made the edit change but forgot to compile. Or link. Or, in Lisp, to zap the faultily specialized method. Or something.

So Paul edited the definition again and checked that NULL_VALUE = SPACES was the right syntax and right spelling and on the right key -- to hell with that, he put it on all three damn keys -- and he saved it and checked the date and created the file again and ran his program again and you know it did not run any faster or I would not be telling this story.

OK, time to get serious. Or if he was good he did this without all the huffing and puffing of the preceding paragraph. He Just Typed In "rms/analyze sdb.dat". And RMS looked at the file itself (not the script used to create it) and confirmed that "NULL_VALUE = SPACES" was operative for all indexes.

Momma don't let your kids grow up to be programmers.

What comes next is hard to convey. I can tell you but if you have not worked on this code or (we will learn) run this batch application it is hard to convey how much blood, sweat, tears, CPU time, and delayed nightly batch closes for how many years resulted from Paul's not first solving the failure of NULL_VALUES=YES.

Well, maybe this is a fair glimpse of the enormity that followed: the problem got sorted out only because the head of operations and I got to talking one day and something reminded him and next thing I know he is pretty much down on his knees begging me to find some way to eliminate the two-hour merge step that held up the nightly close every night. 

"It just sits there for two hours," he groaned. "It kills us every night. Please, if you can, please, do something to make this go away."

Whoa. I had inherited this system and been asked to enhance it but no one had said a word about this. The code was far and away the best OPC I had ever dealt with so everything got the benefit of the doubt, including the (soon-to-be explained) two hour merge. As in, if it is there, it must be there for a good reason. What was not there was The Story of the Unsolved Failure of NULL_VALUE=SPACES, but even if it had been I would have taken that at face value, too, because the NULL_VALUE option was unknown to me. But enough of this flash forward, let's get back to poor Paul.

NULL_VALUE was not working as it should. Software is like that. Good programmers do not let bad software stop them. Plan B. A rule is born: Thou shalt not write new securities to the securities database where the massive duplicates will make each write take twenty seconds. Paul decides to write them to a second file initialized empty on each run. Since we only got dozens of new securities in one batch, that file would never have the massive count of duplicates and writes would be lightning fast. Then we just do a sort/merge at the end of the batch to combine the new securities in with the old. Oops. "Just."

The funny "you can run but you cannot hide" moral within the moral being that I did the calculations one day and worked out that twenty seconds times the average number of new securities in a day was exactly as long as the sort/merge that was just killing the folks down in operations. And I bet Paul realized that but only after writing all the crazy code he had to write to work with two files at once as if there were only one file and at that point he just gave up and moved the thing into production. Speaking of crazy code...

You should have seen it. Looking back I cannot recall why it should have been so hard, but I did overhaul that code and I was forever tripping over it. The idea is simple. To look up a security to see if we already have it, first look in the real DB and if it is not there look in the daily "new stuff" DB and if it is not there, ah, it is new. If it is, update it. Just remember to update the right file, because we can get data from two sources about the same new security.

Piece of cake, right? A bottleneck function for all reads and updates... anyway, it seemed like the issue was always getting underfoot as I worked, and just looking at the code one saw again and again this check here/there code, and both Paul and I were the kind of engineers always on the lookout for ways to make code non-redundant. I would think my memory was faulty but I also remember eliminating Paul's Plan B after solving his failure first and that was no picnic. It just permeated the application.

So what was the first failure and how did it get solved? First, I had noticed the issue myself while using Datatrieve to add a record to the securities DB for test purposes. I hit enter and thought I had crashed the system because it went away and never came back and like every egomaniacal programmer out there I always assumed that whenever a system stopped responding the last thing I had done must have broken it so there I sat in dread for twenty seconds until the system finally responds. Wow. Twenty seconds? And then I guess I added a record specifying all three keys and it responded instantly.

But this idea of null values not being recorded in an index was new to me, and we did not have the Internet back then where I could just ask the ether what was going on so it was only a coincidence that just after the guy in operations had begged me for a fix that I was visiting with the lads from a prior contract and I moaned that RMS sucked because it could not handle files with hundreds of thousands of records and they laughed at me and said they were handling millions with RMS.

I can actually remember the look on my face, a neat trick when you think on it.

I haul ass back to work and pull out the RMS reference manual and I can tell you that dead trees aside there is one good thing about paper documentation: right above the entry for NULL_VALUES close enough to catch my eyes was the entry for NULL_KEYS.

Yep. You need to specify both. Paul had specifed NULL_VALUE=SPACES. He had not specified NULL_KEYS=YES. The default for NULL_KEYS? Guess.

I kinda wretch inside even now thinking about the astonishing amount of money, work, debugging, and delayed batches that followed from one simple failure to understand one broken thing.

The meta-lesson shared with "Solve the First Problem"? In programming, never deal with the unknown. This game is hard enough.

Epilogue

The punchline is that I never solved the first failure from Scene I of this tragedy. As my father used to say, "Do as I say, not as I do." We did have a deadline, and I did narrow down the location of the problem in a way that reassured me somewhat that it would not jump up to bite the new code in the rear end. And even in its breech the law is confirmed: we do need to address the underlying problem which I have some confidence I now understand because it still presents problems for the software but it will go away only when bigger problems are solved and they are much bigger so I am keeping my sights set on them.

The Road to qooxdoo Part III: Why It Rocks

In the episodes one and two we learned all about how a Lisp God and Interwebby know-nothing found a way to support a client needing an enterprise Ajax Web application, said way turning out to be qooxdoo after solid efforts to make jQuery, Dojo, and YUI fill the same bill. But we did not learn very much detail about qooxdoo, we just learned that it comes with lots of doc, lots of examples, a solid community, and that the engineering is first rate. I may not have mentioned this, but it also was dramatically faster/smoother than the others on my acid test: scrolling a datagrid. 

Okay, that is a lot as a general recommendation. I mean we did not learn much detail. Until now. Here are four Big Things that really stand out:

1. HTML? CSS? Never heard of 'em.
First comes a shocker. The qooxdoo user codes no HTML or CSS and needs know nothing about those two disappointments. Just as my Lisp compiler reads my Lisp code and spits out optimized native machine code, qooxdoo takes your qooxdoo/Javascript and produces the HTML and CSS to drive the browser. Upside: qooxdoo developers write less code and never have to worry about browser variability again.

To a Web noob like me there is a second big win here: I do not have to learn HTML, CSS, or the vagaries of the different Web browsers my client might want supported. I am a bit of a rarity in my ignorance of those things but soon I should have a lot of company now that qooxdoo has set the bar as high as it has on insulating the web app developer from lower level technology.

By the way, in case you are wondering: yes, there is a widget (two, actually) that will accept classic HTML.

2. I do declare. Not!
The second thing that hits us is a consequence of the first: we are going to be writing a lot more Javascript than we would be with libraries such as Dojo where authors still write HTML/CSS. With declarative tools we sit in Lisp or Ruby or PHP on the server generating HTML. Not qooxdoo (but see next)  and not YUI.

I am not at all happy about this but I will live and if it ever bothers me enough I will spend a weekend exploring the qooxdoo markup contrib called QxTransformer. Right now I need to catch up on a backlog of functionality the client has had to do without because of the six weeks it took me to identify qooxdoo as The One.

3. OO is not the Grail, but OO rocks
Stendhal said it best in "On Love" in the chapter on Infatuation: (paraphrasing poorly) we do the object of our desire two injustices, first setting them on a pedestal unreasonably high and then, when they inevitably disappoint, setting them too low. OO is not the Grail, it is just a profound advance in serious application development. All by way of introduction of another stunner: qooxdoo's slightly more substantial object model layered atop the core Javascript OO capabilities. 

To be honest this struck me as a bit of a yawn until I decided to start carving up my existing code into distinct qooxdoo classes. This is a very big application and I needed code re-use almost immediately and doing it in HTML with Javascript glue was...well, before qooxdoo I was pretty much starting to build my own abstraction layer atop the core tools--there was no way around it.

But now that I have used qooxdoo's OO to partition my rapidly growing code base I have more than once had that wonderful sense of getting something for nothing after doing a major refactoring (you know, taking all the pieces and throwing them up in the air) and having it Just Work after two or three tweaks. Furthermore, I will not bore you with the details but I have not even started leveraging the full capabilities of qooxdoo's object layer: even more benefit will be reaped moving forward because of some nifty reactive capabilities.

This came as a pleasant surprise. I do more OO in more ways than you can imagine in part because I normally program with Lisp's CLOS which is to normal OO what normal OO is to binary arithmetic but by the time I stumbled onto qooxdoo I had been doing Javascript, HTML, and CSS for a couple of months under a lot of pressure so I never had the time to build a sensible OO abstraction atop Javascripts primitive little OO pretensions. Thus Javascript had endowed me with a bit of child mind and I was able to experience the mundane code structuring win of OO as if for the first time. It rocks.

3. Layout the Wazoo
I may have lied. This may have been the first stunner I felt in my gut. The others I knew would be huge if they panned out, but this was the first one I saw in action working as beautifully as had the other frameworks left me hanging. With other frameworks I could find no way to get built-in layout schemes to use the full window real estate available. That cuts two ways: I either ended up with unused whitespace (and with our app we need all we can get) or my application panels grew so large that the browser itself put up scroll bars, meaning I had to window-scroll my scroll panes around. No, that was not a candidate for production release. Note that to a large degree this is the same as point #1: a framework is not hiding HTML/CSS from me if I am still victim to surprising browser decisions, such as oh gosh that is big let's use a scroll bar here.

How good is qooxdoo at layout? When I popped open the Firebug debugger in FireFox, where by default it begins by seizing application window real estate, the qooxdoo layout engine handled the downsizing impeccably, right down to adjusting scroll bars to accurately reflect the new dimensions.

4. Message In A Bottle

This feature is huge in a small way, by which I mean I could probably roll my own version of qooxdoo's Message mechanism in a an hour but--wait for it--I did not have to. They did. That is huge, the huge thing being that the qooxdoo team really has set out to create a compleat Web application development environment and with such a beast comes pleasant surprises such as their Message mechanism which allows a nice decoupling of widgets: one widget can dispatch "Make it so" and anyone responsible for "it" knows to roll up their sleeves simply by having subscribed to "Make it so".

Beyond the Message mechanism we have the Event mechanism. The documentation of every widget lists the events it might generate and what data the event will carry, and in toto we have an engine with an event model dozens of times richer than pure DOM.

5. I know, I said "4"

Number five is an umbrella win, too detailed to list: as I whined above, this Lisp God is now writing an awful lot of Javascript leveraging more and more qooxdoo all the time. Again and again your highly critical correspondent is finding nothing but good news in qooxdoo. 

Yes, I could make a bug report every other day if I had the time, but a long time ago I figured out that I am not looking for a perfect library, I am looking for a powerful library that is fundamentally sound (so anything broken can be readily fixed) and actively maintained (so things actually get fixed).

Summary

qooxdoo: fast, powerful, complete, rich, well-engineered, and it hides browsers, HTML, and CSS. Great doc, loads of examples, fine community. It's all good.

The Road to qooxdoo - Part II

Last time, we learned how one self-appointed Lisp God with no knowledge of web programming ended up programming the web and then soon enough embarked on a six week grand tour of Javascript frameworks in search of a platform suitable for an enterprise web application and where his tour ended: qooxdoo. That episode ended with a promise of more information on qooxdoo, but first a note on the methodology: there was none.

My first experiment with web programming had thanks to the marvels of Lisp in general and AllegroCL's web tools in particular mushroomed into a full-blown web application aided here and there by jQuery widgets before we slowed down enough to think, Damn, we could benefit from a full-blown Web application framework.

 So I eyeballed the usual suspects and ran them through googlefight and picked the most promising one and used that to redo what I had accomplished to date. Which was: tab controls, select boxes, data grids as views into server-side data stores, tree views, text input, tool tips, pushbuttons, pop-up windows, and probably one or two things I am forgetting.

How did I assess the libraries? We needed a professional look, good performance, and maximum productivity. Sorry if that seems blindingly obvious. The productivity assessment has some beef: I looked for lots of documentation and tons of examples describing a rich set of widgets at once high-level, feature-rich, and yet highly authorable which may not be a word but it means I get to override stuff without resorting to back doors. 

I considered also two metaqualities: on-line support and internals code quality. Having the Lisp HyperSpec an F1 away is just a little faster than asking for the same information on comp.lang.lisp. ie, Community matters. As for code quality, the good thing about open source is not that it is free, it is that I can use the source to understand things at 3am when the community might be quiet; I can add print statements to debug my code; and maybe I can find and fix a bug in the library. But if the code is of low quality good luck with anything more than the print statement insertion.

It was a rough six weeks. Three times in six weeks I logged onto support groups and said, "Hi, this is my first day..." as I rebuilt a relatively feature-rich web app three times while staggering up to speed three times on three environments. All while the client is getting absolutely no new functionality. I feel bad all the while because they need this stuff now. I am letting them down and there is the strong possibility they will be letting me go: I had been brought on to do the Web front-end. I like learning, but not so much with a gun to my head and a time bomb ticking.

Are we having fun yet? Well, yes. The last framework we tried was qooxdoo and it has worked out great. I can now give the client pretty much what they want fast and looking good once they pony up for a graphic artist because I just saw this impressive example of what can be done with their "appearance" framework. But I ain't no designer.

Funny thing, though: qooxdoo  like YUI and Dojo came up short during the evaluation. It had no example of the widget I needed most, a fairly standard widget at that: a datagrid backed by a remote store. Bad sign. Second, qooxdoo did no better on googlefight than does Lisp, and that is pretty bad. Finally, qooxdoo like YUI lacks a declarative programming model [though the declarative QxTransformer has been revived since my original investigation] which means I cannot just sit in my Lisp IDE catching XHRs and tossing back HTML/CSS. This would put a non-trivial dent in my productivity.

So how did qooxdoo prevail? What can I tell you? Bullet charts are useless, expert systems work great as long as expert humans interpret their output, and resumes tell us nothing about the work we will get out of people. 

qooxdoo did have a contributed example of a remote data store which pretty much worked and when I looked at the qooxdoo internals to learn how to make it work better I found very good code and rather quickly had the datagrid I needed. Otherwise qooxdoo did have loads of examples and documentation and decent community (I say "decent", you'll say "great"--my standard is comp.lang.lisp) and then we get to the bottom line.

While resurrecting my app for the third time with qooxdoo, yes, I worked just as hard as I did using Dojo and YUI. But the effort was different. With qooxdoo I was struggling to learn a very nice framework that needs an index: I was slaving for six hours to find the simple six lines of code I had to write. With YUI I was struggling with a framework that did not work all that well and scared me more and more the more I looked inside it. With Dojo the struggle was with a good framework but the struggle was such that I suspected the struggle would never diminish and probably increase exponentially as the application grew in complexity.

As for the declarative thing, hey, I am a programmer, I can write code. And Javascript has a lot of Lisp soul, I am not stuck with something daft like PHP or Python. Sure, having to spend more time in a JS IDE means a ten percent productivity hit but ten percent we can do. The terror we developers dread is the tool nightmare death by a thousand cuts two steps forward two steps back rope drag death. 

I will blog on rope drag sometime soon, but first let's get specific about qooxdoo. In Part III of my Road to qooxdoo.

The Road to qooxdoo

[The title of this rather boring entry derives from a fun little survey I kicked off years ago.]

OK, why should you listen to a Lisper on Javascript frameworks? Because we use Lisp so obviously we have better instincts when it comes to developing applications. In fact, this road would not have ended so well had not Lars, a fellow Lisper developing a killer new Lisp Web programming framework called Symbolic Web (SW) brushed aside my enthusiasm for Dojo and mentioned he found qooxdoo interesting, although for now SW uses a little jQuery. "A little" because the whole idea of Symbolic Web is to do most of your programming in Lisp which of course is the only way to go but SW is still emerging from the sea and I had a requirement now (well, six weeks ago, but there is major new release of SW just out I need to look into but as this Road will reveal I am not likely to change horses now because of what I found in qooxdoo (not that SW will not eventually catch up)). Gasp.

The second question might be how a Lisp god and Interwebby know-nothing ended up doing heads-down intense Interweb development. Well, it was the Lisp credentials. A former client using mostly Lisp was having great luck with a Ruby/Rails front end but was looking for more speed and hoped having a Lisp image serve the pages directly would make for better response.

What happened next was pretty neat, and again Lisp conquers all. I told the client I would look at what they had and what they needed for a few days to decide if even I could help them, because I can do a lot of things but one thing I cannot do is exaggerate how little I knew about Web programming and even static HTML eight weeks ago.

A glance at the existing screens and Rails code had me thinking, "No way", but I was curious anyway about even getting an application to drive a web site so I fired up a tutorial on Web app development using tools from my Lisp vendor of choice, Franz.

How good are Lisp, Franz, the tutorial, and the WebActions tool? In the three days I was supposed to be deciding whether I could help the client I managed to execute a respectable fraction of the first component they needed. Sorry, no screenshots, NDA and all that.

So now there I am with a contract and a browser toolbar sagging from the weight of bookmarks to sites on HTML, CSS, and for the hell of it jQuery though I was not sure why but everyone was jumping up and down about it. And then I needed it.

Well, not jQuery. I needed a razzle-dazzle spreadsheet-like grid presenting data from an arbitrarily large table of data held on the server. Now at this point I had already rolled my own Ajax paging mechanism, but something superficial but nonetheless important for that this being an important application was holding me back: my HTML looked like crap. Meta-holding me back was my sense from surfing the Web that getting good results out of HTML/CSS was a black art requiring years of practice which certainly leaves me out. Enter FlexiGrid. A URL which as I write responds "Please contact the billing/support department as soon as possible."

Which brings me to why I dumped jQuery. What I found was small, yes, and add-ons, yes, but a hodge-podge of add-ons does not a professional development environment make. By the time I punted I had contribs from three people and while each was lovely alone together they looked like, well, a great decorator on a multiple-personality binge. And beyond appearance there came a surprising result: FlexiGrid did not offer a "row selected" event. Meta-worse was the author's response. "Right, no row-selected event." Ooops. Next up was jqGrid and that was fine but I still had Crazed Decorator Syndrome going and now I am looking for the next widget I need and a light went on: we are building an application here, not a Web page/site. It is wonderful that jQuery is small, but small is exactly the wrong thing for developing a sophisticated application with a coherent look and feel never mind coherent engineering underneath so widgets can play well together. Enter Dojo. But first...

In case you are wondering where the client went, they are encouraging my search for a pre-fab solution. Probably seeing the gorgeous FlexiGrid widget compared to my ugly efforts did not hurt, but the client has good instincts anyway (they use Lisp, right?) so they are leaning towards my original assessment that I am overmatched if I am going to try to turn out a hot application on short experience with HTML and CSS even though at this point I have wondered aloud to them how much HTML/CSS I might have mastered in the same time I was spending beating my head against pre-fab, this being the classic failed deal we make with the devil when we reach for 4GL: yeah, wow, a complete browser screen in five minutes. Gee, could we get the first sub-item appearing on the same row with the item? No. The client wants it. It cannot be done. That was Datatrieve on VAX/VMS twenty-five years ago, but nothing has changed. 4GLs get their apparent power by making decisions for me which works only if I (and my client) are not making any decisions.

But no, the client says "find a framework" and I decide to try again with Dojo. Again?

I had looked at Dojo before. The word on Dojo was incredibly good. Of course that word was from their Web site. When I tried to confirm by eyeballing the doc I could not find it. I mean, there were some great links, but all the pages they led to were blank. Next! But now I was back, encouraged by the news that OpenLaszlo had just placed a bet on Dojo. IBM as well, but OpenLaszlo is more important because I know they are geniuses, they have a dataflow hack like mine.

Oooh-ooh! Another reason Dojo looked good was a benchmark I had found showing the thing was fast. I have no idea if the benchmark is valid, but the pictures are gorgeous so I trust it. So why did I dump Dojo?

Well, OK, I did not really. It came in second. The question is why did I keep looking such that I found the eventual winner. Easy: too frickin hard to get to work. The documentation is awful. They know this and are addressing it but did I mention we need this now? Overall Dojo fit the bill of being a Compleat Solution with solid backing and a great future, but after banging my head against the wall over something (ah, it was the six hours I spent trying to figure out the "build" mechanism for a compressed production release of my JS culminating in watching a blurry video that finally showed how to do it) I went for a better way surf (as in there's gotta be) and saw YUI touted as a serious solution.

A quick glance at the enormous volume of well-organized documentation and exhaustive examples and the client and I agreed at once to give it a try. And I will tell you right now YUI has the best graphic designers, because their stuff looks hot. Dojo is a close second there, too. YUI is a complete solution with a big backer, gorgeous style, and more doc than you can imagine. How did it get dumped?

Stuff was not working well. Even with all the doc and examples it was hard to get working. When I dug into the source to see why things were not working, I did not like what I saw. And the death knell: YUI abandons the declarative model. Why is this so bad? Hellooooo, Lisp? Remember?

Up thru Dojo I was programming in Lisp 90% of the time, doing just enough JS to get information back to the Lisp server application where HTML could be generated and sent back, even that Lispily thanks to HTML generating macros. And YUI was going to make me give that up and be as hard to figure out as Dojo and I suspect worse under the hood. So regretfully I informed the client my battles with Dojo would be resuming.

So how does this road end up at qooxdoo?

The weekend was coming up. Weekends are mine, so I decide to follow-up on Lars's hint about qooxdoo. qooxdoo is the ultimate threat to YUI because qooxdoo also says Just Code Javascript, and if I am forced to do that by YUI I may as well do it in qooxdoo where the code is as good as YUI's is bad. (I looked at about fifty lines of it for five minutes, I should know.) Anyway....

qooxdoo is amazing. They need help on the graphic design (never look directly at their tab control, use a mirror) but other than that the engineering is terrific. [My bad: they simply leave anything more than the utilitarian look up to the developer, witness this before/after case study.] The documentation and examples are also abundant here, and the support is great. They even do dataflow! Yes, I wish I could spend all my time in Lisp sending over HTML, but two things on that.

One, there is a side-project that brings a declarative model back into qooxdoo. QxTransformer. I have no idea where that stands, but I did see a note on-line where a developer spoke of it in the past tense. [Now resurrected.]

Two, hey, I can write some Javascript. I did ten years of C before I did fifteen years of Lisp, Javascript is not a problem. And it even has lambda, something this guy still does not understand. Lisp/markup would be better, but when dealing with mission-critical stuff (the client is doing valuable data crunching but does not win until they make it accessible) any team can roll up its sleeves and crank out the work even if they have to do without the brilliance of Lisp.

This entry is getting long and I am getting tired and my bartenders are starting to worry about me so I will leave a detailed praise singing of qooxdoo to my next entry, but allow me a bit of a rant. What a mission-critical effort cannot stand is fundamentally flawed solutions that seem to offer a fast track but in fact bleed developers to death. This can be jQuery with an incomplete solution forcing me to try to build an application out of mom and pop contribs or YUI with fundamentally bad code or some other framework that might be just plain slow.

As a developer I am not looking not to work, I am looking to have my work yield quality results predictably. I might spend four hours trying to figure out how to do something in qooxdoo but once I do it turns out to be insanely simple and work beautifully. Moral: their documentation needs an index! But I am keeping my big yap shut because documentation is a bitch and they have done a ton of it and I do not document my stuff so...I keep my big yap shut. But the point is that my work now yields results reliably and predictably and this will accelerate as I learn the framework and even learn better how to navigate the documentation.

qooxdoo not only fits the bill of being a complete solution well documented with many examples but it also has high-quality code under the hood, which both gives me confidence that I have found the right tool and not incidentally serves as useful further documentation. More next time.

Why Lisp Packages are so Easy (Hard)

Slobodan Blazeski wrote:
> May somebody please explain me with simple words why cl packages are
> hard to use, 'couse I really don't get it?
> The best is probably to hear from somebody who just learned lisp or
> somebody who teaches lisp (do such people exist anymore?) and his/her
> students has problems with packages.

...or a good teacher, defined as someone who remembers what it was like before they knew what they were teaching.

The first problem is one of those immune system deals. Packages have a lot of the same proteins as, say, C dot aitch definition files so the recognition systems of programmers coming to Lisp from other languages (you know, everyone) does its damndest to parse them as dot aitches.

In fact, packages have nothing to with anything but symbols, neither the functions nor global values they might name. More expansively: packages are about mapping string names to first class Lisp symbol objects and dividing this up into discrete mappings called packages so the same string "Hi, Mom!" can map to multiple |Hi, Mom!| symbols in multiple packages.

The second problem is that symbols take us straight to black-belt Lisp in which one must understand the Lisp reader and fancy things like read-time and compile-time and all that. Example:

I remember getting messed over because some symbol was mysteriously appearing the base CL-USER package. I could not for the life of me see where it was being introduced. I turned the problem over to a few trees worth of monkeys and one of them came up with this rule:

"Never use a normal symbol like |Hi, Mom!| in a defpackage form coded in a source file with (in-package :cl-user) at the top which would likely be the default even without that."

Instead, use a string like "Hi, Mom!" or an uninterned symbol like #:|Hi, Mom!|.

Strings are dicey because you might want to export 'banana and be silly enough to code "banana". Lisp will take you seriously and create |banana| instead of BANANA, case sensitivity being what it is except when it is not when Lisp folds 'banana to 'BANANA which is what it does unless you are in Modern (case sensitive) mode.

Did I mention "black-belt"? The fundamental idea is simple, but noobs get thrown off mostly by the non-simular simularity with C and then in the debugging stage by case sensitivity seeming to come and go at random and the whole read-time compile-time thang.

hth, kxo

A Beginners Guide to ASDF (Ha!)

OK, there is at least one thing about which Lispers are not so smug: ASDF (Another System Definition Facility), the de factor standard for the Lisp library equivalent of Makefiles, and my post today was prompted by ANFOBA (Another Noob Effed Over By ASDF) washing ashore on comp.lang.lisp, and how I almost responded there:

Francogrex wrote:

Hi, I am struggling to understand how to work with asdf.

I have been using it for ten years and I am about 50-50 now on getting it to work. I am assured it does a great job in working with a multitude of Lisps/OSes, and I can assure you it is a disaster at working with programmers.

Come back here and post console output freely, you will need help, especially because one of its great triumphs is useless error messages.

Hints:

(1) It will work. You will have to promise it your first-born, but it will work.

(2) No, PB was not joking: you have to separately push locations onto some magical special variable. Leave off a slash (or add a slash, I forget) and ASDF will take your head off, so don't do that.

(3) No, PB was not joking, the syntax is:

(asdf:operate 'asdf:load-op :cl-pdf)

Why is it not (asdf:load-system :cl-pdf)? In my experience, extremely smart programmers think it astoundingly brilliant when "one function does everything!!!!", remembering the ridiculous op-code first parameter left as an exercise.

Since you asked, this is because extremely smart programmers are tone deaf to programmer productivity; they are so smart they do not even notice when they have created something unuseable, they just brain through it.

(4) ASDF has a featurebug: it loads in the wrong order in order to make McCLIM work. Don't ask. If the author of the .ASD file retreated to the desert for forty days to slave over the dependencies this will not be a problem. Otherwise do what I do: take a sledgehammer to the beast:

(asdf:operate 'asdf:load-op :cl-pdf :force t)

That abandons the whole point of having a build system, but it gets ASDF to work. When for the love of god I am just trying to build this frickin thing!!! that is a nice tradeoff of compile-time vs tracking down and short-sheeting the author of ASDF.

(5) On success ASDF emits more messages than an IBM OS/360 core dump. This is so any errors will scroll well out of view and anyway stand out like a healthy thumb from the surrounding...um, other healthy thumbs. Anyway, do what I do: ignore all messages other than a backtrace and Just Try the Hello World(tm).

If you get a backtrace, Just Post It to c.l.lisp.

(6) If hello world fails, f*ck! But at least now you are a Real Lisp Programmer(tm), this is what we live with.

hth, kenny

The Foisting of An Infinite State Machine

"I will not allow you to foist this convoluted scheme on the bank!", Sam yelled.

This was fun in a lot of ways. First of all come on this is a bank, three or four full floors of programmers all enthralled by nothing more technical than getting through the day unfired and back on the train home to their families and here is this aging cardpunch nutjob throwback not only noticing Someone Else's Code but caring?!

Second, I did not make that up. Sam actually used the word "foist" without trying. I use SAT words all the time but I am usually trying, you can see my eyebrows arch a little as my language organ reaches for the thesaurus. Nothing shabby about "convoluted", but "to foist"? And Sam tossed that off in earnest in anger in the heat of diatribe sans affectation and it is not clear to me I have ever before or since heard the word used in vitro. Which brings me to the third fun bit.

Sam was Jewish. Classic Orthodox home-by-sundown-on-Friday Brooklyn Jewish and of course he would be good with words. Right, get all PC on me and denounce me but what can you say?, he used "foist" in a sentence on the fly and I think I win on this. So there he is with this classic Billy Crystal Princess Bride accent yelling at me for foisting convoluted code on the bank.

"I will not allow this!" he cried. "I will stop you!"

Oh, please, tho Sam was a sweetheart. Yeah, "was". I heard he passed, Lord keep him, a beautiful soul. I could have choked him in that moment but at least he was attacking me for Code Quality. What was my sin?

I had been tossed a throw-away job of persuading two applications to exchange information reliably when either app was free to collapse in a heap at any time and I was not to lose any information. Nowadays we just pull packages off the shelf for that but this was a while ago. Oddly enough in my first IT job ever I had had to do the same and it was fun because there was nothing theoretical about it, the two applications did even during development reliably disappear left and right allllll the time. Decnet, PDP-11, RSTS/E, 1981... you understand.

What was different was that somewhere in the intervening ten years I had gotten this crazy idea of creating a domain-specific language to make it easier to develop a software Algebra tutor so I had gotten a book on compiler design because I did not realize I could Just Use Lisp. I do not like reading so I did not get very far but fortunately one of the earliest chapters was on parsing so I ended up learning about finite state machines. Oh, sorry.

This post came up because I am doing some Web2.0 programming these days from Lisp and I wanted to send over Lisp sexprs and since Javascript for some reason lacks a Lisp reader I was going to have to parse it myself and after ten minutes of trying to do it without a finite state machine I realize, damn, I should code up a finite state machine.

I will not in this space try to explain what is a finte state machine but it is really simple and powerful and no matter what level programmer you are you need to have it in your, well, toolbox. I think the example I learned on was a pocket calculater when folks still had those. Your initial state is "initial". Duh. Then what happens? Oh, I got "/". Bzzt! Or just ignore it and my next state is still "initial". Now I get a "1". Great! Save that, my new state is...I dunno..."got a digit". We're on a roll. But wait, what other inputs could I get? A "+". Ummm...well, it seems no different, but we'll say "got a plus sign", and indeed when we are done one of the cool things that can happen is that as we fill in the table we may well discover that the row for the state (did I mention that rows were for states and columns were for inputs? Sorry) ..the row for the state "got a plus sign" might be no different from the row "initial" and we first go "whoa, enlightenment" and then collapse the two rows into one. I digress.

But the idea is that we just break it down into cases. A fun example is that we now know whether a "+" is addition or a plus sign (or an error) without really trying, because we have aggrssively exploded the analysis into all these different states like "just got a digit" or "just got a left parens" or...yeah, it goes on, but guess what? Here comes the name! "Finite State Machine"! There are only so many!!! You may think there are a kabillion but after ten you are done, maybe after twenty. And you can collapse inputs (1-9 become "positive digit") and so you have a finite number there. Do you now have 150 cases to handle? Yeah, and each one you can code in your sleep. The alternative is one hyperglutionous mass of conditionals, tests, and branches that will fail on the one millionth execution because of a first time path execution.

Back to Sam.

I have realized that the reliable exchange between two autonomous systems permitted to fail at any point will best be solved using my old friend the finite state machine and I have enthusiastically shared this with Sam. Somehow I am now being denounced for using an algorithm learned from the top book in compiler design. Specifically, I am being denounced for foisting. I will be stopped. The bank will be saved from my convoluted scheme.

Ooooookayyyyyyy.

Well, what can one anachronistic lost in the Torah yamaha wearing card punching goto coding nut job do? Would you believe he could force a code review? Nay, a defense. Sam forced a frickin defense of about thirty lines of the best simplest most powerful correct code that bank had ever seen. Mind you Sam is also the man who produced my favorite line of code ever, using the Vax Basic statement modifier mechanism:

return if flag

That's right, an action at a distance code teleportation straight out of the middle of seventy line for loop using a flag he had named.... flag. You can imagine my reaction. And this guy is challenging my code?

If you are not yet suitably horrified, understand that novice programmers were regularly crashing production runs because no one anywhere was looking at their code and they were going to call a meeting of like twelve programmers to review my foistation of convolution upon the bank? Sorry, my ire is showing. Chase cut to.

Code review/defense: twenty minutes, here it is, any fucking questions?

Yes that was pretty much the tone of my presentation -- I like to consider people skills one of my strengths. And then Sam spoke and you will see why I loved him. At pain of repetition, probably no one else on those four floors would have even listened to me as I bragged on my FSM, but go to the mattresses to stop what he thought was bad code? One in a million. And one in ten million in my experience would have said what he said.

"OK," Sam he said. "That looks fine."

Of course now I am pissed off because I am just getting my turbines spun up for a full bore antler to antler knock down drag out kickin and a gougin in the mud and the blood and the beer round fifteen thrilla in manila finish and Sam is saying...fine?

"That is simple," Sam continued, gesturing towards my whiteboard diagram. "This infinite state machine idea is complicated, but the way it works is simple."

The beauty of that being......no. I would only ruin it.

RIP, Sam. RIP.

URBBR #2: How Bad Is This Book?

[The following has been... well, only the names have been changed to protect the endangered toads.]

We have a One-Paragraph Test for fiction. Not a complicated test, it involves reading the first paragraph and then deciding. If We are really undecided We can read a second paragraph, but that is a Bad Sign.

For non-fiction We look at the blurbs the author managed to suck out of unwilling blurbers. Well, OK, We do not really want the blurbs to be from blurbers, We want them to be reviews and We know they have cherry-picked the review sentences so everything gets discounted about an order of magnitude. 

And there are two such reductions applied, the second for the source of the review. Kirkus Review of Books... well, just quoting them gets a book back onto the shelf. In the wrong section, upside-down, binder in.

Here is the Gold Standard for non-fiction back covers (hmm, the URL goes to the whole book, be clever and click "Back Cover"): The Glory of Their Times. Yes, you need to read this book, and no I do not give a rat's ass if you like baseball just read it.

We turn now to a book I have been asked to review, which will go unmentioned for the same reason I restacked it binder-in:

"This work strikes a balance between the pure functional aspects of Blub and the object-oriented and imperative features that make it so useful in practice, enable .NET integration, and make large-scale data processing possible."

The clever unwilling blurber always manages to say something nice by talking about something else, in this case the language .... instead of the book for 90% of the blurb. What does he say about the book? 

It strikes a balance. Wow. How big a put-down is that? Note that the reviewer did not say "a perfect balance" or even "a good balance". And what the Hell does balance buy me anyway?

The good news is that We cannot discount that 90%, Our micrometer does not go that low. But We better check the author of the blurb.

—John Doe, PhD, Researcher, Blubber Ltd.

Oh. The people selling Blub.

The neat thing is that We now know the book is awful and that John was slowest to hide under his desk when the marketing people came around looking for a recommendation.