Tilton's Un-Law: Treasure the First Problem

Here is a good one from the trenches.

Once upon a time I said that when a multitude of problems present themselves we should decide which seems most like the first (independent of anything else the first observed, etc) and solve that first.

Perhaps I should have said identify the fault underlying the first problem.

This morning I am hacking away at Tilton's Algebra getting ready for a big pilot test and looking at several observed problems trying to help the student factor -x^2-2x-1, which can be solved as -(x2+2x+1) then -(x+1)(x+1) and finally -(x+1)^2.

• The app will not accept -(x+1)^2 as the answer. It says I can do more work.
• When then asked for a hint, it does not have one to offer. Wait, you said I could do more work!
• When asked for a hint after -(x+1)(x+1) it says it does not have one to offer, either.
• If I then ask if -(x+1)(x+1) is the answer, it says no (which is fine but again inconsistent with not having a hint to offer.)
• Final observed oddity: on a different problem such as -x^2-5c-6 -> -(x+2)(x+3) it will accept that as the answer.

So what is the first observed problem? It is a tie:

• If I first ask for a hint on -(x+1)(x+1), it has none to offer.
• If I first say -(x+1)^2 is the answer, it says I can do more work.

So here I would say the failure to hint comes first, because I could reasonably ask for a hint before reaching what I think is the answer. Put another way, it is not so much the order actually encountered, it is more that I can see a problem exists as early as when I might ask for a hint.

And the underlying fault is quickly found:

The hint mechanism looks at where the student has gotten in a solution to decide what to suggest next. In this case, the engine thinks the student is just plain wrong, because it itself came up with a weak answer: (x+1)(-x-1). 

But why did it accept -(x+1)(x+1)? Aha! Big story begins: this is an educational app, designed mostly with struggling, unhappy students in mind. It does so with an expert system engine programmed to be able to do Algebra. This engine sometimes goes wrong, as in this case. When that happens it tells the student they are wrong when they are not. I for one consider that a Deadly Sin.

It occurred to me that simple numerical methods would be able to determine that a "wrong" entry by the student was consistent at least with the original problem, so a failsafe mechanism was introduced to let slide anything that looked wrong but was not, in the hope that it would work out in the end.

That trick actually fools the hinting mechanism, which looks back to the last correct step to decide what to hint. Unfortunately the safeguard decided to say a step (mistakenly) considered wrong should be labelled correct. As we learned in the movies 2001 and 2010, it is not wise to deceive software. Better would have been to flag the step accurately as "wrong but consistent", so the hinting engine would not try to hint off it.

And in the case of -x^2-5x-6 the safeguard does succeed at least when it comes to accepting the answer. But in neither case is it able to offer a hint once the student enters, say, -(x+2)(x+3) because it thinks they are just wrong and as noted above the engine looks to see where they are in the solution. In this case it decides nowhere, so it has no hint to offer.

Interestingly, when asked if -(x+1)(x+1) is the answer it says no because it sees it can be rewritten as -(x+1)^2! This then reveals a second fault:

The engine when looking for a hint and checking if they are done looks in two different places, so it can say "not done" then have no hint to offer. This sin is not as bad, but it could be deadly: the student will lose confidence in the software, at which point no matter what mistake they make they will think the app is at fault.

Re that last: helluva war story from the enterprise trenches when a single bug trashed insurance enrollment data significantly but not so badly that it got noticed immediately. By the time the complaints had rolled in and enough investigated to reveal the corruption reversion to a backup was not viable. Users had to deal with complaints for weeks while a fix was found, so once it was they continued to question the data for an additional month or two before confidence was restored.

 Now it may well be that fixing the original fault (the engine's answer of (x+1)(-x-1) will eliminate all the other misbehavior, but remember the spirit behind the failsafe: if we can avoid abusing the user when the engine goes wrong (as it may) we should.

Now in the past I have come this way before and simply fixed the engine and moved on, but then it was impossible later to design a safeguard! The engine flaw has to be authentic, because a fix for a contrived flaw might well not work on the real thing. And engine flaws are rare enough that I just forget the whole thing until a day like today comes along.

So this time I am not fixing the first problem, Go-live is approaching and soon living breathing struggling unhappy students will be at my app's mercy. I am treasuring this problem for the stress it puts on the rest of my app, because I accept that the expert engine will not always be so expert.

Only after all the downstream misbehavior has been addressed will the first problem be fixed.

 

Wow, even Lispers hate Lisp!

Not sure yet what is going on here but someone on comp.lang.lisp just made an interesting post, interesting because it seems to reflect a substantial and well-informed effort (nice!) all to denigrate -- wait for it -- a computer language. (Really?)

Not sure how that works. I think Python bites so I do not use it, I do not labor over detailed treatises to post on comp.lang.python. But it is a great treatise so I will promote it some here.

We have perhaps yet another measure of how great is Lisp: even its haters are its fans. They come to boo, but come they do. Like Gorgeous George and his understudy, Muhammad Ali, Lisp puts ticket buyers in seats as much to be hated as used.

WJ on comp.lang.lisp wrote:

Worshippers of CL (COBOL-Like)

Dying is easy, comedy is hard. Humor must have within it some truth for it to work. The laugh comes from the accuracy followed by a sharp turn to a put-down, the sharp turn being the key. If Lisp is not like COBOL there is no sharp turn and the put-down becomes mere name-calling. Sticks and stones and all that.

Please try not to try to be funny again.

are the most slavish punks that
the world has ever known.  The hypertrophied CL hyper-spec is
their Holy Bible.

Their are the worst enemies that Lisp has.  (And they never win.
Only the most mediocre of programmers are willing to slave over
the CL hyper-spec.) 

We need the spec when we want to get fancy with format, 'mkay?

And yes, it is actually a Religion to these degenerates.
Don't believe me?

Kenny Tilton:
We are the Priesthood.  Offerings of incense or cash will do. 

I take it back. I have now worked with some pretty awful Lisp programmers. They are better than any non-Lisp programmers, but any incense or cash should go to me alone.

Here's what the experts say about COBOL Lisp:

"an unwieldy, overweight beast" 

The only thing I do not use is series.  Hmmm, I should go look that up.

"intellectual overload" 

For most of you, yes. Lisp is for smart people.

"did kill Lisp" 

That explains the smell. Sorry, I forget the author of that one.

"A monstrosity"
"ignores the basics of language design"
"killed Lisp"
"sucks"
"an aberration"
"the WORST thing that could possibly happen to LISP"
"incomprehensible"
"a nightmare"
"not commercially viable"
"no future"
"hacks"
"unfortunate"
"bad"

In context:


Guy L. Steele, Jr., July 1989: 

He went from Lisp to Constraints and butchered that, from there to Java and can take a lot of credit for that steaming pile of turd, and has now abandoned that failure to work on... Fortran?

I think we may usefully compare the approximate number of pages
in the defining standard or draft standard for several
programming languages:

  Common Lisp   1000 or more
  COBOL          810
  ATLAS          790
  Fortran 77     430
  PL/I           420
  BASIC          360
  ADA            340
  Fortran 8x     300
  C              220
  Pascal         120
  DIBOL           90
  Scheme          50 

50? Check again.

-----


Brooks and Gabriel 1984, "A Critique of Common Lisp":

Every decision of the committee can be locally rationalized
as the right thing. We believe that the sum of these
decisions, however, has produced something greater than its
parts; an unwieldy, overweight beast, with significant costs
(especially on other than micro-codable personal Lisp
engines) in compiler size and speed, in runtime performance,
in programmer overhead needed to produce efficient programs,
and in intellectual overload for a programmer wishing to be
a proficient COMMON LISP programmer. 

Yeah, Graham said that, too: a language has to fit in your head. I am thinking he did not program enough to learn it really well*. And he prolly did not have the hyperspec an F1 away either.

* I am pretty sure a constraint on a language I intend to use all the time should not be that I have to be able to remember it all when I am not using it all the time.

-----


Bernard Lang:

Common Lisp did kill Lisp. Period. (just languages take a
long time dying ...)

You know Lisp is the perfect language precisely because it lost its momentum (died) and lives on. Other languages need their Next Big Thing momentum.

It is to Lisp what C++ is to C.  A
monstrosity that totally ignores the basics of language
design, simplicity and orthogonality to begin with.


-----

Gilles Kahn:

To this day I have not forgotten that Common Lisp killed
Lisp, and forced us to abandon a perfectly good system,
LeLisp. 

The French are still pissed off about Lance Armstrong!

-----


Paul Graham, May 2001:

A hacker's language is terse and hackable. Common Lisp is not.

The good news is, it's not Lisp that sucks, but Common Lisp.

Historically, Lisp has been good at letting hackers have their
way. The political correctness of Common Lisp is an aberration.
Early Lisps let you get your hands on everything.

A really good language should be both clean and dirty:
cleanly designed, with a small core of well understood and
highly orthogonal operators, but dirty in the sense that it
lets hackers have their way with it. C is like this. So were
the early Lisps. A real hacker's language will always have a
slightly raffish character. 

Come on, we have "goto". What else does he want?

Organic growth seems to yield better technology and richer
founders than the big bang method. If you look at the
dominant technologies today, you'll find that most of them
grew organically. This pattern doesn't only apply to
companies. You see it in sponsored research too. Multics and
Common Lisp were big-bang projects, and Unix and MacLisp
were organic growth projects.


-----

Jeffrey M. Jacobs:


I think CL is the WORST thing that could possibly happen to LISP.
In fact, I consider it a language different from "true" LISP.

  *****

Common LISP is the PL/I of Lisps.  Too big and too
incomprehensible, with no examination of the real world of
software engineering.

...  The CL effort resembles a bunch of spoiled children,
each insisting "include my feature or I'll pull out, and
then we'll all go down the tubes".  Everybody had vested
interests, both financial and emotional.

CL is a nightmare; it has effectively killed LISP
development in this country.  It is not commercially viable
and has virtually no future outside of the traditional
academic/defense/research arena. 

Can I respond to all these "CL killed Lisp" quotes by pointing out that they are wrong? I am developing The World's Greatest Algebra software on Windows 8 (after 7 and Vista and XP), pushing to GitHub, pulling onto Linux and away we go*. Standards avoid the disaster of Scheme, which brilliantly recreated the problem CL had to fix: disparate implementations leading to unshareable code. *Okay, I am using compilers from the same vendor, but I am also using a ton of open source written mostly for SBCL.

Common Lisp worked, but it got the blame for Minsky's (inter alia) over-promising and under-delivering on AI.

-----

Paul Graham:

Do you really think people in 1000 years want to be
constrained by hacks that got put into the foundations of
Common Lisp because a lot of code at Symbolics depended on
it in 1988? 

Constrained? Where? Who? How? I am writing amazing code and having a blast between bong hits. Where's the problem?

-----

Daniel Weinreb, 24 Feb 2003:

Having separate "value cells" and "function cells" (to use
the "street language" way of saying it) was one of the most
unfortunate issues. We did not want to break pre-existing
programs that had a global variable named "foo" and a global
function named "foo" that were distinct.  We at Symbolics
were forced to insist on this, in the face of everyone's
knowing that it was not what we would have done absent
compatibility constraints. It's hard for me to remember all
the specific things like this, but if we had had fewer
compatibility issues, I think it would have come out looking
more like Scheme in general. 

I think DW meant "looking more like Scheme" as a good thing. Ouch. I used Arc for a while and it had one namespace and it was not fun.

-----

Daniel Weinreb, 28 Feb 2003:

Lisp2 means that all kinds of language primitives have to
exist in two versions, or be parameterizable as to whether
they are talking about the value cell or function cell. It
makes the language bigger, and that's bad in and of itself. 

I think some of these smart guys are thinking themselves into to many rules and regulations and way too much "should". I am not that bright so I am free to just get on with the programming, and when one is programming one knows what works and what does not. One namespace does not.

-----

Paul Graham:

I consider Loop one of the worst flaws in CL, and an example
to be borne in mind by both macro writers and language designers. 

Except there is no non-Loop code posted to comp.lang.lisp that I cannot make clearer and faster with loop. And it took me a very long time to come around to loop, so I should know. Yes, the syntax can make you weep*. Until you learn it. Then it is an undisputable win.  * Hey, it is a DSL. Ya gotta learn it, like any language. And it is a DSL for iteration, something that comes up more than a little in programming, so the effort (and the DSL itself) are justified.

-----

Dan Weinreb, one of the designers of Common Lisp:

... the problem with LOOP was that it turned out to be hard to
predict what it would do, when you started using a lot of
different facets of LOOP all together. This is a serious problem
since the whole idea of LOOP was to let you use many facets
together; if you're not doing that, LOOP is overkill. 

I wonder if Dan wrote enough code. Loop never surprises me, now that I have learned not to close over loop variables.

-----


From: John Foderaro
Newsgroups: comp.lang.lisp
Subject: Re: the "loop" macro
Date: Sun, 26 Aug 2001 10:51:26 -0700

I'm not trying to join a debate on loop.  I just wanted to present
the other side of [the issue so that] the intelligent people can
then weigh the arguments on both sides.

I'm not suggesting that loop can be fixed either by adding
parenthesis or coming up with ways of indenting it to make it
understandable.  It's a lost cause. 

Hey, that's the guy that did IF*, a DSL for conditions!

Why does all the above miss its mark? Because it is 2013 and all that was written when CL got created back in the eighties and we are still using Lisp and when we do we use Common Lisp. All those quotes are by people who saw the politics and the big spec and were unhappy about the compromises and the size of the resulting language. But I use it all (except series) so they were wrong about the size.

Meanwhile the raison d'etre of unification was achieved. Yobbos are writing open source on SBCL and I am going to use it to change the way the world learns math, atop AllegroCL. A brave attempt to resume the fragmentation (scheme) failed. Mostly because they were wrong about language design, but also because of -- wait for it -- fragmentation. Perfect.

Fortune Cookie File Number Two

Wow, what an effort! Nice followup to part one! Thx, Madhu!


%
Are you trying to win a Moron Contest*, or did I miss a joke?
-- Kenneth Tilton
%
[Quoting some "Bell Labs engineer" from the Newark Star Ledger]

"The hardest part for me was realizing I was being tolerated by all the people I had been tolerating."
-- Kenny
%
That is a metaquestion. No metaquestions allowed under Obama. Note that this is a metametaanswer, it's like minus signs, ya got two ya ain't got any. With me so far?
-- Kenneth Tilton
%
Let's decide on the shape of the table before deciding if the discussion to decide if free open software is ethical is going anywhere.
-- Ken Tilton
%
Pioneers do not wait for the rest areas on the interstate highway to
have sushi bars.
-- Kenny
%
The Mac I have now sounds like a 747 turbine after spinning up, can't
even hear The Voices let alone program a computer.
-- Kenny <490ba086$0$5643$607ed...@cv.net>
%
What part of Anna Kournikova do you not understand?
-- Kenny
%
That will be well-received, and when Anna Kournikova sends you an
email asking you to come spend a week with her you complain that she misspelled Tahiti and the airline tickets she sent were not first-class and write back asking if she knows anyone prettier you could stay with?
-- Ken Tilton
%
I think Celtk just needs Cells. This is like saying that if you want
Anna Kournikova to have your baby then you have to sleep with her.
-- Ken Tilton
%
Anna played tennis?
-- Ken Tilton
%
Muhammed ibn Musa al-Khowarizmi! He wrote the book on Algebra!
Literally. Not sure where I can get a picture, tho. Maybe I can give
Anna a beard...
-- Ken Tilton
%
I feel a naggum coming on. When you think you know what I am thinking try to stop thinking.
-- Kenny
%
[...] you should -- I feel a naggum coming on -- stop trying to impose
your prior understanding on a new experience. You are like an
American tourist landing in Kinshasha and going in search of a Burger
King. Sadly they do not have to go far.
-- Kenny
%
I feel a Naggum coming on. Before you contradict me or challenge me or quote me, please make sure you are not in fact talking about an inferior model of me rooting around somewhere in your cortex (and what a scary image that is).
-- Ken Tilton
%
New Yorkers are not offensive. The joke is a good one (The proper way to ask for directions in NYC? "Excuse me, can you tell me the way to Lincoln Center, or should I go fuck myself?") but the reality is New Yorkers are so nice they will gladly give you directions whether or not they know the way.
%
[...] the mentally ill are often the most compassionate because of what they have endured.
-- Kenny
%
Instead you are like a new yorker asked for directions and before they
can even get out the words "...to Carnegie Hall" you have responded
"Use Google Maps. And rent the French Connection. The chase scene
covers most of Manhattan."
-- Ken Tilton

[Actually it was shot in Brooklyn -- rc]
%
Your other mistake is thinking I was planning a 500-page treatise
complete with suggested legal forms. I am an American, we have Cliff
Notes for haiku.
-- Ken Tilton
%
I will be honest here, I am just a humble application programmer, and as an American I barely know where Monte Carlo is because we only need to know where is Las Vegas.
-- Ken Tilton
%
> "If Lisp is so great why don't libraries, etc. exist for it like
> they do for Ruby, Python, ...".

You are wondering why Shakespeare never conceived a hit game show like
Deal Or No Deal, and why Tiger Woods sucks at miniature golf. Why
Pavarotti never has and never will make it to the Billboard Pop 100,
and why Dale Earnhart got fired after one week driving a taxi in NYC
(he could only make left turns).
-- Ken Tilton
%
The first time you run into something is only the first time you will run into it.
-- Ken Tilton
%
Then again, I do not see why you even want to see it, you already have that sanctimonious glow from using free as in how we defined it software.
-- Ken Tilton
%
What I saw was a defense of Java as being halfway to Lisp and the bit about him having a chart trying to close all the possible holes where behavior was unspecified. True Lispers laugh in the face of unspecified. Hell, we pay extra for it.
-- Ken Tilton
%
Meanwhile Steele famously claims Java is halfway to Lisp. Perhaps he
meant starting from the stone axe?
-- Ken Tilton
%
Apparently these geniuses think it matters one whit whether the spec
in its entirety can be carved on the head of a pin.
--Ken Tilton
%
Now there's a language design principle. Another good one is that you
can sing it to the tune of Camptown Races.
-- Ken Tilton
%
"...Those who can't do, teach. Badly. I am reminded of the New Math,
which made Principia Mathematica (?) a first-grade textbook."
-- Ken Tilton
%
".... Never heard from again, though rumor has it Steele found work
as a tech writer for Sun."
-- Ken Tilton
%
[on motivation to finish his product]

Not to worry, I have a friendly letter here from the IRS asking when
they might expect to see last year's taxes, those tend to focus the
mind wonderfully as well. :)
-- Ken Tilton
%
wants to follow The One True Lisp Way and trust us to know what we are
doing, so compliance here would be compliance for it's own sake.

I must need a drink, that last word looks like a Japanese malt brew.
-- Ken Tilton
%
Had you responded here instead of by email I could have eviscerated you in public (the only thing I really enjoy, the only thing that sets me apart from serial killers).
-- Ken Tilton
%

> And thank you all, (every #'primep ...) worked great. I found it is
> a very nice and helpful place here!

Just wait until you put a parens on its own line, the honeymoon will
be over fast.
-- Ken Tilton
%
> I try never to memorize what I can just look up.

Right. I never memorized C precedence, I dog-eared that one page in
K&R and/or threw in a pair of air-bag parens and skipped the lookup
altogether,
-- Ken Tilton
%
But that code is quite solid and close to Deeply Correct. I know because it has not changed much in years and handles new requirements effortlessly, generally by /taking out/ code that was enforcing disciplines which turned out not to be necessary (and in Lisp we hold inalienable the right to shoot off our own toes).
-- Ken Tilton
%
In the end I remembered that I have never let a concern for accuracy
get in the way of my rants, way too much work.
-- Ken Tilton
%
hard-charging newbies such as yourself landing in Lispville dumbfounded by all the dust, cobwebs, rust, and neglect giving the boot to the war-weary, disheartened, parentheses-mocked old soldiers rolling up your sleeves and setting about dragging the damn language out of the seventies and into the 21st century just in time for the asteroid to hit. What was the question?
-- Ken Tilton
%
You want there to be a problem, just like the strong static typers
want there to be a problem. Unfortunately for all you finger shaking,
rule making, strait jacket wearing school marms we have a nonexistence
proof of craploads of great code being written without a problem in
spite of your sky is falling obsessive compulsive gnashing of the
teeth.
-- Ken Tilton

%
>> Wow, that is two non-required requirements in a week. Me, I am
>> looking for a transmission that can go from forward to reverse at
>> fifty miles an hour without self-destructing. I don't have a need
>> for this, I am just looking for it.
> Easily done. Not so easy is to allow any human passengers to
> survive the event.

Reminds me of the guy I met who said he and his buddy agreed at
sixty-five miles an hour to find out what would happen if they applied
the parking brake. Let's just say it is a good thing that they had
agreed on it, and that the rental car company did not ask how their
car ended up upside down.
-- Ken Tilton
%
If you have to use so many big words, you must be wrong.
--Ken Tilton
%
Please follow up, I want to see if my killfile is working.
--Ken Tilton
%
Yeah, yeah, it was just a rant, you never want those held back by
concerns over accuracy. The sexp/mexp thing esp. suggests divine
inspiration might be a better model than alien arrival.
-- Ken Tilton
%
Unlike the inability of a deliberate mention of Hitler to function as
would an emotionally honest invocation of same to signal the end of a
flamewar, one can apparently climb up on top of the nearest car hood
and announce one is starting a flamewar just to irritate people and a
crowd will immediately form to argue with one over doing so.

I once saw a nature special in which some insect or other dragged some other dead insect somewhere then turned around and dug a whole for it to bury it but the researchers moved the dead insect a bit while it was digging so it had to drag it back but while it did they filled in the hole and back and forth this insect went indefinitely until a PETA sniper took out the researchers. Where was I?
-- Ken Tilton
%
>>I realize other people prefer other environments, they are just
>>mistaken. My ideal setup happens to be the best, hands down.

Wow, I am really out on a limb there. It would be pretty easy to take me down by naming a superior or even near equal environment. Or you could back down in the face of my confidence and resort to, I don't know, name-calling?

> You're clearly deluded.

Understood.
-- Ken Tilton
%
I am not, really. I will do a year or two of Algebra and then have
enough money to open a bar, do what I really want, tend bar.
-- Ken Tilton
%
Mind you I /have/ AG, but there is no point in doing cells-rdf if the
rest of you food-stamp licking, government cheese eating, thrift
shopping oooh-its-gotta-be-free pikers won't be able to benefit from
my unceasing thankless toil on your behalf.
-- Ken Tilton
%
> Cells or Cello might be the solution. But getting mad at people won't
> help.

Thanks for your concern!

Not to worry, I abuse these yobbos for fun, not out of anger. And I
am not sure c.l.l would know what to do with KGK (Kindler Gentler
Kenny), but...
-- Ken Tilton
%
Otherwise, sorry Karl, we are discussing the best way to learn Lisp
(free ACL trial on win32), not the best way to resurrect your six-feet
under commie pinko socioeconomic theory.
-- Ken Tilton
%
> Can someone with a bigger brain than me please elucidate.

I think the problem is not brain-size so much as your admirable attempt to understand a tool by reading about it. That puts you at the mercy of technical writers, who combine an inability to program at all well with an inability to write.
-- Ken Tilton
%
> check this out buddies

I have a buddy?!!! woo-hoo! I thought I had alienated everyone with
my sarcasm!!!!
-- Ken Tilton
%
Tell us more about your home planet.
-- Ken Tilton
%
I'm gonna hurl. Come on, everyone, newsgroup hug....
-- Ken Tilton
%
(a) You seem to be unaware of the Laws of Conservation of Hyphen
Momentum: according to the CLHS, a term in hyphen motion tends to
remain in motion, a term at hyphen rest tends to remain at rest.

(b) Really, the worst thing you can do in CL is use a macro where a
function would do. The Pope does not sudo ex cathedra to say, "It's
not the heat, it's the humidity."
-- Ken Tilton
%
Was the author writing under the cloak of infallibility, channeling
the word of G*d, and is our Talmudic interpretation of those awkward
words precise? The legislative history shows that CL got designed to
address concerns of The Big Customer over language fragmentation, so
it is hard to imagine an intent other than to define one language.
The next step was a pretty tight ANSI standard language specification.

Too easy? (I know, it is more fun being contrarian.)
-- Ken Tilton
%
The language lawyers cannot save you. I am your only hope. I am a
simple application programmer.
-- Ken Tilton
%
Fine, bring me a single malt, a pint of amber back, a wedge of cheddar and some saltines. And we'll need more napkins before we're done with this design.

ps. Oh, and another jar of dijon, and ask the redhead under the moose head if she would like to join us.
-- Ken Tilton
%
> What is a `hacker', or `programmer', or `computer scientist'?

The last two were dragged to their death in the last thread. Hacker is a term used by us computer geeks in a desperate attempt to glamorize our bit-ridden asses, as if the best of us will ever get laid as often as the tone-deaf, rhythym-blind bassist of a third rate cover band on Long Island, let alone the rock stars we pose as when we call ourselves hackers. Paul Graham, who I generally greatly admire and hope will because I said that fund my start-up but more lavishly than he does those Y-Combinator conscripts, drove a stake through the heart of the term here: [snip]
-- Ken Tilton
%
I actually had a business card that just said "Programmer". Got everyone quite upset, they wanted "Systems Analyst" or "Software Engineer" or "Database Administrator" or something. My point was that one cannot program a computer effectively without doing all those things, so "Programmer" was sufficient.
-- Ken Tilton
%
Now can we get back to name-calling? Stop trying to civilize this
brawl.
-- Ken Tilton
%
>> I always tell youngsters it is OK to take one year off before grad
>> school, but for the love of god don't take two.
> I think this depends on the person.

Never look a gift joke in the mouth.
-- Ken Tilton
%
> My theory is that is we bought and open-sourced [...] we could get
> the community to rally around that one,....

The idea of this Lisp community "rallying" is about as conceivable as
a hootenany down at the cemetery.
-- Ken Tilton
%
It is the mouse that feels bad, not the cat playing with it.
-- Ken Tilton
%
The big mistake is thinking Lisp is going to grow by first being
adopted in Tall Buildings. They are the drones, the lemmings, the
sheep. They follow where We the Blessed Gurus lead them. But this
time it is to the slaughterhouse, because the world needs only fifty
Lisp programmers to write All the Code.
-- Ken Tilton
%
... listening instead of yapping? Read my frickin lips: I am talking
about actual tall-office design reviews in which well-paid engineers
... pragmatically suggested design disasters because they objected to
anything a pet rock (sorry, Rockie) could not code.
-- Ken Tilton
%
Anyway, if one has not programmed heads down for three years one likely does not know much about design. I am sure I write more code in a year than academics write in a lifetime, because we are doing different things. Hell, they have the sorry task of trying to pretend there /is/ such a thing as computer science. If there was, wouldn't everyone be using Lisp?
-- Ken Tilton
%
Unfortunately I still do not understand the question, and I am a
frickin genius, I have a three-digit IQ, 50% more than 2.
-- Ken Tilton
%
Well the trick of folks like H.. is to listen just enough so that one can respond to a direct hit with multiple non-pointing counterpoints, each more retarded than the last and each stated in an artfully needling fashion guaranteed to make the sanest NG denizen continue the thread, as if the /next/ direct hit will achieve any more than the last.

It's like playing paintball with a guy who keeps running around and
shooting, covered head to toe in paint.
-- Ken Tilton
%
> deal more. But after seeing your behaviour in cll you can be
> assured that I will never, ever consider any functional programming
> language. I can just do without languages that attract that kind of
> behaviour.

-- tim (and yes, before you respond, that is one reason I
don't use CL so much any more as well.)

Abandoning something wonderful because of who else uses it makes
perfect sense. Why did I give up sex? One word: Joey Buttafucco.
-- Ken Tilton
%
Never been to a code review, have you? You are blessed. The worst crap in the world gets protected by the manager because he is the only person in the room more clueless than the author of the crap and dies when the author dies because the author is in effect a buoyancy device for the in effect non-swimmer manager. But I digress.
-- Ken Tilton
%
I cannot write correct code to save my life, I just throw out any bad code. Only trick there is to distract the author with a banana while deleting their code.
-- Ken Tilton
%
You know, the oil companies have developed a car that runs on carbon dioxide and has like 800 horsepower. Where you can buy one is another question.
-- Ken Tilton
%
Me, I saw the "Microsoft Research" oxymoron and did not get much
further. Unless by research they mean using Google to find out what
ideas other people have successfuly developed and commercialized so
they can copy it badly and crush them.
-- Ken Tilton
%
> Yawn. You must be a riot at parties.

And you must be the life of a funeral.
-- Ken Tilton
%
The way to get this going is to post here an especially good RQ
question and your Lisp solution, see if you can drum up interest. If
it takes off, you start a Web site or something. If not, the ball
game comes on at two.
-- Ken Tilton
%
ps. I agree, the "fingers will be chopped off" sign should not have
been Comic Sans. :)
-- Ken Tilton
%
Well whaddya know. I do GUIs, leave file work to the chimps. You win
a banana.
-- Ken Tilton
%
Can we continue this over on comp.lang.turing.complete?
-- Ken Tilton
%
...the only thing that matters is Becoming the Latest Thing. What is
the latest thing? The (a) new thing (b) being recommended (c) by
Famous People (d) in respectable places
-- Ken Tilton
%
The operator helpfully suggests that I could avoid this problem by simply saying that my mother's maiden name is YOBBO, no one will make fun of me. He also takes care of confirming the purchase over the phone, while I try to figure out how to sell "Hi, I need to change my mother's maiden name..." to the next operator.
-- Ken Tilton
%
So I am working in a lab? That would explain all the beakers.
-- Ken Tilton
%
That rose girl is drying up and a few volcanos need sweeping.
-- Ken Tilton
%
Grapette this: no, Jodie Foster is /not/ responsible for John Hinckley
shooting three people including the President. The correct question
was "Who is John Hinckley?".

Really, kiddies, it is OK to blame the perpetrator. Not that I do in
this case. The OP is clearly an unhappy puppy deserving nothing other
than compassion for the demons that drive them to randomly attack
Usenet blowhards like me.
-- Ken Tilton
%
And Uhl says Kenny has made the South forget the Civil War, not sure
how I could top that. The hounds are exhausted, smiling in their
sleep. It's all good -- but someone has to talk Bubba and Jethro down
from their sniper nests.
-- Ken Tilton
%
Do they have the distinction between a priori and a posteriori in your
banjo-pickin, moonshine-slurping, carcinogen-growing,
basketball-playing neck of the woods? How about de jure vs. de facto?
Simular.
-- Ken Tilton
%
That /was/ a despicable and wholly unjustified ad stateum low-blow, a
cheap shot deliberately designed to make me look bad. It crosses a
shocking line, beneath contempt, really. Clearly I had cut in my
flamethrower after-booster and lost all reason or sense of decency.
It almost makes me wonder if I was even serious...
-- Ken Tilton
%
See, this is what happens when you get all riled up and change the
subject to whether Kenny is consistent or not, you get so emotional
you cannot read straight. And my inconsistency varies quite a bit, so
I do not know if you can build an argument on that anyway.

ps. Random problem cloning is going well, but obviously slowly enough
to have me dashing here to hide pretty regularly. :) What kind of Lisp
did you write today?
-- Ken Tilton
%
MWUAHAHAHAHAHHAHAA!!!!! Yes, palindromeloopstate set to 1 makes the
movie run endlessly forth and back. Picture looks fine running
backwards, but instead of the spoken words coming out in reverse,
well, it is just unrecognizable noise. Working on that now...
-- Ken Tilton
%
Fine, but anyone who uses Usenet knows that that is not how one says
"Thanks". Which is why I felt safe moving directly to defcon 3, just
to see how big an *sshole we had on board. His defcon 1 "Rot in hell
jackass" response merely, well, QED.
-- Ken Tilton
%
[...] the serious answer comes from the Tao Te Ching, lessee, on about
every other page:

"A man on tiptoe cannot walk easily."

Or:
"Never trying to impress, their being shines forth Never
saying 'this is it', people see what the truth is -- Never
boasting, they leave the space in which they can be valued ...
And since they never argue, no one argues with them either."

Lao Tzu clearly would be no fun in a flame war.
-- Ken Tilton
%
"Gotcha" is like analogies, they lead to arguments about the
gotchaness of the gotcha, an exponential explosion guaranteed.
-- Ken Tilton
%
Despite copious opinion to the contrary, I speak not as Grand Poobah
of Common Lisp and have not the power to marshall our forces to
maximize the benefit of their labor. I can only pause between rides
to town on my pushbox to wonder aloud in camp why a serous chunk of
our tribe is over there under the tree working on the wheel and
axle -- ones no better than the one on my pushcart.

Kenzo: "Dudes, it's been twenty-five years, prices down at Throg's
Wheel & Axle aren't all that bad. Why not work on a mast and sail?"
-- Ken Tilton
%
A subtle execution of the tip of a tongue pressed against the upper
teeth with sprays of spittle coming out either side probably is not
what you had in mind.

Hmmm. Then we change the spelling to Lithp, and never have to hear
that stupid joke again. Our slogan can be "Thay it loud, thay it
proud."*, and we already have the frickin lambda.

Or "Out With Lithp!".
-- Ken Tilton
%
Barker: "Stairway to Heaven! Open to all! Come on up! One-day
amnesty!"

Kenny: "No elevator?"
-- Ken Tilton
%
No, I made a few changes and sent it to the Copyright Office.

For Mom I am holding out for the cover of Wired. (She already takes
America's Most Wanted.)
-- Ken Tilton
%
"you also physically incapable of understanding that your opinions are
not the words of God". Nice comeback on originality! Physical
understanding? What part of the mind-body problem do you not
understand?
-- Ken Tilton
%
No, I would mock you for being so locked into mob-rule and aggression that you think my insistence on walking to a different drummer entails also abusing anyone not conforming to my non-conformity.
-- Ken Tilton
%
"wet feet" might not do justice to the 3D learning curve -- that
subsonic rumble shaking your kayak is Niagara Falls.
-- Ken Tilton
%
"Feeling no pain, Kevin?"
"Sorry?"
"You just came out of the women's rest room."
"Look, I had to take a leak. Odd, no urinals, just sit-downs."> & no
'bouncers' made any approach..

It's crowded, they are still working their way through the crowd.
Don't worry, we'll explain about the Lisp "high".
-- Ken Tilton
%
er, um... no. OO is about managing huge wadges of code in huge
systems that will be spending most of their lives waiting on disk I/O
so WTF cares about OO overhead, we need to manage these huge
codebases!!! ie, OO is for lazy-ass mo-fos who cannot be bothered to
toss of a few dozen lines of code and reinvent "objects" in a fashion
screamingly optimized for their application.
-- Ken Tilton
%
So it samples the pitch and guesses at rhythm and provides tutoring
similar to what I would get from a good music teacher? Astonishing.
AI has been solved! Stop the presses on the emasculation of George
Bush, we have real news!!!
-- Ken Tilton
%

> if Clisp is so good where are the commonly used apps?

If George and Barbara had such great sex, how did they produce Jeb and Dubbya?
-- Ken Tilton
%
> I like the lizard.

No, you don't, you are just saying that. Think again.

> But then, I like the Geico gecko, too.

Everybody likes the GG. It has an australian accent.

> And I like Common Lisp, too. Guess I'm weird, hunh? ;-}

I was thinking "doomed".
-- Ken Tilton
%
And a mascot like Joe Camel to suck in the kiddies, gotta have a
mascot.
- Ken Tilton
%
You yobs might want to ... oh, what's the use? Lisp is dying. The next generation of Lispniks is over on #lisp worshipping themselves and learning nothing and achieving less. My god, another two years and Slime may be half the power of the ACL IDE. What is the word I am looking for... ah, here it is: PFFFFFFFT!
-- Ken Tilton
%
You French really are pissed off about Lance Armstrong, aren't you?
-- Ken Tilton
%
* reminds me of part of a route description to a rock climb called
Death's Door: "Don't use the jug handle just to the right of the
finger jam, that hold is part of Cakewalk."
-- Ken Tilton
%
I have worked with body shop programmers who could not be bothered to write structured code. Are the concepts of structured programming too hard? Nah, those people just "refused to be bothered" (a direct quote), meaning they were too inured to the pain of spaghetti coding to realize how much "bother" structured programming could save them. They thought spaghetti code was /easier/ because, hey, how hard does one have to think to add another GOTO? It breaks somewhere else? Add another GOTO! C'mon, this is easy! Breaks somewhere else...read my lips: GOTO!
-- Ken Tilton
%

The King is Dead!? Long live... Scala? Clojure?!

Whoa, why wasn't I told Java is closing its doors? I guess I have been out of touch, word seems to be everywhere. I had to go here (blog of the guy who created Groovy) to find out. James is whooping it up over Scala as Java's successor. Yes, the guy who invented Groovy prefers Scala. Quite a bit:

I can honestly say if someone had shown me the Programming in Scala book by by Martin Odersky, Lex Spoon & Bill Venners back in 2003 I'd probably have never created Groovy. -- James Strachan

Damn. So he pretty much invented Groovy by mistake? Did not know about the two-year old Scala? No one mentioned it to him? Groovy got admitted to the standard in the meantime?

Well, Johnathan Edwards reinvented Python Trellis (ergo Cells) without knowing it, and I did not know about Garnet's KR or constraints -- but Groovy got adopted as official Java! You think Scala might have come up over coffee. Anyway...

Steele said Java brought the world half-way to Lisp. I do not think Lisp means what he thinks it means. Proof might be how hard it is for folks to climb out of the pit of javathink. If Java had been a stepping stone to Lisp it would have made the next step easier, not harder. But Java still cannot do closures. Please. And a quick look at closures in Scala has me thinking, omigod, they call that closures?

Clojure starts to look like a Good Move. I see it mentioned in writings on the death throes of Java and that is a big marketing win. The superwhacky thing here is that both Scala and Clojure are syntactically discontinuous from Java. Folks always thought successors had to have syntax similar to the succeeded though Dylan should have served as cautionary counter-evidence.

No, it is not the syntax. The necessary bridging element seems to be....wait for it...the Java runtime! How did that tail end up wagging the language adoption dog? But Clojure gets the nod along with Scala just for sitting atop the JRE! You people scare me.

Well, if Steele were right Clojure would prevail over Scala. Right now googlefight has Scala winning five to one. Maybe Rich Hickey can move 17% of the world 90% of the way to Lisp?

To write, or not to write?

>> By the way, to change the subject a little, who was it who said,

>> "there are no dead languages, only dead minds"?

>

> Dunno, but to change the subject even more, Socrates objected to writing

> since it deprives an idea of a mind in which it can "live". So yeah.

Interesting. "Free writing" is a form that lives within a mind but also create a permanent record and slow the mind down enough to achieve more coherence so the mind can work out hard problems. Comedy writing is necessary to trigger a laugh response because every word matters, but then the words must be delivered as if they were coming live from a mind. Exceptions are improv and semi-improv such as Eddie Izzard, of which Mr. Socrates would approve because they arise within a living mind.

I get a lot of complaints about the writing in this blog because I deliberately write as chaotically as I think. Other times I found I gave a much better talk if I read from something written beforehand precisely because otherwise the living mind is too chaotic to get the talk done in anywhere near the time available.

I was just getting ready to videotape an improvised bit to get the good bits to then pull into a fixed, written bit because I am finding good stuff comes out only if the mind is not slowed down as by free writing.

The question is whether Eddie Izzard is lazy, or if Socrates is right on this. Does Izzard do better by capturing his improv and distilling it down to a precise fixed bit, or does he do worse? Or does he just lack the ability to deliver the prepared as if it were unprepared.

We're getting pretty close to talking about programming in Lisp vs NotLisp now. Lisp programming unconstrained by static typing and blessed with a rich library once that library is mastered such that it is all at the programmer's fingertips allows the code to flow freely yet mostly correctly from a live mind even as that mind is forming the solution the code embodies. Diagram that.

I Feel A Naggum (RIP) Coming On: Quads

I sometimes begin c.l.l rants with "I feel a naggum coming on...". What is a naggum? Normally:

naggum (n): A rant along one of Erik Naggum(1965-2009)'s themes.

That might be self-referentially hopeless which is fine because that is not what I am talking about, I just thought that would be a clever title. In this case a "naggum" is a nugget of Erikian technology. First, his specification of what he called quads (see below), and my poor implementation (even further below and good luck even figuring out how to test it) .

kt

#|

From: Erik Naggum (erik@naggum.no)

Subject: Re: XML->sexpr ideas

Newsgroups: comp.lang.lisp

Date: 2004-01-19 04:24:43 PST

* Kenny Tilton

| Of course it is easy enough for me to come up with a sexpr format off

| the top of my head, but I seem to recall someone (Erik? Tim? Other?)

| saying they had done some work on a formal approach to an alternative

| to XML/HTML/whatever.

|

| True that? If so, I am all ears.

Really? You are? Maybe I didn't survive 2003 and this is some Hell

where people have to do eternal penance, and now I get to do SGML all

over again.

Much processing of SGML-like data appears to be stream-like and will

therefore appear to be equivalent to an in-order traversal of a tree,

which can therefore be represented with cons cells while the traverser

maintains its own backward links elsewhere, but this is misleading.

The amount of work and memory required to maintain the proper backward

links and to make the right decisions is found in real applications to

balloon and to cause random hacks; the query languages reflect this

complexity. Ease of access to the parent element is crucial to the

decision-making process, so if one wants to use a simple list to keep

track of this, the most natural thing is to create a list of the

element type, the parent, and the contents, such that each element has

the form (type parent . contents), but this has the annoying property

that moving from a particular element to the next can only be done by

remembering the position of the current element in a list, just as one

cannot move to the next element in a list unless you keep the cons

cell around. However, the whole point of this exercise is to be able

to keep only one pointer around. So the contents of an element must

have the form (type parent contents . tail) if it has element contents

or simply a list of objects, or just the object if simple enough.

Example: 123 would thus be represented by (foo nil "123"),

123456 by (foo nil "123" bar nil "456"), and

123456 by #1=(zot nil (foo #1# "123"

bar #1# "456")).

Navigation inside this kind of structure is easy: When the contents in

CADDR is exhausted, the CDDDR is the next element, or if NIL, we have

exhausted the contents of the parent and move up to the CADR and look

for its next element, etc. All the important edges of the containers

that make up the *ML document are easily detectible and the operations

that are usually found at the edges are normally tied to the element

type (or as modified by its parents), are easily computable. However,

using a list for this is cumbersome, so I cooked up the «quad». The

«quad» is devoid of any intrinsic meaning because it is intended to be

a general data structure, so I looked for the best meaningless names

for the slots/accessors, and decided on QAR, QBR, QCR, and QDR. The

quad points to the element type (like the operator in a sexpr) in the

QAR, the parent (or back) quad in the QBR, the contents of the element

in the QCR, and the usual pointer to the next quad in the QDR.

Since the intent with this model is to «load» SGML/XML/SALT documents

into memory, one important issue is how to represent long stretches of

character content or binary content. The quad can easily be used to

represent a (sequence of) entity fragments, with the source in QAR,

the start position in QBR, and the end position in QCR, thereby using

a minimum of memory for the contents. Since very large documents are

intended to be loaded into memory, this property is central to the

ability to search only selected elements for their contents -- most

searching processors today parse the entire entity structure and do

very little to maintain the parsed element structure.

Speaking of memory, one simple and efficient way to implement the quad

on systems that lack the ability to add native types without overhead,

is to use a two-dimensional array with a second dimension of 4 and let

quad pointers be integers, which is friendly to garbage collection and

is unambiguous when the quad is used in the way explained above.

Maybe I'll talk about SALT some other day.

--

Erik Naggum | Oslo, Norway

Act from reason, and failure makes you rethink and study harder.

Act from faith, and failure makes you blame someone and push harder.

|#

(in-package :ukt)

;;;(defstruct (juad jar jbr jcr jdr)

(defun qar (q) (car q))

(defun (setf qar) (v q) (setf (car q) v))

(defun qbr (q) (cadr q))

(defun (setf qbr) (v q) (setf (cadr q) v))

(defun qcr (q) (caddr q))

(defun (setf qcr) (v q) (setf (caddr q) v))

(defun qdr (q) (cdddr q))

(defun (setf qdr) (v q) (setf (cdddr q) v))

(defun sub-quads (q)

(loop for childq on (qcr q) by #'qdr

collecting childq))

(defun sub-quads-do (q fn)

(loop for childq on (qcr q) by #'qdr

do (funcall fn childq)))

(defun quad-traverse (q fn &optional (depth 0))

(funcall fn q depth)

(sub-quads-do q

(lambda (subq)

(quad-traverse subq fn (1+ depth)))))

(defun quad (operator parent contents next)

(list operator parent contents next))

(defun quad* (operator parent contents next)

(list operator parent contents next))

(defun qups (q)

(loop for up = (qbr q) then (qbr up)

unless up do (loop-finish)

collecting up))

(defun quad-tree (q)

(list* (qar q)

(loop for childq on (qcr q) by #'qdr

while childq

collecting (quad-tree childq))))

(defun tree-quad (tree &optional parent)

(let* ((q (quad (car tree) parent nil nil))

(kids (loop for k in (cdr tree)

collecting (tree-quad k q))))

(loop for (k n) on kids

do (setf (qdr k) n))

(setf (qcr q) (car kids))

q))

#+test

(test-qt)

(defun test-qt ()

(print (quad-tree #1='(zot nil (foo #1# ("123" "abc")

. #2=(bar #1# (ding #2# "456"

dong #2# "789")))))))

(print #1='(zot nil (foo #1# ("123" "abc")

. #2=(bar #1# (ding #2# "456"

dong #2# "789")))))

#+xxxx

(test-tq)

(defun test-tq ()

(let ((*print-circle* t)

(tree '(zot (foo ("123")) (bar (ding) (dong)))))

(assert (equal tree (quad-tree (tree-quad tree))))))

(defun testq ()

(let ((*print-circle* t))

(let ((q #1='(zot nil (foo #1# ("123" "abc")

. #2=(bar #1# (ding #2# "456"

dong #2# "789"))))))

(print '(traverse showing each type and data preceded by its depth))

(quad-traverse q (lambda (q depth)

(print (list depth (qar q)(qcr q)))))

(print `(listify same ,(quad-tree q))))

(let ((q #2='(zot nil (ding #2# "456"

dong #2# "789"))))

(print '(traverse showing each "car" and itd parentage preceded by its depth))

(print '(of data (zot (ding (dong)))))

(quad-traverse q (lambda (q depth)

(print (list depth (qar q)

(mapcar 'qar (qups q)))))))))

;;;(defun tree-quad (tree)

(defun testq2 ()

(let ((*print-circle* t))

(let ((q #2='(zot nil (ding #2# "456"

dong #2# "789"))))

(print '(traverse showing each "car" and itd parentage preceded by its depth))

(print '(of data (zot (ding (dong)))))

(quad-traverse q (lambda (q depth)

(print (list depth (qar q)

(mapcar 'qar (qups q)))))))))

American & Iran: Separated at Birth?

Am I the only one grooving specifically on the fact that Iranians are telling their authority figures to go f*ck themselves? Here is the country we thought we hated but it turns out they are as kick-ass as us when it comes to political freedom, and we utterly respect them for their strength. Omigod, Americans and Iranians are going to get along great!

How to teach math

> On Sat, 09 May 2009 15:30:52 -0400, Kenneth Tilton wrote:

Well, i was not really trolling, I was forking the thread to make fun of  the New Math that tried to get the numeral/number distinction across to five year olds.

Someone responded:

You find it better to start with medieval concepts working gradually on to the mathematics of XIX century, while explaining each next year what was wrong with the things they learnt a year ago?

I said all that? Ma's gonna be right proud. But... 

Funny you should ask. Yes, I suspect the path society took to get to what it knows now about math is the path an individual neuronal mass should follow. ie, kids should encounter zero and roman numerals and place value and algebraic variables in the same order society developed those ideas.  The history of math is your math curriculum guide.

The New Math erred by selecting the logical organization of mathematical concepts as its curricular pole star. Next came Constructivism, which wanted kids to reinvent math. From scratch. Cool idea, but too slow.

Instead, let the history of mathematics dictate the order in which things are directly taught. Maybe go further and teach math as history with less emphasis on computation. Math often advanced when needed to solve real problems. Maybe we can shut up the little devils asking why they need to learn this stuff.

As for explaining all along the way what was wrong with the ideas taught the day before, hey, ever read a book on programming? They typically develop a chunk of code iteratively, presenting ever more improved variations on a primitive original. Come to think of it, ever develop some software? Same thing.

Here's the deal: most folks do not even know zero had to be invented. One understands zero better if one has done without it and then the teacher invents it for you. Something like that.

Cells: The Secret Transcript

The Boss asked me to give the group fifteen minutes on Cells because I have been talking about it for a while as a future better mousetrap for us and then suddenly last week threatened actually to apply it to qooxdoo and the front end.

I forwarded to everyone a link to a reasonably complete yet relatively brief write-up which tells you everything you need to know about Cells. I know that if I were in your shoes I would not have read it so I presume no one has. But I would like to determine how many folks I will be boring to tears if I review said document, so I will first cut to the chase and ask if anyone has any questions based on what they read.

........silence..............

OK. Cells is at once the simplest and hardest thing in the world for programmers to understand. Simple because the idea is just to have slot values of objects work like cells in a spreadsheet, and everyone knows how spreadsheets work. What is hard is understanding that one can program computers this way.

I only have fifteen minutes so: Yes, you can. Program inputs are assigned by good old imperative code to input cells the same way a user types values into a spreadsheet when they are doing what-if analysis or recording, say, actual monthly expenditure into a budget spreadsheet. Intermediate, derived, and aggregate cells compute new values based on those new inputs from predefined rules just as user changes to a spreadsheet propagate to other spreadsheet cells. Observers on cells let the emergent working model manifest its decisions with more good old fashioned imperative code, usually by simply updating the screen or playing a sound or controlling some external device over a serial port or updating a database.

Boom, we're done. Why is it so great? What part of the superiority of functional and declarative paradigms should I explain first? As I outlined in the material you did not read, a lot of things have to happen when a program receives an input. The programmer coding the event handler has to look at the event and decide all the things that have to happen in light of that event, and any things that follow from those first things. Not only must they reliably see to all those things, but they must do them in the right order. The analogy to a real spreadsheet is quite strong, if you imagine hand-implementing a spreadsheet with old-fashioned pencil and paper.

So the first things Cells does is eliminate a lot of work and thus a lot of bugs. Because the work eliminated is tedious, Cells also makes programming a lot more fun. But there is more.

The declarative paradigm means I always know why a slot has a certain value, because all the logic appears in one place, in the rule assigned to that slot. Without Cells any number of lines of code may have assigned a value to a particular slot and a unified deriving rule certainly cannot be divined even if one were to track them all down.

There is more. Most people hate OO because it never quite panned out. Objects turned out not to be reusable. One of the nicest features of Cells is that two different instances can have different rules for the same slot. That makes objects reusable. Yayyyyyyy.

I did not use Cells in the Kleaner because that was more of a straight calculation running from start to finish. I like to decribe the role for Cells being in any situation where one has an unpredictable stream of data and one is keeping a model with a sufficiently large amount of internal state consistent with that stream of inputs. Two examples being a GUI and a RoboCup client. The Kleaner worked by compiling statistics from a fixed store and then translating exactly once dirty data into corrected data.

Where Cells would have been useful would have been in implementing Phil's ideas about an ongoing stream of data leading to rediscernment of things previously discerned. Cell rules would take new raw inputs and propagate them over to tables of probabilities which would then reach out to existing cleaned data and possibly redecide from the original raw state a new cleaned state.

So no, I do not use Cells for everything, but in this case it was only because the full functionality had not been addressed.

A fun note is that I have in the past applied Cells to a database, specifically the old AllegroStore persistent CLOS database. This works two ways. One is that a user can be looking at a screen and as the underlying data changes the screen changes. That may sound like old news but with Cells one doe not have to write any code to make it happen. One just says "this view shows this users overdue books" and when the date changes the overdue status on every book gets updated and a new book appears in the list on the screen if someone happens to be looking, simply by someone having written code to list overdue books on the screen as if it were an unchanging value.

The other thing that happens is hinted at above. Things like overdue books and amount of fines owed and paid can be calculated from scratch by reading a users entire history of checkouts and returns, but sometimes it is useful to record such derived values in the database and update them incrementally as books are checked out and returned. We can have code in programs do it and hope they run at the right time, or we can have the code in the database (as datapoints mediated by Cells) and be sure they run and run immediately. We get timeliness of data, efficiency, and we still get consistency even as we introduce redundancy.

I left something out. That is bad because the thing I left out is the thing I am planning to do with Cells on my own anyway and then apply to the FE. Cells makes it dead easy to drive a separate framework from Lisp. In my note I mentioned tcl/Tk and Gtk. These are two killer C GUI frameworks with their own homebrewed little object models. We want to program in Lisp, and we want our models driven by Cells for all the reasons above. No problem. We build a model out of instances of CLOS classes mapping isomorphically onto Tk or GTk classes and use Cell observers to pipe information (thru an FFI or even literally a pipe) to the C library or runtime to drive there the creation and animation of C instances.

Works great, and one amazing programmer Peter Hildebrandt pulled off a trifecta in which he had Cells driving and driven by both GTk and a C physics engine, name forgotten.

For a while I kinda marvelled at how Cells could be so useful for such disparate activities, and do so in the same application, the two activities being building an application model and having some other programming framework dance to that model's tune.

I figured it out in time for ECLM 2008, not they were able to understand me. I opened by telling them that Cells was the single most powerful library they could use, because Cells is about change and nothing is more fundamental than change.

Programming is hard because like someone doing a spreadsheet on paper we programmers end up with the burden of propagating change thoughout our models. It is tedious work, it must be done reliably, and there is a lot of it as internal program state multiplies, exponentially a lot. This exponential growth in interdependence of program state is what led Brooks to declare that a silver bullet was not only unlikely to be found but that it would be impossible to find; he felt the complexity was ineluctable because as states multiply there is nothing that can be done to avoid the explosion of interdependence.

I wrote to Dr Brooks recently and asked him if he had ever looked at dataflow. He said he was familiar with the concept, but no. Oops.

Tinkering

Spring cannot come soon enough for Bobi, who is losing it badly these days on comp.lang.lisp:

Slobodan Blazeski wrote:

Dear board members

I'm baseball player for a several time periods (days,

moths ,years,decades) I've noticed that interest in baseball is

dwindling, and baseball is becoming less and less relevant and will

soon become extinct with only baby boomers supporting it, and even

those are either going to die or switch to golf. In order to save our

favorite sport I propose we make drastic changes and adapt more modern

things like:

a. Playing on the beach sand wearing swimwear like in beach

volleyball, very modern sport. Check Thiobe for growth rate

b. Replacing bats  with  hockey sticks. Note that hockey is popular in

many world countries and we should think international

c. Including  24-Second Shot Clock like in NBA that will make our

sport more lively and fast paced

d. Square playing fields should be replaced with the more common

rectangular one like found in many popular sports : soccer, football,

tennis etc

Including this will make baseball prosper.

 

very truly yours

 

Concerned Semi-Ex Baseball Player

Avenue of delusional weirdos Number 23

Bobi may not be as crazy as he thinks he is. Baseball suffered extreme popularity anxiety in the late Sixties and did indeed tinker with the game. Thinking more offense would attract more fans the pitching mound was lowered so pitchers did not get extra energy into the ball from falling into a pitch. The American League adopted the designated hitter to eliminate the 11% nil pitcher from batting lineups (eliminating as well an awful lot of interesting strategy). They avoided the salary caps of the NBA and instituted free agency (well, no, they lost a lawsuit) which allowed bigger markets like NYC, Boston, and LA to buy better teams, and bigger markets are always good for ratings. Minnesota fans will follow the Dodgers, Los Angeles fans will not follow the Twins. 

The changes went beyond the playing field. Ballparks added mascots and a disgusting cacophony of party music between innings so loud you can barely talk, and limited alcohol sales late in games to make the experience more family-friendly cuz you know how the losing fans get in their third hour of drinking.

Now baseball is hugely popular again so tinkering with grand institutions can work. Right?

Wrong. In the end, baseball is just a great game: multi-dimensional and deep. Quality tells, and which quality one emphasizes matters. Hockey and basketball have non-stop action and are fading in popularity, while baseball and football like great music have a variety, a rhythm, a balancing of quiet against intense. Baseball has the pitch, football has the snap. All scales from small to large from inning or drive to the game or season always and invariably end up condensed into one point of explosive tension when the pitcher releases or the center snaps the ball.

Intense without quiet merely exhausts. A boxing match with two brawlers spurning defense landing bombs back and forth brings the crowd to its feet but those who love the sport do so for its nickname, The Sweet Science. They still talk about one genius of defense who won a round without throwing a punch. Between evenly matched fighters one solid punch (forget the knockout, the cartoon haymakers of Rocky n) brings the crowd screaming to its feet, the culmination of rounds of careful, tentative, mutual exploration. A single knockdown becomes a cause for pandemonium and one punch knockouts almost do not happen between the best and when they do they are talked about for a long time. I digress.

Tinkering. Basketball has all the action in the world and now faces its own popularity crisis. Racism is one factor, another is probably the salary cap that has San Antonio in the championship series instead of New York. Another problem: poor defense, and a twenty-point lead does not mean anything. 

But worst of all is the lack of dimensionality. There just is not that much to these games to argue about over the water cooler. Baseball? Boston still talks about the time Grady Little [thx, Xach. ed.] left Pedro Martinez in one inning too long against the Yankees in game seven of the ALCS. Come on, he had thrown a hundred pitches! Everyone knows Pedro is useless after a hundred pitches! You just never hear anything like that about hockey or basketball, which both boil down to great athletes pretty much just playing run and gun.

Baseball never needed tinkering, though tinker they did. The fundamental quality of the game first ensured its survival throught the hard times when fans strayed for the quick fix of non-stop hockey and basketball action. Now the richness, subtlety, and sophistication of the game has some stadiums selling out most games of the year of a very long season.

Moral for Lisp left as an exercise.

Tilton's Law: Solve the Failure First

The team was at my throat.

"Just use the new search!," they bellowed.

The mission critical, project saving, do or die demo to upper management was eight hours away and we had not even begun the always dicey process of moving the software from the development system to one within reach of the Demomeister, and I was trying to find out why the old search was so slow.

"Soon," I replied.

We had a new search I was told was a screamer but I continued poking around putting in metrics trying to figure out why the old search was so slow. Had we not been a virtual remote telecommuting team I would not have lived to tell this tale but we were so they had no choice and I reassured then that "soon" meant ten minutes and they shut up.

Why was I still trying to understand the perplexing sloth of the old when a whole new replacement module was available and working fine and pretty much the demo on which all our jobs and a cool project depended was coming on like freight train?

Tilton's Law: Solve the failure first.

Early on we learned the other side of that coin: Solve the first problem. The commonality is...no, let's do the war story first, war stories are more fun than preaching.

Back we go a quarter of a century to my first contract with a client who would become my sole recurring client for the next decade. I was being hired to take over maintenance of an application whose author had been one of the first to die of AIDS. I was reminded of the whole business by a conversation with another developer recently about the nature of working on OPC. Other People's Code.

In my IT career I have worked always at the poles of software development, either writing new code or performing massive overhauls of OPC, never that relaxed zone between in which one simply maintains and extends in small ways a long-lived system. The second pole (OPC overhauls) always seemed to me an intimate one-way encounter with some anonymous predecessor, an encounter usually involving me roundly and steadily cursing them out. You can imagine then how eerie it was working on this system from this predecessor who was not so anonymous this time, especially when I learned that the poor guy was in bad shape during one stint but needed the money and so worked on the code I was now working on even as his fate rose up to meet him (this well before the days of the cocktails of today that make ones fate less certain). This guy I do not remember cursing out so much.

But I digress. Our lesson today is how to piss off your coworkers by insisting on solving a failure first, by which I mean even if you do decide to punt on X make sure you understand how X failed. I am not alone in this. In 2001 the movie when the crew determines that the unit Hal said was no good was fine he says fine let's put it back in and let it fail. Sure, he was really looking for a way to kill the crew but we learned in 2010 that Hal was just a computer system and I think the bit about putting the supposedly OK/not OK system back in to see if it failed was one of Hal's systems working nominally in accordance with Tilton's Law: we need to understand broken things.

And now at long last, my unsolved failure. My predecessor's, actually. The application was a securities database with a nightly feed of data applied to the cumulative DB by a batch program. This is late 80s, primitive stuff. A security could have three IDs because three groups were tracking securities and each had their own ID system. We had tens of thousands of records in our VAX/VMS RMS file, and a separate RMS key for each of the three possible IDs. So far so yawn. Here comes the fun part.

Two of the IDs were populated all the time. The other one was populated five percent of the time. Big deal, right? Right, very big deal, the poster boy for Solve the Failure First. What happened was this quesswork reconstruction:

My predecessor Paul (I picked "Paul" because it easier to type than predecessor) had a problem. His program ran an initial test load of a hundred securities in a few seconds. Fine. Everything looked good. So then he ran it against a full daily feed, which would include news of every security traded that day so it would be -- OK, I confess I completely forget even the order of magnitude, let's say tens of thousands and declare up front that that is idiotic and I am sorry, but here is what happened: the damn thing ran forever. There probably was no immediate specific great mystery because Paul probably had the program printing something (a count, the last ID recorded, something) right to his VT-100 console as it went and he could see that the program had started out zooming along but then gradually got slower and slower until just adding one security to the database (and this is just good old ISAM, mind you) took... wait for it... twenty seconds. Oh. My. God. What on earth is happening?

Paul got a clue. Every once in a while two records were written out bang-bang, as fast as at the start. Dig dig dig puzzle puzzle...ah, there it is. Any record for which we have all three IDs is written out in nothing flat. Any record (you know, the ninety-five percent) with just two will (by the end of the run) be written out three per minute, 180/hour, or 1000/fuggedaboutit.

Paul realized what was going on. The ISAM file system had no problem storing data with duplicate keys, which was a good thing because Paul was storing a whole lot of data with one key 95% the same: spaces. Poor ISAM it seemed was chugging thru all the duplicates looking for the last one after which it would record the latest duplicate. And apparently it took twenty seconds back then to walk (effectively) the entire index of a hundred-thousand record file.

Now the good news is that we would never need to look something up using spaces as the key value sought, so....what can we do? Paul was no slouch. He popped open the RMS reference manual and to his delight discovered he was not the first to pass this way and gleefully added the option "NULL_VALUE=SPACES" (translated: "if the value is spaces, Just Don't Index this record on this key") to the key definitions in the file definition script he was using to initialize the file and recreated the file and re-ran the program from scratch.

The change did not help. At all. I think we all know that feeling, as visceral as a dropping elevator.

There it was, the option explicitly intended to solve the problem he had explicitly encountered, and it did not change a thing. Impossible. But this happens to us programmers all the time. We know what to do. Compile the damn code, because we made the edit change but forgot to compile. Or link. Or, in Lisp, to zap the faultily specialized method. Or something.

So Paul edited the definition again and checked that NULL_VALUE = SPACES was the right syntax and right spelling and on the right key -- to hell with that, he put it on all three damn keys -- and he saved it and checked the date and created the file again and ran his program again and you know it did not run any faster or I would not be telling this story.

OK, time to get serious. Or if he was good he did this without all the huffing and puffing of the preceding paragraph. He Just Typed In "rms/analyze sdb.dat". And RMS looked at the file itself (not the script used to create it) and confirmed that "NULL_VALUE = SPACES" was operative for all indexes.

Momma don't let your kids grow up to be programmers.

What comes next is hard to convey. I can tell you but if you have not worked on this code or (we will learn) run this batch application it is hard to convey how much blood, sweat, tears, CPU time, and delayed nightly batch closes for how many years resulted from Paul's not first solving the failure of NULL_VALUES=YES.

Well, maybe this is a fair glimpse of the enormity that followed: the problem got sorted out only because the head of operations and I got to talking one day and something reminded him and next thing I know he is pretty much down on his knees begging me to find some way to eliminate the two-hour merge step that held up the nightly close every night. 

"It just sits there for two hours," he groaned. "It kills us every night. Please, if you can, please, do something to make this go away."

Whoa. I had inherited this system and been asked to enhance it but no one had said a word about this. The code was far and away the best OPC I had ever dealt with so everything got the benefit of the doubt, including the (soon-to-be explained) two hour merge. As in, if it is there, it must be there for a good reason. What was not there was The Story of the Unsolved Failure of NULL_VALUE=SPACES, but even if it had been I would have taken that at face value, too, because the NULL_VALUE option was unknown to me. But enough of this flash forward, let's get back to poor Paul.

NULL_VALUE was not working as it should. Software is like that. Good programmers do not let bad software stop them. Plan B. A rule is born: Thou shalt not write new securities to the securities database where the massive duplicates will make each write take twenty seconds. Paul decides to write them to a second file initialized empty on each run. Since we only got dozens of new securities in one batch, that file would never have the massive count of duplicates and writes would be lightning fast. Then we just do a sort/merge at the end of the batch to combine the new securities in with the old. Oops. "Just."

The funny "you can run but you cannot hide" moral within the moral being that I did the calculations one day and worked out that twenty seconds times the average number of new securities in a day was exactly as long as the sort/merge that was just killing the folks down in operations. And I bet Paul realized that but only after writing all the crazy code he had to write to work with two files at once as if there were only one file and at that point he just gave up and moved the thing into production. Speaking of crazy code...

You should have seen it. Looking back I cannot recall why it should have been so hard, but I did overhaul that code and I was forever tripping over it. The idea is simple. To look up a security to see if we already have it, first look in the real DB and if it is not there look in the daily "new stuff" DB and if it is not there, ah, it is new. If it is, update it. Just remember to update the right file, because we can get data from two sources about the same new security.

Piece of cake, right? A bottleneck function for all reads and updates... anyway, it seemed like the issue was always getting underfoot as I worked, and just looking at the code one saw again and again this check here/there code, and both Paul and I were the kind of engineers always on the lookout for ways to make code non-redundant. I would think my memory was faulty but I also remember eliminating Paul's Plan B after solving his failure first and that was no picnic. It just permeated the application.

So what was the first failure and how did it get solved? First, I had noticed the issue myself while using Datatrieve to add a record to the securities DB for test purposes. I hit enter and thought I had crashed the system because it went away and never came back and like every egomaniacal programmer out there I always assumed that whenever a system stopped responding the last thing I had done must have broken it so there I sat in dread for twenty seconds until the system finally responds. Wow. Twenty seconds? And then I guess I added a record specifying all three keys and it responded instantly.

But this idea of null values not being recorded in an index was new to me, and we did not have the Internet back then where I could just ask the ether what was going on so it was only a coincidence that just after the guy in operations had begged me for a fix that I was visiting with the lads from a prior contract and I moaned that RMS sucked because it could not handle files with hundreds of thousands of records and they laughed at me and said they were handling millions with RMS.

I can actually remember the look on my face, a neat trick when you think on it.

I haul ass back to work and pull out the RMS reference manual and I can tell you that dead trees aside there is one good thing about paper documentation: right above the entry for NULL_VALUES close enough to catch my eyes was the entry for NULL_KEYS.

Yep. You need to specify both. Paul had specifed NULL_VALUE=SPACES. He had not specified NULL_KEYS=YES. The default for NULL_KEYS? Guess.

I kinda wretch inside even now thinking about the astonishing amount of money, work, debugging, and delayed batches that followed from one simple failure to understand one broken thing.

The meta-lesson shared with "Solve the First Problem"? In programming, never deal with the unknown. This game is hard enough.

Epilogue

The punchline is that I never solved the first failure from Scene I of this tragedy. As my father used to say, "Do as I say, not as I do." We did have a deadline, and I did narrow down the location of the problem in a way that reassured me somewhat that it would not jump up to bite the new code in the rear end. And even in its breech the law is confirmed: we do need to address the underlying problem which I have some confidence I now understand because it still presents problems for the software but it will go away only when bigger problems are solved and they are much bigger so I am keeping my sights set on them.