Thursday, November 03, 2005

I've just started The Mystery of Capital, by Hernando de Soto, and so far I don't think it's a bad read. I confess that I don't know much about economics, so it's sometimes disorienting. The premise is that formal property systems in developing nations exclude the poor, who hold the vast majority of assets in those nations. The poor have no access to the formal property systems, because the assets they own are owned illegally or because the legal system is too cumbersome to use. Lacking an entry into the formal property system, there is no way to use these assets for collateral. The statistics de Soto and his team use to illustrate the massive impediment to property ownership in developing nations is enlightening. My biggest question so far is whether the "formal property" argument can explain conditions in inner-city slums, and I suspect the answer is that inner-city slums in developed nations are caused by fundamentally different forces than shanty towns on the outskirts of a sprawling third-world metropolis. The author notes that land value in shanty towns is sometimes as high or higher than land in the legally-owned areas of town and going up, whereas land value in inner-city slums is wretched and deteriorating quickly. Hopefully more reading will clear up some of my economic confusion.
I'm releasing version 1.0 of my clisp SQLite bindings for SQLite 2.8. If you've never used SQLite before, it's an embeddable SQL (duh) database. Check out SQLite. You can download my clisp binding at An ASDF/ASDF-Install-able version is Real Soon Now (tm). The bindings offer two levels of support: a low-level binding which is directly tied to the FFI (package SQLITE.RAW) and a more friendly, lisp-like interface built atop it (package SQLITE). Here's a glimpse of how it works:
[1]> (use-package :sqlite)
=> T
[2]> (setf my-db (sqlite-open #"~/tmp/test.db"))
=> #<SQLITE::DATABASE #P"/home/scottw/tmp/test.db" :OPEN>
[3]> (sql my-database "malformed sql")
*** - sql error 1: "near \"malformed\": syntax error. Offending query:
        "malformed sql"
Break 1 [4]> :r1
[5]> (sql my-database "create table bar (a, b)")
=> NIL
[6]> (sql my-database "insert into bar values (1, 2)")
=> NIL
[7]> (sql my-database "insert into bar values ('a', 'x')")
=> NIL
[8]> (sql my-database "select * from bar")
=> (("1" "2") ("a" "x"))
[9]> (sqlite-close my-database)
=> #<SQLITE::DATABASE #P"/home/scottw/tmp/test.db" :CLOSED>
[10]> (with-open-db (db #"~/tmp/test.db")
        (with-query (db (a b) "select a, b from bar")
          (format nil "Processing a row with a=~s, b=~s.~%" a b)))
Processing a row with a="1", b="2".
Processing a row with a="a", b="x".
=> T
So far as I've used it, I find with-query to be one of the most efficient and useful of the sql forms supplied with the binding. I optimized it for relatively high throughput (though for some reason, clisp creates 3 strings for every column in the row, odd), so it's fast in tight loops where your lisp must process a lot of data (it can handle a million 3-column rows in less than a second on my machine). Since the body of with-query is executed once for each row as the row is returned, it avoids building up potentially large lists, the way the plain SQL call is likely to. I find this form so particularly useful because it binds the columns from each result row to the specified variables, making them much more accessible. There are a few outstanding issues. The first is the lack of a with-transaction. Inserts into the database are much more efficient when wrapped in a transaction, but I haven't yet worked out how to wrap the body of with-transaction so that "rollback" is called if the body is left abnormally, and "commit" is called otherwise. I suspect some trickery with unwind-protect and catch is going to be necessary, though it's not obvious to me how to do it, so I'm delaying a while. Next, I plan to explore the MOP by building a Class::DBI-style object mapper. CLSQL already does this, but a) it doesn't compile out of the box with clisp, and b) I want to make my own.

Tuesday, November 01, 2005

I got my copy of the Art of the Meta-Object Protocol in the mail today! Already it's more well-written and accessible than I could possibly have imagined, and is so far answering several questions in fields I hadn't related to the MOP (like how classical object models can be integrated with the generic object models used in CLOS, and how the MOP relates to the "essential difficulty" of software production). Thinking over the essential difficulty of programming today and the most promising (though non-silver) bullets, I was trying to decide how Generators, Patterns, and metaprogramming are related to one another in the attack on the essence of difficulty in programming. I've been thinking a lot, especially about Paul Graham's assertion that "When I see patterns in my programs, I consider it a sign of trouble. The shape of a program should reflect only the problem it needs to solve. Any other regularity in the code is a sign, to me at least, that I'm using abstractions that aren't powerful enough-- often that I'm generating by hand the expansions of some macro that I need to write." Perhaps more stewing on this idea is necessary before coming to a conclusion, but I'm tempted to agree, and also to point out Graham's use of Generators (in the form of macros) to help him accomplish some metaprogramming (someone who deserves an attribution but whose name I can't remember once said that metaprogramming is the art of engineering the language upward to meet it's intended use).
Perl and Python, Lisp and TeX: Fundamental differences in user communities. I have a bias against Lisp and Tex documentation because I have always found it difficult to come by; hence I have always regarded these languages as poorly documented, even though I'm a fairly active and excited user of both. On the difference of availability of documentation, I think the origins of the languages are extremely important. Both Lisp and TeX have a rather more academic user group (not that they are any more capable than the Perl and Python crowd, but when you think practical programming, you think Perl, not TeX), whereas Perl and Python are grassroots efforts, so to speak. The very definition and elaboration of the languages themselves reflect this. If you want good documentation of TeX, buy the TeXbook, which I've heard is quite a rigorous bible of TeX. Consequently, anybody who owns the TeXbook is unlikely to be constantly publishing the little tidbits of information and procedure that float around. When I first began using LaTeX, I had a hell of a time finding any documentation for it, and now that I know exactly what to search for, I have a slightly less hard time, but by no means is it as easy as CPAN. The same is true with Lisp. The official common lisp spec has to be purchased from ANSI, and if you want to delve under the hood with the MOP, you're pretty much out of luck unless you buy the AMOP. Contrast this with the Python and Perl world: if you've been to CPAN, if you've read the perldocs, or browsed, you already know what I'm talking about. The TeX group has CTAN, which is similar to CPAN, and common lisp has cliki, which is also similar, but these two repositories are fundamentally different than CPAN—it's the documentation! The CTAN and cliki are only (or at least mostly) code repositories. They allow for quick and easy downloading of code, but provide no support for the user. I posit that it is exactly because the languages Perl and Python grew up in the bright sun of public scruitiny that they are so well documented on the Internet. As single individuals attempt to document a language, they must surely fail to document all of it. The Internet responds by producing stacks of How-To articles that guide the novice user. In the more academic world, the novice is guided to a book. The book may be perfectly instructive, but the community dialog, that searchable collective experience, is not available. I think this accounts for a significant difference between the Perl and Python, Lisp and TeX crowds.