Evan Lenz

My MIDI-to-USB adapter arrived via UPS today, so I hooked it up to the Roland RD-600 that I’m borrowing from my wonderful neighbors. I’m pleased to see that it works right out of the box and I only needed to make one small modification to my code to make (what’s implemented of) the video game to work. (Whereas previously it was only expecting MIDI note off events to appear as “note on” with volume 0, now it will handle explicit “note off” events as well.)

(Please forgive the low recording quality. I was using one hand to hold the camera and one hand to play the keyboard.) As you can see, there’s a fairly significant delay between the sound (which is pretty immediate) of me striking a key and the visual (green or red) note response that appears on the screen. This is something I have to work on before the game will be close to release-worthy.

Now that I have a functioning MIDI keyboard, I’m really motivated to get some real music in there (instead of these hard-coded notes that rise up the screen).

Over Christmas break, I got inspired to work on the Klavarskribo game some more. The video below shows the color coding I mentioned in my last post. When a currently-played piano key overlaps with a supposed-to-be-played piano key, the note is green for the duration that it’s supposed to be played. If you play a wrong note or if you hold the note for longer than it’s supposed to be played, it will be red. Any supposed-to-be-played notes that you don’t play will simply be black, until you hit them and then they’ll turn green. So the object of the game, at least as currently realized, is to make things as green as possible.

You’ll notice that the notes rising on the screen (which include an ascending chromatic scale and some random chords) have nothing to do with the live MIDI input. It’s as if the user is ignoring the notes on the screen and instead playing a Chopin waltz. So you end up seeing mostly red notes with a couple of green notes where the two happen to line up. That’s because I don’t yet have an actual MIDI keyboard hooked up. I’m waiting for a MIDI-to-USB adapter to come in the mail on Monday…

I still don’t have a name for this game I’m working on, so I’m not sure how to refer to it yet. Also, I’m still on the inept side when it comes to video publishing, so this first offering is of pretty low quality.

What this shows is live MIDI input (from a keyboard, eventually). What it doesn’t show is the rising notes of a score, which I’m still working on. This is an early video, so things have already changed quite a bit from what you see here. But at least it conveys some idea of what it will look like (piano keyboard at the top of the screen, Klavarskribo groups of 3 and 2 vertical lines down the screen). I’ve since removed the stems from the live (currently-being-played) notes, but they will remain on the notes that rise up the screen. As with Dance Dance Revolution’s arrows, when a note hits the top of the screen, that’s when you’re supposed to play it. I’ve also added color-coding to the notes: red for wrong note, green for correct note, and black for missed note (not shown in the above video). The aim is to provide immediate visual feedback as to how closely you’re matching the prescribed notes as they hit the top of the screen.

Steven Wilson commented on my last blog post. I went to his site and found a remarkably similar thing that he’s working on, along with a video showing the system in action. It doesn’t use Klavarskribo notation; it looks more like Synthesia’s vertical bars (with length indicating note values). But unlike Synthesia (and like what I’m doing), the keyboard is at the top of the screen (as opposed to the bottom of the screen). The link to the video on Steven’s page appears to be disabled at the moment though.

In my almost-daily scouring of the Web for Klavarskribo-related information (in English, as I don’t yet speak Dutch), I came across a thread on the Cakewalk forums where Johannes Drinda was proposing a Klavar-like notation for MIDI editing. (I’m not a Cakewalk user, so I don’t really know the context.) Anyway, I admired this whistling philosopher’s persistence and enthusiasm, so I emailed him.

In his response, he pointed me to the Janko keyboard design. Quoting him (with his permission): “What a shame, that true revolutionary notions, such as Klavarskribo and Janko keyboard pattern have almost vanished. They are indeed invaluable tools to create a system, which offers/ enables any musically minded (hobby/ home) musician to play MIDI instruments the easiest way.” So he shares my passion to reduce the cognitive overhead that’s built into conventional notation (and/or instruments). He also has created instructions for a DIY Janko keyboard overlay project (PDF file), which I found quite impressive.

That led me to some YouTube videos about the Chromatone, which is based on the Janko keyboard. Pretty interesting stuff. If nothing else, it helps situate my understanding of the traditional piano keyboard. But I’m keeping my mind open to what role it might play in my investigations. I’m also keeping an eye on what Thumtronics is doing, and I subscribed to its founder’s blog, which has a bunch of fascinating reading.

If you have experience with Klavarskribo or are particularly interested in it, please contact me. I’m hoping to get feedback once I’m able to get an initial release of this game out, and more ideas and suggestions in the meantime.

I’ve been steadily learning to read and play piano music in Klavarskribo notation. I still feel like a beginner but the pace of learning is good. New observations are coming fast and furious.

From the start, I wanted to have a computer game to teach me Klavarskribo interactively a la Dance Dance Revolution. There are some games out there that come close: Piano Wizard and Synthesia. I even crashed the Synthesia forums to see if Nicholas Piegdon would consider adding support for Klavarskribo notation. He was very gracious, but he’s also got his plate full with lots of feature requests, including support for displaying traditional notation.

I couldn’t wait, so I’ve started to cobble something together, working on it on and off over the last couple of months. It’s not close to being ready to release. It’s more of a test to help me visualize the particulars and assess its feasibility. So far, I’m encouraged that I’m on the right track.

Here’s a screenshot of an early version:

I will post a video once I figure out the best way to do that using WordPress and/or YouTube. Also, I plan to blog more about this throughout the rest of this first month of 2008.

I want a DDR-like game for Klavarskribo. I woke up with more ideas this morning. I’m just going to spew them out so I don’t forget any of them.

I’ve often complained about common (traditional) notation as like reading “assembly language” for music. It leaves so many intentions of the composer implicit. Yes, certain things can be easily ascertained, like the key of the music (from the key signature) and that a certain passage should be repeated (from the repeat signs). But there are so many other structures and patterns that I wish were explicit in order to save me a lot of time as a learner/performer of the music. In programming, I would never dream of copying and pasting a whole ream of code just because I want the same thing to happen in another context except for one little change in the middle. Yet in music notation, that is the order of the day. As a student of the piece, I have to manually double-check that, yes, everything in this section is exactly the same as that other section, except for this little riff right here. Or perhaps it’s exactly the same, but it’s in a different key. If I were writing software, I would store the current key in a local variable. Then, if I wanted to do exactly the same thing but in a different key, I would just call the same function, initializing the key accordingly. Or I’d compose the function with a “transpose” function. For Pete’s sake, I wouldn’t copy and paste all of my code, and then update all the hard-coded pitches, when they’re all the same fixed interval from a sequence of pitches I’ve written elsewhere. But in music that’s how it’s been done for a thousand years or so (except that “copy and paste” was a bit more painstaking).

So that’s why I liken common notation to assembly language. How does Klavarskribo improve on this? It doesn’t. If anything, things are much worse, i.e. if you take “assembly language” pejoratively. But I’m starting to recognize Klavarskribo as a “better assembly language” for music. If you’re going to leave structures and patterns and intentions implicit, then you might as well go the whole way, and encode the notes in a normalized, clean-slate sort of way. In Klavarskribo, there’s no distinction between G-sharp and A-flat, for example. Likewise, there’s no key signature. Klavarskribo gives you a notation that strips music of its theory.

Why would I want this? Wasn’t I just complaining about all the effort it takes to figure out a piece of music when the composer leaves so many intentions unspecified? I do hate that process of figuring out what shouldn’t have to be figured out and is only that way because the notation doesn’t support it. My attitude when writing a book is that it’s much better for me to put in the extra effort upfront in researching a technology or specifying a behavior, etc. Otherwise, that extra effort will have to be multiplied a thousand-fold, leaving it up to my readers to repeat that work when I could have saved them all the trouble. As an author, I want to save my readers all that trouble and throw them as many bones as I have at my disposal. I don’t blame composers; they’re just using the traditional medium they have. But I do long for musical representations that make more intentions explicit.

That still doesn’t answer why I like Klavarskribo. Correct, Klavarskribo doesn’t help in that arena at all, except perhaps to provide a cleaner slate from which to build. By removing all theory (other than the assumption of a 12-note world), it creates a clean, not-tonally-biased notation. I suppose for atonal music it does help, because it removes lots of misleading hints (accidentals that aren’t really accidentals, etc.).

Right now, the reason I like Klavarskribo is its great promise for enabling easier sight-reading. And not just sight-reading, but reading a piece for the first time which I ultimately intend to play from memory. I’ve always been a terrible sight-reader. There are so many variations to digest when sight-reading a piece. I won’t lie. The key signatures do help, especially if there aren’t a lot of accidentals and I’ve been practicing my scales. “Okay, I won’t be playing any of those notes that aren’t in the key of D-flat.” That certainly simplifies things. So I do have some unanswered questions about how Klavarskribo will fare without highlighting accidentals. But I see a lot of potential in totally equalizing the keyboard landscape and removing fear of the black keys for beginners. Heck, you don’t need to know anything about music to start reading Klavarskribo. Common notation is laden with concepts like “keys” and “sharps” and “flats” and “naturals” and “clefs”. I’ve been playing piano since I was six years old, I have a professional music degree, and I still suspect that I will never completely overcome the cognitive overhead of all these potential combinations and variations that are hard-wired into traditional notation.

To me, sight-reading is a different species of musicianship. I watch proficient sight-readers and I’m amazed at how unquestioning they are about what they see. You could throw a bunch of “wrong” notes in the manuscript and they would keep on playing without skipping a beat. Perhaps they’d have some mild bemusement, but they won’t get derailed like I would. There is more of a direct relationship between what they see on the page and what their fingers do on the keyboard. Their ears don’t get in the way, and I mean that in the best way possible. For me, all the notational clutter clogs up that channel. And I suspect that’s true for a lot of people.

In the computer game I’m envisioning, you would get visual feedback on what you’re doing, just like you do in DDR. In fact, there should be a free-play mode where what keys you play show up instantly in Klavar notation on the screen. That way, you can start with what you’re thinking about musically, e.g. a C-major chord, and then instantly see what it looks like in Klavar. It seems like that way you could start hooking up the neural connections without any effort. Play around and see the notes appear on the screen. “So that’s what Klavar notation looks like.” (With traditional notation, you’d have to first answer all these questions like what clefs to use, what key are we in, is this a G-sharp or A-flat, etc.) Rather than reading music, you’re watching it appear on the screen in response to what you do, and the correspondence between the keys you hit and what you see is quite natural and obvious.

In the play-along mode, you would have the notation scroll up the screen, like a piano roll, and just like the arrows do in DDR. This is where you test your sight-reading ability, getting immediate visual feedback on the screen about how you’re doing, just as you do in DDR with messages like “PERFECT!”, “GOOD!”, etc.

I would also want a silent-play mode where your ears truly have no opportunity to get in the way. You would still know when you hit a wrong note based on the visual feedback, but you won’t get derailed by weird-sounding notes, whether right or wrong.

Heck, I’d maybe even want a random mode, where you can really push the bar on this “mindless”, or rather unconscious, intuitive connection being forged between your eyes and fingers.

Klavarskribo and accompanying learning tools are only one piece of the picture that I envision for refactoring music representation. In software engineering, refactoring means “improving the design of existing code”–without changing its behavior. To me, refactoring a piece of music would mean changing the representation–or adding multiple representations, while leaving the content unchanged. All for the purpose of making the piece easier to comprehend, and at multiple levels. More on that later…

Back in college, I majored in piano performance and have mostly been out of practice since then. I tell people that I’ve always had a love/hate relationship with the piano. Recently I picked up a Brahms Intermezzo I learned years ago, so I could play it at the memorial service for my wife’s grandmother. This has got me back to wanting to learn more music again. But I always run into the same mental difficulties when learning music. I long for a more efficient way to load musical information into my brain/body. Deciphering traditional music notation has never been fun for me. Some cases are worse than others. Atonal music is particularly frustrating to learn from the traditional notation.

Consider the first phrase of #2 from Ned Rorem’s Eight Etudes for Piano.

These chords look like a bunch of triads, but that completely obscures what’s really going on. They’re actually very dense chords, and it would be folly to try to play triads with each hand. I got so frustrated after spending a little time trying to (re-)learn this piece, so I decided I needed some way to capture the information after decoding what was written, rather than traversing it repeatedly and hoping some of it would stick, when in reality it would keep falling right out of my brain because I had no way to easily read it.

Short of having someone play it for me repeatedly, so I could watch the correct keys being depressed (or short of owning a Disklavier which could also do that for me), I fired up Vim and started making some ASCII art:

| | |   | |   | | |   | |   | | |
 _ _ _ _ _ o o o _ o o o _ _ _ _


| | |   | |   | | #   | |   | | |
 _ _ _ _ o o o _ o _ o _ _ _ _ _


| | |   # #   | # |   | |   | | |
 _ _ _ _ _ o _ _ o o _ _ _ _ _ _


| | |   | |   | | |   | |   | | |
 _ _ _ _ _ o o o _ o o o _ _ _ _

The rhythm is easy enough to read with traditional notation, so I ignored that part. I just wanted to capture the notes of the chords. The vertical bars provide a template representing the black-key landscape of the keyboard. The circles are white key notes, the # signs are black key notes. After seeing this on the screen, it reminded me of Klavarskribo (Esperanto for “keyboard writing”), which was invented in 1931. I’ve never investigated Klavarskribo very closely. For some reason, I never got inspired by it before today. But now I have this feeling that I’m going to become the next Klavarskribo evangelist. It’s like DDR notation for piano. (Why not?)

Thanks to the fantastic and free KlavarScript software, I mocked up the corresponding phrase (including rhythm this time) in Klavarskribo:

Learning Klavarskribo will be challenging, but I have this feeling that after just a little effort the floodgates will open for me. Most of what I’ll need to do is un-learn what I already know about traditional notation. I see an analogy with imperative programmers trying to learn Haskell (or XSLT) for the first time. They’re trying to overcome “being brain-damaged by years of imperative programming” (their words, not mine). Thankfully, I didn’t have that obstacle in the programming world, because XSLT is where I started. But with music notation, I indeed will need to overcome years of brain damage, not to mention psychological distress, inflicted by traditional notation.

I thought I’d post a brief answer to the question I raised in my last post, which was “is there an analogous higher-order function [like foldr] for recursion on numbers?”

As I pointed out, you can define product using foldr, like this:

product = foldr (*) 1

What I wanted is another function, say foldrNum, that would allow me to define factorial using the same idea:

factorial = foldrNum (*) 1

As it turns out, all I need to do is make some slight modifications to the foldr definition, which looks like this:

foldr            :: (a -> b -> b) -> b -> [a] -> b
foldr op v []     = v
foldr op v (x:xs) = op x (foldr op v xs)

I’ll keep foldrNum general enough so that, like foldr, it can return a value of any type (b), but unlike foldr, which can process a list of any type (a), I think it’s essential to the idea of foldrNum that it only operate on a non-negative integer, which is what the recursion is based on. First, I’ll show the definition of foldrNum I came up with. Then I’ll compare it line-by-line with foldr so that we can see exactly what’s the same and what’s different.

Here’s foldrNum:

foldrNum           :: (Int -> b -> b) -> b -> Int -> b
foldrNum op v 0     = v
foldrNum op v (n+1) = op (n+1) (foldrNum op v n)

The similarities are obvious, but I like to make things as obvious as possible. First, the types of each function:

foldr               :: (a   -> b -> b) -> b -> [a] -> b
foldrNum            :: (Int -> b -> b) -> b -> Int -> b

Now, the base case:

foldr    op v []     = v
foldrNum op v 0      = v

Finally, the recursive case:

foldr    op v (x:xs) = op x     (foldr    op v xs)
foldrNum op v (n+1)  = op (n+1) (foldrNum op v n)

The natural question arising from this is “How do I generalize foldr and foldrNum, since they’re so much alike?” Joshua Ball anticipated that question in his comment on my last post, although I confess I haven’t fully wrapped my head around his solution yet.

I think this is enough for today. So many possible directions to go in. I must content myself with one small piece at a time. 🙂

I’m writing this post as a way of moving myself forward at least one notch through Graham Hutton’s Programming in Haskell, which I’ve been very slowly, occasionally working through. I’m in the middle of chapter 7 right now.

The thing that slows me down (apart from having 3 kids at home including a baby) is that I’m distracted by the learning process itself. I don’t want to too easily forget the false or imperfect theories that I form along the way. A teacher’s job is to help bridge the gap between imperfect models in the student’s mind and models that are more useful and effective. But the only way they can do that is to be familiar with such inaccurate models, as well as to value them as important stages of the student’s learning (rather than reject them out of hand as is the modus operandi of schools today, but I digress). I may want to teach this stuff some day, and this is my only chance to observe the learning process first-hand, at least as a beginner. I need to keep a log of my learning, or at least parts of it, in order to keep me moving forward with the assurance that my intermediate stages of learning will not be forgotten forever.

Quick disclaimer: This post is not the first of a series of Haskell posts, at least not any planned series. I hereby give myself permission to not post on Haskell ever again, or for that matter, to drop it in favor of learning something else (although given how much I keep getting drawn back to it, I’m pretty sure that won’t happen).

Recursion patterns

In chapter 6 (on recursion), I noticed there are basically two things you could base recursion off of, or two things to “recurse” on:

• a list, or
• an integer

Recursion on an integer

The basic pattern for recursion on an integer is evident in the definition of the factorial function:

factorial 0     = 1
factorial (n+1) = (n+1) * factorial n

The factorial of 0 is just 1 (the identity for multiplication). The factorial of a positive integer (represented by (n+1)) is that integer ((n+1)) multiplied by the factorial of one less than that integer (n). The function is invoked recursively until the numbers run out and the termination case is reached (an argument of 0).

Recursion on a list

The basic pattern for recursion over a list is evident in the definition for the product function:

product []     = 1
product (n:ns) = n * product ns

The product of an empty list is just 1 (again, the identity for multiplication). The product of a non-empty list (represented by (n:ns)) is the result of multiplying the head of the list (n) times the product of the tail of the list (ns). The function is invoked recursively until the list runs out and the termination case is reached (an empty-list argument).

Capturing the list-recursion pattern

I’m only partly through chapter 7 (on higher-order functions), but I’ve now been introduced to the foldr library function, which captures the pattern evident in the product function, which uses recursion over a list. Here it is:

f []     = v
f (x:xs) = x ⊕ f xs

The v stands for some base value (which was 1 in the product function). The circled-plus symbol (⊕) stands for some operator (which was * in the product function). The foldr function takes those two variables (a base value and an operator) and automatically constructs a function for you that does the above. So you can now define product more succinctly (and without using explicit recursion):

product ns = foldr (*) 1 ns

Or even more succinctly, and thanks to Haskell’s function currying:

product = foldr (*) 1

What about capturing the number recursion pattern?

The open question in my mind is: is there an analogous higher-order function for recursion on numbers?

What wasn’t explicitly mentioned in the book is the idea that an integer can be thought of as representing a list of numbers, starting at that number and counting down to 1. If that’s the case, then I could just use foldr for recursion on a number, provided that I first turn that number into a list of numbers counting down to 1.

Using that idea, here’s another way to define factorial:

factorial n = foldr (*) 1 [n,(n-1)..1]

Of course, in the case of multiplication, the order of the list of arguments doesn’t matter, so the list doesn’t have to count down. It could start at 1 and count up to the integer instead:

factorial n = foldr (*) 1 [1..n]

Anyway, that’s where I’m at with these half-baked observations. I’m sure I’ll find better or more appropriate abstractions as well as more interesting insights. I’ll be able to look back at this post and say “Oh, what you want is the ____ function!”

A Vim/Dvorak tutorial. First, QWERTY is designed to slow typists down (so early typewriters wouldn’t jam). Then, Vi was designed to make keystroke editing commands easy on QWERTY keyboards. Meanwhile, the Dvorak keyboard was designed to enable faster typing and prevent carpal tunnel.

How bizarre then to see a tutorial for Vi keystrokes on a Dvorak keyboard:

Just look at where those up/down/left/right arrow keys are.

I have to wonder if this person really uses Vi. Maybe the joke’s on me. 🙂

The original one it’s based on is quite cool though.

Priscilla Walmsley was kind enough to point me to a post from Jim Garrison on comp.text.xml regarding XSLT 1.0 Pocket Reference, which I poured my heart and soul into last year. This was actually the first feedback I’ve gotten from anyone who has used the book. Needless to say, it made my day:

First, a big thank you to Evan Lenz for the O’Reilly XSLT 1.0 Pocket Reference. This is an amazing feat of distilling all you really need to know about XSL into 170 pocket-sized pages. What’s more amazing is that an experienced developer with little prior XML/XSL experience can actually learn enough from this little gem to write competent XSL.

A reference manual AND quality tutorial in 1/20th the space (and dead trees) of most tech books these days. I’ve recommended this to several of my colleagues who had to get up to speed on XSL for a new project, and the reaction from them is the same as mine…. This was EXACTLY what I needed.

So, Evan, when is the version for XSL 2.0 coming out?

As I already told Jim, such is not in the works right now, but if it were, I hope he wouldn’t hold me to keeping it within 170 pages again!

I was reading in Chapter 6 of Practical Common Lisp about dynamic (a.k.a. “special”) variables and it hit me that this is something that I have longed for in XSLT.

I generally favor using template rules, and hence xsl:apply-templates, over xsl:for-each whenever possible. But sometimes it’s a pain—particularly when you have to pass parameters around.

An example using <xsl:for-each>

Take this contrived piece of code, for example:

<xsl:template match="foo">
  <xsl:variable name="x" select="string(@x)"/>
  <xsl:variable name="y" select="string(@y)"/>
  <xsl:for-each select="document('some-doc.xml')//bar">
    <my-element x="{$x}">
      <xsl:for-each select="document('another-doc.xml')//bat">
        <another-element y="{$y}"/>
      </xsl:for-each>
    </my-element>
  </xsl:for-each>
</xsl:template>

Okay, not so bad. But it’s inflexible.

The template rule version

If I wanted to convert this to a solution using template rules, I’d end up with this monstrosity:

<xsl:template match="foo">
  <xsl:variable name="x" select="string(@x)"/>
  <xsl:variable name="y" select="string(@y)"/>
  <xsl:apply-templates select="document('some-doc.xml')//bar">
    <xsl:with-param name="x" select="$x"/>
    <xsl:with-param name="y" select="$y"/>
  </xsl:apply-templates>
</xsl:template>

<xsl:template match="bar">
  <xsl:param name="x"/>
  <xsl:param name="y"/>
  <my-element x="{$x}">
    <xsl:apply-templates select="document('another-doc.xml')//bat">
      <xsl:with-param name="x" select="$x"/>
      <xsl:with-param name="y" select="$y"/>
    </xsl:apply-templates>
  </my-element>
</xsl:template>

<xsl:template match="bat">
  <xsl:param name="x"/>
  <xsl:param name="y"/>
  <another-element y="{$y}"/>
</xsl:template>

UGH! It’s bad enough to have to pass parameters around. Add in XSLT’s syntax for parameter-passing and it becomes excruciating. So… in this case, the solution using <xsl:for-each> has to be preferred, because the variables stay within lexical scope and require no parameter-passing. The only reason to go with template rules is when you really do need the flexibility afforded by the latter solution (e.g. enabling a template rule to be overridden, etc.).

Using a global variable is often the solution (and a perfectly acceptable one), but in this case, the values may vary for each <foo> element that is processed, and I don’t know upfront how many <foo> elements there will be. So that won’t work here.

Enter dynamic variables. As explained in the aforementioned chapter on Lisp, a dynamic variable is a global variable that can have multiple bindings. You have the option of shadowing it in a given lexical context and—here’s the kicker—for the full extent of any calls you make to other functions. For XSLT, just replace “functions” with “templates”.

The fanciful template rule version

XSLT doesn’t have such a thing, but if it did, it might look something like this:

<xsl:variable name="x" special="yes"/> <!-- made-up "special" attribute -->
<xsl:variable name="y" special="yes"/>

<xsl:template match="foo">
  <xsl:let name="x" select="string(@x)"> <!-- made-up "let" element -->
    <xsl:let name="y" select="string(@y)">
      <xsl:apply-templates select="document('some-doc.xml')//bar"/>
    </xsl:let>
  </xsl:let>
</xsl:template>

<xsl:template match="bar">
  <my-element x="{$x}">
    <xsl:apply-templates select="document('another-doc.xml')//bat"/>
  </my-element/>
</xsl:template>

<xsl:template match="bat">
  <another-element y="{$y}"/>
</xsl:template>

To me, this would be ideal. No parameter-passing required.

XSLT 2.0 offers a very good compromise

XSLT 2.0 introduces what are called “tunnel parameters”. I’ve known about these for a while, but my new acquaintance with dynamic variables in Lisp is helping me understand where these fall in the spectrum of programming language features. If the above example shows what a “global dynamic variable” looks like, then tunnel parameters represent what you would call a “local dynamic variable”. So here’s the bona fide XSLT 2.0 version of the above, without resorting to any made-up syntax:

<xsl:template match="foo">
  <xsl:apply-templates select="document('some-doc.xml')//bar">
    <xsl:with-param name="x" select="string(@x)" tunnel="yes"/>
    <xsl:with-param name="y" select="string(@y)" tunnel="yes"/>
  </xsl:apply-templates>
</xsl:template>

<xsl:template match="bar">
  <xsl:param name="x" tunnel="yes"/>
  <my-element x="{$x}">
    <xsl:apply-templates select="document('another-doc.xml')//bat"/>
  </my-element/>
</xsl:template>

<xsl:template match="bat">
  <xsl:param name="y" tunnel="yes"/>
  <another-element y="{$y}"/>
</xsl:template>

With tunnel parameters, you don’t have to declare a parameter unless you need access to it. And the only time you have to explicitly pass the parameters is when you initialize them. After that, you can just trust that they will be there for you when you do need them, later on down the call stack.

Since this is just a simple example (with only two template rules being invoked), it doesn’t do justice to how helpful tunnel parameters actually are. Imagine 10 more template rules that get invoked between the first and last one. Most of those won’t require any xsl:param declarations. In other words, the final example above is much better than it looks. Conversely, the first template rule example above (the XSLT 1.0 solution) is much, much worse than it looks. Imagine having to type out <xsl:param>or <xsl:with-param> 40 times in a row. That’s essentially what you’d have to do.

Conclusion

I may actually write some programs in Lisp at some point, but so far the insights I’m getting into XSLT alone are worth the effort of learning it.

Wouldn’t it be nice to be able to interact with Lisp sub-expressions? I should be able to, say, hover over a sub-expression (having perhaps set some default input data) and see the dynamic result of that sub-expression. That would make it easier to understand expressions, debug them, etc. Maybe the REPL is all that’s really necesary and I just haven’t learned it well enough yet. But it would be really handy to interactively explore the list transformations that happen as an expression (that I’ve already typed out) is composed. I couldn’t imagine a faster way to explore it than by hovering over sub-expressions and seeing the result in another window.

Has anyone built anything like this?

From the same Lisp tutorial I’m reading:

This “back-quoting” technique is a great feature in Lisp- it lets us write code that looks just like the data it creates. This happens frequently with code written in a functional style: By building functions that look like the data they create, we can make our code easier to understand and also build for longevity: As long as the data doesn’t change, the functions will probably not need to be refactored or otherwise changed, since they mirror the data so closely. Imagine how you’d write a function like this in VB or C: You would probably chop the path into pieces, then append the text snippets and the pieces together again- A more haphazard process that “looks” totally different from the data that is created and probably less likely to have longevity.

This is one of the things I’ve always liked about XSLT. The template rules directly show the XML that is being created. Attribute value templates (AVTs) are helpful in this regard too. In fact, I try to use them whenever possible. I try to avoid using xsl:attribute (unless there’s no other way to do it, i.e. when the attribute’s presence is conditional). For example, when you have to use an instruction to get an attribute’s value, this may seem the most natural approach:

<para>
  <xsl:attribute name="lang">
    <xsl:apply-templates mode="get-lang-value" select="."/>
  </xsl:attribute>
  ...
</para>

But I far prefer this approach even if it does take a few more characters:

<xsl:variable name="lang">
  <xsl:apply-templates mode="get-lang-value" select="."/>
</xsl:variable>
<para lang="{$lang}">
  ...
</para>

That way, when reading the code to figure out what it does, your eye can just skip by the variable definition (it’s just a variable; it doesn’t do anything). After that, it’s easy to see what we’re creating. The first example requires you to look more closely and to think about it for a second, ugh.

Now this is just a small version of a much bigger problem with understanding XSLT stylesheets, especially ones that use a lot of template rules. They can be so dynamic that you absolutely must see an example source document (input data) to understand what the stylesheet does. Integrating dynamic execution profiles into development via multiple code views and software visualization—those are some of my research interests…

I took a detour from Seibel’s book to try out this neat little Head First-like Lisp tutorial.

In it, there’s some sample data:

(setf *map* '((living-room (you are in the living-room of a wizards house. there is a wizard snoring loudly on the couch.)
                           (west door garden)  
                           (upstairs stairway attic))
              (garden (you are in a beautiful garden. there is a well in front of you.)
                      (east door living-room))
              (attic (you are in the attic of the wizards house. there is a giant welding torch in the corner.)
                     (downstairs stairway living-room))))

And a function for accessing part of that data:

(defun describe-location (location map)
  (second (assoc location map)))

The second function grabs the second item in the list returned by (assoc location map). What bothers me initially is that the order in which I have to write this seems to be the reverse of the order in which I would think through the problem. For example, if I wanted to get the description of the living room, I would think “First, get the *map*. Then, look inside it for the living room. Finally, grab the second item in the living-room list.” This is the way it works in XPath. If the sample data was XML, then an XPath expression for getting the living room description might look like this:

/map/*[@name=$location]/*[2]

Start with the map, then get location, then get the second item inside that. (Also, one nice thing about XPath 2.0 is that path expressions are generalized for function application. You can write get-map()/get-location()/get-second-item(), which works the same way in this case because each step returns a 1-item list.)

Anyway, I’m sure there’s probably another way to write such Lisp expressions that doesn’t require me to think backwards and—worse yet—keep all the steps in my head at once. Maybe an object-oriented approach is all that’s needed a la map.getLocation().getSecond() (like in Java). Short of that, I could always bind temporary results to variables and build up the function calls from that. But I really don’t like being forced to create variables just so I can write the code in the order that seems natural.

Maybe I just need to reverse my thinking. Maybe this has something to do with what Paul Graham refers to as “bottom-up programming”. I don’t know.

Speaking of the Head First series, Kathy Sierra just posted “The Clueless Manifesto”. In the spirit of that, I intend to ask even more dumb questions as I continue to learn Lisp. I may want to teach this stuff some day, and this will be a record of what it was like for me when I was a beginner.

In bed this morning, I was contemplating all I learned about Lisp last night. XSLT’s most obvious analog to Lisp macros is its ability to generate stylesheet code, since XSLT is in XML syntax.

But if data is really code, then XSLT template rules are also like macro definitions. An element in the source document is processed and thereby expanded, causing the code in the body of the corresponding template rule to be executed.

At one level, template rules seem more powerful than Lisp macros. A macro is defined by a name and resolved via name-lookup, whereas a template rule is defined by a pattern and resolved via pattern-matching. I don’t think I’ve encountered this level of indirection yet in my two days of learning Lisp. I’m curious what the best analog to this pattern-based function resolution is in Lisp. Maybe the best analog is to be found in particular Lisp functions, such as remove-if-not. Perhaps XSLT’s core processing model is like just another Lisp function.

An obvious difference is that Lisp macros are expanded recursively, whereas template rules are just applied once. I’ve wanted a way to apply them recursively sometimes (well, certain template rules in my stylesheet if not all of them). This isn’t built in to XSLT. Either you have to store a temporary result in a variable and then process the results again, or you have to define a pipeline process outside of XSLT to invoke the XSLT processor. I’ve contemplated how a more automatic recursive application of template rules might look or might be built into an XSLT-like language.

Anytime you learn something new, you’re going to compare it to what you already know. So far, it seems like having an XSLT background is going to be helpful in my learning of Lisp.

Got Aquamacs, Allegro CL, and SLIME up and running on my iMac at home. I can’t believe I’ve already been using Vim for 5 years now. Emacs is certainly another world. But I am motivated one way or another to learn Lisp, and learning Emacs seems to go hand in hand with that. So far, Viper mode is my friend. But I’ve already gone so far as to swap my Capslock and Control keys, as recommended in “Effective Emacs”.

So far, I’ve worked through chapter 3 of Peter Seibel’s book and am now reading chapter 4. That macros are a way of life in Lisp is exciting. I’ve gotten a glimpse of their power with XSLT, but I’m looking forward to taking it to the next level.

I’ll be giving a tutorial on XSLT 1.0 at the O’Reilly Open Source Convention this August.