Evan Lenz

This is how the PianoNinja is feeling tonight after suffering a difficult blow:

We had a little setback involving lost code. Ironically, this happened when I was trying to set up version control to guard against this very thing. I’ve posted a message to the user’s list for the IDE that failed to give me a warning before deleting all this naive user’s code, in hopes that similar mistakes by other users can be prevented. Hindsight is 20/20. I should’ve backed it up before trying to…back it up. It would also have been good to have Time Machine set up already, but I hadn’t done that either.

Thankfully, I did still have an older version of the code from before I switched over to the development environment that I’m now using, so I don’t have to start totally all over. And you can be sure that I have already secured that code into version control on a server machine so that I can continue from now on from a solid base.

How bad is it? Well, we were on Video #7. Let’s just say we’ve now reverted back to Video #3. In other words, all the code I wrote in the last two weeks is gone.

Maybe this is a blessing in disguise. Sure, I learned a lesson and I won’t make this mistake again. But the blessing might be that my continued dependency on MIDI files as the game’s underlying format is now that much less alluring. The MIDI crutch has been snatched out from under me. I don’t need to wrest myself from the MIDI code so I can move onto a better way. The code has wrested itself from me…

I was going to add bar lines tonight. Oh well, those can wait.

Before you know it, the ninja will be feeling better than ever.

Okay, this post is just to complete the thought I’ve been exploring over the last couple of posts. I wasn’t up for much heavy thinking tonight, so I decided to just download a free MIDI editor and normalize the durations of the notes in the Chopin Waltz MIDI file so that it had no more staccato releases in the first section of the piece. Now the visual note releases reflect the actual notated durations (except that some Klavarskribo continuation dots are still missing; I haven’t implemented those yet). I wanted to see if this helped make it easier to “see” the beat:

Video #7: Notation-consistent note releases (Quicktime streaming)

I think it works pretty well for this piece of music. The sudden flash of each note release now coincides with the attack of the next note. I think it now looks smooth and rhythmic, which is what I was hoping for.

I haven’t made any progress since yesterday’s report, but I did finally track down our camera. This video shows what I was talking about yesterday: the MIDI Note OFF events are now visible.

Video #6: Note OFF events visualized (Quicktime streaming)

The game isn’t making any sound in this video, because I’m not playing along on the MIDI keyboard. It’s easier to see the note releases when I’m not playing along and adding all the green and red colors. The piece is the same one I’ve been using for the last few demos: Chopin Waltz in E-flat, Opus 18.

The molasses effect of lingering notes is gone, so that’s good. But the note releases are accented too much, in that the suddenness of the notes’ disappearance tends to be the most visually striking movement that’s going on. It’s also dependent on the interpretation of the person who rendered the performance in the MIDI file. My theory is that if performance-dependent note releases are ignored and instead the actual note value is reflected (regardless of whether the note should be played staccato), then the accented note releases will actually help rather than hinder the feeling of the beat, since they’ll most often coincide with the attack of the next note. I’m looking forward to testing my theory out (and then moving on past this academic theorizing).

Today, I tried to enable what I talked about in my last post about PianoNinja. I was able to get the MIDI Note OFF events to be reflected in each note’s visible duration: how long it remains stationary on its piano key before disappearing. (I would have posted a video, but I can’t seem to find the camera right now, and I really need to be getting to bed earlier anyway.) I have a couple of observations:

• It’s much nicer than before: crisp and clean releases, but…
• the effective duration doesn’t necessarily correspond to the actual note value.

A MIDI file contains a rendition of a piece, not (necessarily) the authoritative musical information you’d need to reconstruct a score. Staccato durations, for example, get interpreted as short notes that obscure the actual notated value. I’m coming to terms with what I want PianoNinja to do: display a score (in Klavarskribo notation) that can be relied upon as containing the more-or-less canonical information that makes up the piece. While there might be many and varied MIDI files for the same Chopin Waltz, I don’t want PianoNinja to be subject to those variations.

So while I’m glad that I got the MIDI Note OFF events to be reflected, the associated note-vanishing is still a bit jumpy-looking, since they don’t always coincide with the attack of the following note. Before diving into using MusicXML instead, I might try to see what sorts of MIDI file quantization I could do to stretch each duration out for its full note value. I’m going for the path of least resistance here in keeping this project moving forward—without compromising the steady vision I have for what PianoNinja can be.

As I play around with PianoNinja, I’ve noticed so far that it’s fairly easy to get out of sync with the notes that I’m supposed to be playing. I tend to speed up when playing slowly, for example. (You might have noticed this in the last video I posted). Part of the problem is that I don’t have any strong cues as to where the beat falls. In games like Dance Dance Revolution, there’s a very strong beat—you simply dance to the music that you hear. Ultimately, I think PianoNinja will need auditory cues as well, whether as simple as a metronome sound or as complex as a full accompaniment. (That would make it a lot more fun too.)

But before I add any auditory hints, I want to see how far purely visual cues can get me. Of course, I’ve made the decision to go with Klavarskribo, so the question as to what visual cues to use has largely already been answered for me. (Basically, use solid and dotted bar lines to indicate the primary and secondary beats, respectively.) But what Klavarskribo doesn’t answer is how to display the notes once they reach the point where they’re supposed to be played.

The simplest approach would be to just keep scrolling the manuscript right off the screen, just as if it was a roll of sheet music continuing to unwind. In other words, apart from the scrolling movement, the display would be purely static. You’d know when to play the notes only according to when they cross a given (horizontal) line, but then they’d keep on scrolling off the screen and out of sight. In that case, perhaps the most natural place for that line would be the middle of the screen rather than the top. That way, you’d be able to see the notes come and go rather than immediately scroll off the screen at the top.

I’ve decided to take a more dynamic approach, placing that line at the top of the screen and representing it with a picture of a piano keyboard, so that, for one thing, the relationship between the notation and the piano keyboard is immediately obvious (I hope). The dynamic aspect is that the visible note then freezes and remains stationary on its corresponding piano key for the duration that it’s supposed to be played, while all along the rest of the music continues to scroll upward. Early on, I was experimenting with a grayscale fade-out approach for representing the decay of the note’s sound, but I found that keeping it solid and stationary and then suddenly making it disappear worked nicely—especially when followed immediately by another note in the same hand. The combination of the new note stopping in its tracks and the old note vanishing in the same instant creates a certain visceral perception of the rhythm. At least that’s my suspicion and hope; I haven’t fully put it to the test. Right now, the durations are all hard-coded to the length of one beat, which doesn’t fit well with the Chopin Waltz I’ve been using in my demos. The notes all linger far too long, giving a molasses-like visual effect which doesn’t help you feel the beat at all. So I think I’ll focus next on taking care of those note durations.

With this approach, the most “attack-like” movement that you see is when a note ends, not when it begins. What makes it work is the fact that the disappearance of one note often coincides with the attack of the next. For that reason, it may turn out to work well for some kinds of music, but not so well for others.

But now I’m just speculating. First I’ll get it to work correctly and try several pieces of music with it. If I still find it to be lacking, then maybe I’ll add some kind of “flash” to each note’s attack. I’m hoping that won’t be necessary though. I like the idea of a visually spare interface that still packs a punch—like Klavarskribo itself, and, I hope, like the new PianoNinja logo.

I’m trying to figure out what file format(s) to support in PianoNinja. Synthesia has gone the route of “blind devotion to MIDI files” and its attendant pitfalls when it comes to notating music. I’ve already gotten a glimpse of these drawbacks now that I’ve tried to add the left-hand/right-hand distinction to PianoNinja. Some MIDI files separate the hands into two different tracks, and some don’t. And there doesn’t seem to be a standard way of doing this. Also, there are other visual cues built into Klavarskribo that I want to include, like bar lines, note beams, rests, continuation dots, etc. These aren’t going to be easy to derive from simplistic MIDI ON and OFF events, especially since the best MIDI files as far as performance goes would tend to be the worst as far as deriving notation goes. That’s my suspicion anyway. I get the impression that there are features of MIDI files that could allow them to provide notational hints, but those are optional and so they can’t be totally relied on.

MusicXML, on the other hand, is designed for representing musical information that can be notated. The biggest disadvantage I see of using MusicXML is that it’s not nearly as widely available as MIDI files are. Where do I go to find MusicXML files anyway? Do I have to purchase them? Do they only exist transiently on their path from Finale to Sibelius or vice versa? If I can get sufficiently past that hurdle, then it seems that I should go with MusicXML.

But there might be some other disadvantages to MusicXML also. Traditional notation requires you to make certain decisions about things like key signatures, whereas key signatures are optional in Klavarskribo (represented by a circle or diamond at the beginning of the piece on the relevant line for major or minor keys, as well as other shapes for different modes IIRC). Are they optional in MusicXML? Or does MusicXML force you to make certain distinctions that Klavarskribo doesn’t require? If MusicXML files were as abundant as MIDI files, I wouldn’t worry so much about this. I’d just take what I need and leave what I don’t need. But if I want to promote the proliferation of PianoNinja music across the Web, would MusicXML raise the bar too high? The purist in me doesn’t want to have to decide whether a note is B-flat or A-sharp in the underlying format when that distinction may not ever appear on the screen in PianoNinja.

So maybe what’s needed is a new “KlavarML” format, along with a converter from MusicXML to KlavarML (using XSLT of course). I presume there are already converters from MIDI to MusicXML (insofar as they’re able to). I don’t want to reinvent the wheel. But I also don’t want to be held back by unnecessary technological limitations. Both MIDI and MusicXML provide too much information. MIDI has lots of performance/interpretation-specific information that’s not relevant to Klavarskribo (although theoretically could be relevant to PianoNinja insofar as the scroll speed could dynamically vary, but I digress). MusicXML makes distinctions that Klavarskribo and PianoNinja do not. And MIDI doesn’t provide enough information.

Whether a “KlavarML” becomes an interchange format or an internal format specific to PianoNinja, I still like the idea. At the very least, the exercise of designing it would help clarify the whole domain of what it is I need to represent in PianoNinja. And it would also give me another opportunity to use RELAX NG.

Whereas in the last video all the note stems pointed to the right (which made it difficult to tell which hand was supposed to play which notes), now I’ve associated MIDI events in one track with the left hand, and the other track in the right hand. For those of you who aren’t familiar with Klavarskribo (which is probably 99% of the population), if a stem points to the left, then that means the note is meant to be played with the left hand. If it points to the right, use the right hand.

PianoNinja Video #5: both hands (Quicktime streaming)

My 7-year-old son was holding the camera, while I was using both hands to play. He was trying his best to hold it still.

Tonight I got some scrolling notes to appear on the screen in response to MIDI events being played back from a MIDI file. Observe:

There are a number of deficiencies still:

• no bar lines (that aren’t irrelevant and hard-coded still)
• no note durations (also hard-coded)
• no right-hand/left-hand distinction (all hard-coded to the right)
• stems are too short in some cases, erroneously breaking the chord up
• no beams for grouping related notes in a line

Despite these deficiencies, I’m really stoked. I’m starting to get a sense of what it will be like to use this game and to practice sight-reading with it (using Klavarskribo notation, of course). My vision is starting to be realized!

Earlier this month, I decided to come up with a name for the Klavarskribo video game I’m working on, so I wouldn’t have to keep calling it the “Klavarskribo game”. I was planning on making the name public when I had a first version ready to download. But last night I came up with the coolest logo for it that I just couldn’t wait to share it. (You’ll have to be the judge; perhaps I’m just sleep-deprived.)

I’m no graphic designer, so I didn’t think I’d actually come close to anything I’d be happy with. I just thought I’d sketch out some concepts. Here’s what I started with:

I did a quick Google search for drawing programs for OS X and downloaded a trial copy of EazyDraw. Within an hour, I was liking how things were going, so I purchased a license. I added some finishing touches today, and here’s the final result:

Why “Piano Ninja”? Well, for one thing, ninjas are cool. My 7-year-old has taken a particular liking to them lately, so I already have ninjas on the brain. I wanted to juxtapose the word “piano” with something indisputably cool (ninjas), since, let’s face it, “practicing piano” and “piano lessons” don’t traditionally evoke the same sense of enthrallment someone gets when absorbed in a video game (such as Guitar Hero). Also, I already mentioned that I don’t plan to knock on many piano teachers’ doors anytime soon (I’m no glutton for punishment), so I’m not terribly concerned with making the name sound “respectable” to the music education world. Finally, I like the stealth component that’s built into the name.

Now I’ve got no excuse but to get moving on this game (heh, as if not having a logo is what was holding me back). Anyway, I’ll continue to chronicle my progress as I’ve been doing so far.

I discovered and joined the Google group for the MNMA. Now I’m reading lots of interesting discussions about alternative notations, 7-5 keyboards (traditional piano keyboard: 7 white, 5 black) vs. 6-6 keyboards (like Janko and Chromatone). One thing I really like about the general feel of these discussions is the openness with which people greet each other’s ideas. Rather than clamoring to promote their pet notation systems at the expense of others, they generally seem to embrace new ideas as contributing to progress. They seem to embody their declared purpose well:

The MNMA is an international nonprofit organization dedicated to exploring ways to make reading and playing music easier to learn and enjoy. This is our forum for open discussion of music notation improvement, alternative music notations, and related topics.

One thing I think is missing in the discussions is an explicit emphasis on use cases. Klavarskribo and some of the other ANs (which in the MNMA forum is shorthand for “alternative notation”) as well as TN (”traditional notation”) are each what I would call a general-purpose music notation system: a single system designed to support pretty much any piece of Western music, and any activity relating to that piece of music. But I wouldn’t assume that the same notation would be equally well-suited to the very different activities of sight-reading, learning a piece for the first time, and analyzing a piece, to name a few. Similarly, I wouldn’t assume that a given notation would serve equally well for all pieces of music. In one of the MNMA forum discussions, someone mentioned that:

One person on the LilyPond list wanted to see Bartók in a chromatic
staff in order to see the symmetry. With just accidentals, and no
vertical change of position, the symmetry is hidden, just as TN hides
the symmetry.

Each piece has a logic of its own. A one-size-fits-all approach to notation is guaranteed to obscure much of that logic. The extent to which the composer’s structural intentions are clearly revealed will be haphazard if the notation is designed or chosen independently of the piece being represented.

So along with “use cases” and “refactoring”, I’d like to borrow another term from software engineering: “domain-specific languages”. But let’s modify that slightly: PSLs (piece-specific languages). And PSLs would have a leg up on DSLs, because we wouldn’t necessarily have to make them dumb enough for a computer to understand.

I don’t have much to update tonight about my Klavarskribo game progress. I’m currently trying to find a free Java profiler that will help me isolate whatever code inefficiencies are causing the slight delay between pressed piano keys and visible feedback on the screen. Yes, I’m using Java for this project. Java wasn’t my first choice, but since I have some experience with it and since it has built-in MIDI support and books about game development with starter code to work from, I found that it was my fastest way to get started. Before that, I was contemplating using languages that I’d rather learn more about, like Haskell and Q. But for me right now, the elegance of those languages would be distracting. My current focus is on getting the thing to work, not writing beautiful code. I’m not sure if you could call what I have “working” yet, but I have definitely written–rather, splatted–plenty of non-beautiful code onto the screen already. So I’m right on track.

My MIDI-to-USB adapter arrived via UPS today, so I hooked it up to the Roland RD-600 that I’m borrowing from my wonderful neighbors. I’m pleased to see that it works right out of the box and I only needed to make one small modification to my code to make (what’s implemented of) the video game to work. (Whereas previously it was only expecting MIDI note off events to appear as “note on” with volume 0, now it will handle explicit “note off” events as well.)

(Please forgive the low recording quality. I was using one hand to hold the camera and one hand to play the keyboard.) As you can see, there’s a fairly significant delay between the sound (which is pretty immediate) of me striking a key and the visual (green or red) note response that appears on the screen. This is something I have to work on before the game will be close to release-worthy.

Now that I have a functioning MIDI keyboard, I’m really motivated to get some real music in there (instead of these hard-coded notes that rise up the screen).

Over Christmas break, I got inspired to work on the Klavarskribo game some more. The video below shows the color coding I mentioned in my last post. When a currently-played piano key overlaps with a supposed-to-be-played piano key, the note is green for the duration that it’s supposed to be played. If you play a wrong note or if you hold the note for longer than it’s supposed to be played, it will be red. Any supposed-to-be-played notes that you don’t play will simply be black, until you hit them and then they’ll turn green. So the object of the game, at least as currently realized, is to make things as green as possible.

You’ll notice that the notes rising on the screen (which include an ascending chromatic scale and some random chords) have nothing to do with the live MIDI input. It’s as if the user is ignoring the notes on the screen and instead playing a Chopin waltz. So you end up seeing mostly red notes with a couple of green notes where the two happen to line up. That’s because I don’t yet have an actual MIDI keyboard hooked up. I’m waiting for a MIDI-to-USB adapter to come in the mail on Monday…

I still don’t have a name for this game I’m working on, so I’m not sure how to refer to it yet. Also, I’m still on the inept side when it comes to video publishing, so this first offering is of pretty low quality.

What this shows is live MIDI input (from a keyboard, eventually). What it doesn’t show is the rising notes of a score, which I’m still working on. This is an early video, so things have already changed quite a bit from what you see here. But at least it conveys some idea of what it will look like (piano keyboard at the top of the screen, Klavarskribo groups of 3 and 2 vertical lines down the screen). I’ve since removed the stems from the live (currently-being-played) notes, but they will remain on the notes that rise up the screen. As with Dance Dance Revolution’s arrows, when a note hits the top of the screen, that’s when you’re supposed to play it. I’ve also added color-coding to the notes: red for wrong note, green for correct note, and black for missed note (not shown in the above video). The aim is to provide immediate visual feedback as to how closely you’re matching the prescribed notes as they hit the top of the screen.

Steven Wilson commented on my last blog post. I went to his site and found a remarkably similar thing that he’s working on, along with a video showing the system in action. It doesn’t use Klavarskribo notation; it looks more like Synthesia’s vertical bars (with length indicating note values). But unlike Synthesia (and like what I’m doing), the keyboard is at the top of the screen (as opposed to the bottom of the screen). The link to the video on Steven’s page appears to be disabled at the moment though.

In my almost-daily scouring of the Web for Klavarskribo-related information (in English, as I don’t yet speak Dutch), I came across a thread on the Cakewalk forums where Johannes Drinda was proposing a Klavar-like notation for MIDI editing. (I’m not a Cakewalk user, so I don’t really know the context.) Anyway, I admired this whistling philosopher’s persistence and enthusiasm, so I emailed him.

In his response, he pointed me to the Janko keyboard design. Quoting him (with his permission): “What a shame, that true revolutionary notions, such as Klavarskribo and Janko keyboard pattern have almost vanished. They are indeed invaluable tools to create a system, which offers/ enables any musically minded (hobby/ home) musician to play MIDI instruments the easiest way.” So he shares my passion to reduce the cognitive overhead that’s built into conventional notation (and/or instruments). He also has created instructions for a DIY Janko keyboard overlay project (PDF file), which I found quite impressive.

That led me to some YouTube videos about the Chromatone, which is based on the Janko keyboard. Pretty interesting stuff. If nothing else, it helps situate my understanding of the traditional piano keyboard. But I’m keeping my mind open to what role it might play in my investigations. I’m also keeping an eye on what Thumtronics is doing, and I subscribed to its founder’s blog, which has a bunch of fascinating reading.

If you have experience with Klavarskribo or are particularly interested in it, please contact me. I’m hoping to get feedback once I’m able to get an initial release of this game out, and more ideas and suggestions in the meantime.

I’ve been steadily learning to read and play piano music in Klavarskribo notation. I still feel like a beginner but the pace of learning is good. New observations are coming fast and furious.

From the start, I wanted to have a computer game to teach me Klavarskribo interactively a la Dance Dance Revolution. There are some games out there that come close: Piano Wizard and Synthesia. I even crashed the Synthesia forums to see if Nicholas Piegdon would consider adding support for Klavarskribo notation. He was very gracious, but he’s also got his plate full with lots of feature requests, including support for displaying traditional notation.

I couldn’t wait, so I’ve started to cobble something together, working on it on and off over the last couple of months. It’s not close to being ready to release. It’s more of a test to help me visualize the particulars and assess its feasibility. So far, I’m encouraged that I’m on the right track.

Here’s a screenshot of an early version:

I will post a video once I figure out the best way to do that using Wordpress and/or YouTube. Also, I plan to blog more about this throughout the rest of this first month of 2008.

I want a DDR-like game for Klavarskribo. I woke up with more ideas this morning. I’m just going to spew them out so I don’t forget any of them.

I’ve often complained about common (traditional) notation as like reading “assembly language” for music. It leaves so many intentions of the composer implicit. Yes, certain things can be easily ascertained, like the key of the music (from the key signature) and that a certain passage should be repeated (from the repeat signs). But there are so many other structures and patterns that I wish were explicit in order to save me a lot of time as a learner/performer of the music. In programming, I would never dream of copying and pasting a whole ream of code just because I want the same thing to happen in another context except for one little change in the middle. Yet in music notation, that is the order of the day. As a student of the piece, I have to manually double-check that, yes, everything in this section is exactly the same as that other section, except for this little riff right here. Or perhaps it’s exactly the same, but it’s in a different key. If I were writing software, I would store the current key in a local variable. Then, if I wanted to do exactly the same thing but in a different key, I would just call the same function, initializing the key accordingly. Or I’d compose the function with a “transpose” function. For Pete’s sake, I wouldn’t copy and paste all of my code, and then update all the hard-coded pitches, when they’re all the same fixed interval from a sequence of pitches I’ve written elsewhere. But in music that’s how it’s been done for a thousand years or so (except that “copy and paste” was a bit more painstaking).

So that’s why I liken common notation to assembly language. How does Klavarskribo improve on this? It doesn’t. If anything, things are much worse, i.e. if you take “assembly language” pejoratively. But I’m starting to recognize Klavarskribo as a “better assembly language” for music. If you’re going to leave structures and patterns and intentions implicit, then you might as well go the whole way, and encode the notes in a normalized, clean-slate sort of way. In Klavarskribo, there’s no distinction between G-sharp and A-flat, for example. Likewise, there’s no key signature. Klavarskribo gives you a notation that strips music of its theory.

Why would I want this? Wasn’t I just complaining about all the effort it takes to figure out a piece of music when the composer leaves so many intentions unspecified? I do hate that process of figuring out what shouldn’t have to be figured out and is only that way because the notation doesn’t support it. My attitude when writing a book is that it’s much better for me to put in the extra effort upfront in researching a technology or specifying a behavior, etc. Otherwise, that extra effort will have to be multiplied a thousand-fold, leaving it up to my readers to repeat that work when I could have saved them all the trouble. As an author, I want to save my readers all that trouble and throw them as many bones as I have at my disposal. I don’t blame composers; they’re just using the traditional medium they have. But I do long for musical representations that make more intentions explicit.

That still doesn’t answer why I like Klavarskribo. Correct, Klavarskribo doesn’t help in that arena at all, except perhaps to provide a cleaner slate from which to build. By removing all theory (other than the assumption of a 12-note world), it creates a clean, not-tonally-biased notation. I suppose for atonal music it does help, because it removes lots of misleading hints (accidentals that aren’t really accidentals, etc.).

Right now, the reason I like Klavarskribo is its great promise for enabling easier sight-reading. And not just sight-reading, but reading a piece for the first time which I ultimately intend to play from memory. I’ve always been a terrible sight-reader. There are so many variations to digest when sight-reading a piece. I won’t lie. The key signatures do help, especially if there aren’t a lot of accidentals and I’ve been practicing my scales. “Okay, I won’t be playing any of those notes that aren’t in the key of D-flat.” That certainly simplifies things. So I do have some unanswered questions about how Klavarskribo will fare without highlighting accidentals. But I see a lot of potential in totally equalizing the keyboard landscape and removing fear of the black keys for beginners. Heck, you don’t need to know anything about music to start reading Klavarskribo. Common notation is laden with concepts like “keys” and “sharps” and “flats” and “naturals” and “clefs”. I’ve been playing piano since I was six years old, I have a professional music degree, and I still suspect that I will never completely overcome the cognitive overhead of all these potential combinations and variations that are hard-wired into traditional notation.

To me, sight-reading is a different species of musicianship. I watch proficient sight-readers and I’m amazed at how unquestioning they are about what they see. You could throw a bunch of “wrong” notes in the manuscript and they would keep on playing without skipping a beat. Perhaps they’d have some mild bemusement, but they won’t get derailed like I would. There is more of a direct relationship between what they see on the page and what their fingers do on the keyboard. Their ears don’t get in the way, and I mean that in the best way possible. For me, all the notational clutter clogs up that channel. And I suspect that’s true for a lot of people.

In the computer game I’m envisioning, you would get visual feedback on what you’re doing, just like you do in DDR. In fact, there should be a free-play mode where what keys you play show up instantly in Klavar notation on the screen. That way, you can start with what you’re thinking about musically, e.g. a C-major chord, and then instantly see what it looks like in Klavar. It seems like that way you could start hooking up the neural connections without any effort. Play around and see the notes appear on the screen. “So that’s what Klavar notation looks like.” (With traditional notation, you’d have to first answer all these questions like what clefs to use, what key are we in, is this a G-sharp or A-flat, etc.) Rather than reading music, you’re watching it appear on the screen in response to what you do, and the correspondence between the keys you hit and what you see is quite natural and obvious.

In the play-along mode, you would have the notation scroll up the screen, like a piano roll, and just like the arrows do in DDR. This is where you test your sight-reading ability, getting immediate visual feedback on the screen about how you’re doing, just as you do in DDR with messages like “PERFECT!”, “GOOD!”, etc.

I would also want a silent-play mode where your ears truly have no opportunity to get in the way. You would still know when you hit a wrong note based on the visual feedback, but you won’t get derailed by weird-sounding notes, whether right or wrong.

Heck, I’d maybe even want a random mode, where you can really push the bar on this “mindless”, or rather unconscious, intuitive connection being forged between your eyes and fingers.

Klavarskribo and accompanying learning tools are only one piece of the picture that I envision for refactoring music representation. In software engineering, refactoring means “improving the design of existing code”–without changing its behavior. To me, refactoring a piece of music would mean changing the representation–or adding multiple representations, while leaving the content unchanged. All for the purpose of making the piece easier to comprehend, and at multiple levels. More on that later…

Back in college, I majored in piano performance and have mostly been out of practice since then. I tell people that I’ve always had a love/hate relationship with the piano. Recently I picked up a Brahms Intermezzo I learned years ago, so I could play it at the memorial service for my wife’s grandmother. This has got me back to wanting to learn more music again. But I always run into the same mental difficulties when learning music. I long for a more efficient way to load musical information into my brain/body. Deciphering traditional music notation has never been fun for me. Some cases are worse than others. Atonal music is particularly frustrating to learn from the traditional notation.

Consider the first phrase of #2 from Ned Rorem’s Eight Etudes for Piano.

These chords look like a bunch of triads, but that completely obscures what’s really going on. They’re actually very dense chords, and it would be folly to try to play triads with each hand. I got so frustrated after spending a little time trying to (re-)learn this piece, so I decided I needed some way to capture the information after decoding what was written, rather than traversing it repeatedly and hoping some of it would stick, when in reality it would keep falling right out of my brain because I had no way to easily read it.

Short of having someone play it for me repeatedly, so I could watch the correct keys being depressed (or short of owning a Disklavier which could also do that for me), I fired up Vim and started making some ASCII art:

| | |   | |   | | |   | |   | | |
 _ _ _ _ _ o o o _ o o o _ _ _ _

| | |   | |   | | #   | |   | | |
 _ _ _ _ o o o _ o _ o _ _ _ _ _

| | |   # #   | # |   | |   | | |
 _ _ _ _ _ o _ _ o o _ _ _ _ _ _

| | |   | |   | | |   | |   | | |
 _ _ _ _ _ o o o _ o o o _ _ _ _

The rhythm is easy enough to read with traditional notation, so I ignored that part. I just wanted to capture the notes of the chords. The vertical bars provide a template representing the black-key landscape of the keyboard. The circles are white key notes, the # signs are black key notes. After seeing this on the screen, it reminded me of Klavarskribo (Esperanto for “keyboard writing”), which was invented in 1931. I’ve never investigated Klavarskribo very closely. For some reason, I never got inspired by it before today. But now I have this feeling that I’m going to become the next Klavarskribo evangelist. It’s like DDR notation for piano. (Why not?)

Thanks to the fantastic and free KlavarScript software, I mocked up the corresponding phrase (including rhythm this time) in Klavarskribo:

Learning Klavarskribo will be challenging, but I have this feeling that after just a little effort the floodgates will open for me. Most of what I’ll need to do is un-learn what I already know about traditional notation. I see an analogy with imperative programmers trying to learn Haskell (or XSLT) for the first time. They’re trying to overcome “being brain-damaged by years of imperative programming” (their words, not mine). Thankfully, I didn’t have that obstacle in the programming world, because XSLT is where I started. But with music notation, I indeed will need to overcome years of brain damage, not to mention psychological distress, inflicted by traditional notation.

I thought I’d post a brief answer to the question I raised in my last post, which was “is there an analogous higher-order function [like foldr] for recursion on numbers?”

As I pointed out, you can define product using foldr, like this:

product = foldr (*) 1

What I wanted is another function, say foldrNum, that would allow me to define factorial using the same idea:

factorial = foldrNum (*) 1

As it turns out, all I need to do is make some slight modifications to the foldr definition, which looks like this:

foldr            :: (a -> b -> b) -> b -> [a] -> b
foldr op v []     = v
foldr op v (x:xs) = op x (foldr op v xs)

I’ll keep foldrNum general enough so that, like foldr, it can return a value of any type (b), but unlike foldr, which can process a list of any type (a), I think it’s essential to the idea of foldrNum that it only operate on a non-negative integer, which is what the recursion is based on. First, I’ll show the definition of foldrNum I came up with. Then I’ll compare it line-by-line with foldr so that we can see exactly what’s the same and what’s different.

Here’s foldrNum:

foldrNum           :: (Int -> b -> b) -> b -> Int -> b
foldrNum op v 0     = v
foldrNum op v (n+1) = op (n+1) (foldrNum op v n)

The similarities are obvious, but I like to make things as obvious as possible. First, the types of each function:

foldr               :: (a   -> b -> b) -> b -> [a] -> b
foldrNum            :: (Int -> b -> b) -> b -> Int -> b

Now, the base case:

foldr    op v []     = v
foldrNum op v 0      = v

Finally, the recursive case:

foldr    op v (x:xs) = op x     (foldr    op v xs)
foldrNum op v (n+1)  = op (n+1) (foldrNum op v n)

The natural question arising from this is “How do I generalize foldr and foldrNum, since they’re so much alike?” Joshua Ball anticipated that question in his comment on my last post, although I confess I haven’t fully wrapped my head around his solution yet.

I think this is enough for today. So many possible directions to go in. I must content myself with one small piece at a time.

I’m writing this post as a way of moving myself forward at least one notch through Graham Hutton’s Programming in Haskell, which I’ve been very slowly, occasionally working through. I’m in the middle of chapter 7 right now.

The thing that slows me down (apart from having 3 kids at home including a baby) is that I’m distracted by the learning process itself. I don’t want to too easily forget the false or imperfect theories that I form along the way. A teacher’s job is to help bridge the gap between imperfect models in the student’s mind and models that are more useful and effective. But the only way they can do that is to be familiar with such inaccurate models, as well as to value them as important stages of the student’s learning (rather than reject them out of hand as is the modus operandi of schools today, but I digress). I may want to teach this stuff some day, and this is my only chance to observe the learning process first-hand, at least as a beginner. I need to keep a log of my learning, or at least parts of it, in order to keep me moving forward with the assurance that my intermediate stages of learning will not be forgotten forever.

Quick disclaimer: This post is not the first of a series of Haskell posts, at least not any planned series. I hereby give myself permission to not post on Haskell ever again, or for that matter, to drop it in favor of learning something else (although given how much I keep getting drawn back to it, I’m pretty sure that won’t happen).

Recursion patterns

In chapter 6 (on recursion), I noticed there are basically two things you could base recursion off of, or two things to “recurse” on:

• a list, or
• an integer

Recursion on an integer

The basic pattern for recursion on an integer is evident in the definition of the factorial function:

factorial 0     = 1
factorial (n+1) = (n+1) * factorial n

The factorial of 0 is just 1 (the identity for multiplication). The factorial of a positive integer (represented by (n+1)) is that integer ((n+1)) multiplied by the factorial of one less than that integer (n). The function is invoked recursively until the numbers run out and the termination case is reached (an argument of 0).

Recursion on a list

The basic pattern for recursion over a list is evident in the definition for the product function:

product []     = 1
product (n:ns) = n * product ns

The product of an empty list is just 1 (again, the identity for multiplication). The product of a non-empty list (represented by (n:ns)) is the result of multiplying the head of the list (n) times the product of the tail of the list (ns). The function is invoked recursively until the list runs out and the termination case is reached (an empty-list argument).

Capturing the list-recursion pattern

I’m only partly through chapter 7 (on higher-order functions), but I’ve now been introduced to the foldr library function, which captures the pattern evident in the product function, which uses recursion over a list. Here it is:

f []     = v
f (x:xs) = x ⊕ f xs

The v stands for some base value (which was 1 in the product function). The circled-plus symbol (⊕) stands for some operator (which was * in the product function). The foldr function takes those two variables (a base value and an operator) and automatically constructs a function for you that does the above. So you can now define product more succinctly (and without using explicit recursion):

product ns = foldr (*) 1 ns

Or even more succinctly, and thanks to Haskell’s function currying:

product = foldr (*) 1

What about capturing the number recursion pattern?

The open question in my mind is: is there an analogous higher-order function for recursion on numbers?

What wasn’t explicitly mentioned in the book is the idea that an integer can be thought of as representing a list of numbers, starting at that number and counting down to 1. If that’s the case, then I could just use foldr for recursion on a number, provided that I first turn that number into a list of numbers counting down to 1.

Using that idea, here’s another way to define factorial:

factorial n = foldr (*) 1 [n,(n-1)..1]

Of course, in the case of multiplication, the order of the list of arguments doesn’t matter, so the list doesn’t have to count down. It could start at 1 and count up to the integer instead:

factorial n = foldr (*) 1 [1..n]

Anyway, that’s where I’m at with these half-baked observations. I’m sure I’ll find better or more appropriate abstractions as well as more interesting insights. I’ll be able to look back at this post and say “Oh, what you want is the ____ function!”