It was the technology that originally caught my interest: I saw the MarkMail demo at an XML conference, and one thing led to another. The company was looking to expand the product beyond the core database–they had plans for something called a “utility layer” though in reality it wasn’t really a utility nor a separate layer. It started with Search API, though the very first piece of code I wrote was an RDFa parser.

But what’s really held my interest for these years is a truly unmatched set of peers. This place is brimming with brilliant minds, and that keeps me smiling every day on my way in to work.

Which leads my thoughts back to semantics again. This push in a new direction has a lot of echoes with the events that originally brought me on board. This is going to be huge, and will move the company in a new direction. Stay tuned. -m

It was an amazing experience. I was responsible for sifting through the huge amounts of public data–both in RDF formats and on public web pages–and writing the semantic code to pull everything together, culminating in those ten minutes on stage.

Picture this: just behind the big stage and the projected screens was a hive of impressive activity. I counted 8 A/V people backstage, plus 4 more at the back of the auditorium. The conference has reached  a level of production values that wouldn’t be vastly different if it was a stadium affair. So in back there’s a curtained-off “green room” with some higher-grade snacks (think PowerBars and Red Bull) with a flatscreen that shows the stage. From back there you can’t see the projected slides or demos, but if you step just outside, you’re at the reverse side of the screen, larger-than-life. The narrow walkway leads to the “chute”, right up the steps onto the main stage. As David Gorbet went through the opening moments of his talk in fine form, I did some stretches and did everything I could think of to prepare myself.

Then he called me up and the music blasted out from the speakers. I had been playing through my mind all the nightmare scenarios–tripping on the stairs and falling on my face as I come onstage (etc.)–but none of that happened. I’ve done public speaking many times before so I had an idea what to expect, though on a stage like that the lights are so bright that it’s hard to see beyond about the third row. So despite the 300-400 people in the room, it didn’t even feel much different than addressing an intimate group of peers. It was fun. On with the demos:

The first showed our internal MarkMail cluster with a simple ‘infobox’ of the sort that all the search engines are doing these days. This was an icebreaker to talk about semantics and how it works–in this case locate the concept of Hadoop in the database, and from there find all the related labels, abstracts, people, projects, releases, and so on. During the construction of the demo, we uncovered some real world facts about the author of the top-ranked message for the query, including a book he wrote. The net effect was that these additional facts made the results a lot more useful by providing a broader context for them.

The second demo showed improved recall–that is finding things that would otherwise slip under the radar. The existing [from:IBM] query in MarkMail does a good job finding people that happen to have the letters i-b-m in their email address. The semantic query [affiliation:IBM] in contrast knows about the concept of IBM, the concept of people, and the relationship of is-affiliated-with (technically foaf:affiliation) to run a query that more closely models how a person would ask the question: “people that work for IBM” as opposed to “people that have i-b-m in their email address”. This the results included folks posting from gmail accounts and other personal addresses, and the result set jumped from about 277k messages to 280k messages.

At this point, a pause to talk about the architecture underlying the technology. It turns out that a system that already supports shared-nothing scale out, full ACID transactions, multiple HA/DR options, and a robust security model is a good starting point for building semantic capabilities.  (I got so excited at this point that I forgot to use the clicker for a few beats and had to quickly catch-up the slides.) SPARQL code on the screen.

Then the third demo, a classic semantic app with a twist. Pulling together triples from several different public vocabularies, we answered the question of “find a Hadoop expert” with each row of the results representing not a document, as in MarkMail results, but an actual person. We showed location data (which was actually randomized to avoid privacy concerns) and aggregate cost-of-living data for each city. When we added in a search term, we drew histograms of MarkMail message traffic over time and skipped over the result that had no messages. The audience was entranced.

This is exciting work. I had several folks come up to me afterwards with words to the effect of they hadn’t realized it before, but boy do they ever need semantics. I can’t think of a better barometer for a technical keynote. So back to work I go. There’s a lot to do.

Thanking by name is dangerous, because inevitably people get left out, but I would like to shout out to David Gorbet who ran the keynote, John Snelson who’s a co-conspirator in the development effort, Eric Bloch who helped with the MarkMail code more than anyone will ever know, Denis Shehan who was instrumental in wrangling the cloud and data, and Stephen Buxton who patiently and repeatedly offered feedback that helped sharpen the message.

I’ll post a pointer to the video when it’s available. -m

1. A fundamental unit of the DOM (Document Object Model) used in creating rich HTML5 applications.
2. A basic unit of the Semantic Web–a thing you can say stuff about. Some nodes are even unlabeled, and hence ‘blank nodes’.
3. In operations, a node means, roughly, a machine on the network. E.g. “sixteen-node cluster”
4. A software library for event-driven, asynchronous development with JavaScript.

I find myself at the forefront of a growing chorus of software architects and API designers that are fed up with this overloading of a perfectly good term. So I’m happy today to announce node-node:node.node.

The system is still in pre-alpha, but it solves all of the most pressing problems that software developers routinely run in to. In this framework, every node represents a node, for the ultimate in scalable distributed document storage. In addition, every node additionally serves as a node, which provides just enough context to make open-world assumption metadata assertions at node-node-level granularity. Using the power of Node, every node modeled as a node has instant access to other node-node:nodes. The network really is the computer. You may never write a program the old way again. Follow my progress on Sourceforge, the latest and most cutting-edge social code-sharing site. -m

This is a sign that Y is still infested with slackers. From what I’ve seen, a B-or-C-lister will ruthlessly take advantage of a WFH policy. If that dries up, they’ll move on.

If I’m right, the policy will indeed go into effect at Yahoo starting this summer, and after a respectable amount of time has passed (and the slackers leave) it will loosen up again. And Yahoo will be much stronger for it. Agree? -m

I stood up on a platform in front of a room of native signers, and delivered a (pre-prepared) five minute presentation without making a sound. In front of cameras, with my ugly face beamed out to multiple large screens.

That was stressful, though less so then many different public speaking engagements I’ve participated in. It was a different kind of stress. I’m sure I made all kinds of mistakes of which I wasn’t even aware of. ASL books, videos, and web sites tend to focus on particular signs, and vocabulary is one important part of learning the language–but not the only part. A huge amount of the communication comes through facial expression, body shifting and language, and other “non-manual markers.” I’m learning, if slowly.

It’s also helping me in everyday situations, among hearing folks. I’m better able to express myself and I’ve picked up some new gestures (like non-dominant-hand indexing…more on that later), and I tend to, even if in the back of my mind, think about how you’d express such-and-such an idea in ASL, and having thought it through more, better express it in writing or speech.

It’s also helping to finally tame my inner-introvert. When a fundamental part of communication involves displaying play-by-play emotions on your face (and indeed, entire body) it changes you. Better than acting lessons.

What have you done lately to push yourself out of your comfort zone? -m

Relevant. -m

And I’m completely OK with this. I don’t expect to ever get confused for a native signer, which is fine with me. I do expect, however, to become a better communicator–to develop sufficient skill to be clearly understood in ASL. I prefer to think of it like someone with a suave British accent in America. You’d never mistake them for a native and yet they are a joy to converse with. In the right circumstances, they can even grab your attention moreso than someone with a native accent.

This can only do good things for my spoken communication skills as well. It’s a lot like acting classes in some respects, which is a marked departure from my normally taciturn personality. This is encouraging me to quit holding everything inside quite so much, with encouraging results. If you see me walking a little taller, speaking a bit more emphatically, or better conveying emotion to get my point across, now you know what’s behind that. -m

Sometimes I notice that, when I’m corrected–say I make a sign incorrectly and my conversational partner demonstrates the correct way to do it–I often can’t tell any difference between what I was supposed to do and what my hands actually did. This kind of fundamental error in cognition seems to happen all the time with me. My helpful friends tell me that’s a good sign. (no pun intended)

A less-bruising kind of error is the “oops” kind–the instant you commit the error, you know you messed up. This, however, can sometimes throw you off if you get self-conscious about it. A third kind of error is when you know exactly what to do, but your physiology holds you back–for instance the ASL sign for either ’6′ or ‘W’ (made the way most hearing people show a ’3′ on their fingers; thumb holding down the pinky) is difficult for me to make without slowing way down. And to think, only 13 years ago I was playing keyboards in a little garage band. Guess I need some stretches. It’s good to loosen up.

In ASL, though, there’s a weird kind of middle ground. Sometimes people who don’t know Spanish kind of ‘fake it’ — “Yo no speako español” and the like, which has always come across to me as vaguely offensive. Being overly terrified of making a mistake is itself a fourth kind of mistake. ASL is remarkably flexible; even though it’s a complete language, it has aspects based on pantomime and sometimes “classifiers”, where your hands and fingers can stand in for people, vehicles, or many other things of particular shapes/sizes. I watch some very well-made ASL productions that have equally well-made English paragraphs alongside, and the ASL version uses all of these techniques and more. No word-for-word correspondence here: every time, I’m surprised by the versitility of the language. My theory is that for an earnest student, it’d be a lot harder to accidentally come across as offensive or mocking the language in ASL compared to other spoken languages. And thus, I’m probably committing the fourth kind of error too much.

It’s good to loosen up. -m

Hedgehogs are type A personalities who believe in Big Ideas. The are ideologues and go “all-in” on whatever they’re espousing. A great many pundits fall into this category.

Foxes are scrappy creatures who believe in a plethora of little ideas and in taking different approaches toward a problem, and are more tolerant of nuance, uncertainty, complexity, and dissent.

There are a lot of social situations (broadly construed) where hedgehogs seem to have the upper hand. Talking heads on TV are a huge example, but so are many fixtures in the tech world, Malcolm Gladwell, say. Most of the places I’ve worked at have at least a subtle hedgehog-bias toward hiring, promotions, and career development.

To some degree, I think this stems from a lack of self-awareness. Brash pundits come across better on the big screen; they grab your attention and take a bold stand for sometihing–who wouldn’t like that? But if you take pause and think about what they’re saying or (horror) go back an measure their predictions after-the-fact, they don’t look nearly so good. Foxes are better at getting things right.

It seems like we’ve just been through a phase of more-obnoxious-than-usual punditry, and I found this spectrum a useful way to look at things. How about you? Are you paying more attention to hedgehogs when you probably should be listening to the foxes?

-m

for Virgil

But for whatever reason, it didn’t make it into the printed edition. This post is a small step toward letting the world know about someone important to me.

We first met in 1985 or thereabouts. One day while riding my bike through a back-alley, I stopped to look at an equipment rack set outside a spare garage. Virgil came out to give a get-off-my-lawn kind of speech, and somehow we ended up talking about electronics.  This led to discussions about crystal radios, and in a subsequent visit, we built one, he explaining the principles of operation. Virgil, it turns out, was a retired teacher at the North Dakota State School of Science, where he taught AC theory and thermodynamics. I was going through some rough times, and Virgil ended up being a much-needed role model.

Around that time, I had ttempted to build a Heathkit radio set, but couldn’t quite get it working. I brought it to Virgil, and we traced through the schematic diagrams, eventually getting it working. Along the way, Virgil introduced me to all kinds of electronic test equipment, including oscilloscopes and galvanometers that he had hand-wound in his younger days.

The next year, I needed a science project, and I had become fixated on Tesla Coils. Virgil had worked at Westinghouse (but not in overlap with the good N. Tesla) and found this project right up his alley. We used his wood lathe to turn a base for the coil, and a standard lathe to wind a primary and two perfectly-spaced secondary coils on PVC pipe, after which we sprayed them down with insulating paint. We built a high-voltage power supply out of a car battery, ignition coil, and relay-type regulator from the junkyard. The thing would turn out serious spark on the primary side, and at one point, I accidentally made contact with it, knocking me clear off the metal bench I was sitting on. We used a spark gap and high-voltage capacitors from old equipment to make a resonator, and got the coil working. It could light a fluorescent tube from my full arm-span away. It was a smash hit at the science fair, too.

For one so knowledgable about the foundations of technology, he was awfully curmudgeonly about it. He bemoaned the day students started showing up in his class with hand-calculators instead of slide rules. He would never answer the phone (but would speak on it, if you could get his brother to pick up).

We kept meeting on and off, and we would have epic discussions/debates about technology, thermodynamics, perpetual motion machines, higher mathematics, theology, building test equipment, and logic puzzles. He taught me, in short, how to think.

A non-exhaustive list of things he taught me:

• How to build a crystal radio set
• How to troubleshoot a conventional radio (hint–check for signal at the volume control–that will narrow down the problem to either the front-end or back-end)
• How to compute resonant LC circuits
• How to use a slide rule
• How to pick locks
• How to compute power factor and plot phasor diagrams for AC circuits
• The value of good tools and how to care for them
• How to build a Tesla Coil
• How to debate
• Respect for high voltage
• The joy of back-issues of Scientific American
• The trouble with Pascal’s wager
• How to debunk perpetual motion claims
• How (and why) to use a planimeter

On a recent vacation, I went to see Virgil again–now in his 90s. He’s still vigorous and feisty, though his memory is starting to slip a little. It was difficult to come to terms with the possibility that, given the frequency with which I make it to that part of the country, it may be the last time I see him. Since this is posted online, he’ll probably never see it. But if he could speak to each one of you, I think he’d offer advice something like this:

Cherish the people in your life. Treat every meeting as if it might be the one that sets you on a new course–one that you’ll look back at years later in wonder. Don’t worry what others think of you, and never stop learning.

Thank you, Virgil, for all you’ve given me. -m

If I had to pick out a common theme for the release, it’s all about expanding the appeal of the server to reach new audiences. The Java API makes working with the server feel like a native extension to the language, and the REST API makes it easy to extend the same to other languages.

XQuery support is stronger than ever. I liked Ryan Dew’s take on some of the smaller, but still useful features.

This wouldn’t be complete without thanking my teammates who really made this possible. I had the great pleasure of working with some top-notch front-end people recently, and it’s been a great experience. -m

There is, of course, a large body of work on table-driven approaches, several of them listed here (and bigger list), though XQuery seems to have been largely left out of the fun.

In MarkLogic Search API, we implemented a recursive tokenizer. Since a search string can contain quoted pieces which need to be carefully maintained, first we split (in the fn:tokenize-sense, discarding matched delimiters) on the quote character, then iterate through the pieces. Odd-numbered pieces are chunks of tokens outside of any quoting, and even-numbered pieces are a single quoted string, to be preserved as-is. We recurse through the odd chunks, further breaking them down into individual tokens, as well as normalizing whitespace and a few other cleanup operations. This code is aggressively optimized, and it removes any searches for tokens known to not appear in the overall string. It also preserves the character offset positions of each token relative to the starting string, which gets used downstream, so this makes for some of the most complicated code in the Search API. But it’s blazingly fast.

When prototyping, it’s nice to have something simpler and more straightforward. So I came up with an approach using fn:analyze-string. This function, introduced in XSLT 2.0 and later ported to XQuery 3.0, takes a regular expression, and returns all of the target string, neatly divided into match and non-match portions. This is great, but difficult to apply across the entire string. For example, potential matches can have different meaning depending on where they fall (again, quoted strings as an example.) But if every regex starts with ^ which anchors the match to the front of the string, the problem simplifies to peeling off a single token from the front of the string. Keep doing this until there’s no string left.

This is a particularly nice approach when parsing a grammar that’s formally defined in EBNF. You can pretty much take the list of terminal expressions, port them to XQuery-style regexes, add a ^ in front of each, and roll.

Take SPARQL for example. It’s a reasonably rich grammar. The W3C draft spec has 35 productions for terminals. I sketched out some of the terminal rules (note these are simplified):

declare variable $spq:WS     := "^\s+";
declare variable $spq:QNAME  := "^[a-zA-Z][a-zA-Z0-9]*:[a-zA-Z][a-zA-Z0-9]*";
declare variable $spq:PREFIX := "^[a-zA-Z][a-zA-Z0-9]*:";
declare variable $spq:NAME   := "^[a-zA-Z][a-zA-Z0-9]*";
declare variable $spq:IRI    := "^<[^>]+>";
...

Then going through the input string, and seeing which of these expressions match, and if so, calling analyze-string and adding the matched portion as a token, and recursing on the non-matched portion. Note that we need to go through longer matches first, so the rule for ‘prefix:qname’ comes before the rule for ‘prefix:’ which comes before the rule for ‘string’

declare function spq:tokenize-recurse($in as xs:string, $tl as json:array) {
    if ($in eq "")
    then ()
    else spq:tokenize-recurse(
        switch(true())
        case matches($in, $spq:WS)     return spq:discard-tok($in, $spq:WS)
        case matches($in, $spq:QNAME)  return spq:peel($in, $spq:QNAME, $tl, "qname")
        case matches($in, $spq:PREFIX) return spq:peel($in, $spq:PREFIX, $tl, "prefix", 0, 1)
        case matches($in, $spq:NAME)   return spq:peel($in, $spq:NAME, $tl, "name")
        ...

Here, we’re co-opting a json:array mutable object as a convenient way to store tokens as we peel them off. There’s not actually any JSON involved here. The actual peeling looks like this:

declare function spq:peel(
    $in as xs:string,
    $regex as xs:string,
    $toklist as json:array,
    $type as xs:string,
    $triml, $trimr) {
    let $split := analyze-string($in, $regex)
    let $match := string($split/str:match)
    let $match := if ($triml gt 0) then substring($match, $triml + 1) else $match
    let $match := if ($trimr gt 0) then substring($match, 1, string-length($match) - $trimr) else $match
    let $_ := json:array-push($toklist, <searchdev:tok type="{$type}">{$match}</searchdev:tok>)
    let $result := string($split/str:non-match)
    return $result
};

Some productions, like a <iri> inside angle brackets, contain fixed delimiters which get trimmed off. Some productions, like whitespace, get thrown away. And that’s it. As it stands, it’s pretty close to a table-driven approach. It’s also more flexible than the recursive approach above–even for things like escaped quotes inside a string, if you can write a regex for it, you can lex it.

Performance

But is it fast? Short answer is that I don’t know. A full performance analysis would take some time. But a few quick inspections shows that it’s not terrible, and certainly good enough for prototype work. I have no evidence for this, but I also suspect that it’s amenable to server-side optimization–inside the regular expression matching code, paths that involve start-anchored matches should be easy to identify and in many cases avoid work farther down the string. There’s plenty of room on the XQuery side for optimization as well.

If you’ve experimented with different lexing techniques, or are interested in more details of this approach, drop me a line in the comments. -m

I’m speaking on Thursday about exploring large XML datasets. Looking forward to it!

-m

I’m speaking on Wednesday at 3:30 with Product Manager extraordinaire Justin Makeig about big data visualization. If you’ll be at the conference, come look me up. And if you won’t, well, forgive me if I need a few extra days to get back to any email you send this way.

Follow me on Twitter and look for the #MLW12 tag for live coverage.

-m