Archive for the 'software' Category

Friday, August 16th, 2019

Time Management for Senior Engineers

I recently joined LinkedIn as a Senior Staff engineer–an individual contributor role more senior than any of my previous roles. As such, I’ve been inundated with every manner of request for my time. Part of my journey toward handling this situation is writing up my thoughts. Let’s go.

Does time work differently for “senior” engineers? Yeah, pretty much. The way you need to manage your time changes dramatically throughout the course of a career. Take an intern, for example. They’ll have their 6-to-18-week project that they work on, with a manager checking in maybe once a week, give or take. They’re expected to attend customary project planning meetings including daily standups or similar.

Junior engineers have it much the same, though their assignment shifts over time. Gradually, they might start jugging multiple projects, and having to make their own prioritization decisions. In this mode of work, it’s helpful to block out chunks of time on your calendar for focused work. Bonus points for pushing back on excessive meetings.

At some rung of the engineering ladder, part of your job is figuring out what to work on, rather than just getting handed assignments. This complicates time management, in much the same way that Star Trek-style 3D chess is more complicated than ordinary chess. But with great responsibility, comes great power…

Time Domain Mastery Part 1: Delegation. One of my most productive working relationships was with a now-retired Principal Engineer. This guy had achieved a high rank–across thousands of engineers, there were a only a dozen or so PEs at the company. He made a point of never insisting on his way. In fact, he encouraged disagreement. But in those cases where you disagreed, you had better come equipped with data to make your point. That made collaborating with him invigorating as well as productive.

Now, as I look back at those days and see myself inching toward playing the part of the mentor, I also see valuable that must have been for him. He never viewed spending time with up-and-coming engineers as a chore, but rather as a force-multiplier for the team. It will always be the case that the set of things you’d like to do is bigger than the time available to get stuff done. Working with teams, you don’t have to do it all yourself. The more you raise up those around you, the greater the effect.

Time Domain Mastery Part 2: What have you accomplished today? Calendar blocks don’t seem to work any more at these higher rungs. At least in the role as I’ve experienced it, there’s a lot of temporal noise you have to deal with. Keeping afloat requires constant and intense skepticism to avoid the folly of the urgent. (And there’s always something urgent going on.)

One of the main ways that engineering leaders add value to an organization is by providing steady-handed distinction between the urgent and the important. It’s too easy to get caught on the treadmill of the urgent, rushing from one firefight to the next. The solution, of course, is to instead focus on important things, like making sure those fires don’t start in the first place!

To this end, every day you need to strive to have one notable accomplishment. And putting out fires isn’t an “accomplishment”. Concrete improvements to culture, process, architecture, or documentation, are. More than before, your success will hinge on your capability to set aside time at the start of the day, start of the week, and the start of the quarter to think and strategize. Bill Gates famously set aside “think weeks.” That’s not a bad start.

But what we really need to talk about is meetings. Thought experiment time. If you were an Apple engineer in the 90s, and Woz happened to pop in for one of your meetings, how would that have affected the dynamic of the meeting? I picked Wozniak for this example because he was (and is!) widely considered a strong technical player. Most engineers would look forward to the honor of working with him in a technical context. If it was Steve Jobs stopping by to scream at people, then there’d be all kinds of blame and accusation flying around. Yuck.

Politics aside, the higher up the engineering ladder, the bigger of a deal it is for you to be in a meeting. As you climb the ladder, it’s your imperative to make an impact at any meeting that’s worth your time to be in. This could mean asking the obvious questions that others are afraid to ask or to say what everyone is thinking. It might be using your technical capital to insist on sticking to the planned agenda and avoid rathole-ing on unimportant details. Based on your higher level of context, experience, and vision, you can reshape a meeting from a waste-of-everyone’s-time to a sharp-edged tool for your organization.

But not if you keep your nose in your laptop, working on one of your other urgent projects while only half-listening to what’s going on around you.

The most valuable contribution to a meeting might even mean taking notes when nobody else is willing (or able) to. A succinct summary of a complicated discussion can be worth its weight in gold. So is knocking the legs out from a corrosive culture that considers writing stuff down too mundane for senior folk.

The key takeaway–as a senior engineer, every action you take matters. Your actions have second-order effects–for example if junior engineers see you enduring through a useless meeting, they’ll start to think “that’s how things work around here.” Thus, for senior engineers, even second-order effects need to be carefully crafted due to their outsized impact.

You’ve spent a career becoming a leader. Act like it and you’ll be rewarded.

Production note: Version 1.2 Special thanks to Charles Greer, Jason Hunter, Tomomi Imura, and Evan Goer, for helpful feedback. This essay is a work in progress. Please comment below with suggestions for improvement.

Thursday, May 5th, 2016

Unsafe Java

Pop quiz. Why is the following Java 8 code unsafe? UPDATE: this code is fine, see comments. Still good to think about, though.

Entity e = new Entity();
e.setName("my new entity");

To provide some context, Entity is a POJO representing something we want to store in a database. And persistanceLayer is an instance of a Data Access Object responsible for storing the object in the database. Don’t see it yet?

Here’s the function signature of that put method

<E extends AbstractEntity> CompletableFuture<E> put(E newEntity);

Ah, yes, Spring, which because of the @Async annotation will proxy this method and cause it to execute in a different thread. The object getting passed in is clearly mutable, and by my reading of the Java Memory Model, there’s no guarantee that the other thread’s view of the update that happened in the setName method will be visible. In other words, there’s no guarantee the two threads idea of “before-ness” will be the same; who knows what might have been reordered or cached?

I’m having trouble coming to terms with what seems like a gaping hole. Do you agree with this analysis? If so, how do you deal with it? Immutable object classes seem like an obvious choice, but will introduce lots of overhead in other parts of the code that need to manipulate the entities. A builder pattern gets awkward and verbose quickly, as it hoists all that complexity on to the caller.

How do you handle situations like this? I’d love to hear it.

If you produce libraries, please include documentation on what is and isn’t guaranteed w.r.t. async calls. -m

Sunday, July 27th, 2014

Prime Number sieve in Scala

There are a number of sieve algorithms that can be used to list prime numbers up to a certain value.  I came up with this implementation in Scala. I rather like it, as it makes no use of division, modulus, and only one (explicit) multiplication.

Despite being in Scala, it’s not in a functional style. It uses the awesome mutable BitSet data structure which is very efficient in space and time. It is intrinsically ordered and it allows an iterator, which makes jumping to the next known prime easy. Constructing a BitSet from a for comprehension is also easy with breakOut.

The basic approach is the start with a large BitSet filled will all odd numbers (and 2), then iterate through the BitSet, constructing a new BitSet containing numbers to be crossed off, which is easily done with the &~= (and-not) reassignment method. Since this is a logical bitwise operation, it’s blazing fast. This code takes longer to compile than run on my oldish MacBook Air.

import scala.collection.mutable.BitSet
import scala.collection.breakOut
println(new java.util.Date())
val top = 200000
val sieve = BitSet(2)
sieve |= (3 to top by 2).map(identity)(breakOut)
val iter = sieve.iteratorFrom(3)
while(iter.hasNext) {
 val n =
 sieve &~= (2*n to top by n).map(identity)(breakOut)
println(sieve.toIndexedSeq(10000)) // 0-based
println(new java.util.Date())

As written here, it’s a solution to Euler #7, but it could be made even faster for more general use.

For example

  • I used a hard-coded top value (which is fine when you need to locate all primes up to  n). For finding the nth prime, though, the top limit could be calculated
  • I could stop iterating at sqrt(top)
  • I could construct the removal BitSet starting at n*n rather than n*2

I suspect that spending some time in the profiler could make this even faster. So take this as an example of the power of Scala, and a reminder that sometimes a non-FP solution can be valid too. Does anyone have a FP equivalent to this that doesn’t make my head hurt? :-)


Saturday, August 10th, 2013

XForms in 2013

This year’s Balisage conference was preceded by the international symposium on Native XML User Interfaces, which naturally enough centered around XForms.

As someone who’s written multiple articles surveying XForms implementations, I have to say that it’s fantastic to finally see one break out of the pack. Nearly every demo I saw in Montreal used XSLTForms if it used XForms at all. And yet, one participant I conversed with afterwards noted that very little that transpired at the symposium couldn’t have been done ten years ago.

It’s safe to say I have mixed emotions about XForms. One one hand, watching how poorly the browser makers have treated all things XML, I sometimes muse about what it would look like if we started fresh today. If we were starting anew, a namespace-free specification might be a possibility. But with  XForms 2.0 around the corner, it’s probably more fruitful to muse about implementations. Even though XSLTForms is awesome, I still want more. :-)

  • A stronger JavaScript interface. It needs to be possible to incrementally retrofit an existing page using POHF (plain old HTML forms) toward using XForms in whole or in part. We need an obvious mapping from XForms internals to HTML form controls.
  • Better default UI. I still see InfoPath as the leader here. Things designed in that software just look fantastic, even if quickly tossed together.
  • Combining the previous two bullets, the UI needs to be more customizable, and easier so. It needs to be utterly straightforward to make XForms parts of pages fit in with non-XForms parts of pages.
  • Rich text: despite several assertions during the week, XForms can actually handle mixed text, just not very well. One of the first demo apps (in the DENG engine–remember that?) was an HTML editor. The spec is explicitly designed in such a way as to allow new and exciting forms widgets, and a mixed-content widget would be A Big Deal, if done well.
  • Moar debugging tools

During the main conference, Michael Kay demonstrated Saxon-CE, an impressive tour-de-force in routing around the damage that is browser vendors’ attitudes toward XML. And though he didn’t make a big deal of it, it’s now available freely under an open source license. This just might change everything.

Curious about what others think here–I welcome your comments.


Sunday, March 31st, 2013

Introducing node-node:node.node

Naming is hard to do well, almost as hard as designing good software in the first place. Take for instance the term ‘node’ which depending on the context can mean

  1. A fundamental unit of the DOM (Document Object Model) used in creating rich HTML5 applications.
  2. A basic unit of the Semantic Web–a thing you can say stuff about. Some nodes are even unlabeled, and hence ‘blank nodes’.
  3. In operations, a node means, roughly, a machine on the network. E.g. “sixteen-node cluster”
  4. A software library for event-driven, asynchronous development with JavaScript.

I find myself at the forefront of a growing chorus of software architects and API designers that are fed up with this overloading of a perfectly good term. So I’m happy today to announce node-node:node.node.

The system is still in pre-alpha, but it solves all of the most pressing problems that software developers routinely run in to. In this framework, every node represents a node, for the ultimate in scalable distributed document storage. In addition, every node additionally serves as a node, which provides just enough context to make open-world assumption metadata assertions at node-node-level granularity. Using the power of Node, every node modeled as a node has instant access to other node-node:nodes. The network really is the computer. You may never write a program the old way again. Follow my progress on Sourceforge, the latest and most cutting-edge social code-sharing site. -m

Thursday, August 23rd, 2012

Super simple tokenizer in XQuery

A lexer might seem like one of the boringest pieces of code to write, but every language brings it’s own little wrinkles to the problem. Elegant solutions are more work, but also more rewarding.

There is, of course, a large body of work on table-driven approaches, several of them listed here (and bigger list), though XQuery seems to have been largely left out of the fun.

In MarkLogic Search API, we implemented a recursive tokenizer. Since a search string can contain quoted pieces which need to be carefully maintained, first we split (in the fn:tokenize-sense, discarding matched delimiters) on the quote character, then iterate through the pieces. Odd-numbered pieces are chunks of tokens outside of any quoting, and even-numbered pieces are a single quoted string, to be preserved as-is. We recurse through the odd chunks, further breaking them down into individual tokens, as well as normalizing whitespace and a few other cleanup operations. This code is aggressively optimized, and it removes any searches for tokens known to not appear in the overall string. It also preserves the character offset positions of each token relative to the starting string, which gets used downstream, so this makes for some of the most complicated code in the Search API. But it’s blazingly fast.

When prototyping, it’s nice to have something simpler and more straightforward. So I came up with an approach using fn:analyze-string. This function, introduced in XSLT 2.0 and later ported to XQuery 3.0, takes a regular expression, and returns all of the target string, neatly divided into match and non-match portions. This is great, but difficult to apply across the entire string. For example, potential matches can have different meaning depending on where they fall (again, quoted strings as an example.) But if every regex starts with ^ which anchors the match to the front of the string, the problem simplifies to peeling off a single token from the front of the string. Keep doing this until there’s no string left.

This is a particularly nice approach when parsing a grammar that’s formally defined in EBNF. You can pretty much take the list of terminal expressions, port them to XQuery-style regexes, add a ^ in front of each, and roll.

Take SPARQL for example. It’s a reasonably rich grammar. The W3C draft spec has 35 productions for terminals. I sketched out some of the terminal rules (note these are simplified):

declare variable $spq:WS     := "^\s+";
declare variable $spq:QNAME  := "^[a-zA-Z][a-zA-Z0-9]*:[a-zA-Z][a-zA-Z0-9]*";
declare variable $spq:PREFIX := "^[a-zA-Z][a-zA-Z0-9]*:";
declare variable $spq:NAME   := "^[a-zA-Z][a-zA-Z0-9]*";
declare variable $spq:IRI    := "^<[^>]+>";

Then going through the input string, and seeing which of these expressions match, and if so, calling analyze-string and adding the matched portion as a token, and recursing on the non-matched portion. Note that we need to go through longer matches first, so the rule for ‘prefix:qname’ comes before the rule for ‘prefix:’ which comes before the rule for ‘string’

declare function spq:tokenize-recurse($in as xs:string, $tl as json:array) {
    if ($in eq "")
    then ()
    else spq:tokenize-recurse(
        case matches($in, $spq:WS)     return spq:discard-tok($in, $spq:WS)
        case matches($in, $spq:QNAME)  return spq:peel($in, $spq:QNAME, $tl, "qname")
        case matches($in, $spq:PREFIX) return spq:peel($in, $spq:PREFIX, $tl, "prefix", 0, 1)
        case matches($in, $spq:NAME)   return spq:peel($in, $spq:NAME, $tl, "name")

Here, we’re co-opting a json:array mutable object as a convenient way to store tokens as we peel them off. There’s not actually any JSON involved here. The actual peeling looks like this:

declare function spq:peel(
    $in as xs:string,
    $regex as xs:string,
    $toklist as json:array,
    $type as xs:string,
    $triml, $trimr) {
    let $split := analyze-string($in, $regex)
    let $match := string($split/str:match)
    let $match := if ($triml gt 0) then substring($match, $triml + 1) else $match
    let $match := if ($trimr gt 0) then substring($match, 1, string-length($match) - $trimr) else $match
    let $_ := json:array-push($toklist, <searchdev:tok type="{$type}">{$match}</searchdev:tok>)
    let $result := string($split/str:non-match)
    return $result

Some productions, like a <iri> inside angle brackets, contain fixed delimiters which get trimmed off. Some productions, like whitespace, get thrown away. And that’s it. As it stands, it’s pretty close to a table-driven approach. It’s also more flexible than the recursive approach above–even for things like escaped quotes inside a string, if you can write a regex for it, you can lex it.


But is it fast? Short answer is that I don’t know. A full performance analysis would take some time. But a few quick inspections shows that it’s not terrible, and certainly good enough for prototype work. I have no evidence for this, but I also suspect that it’s amenable to server-side optimization–inside the regular expression matching code, paths that involve start-anchored matches should be easy to identify and in many cases avoid work farther down the string. There’s plenty of room on the XQuery side for optimization as well.

If you’ve experimented with different lexing techniques, or are interested in more details of this approach, drop me a line in the comments. -m

Sunday, January 15th, 2012

Five iOS keyboard tips you probably didn’t know

Check out these tips. The article talks about iPad, but they work on iPhone too, even an old 3G.

One one hand, it shows the intense amount of careful thought Apple puts into the user experience. But on the other hand, it highlights the discovery problem. I know people who have been using iOS since before it was called iOS, and still didn’t know about these. How do you put these kinds of finishing touches into a product and make sure the target audience can find out about them? -m

Wednesday, January 5th, 2011

Why I am abandoning Yahoo! Mail (and why you should too)

This is a non-technical description of why Yahoo! Mail is unsafe to use in a public setting, and indeed at all. I will be pointing people at this page as I go through the long process of changing an address I’ve had for more than a decade.

What’s wrong with Yahoo Mail?

A lot of web addresses start with http://–that’s a signal that the “scheme” used to deliver the page to your browser is something called HTTP, which is a technical specification that turns out is a really good way to move around web pages. As the page flows to the browser, it’s susceptible to eavesdropping, particularly over a wi-fi connection, and much more so in public, including the usual hotspots like coffee shops, but also workplaces and many home environments. It’s the virtual equivalent of a postcard. When you’re reading the news or checking traffic, it’s not a big deal if someone can sneak a glance at your page.

Some addresses start with https://–notice the extra ‘s’ which stands for “secure”. This means two things 1) that the web page being sent over is encrypted, and thus unavailable to eavesdroppers, and 2) that the people running the site had to obtain a certificate, which is a form of proof of their identity as an organization (that they’re not, say, Ukrainian phishers). Many years ago, serving pages over https was considered quite expensive in that servers needed much beefier processors to run all that encryption. Today, while it still requires extra computation, it’s not as big of a deal. Most off-the-shelf servers have plenty of extra power. To be fair, for a truly ginormous application with millions of users like Yahoo Mail, it is not a trivial thing to roll out. But it’s critically important.

First, to dispel a point of confusion, these days nearly every site, including Yahoo Mail, uses https for the login screen. This is the most critical time when encryption is needed, because otherwise you’d be sending your password on a postcard for anyone with even modest technical skills to peek at. So that’s good, but it’s no longer enough. Because sites are written so that you don’t have to reenter your password on every single new page, they use a tiny bit of information called a “cookie” in your browser to stay logged in. Cookies themselves are neither good nor bad, but if an eavesdropper gets a hold of one, they can control most of your account–everything that doesn’t require re-entering a password. In Yahoo Mail this includes reading any of your messages, sending mail on your behalf, or even deleting messages. Are you comfortable allowing strangers to do this?

As I mentioned earlier, new, more powerful tools have been out for months that automate the process of taking over accounts this way. Zero technical prowess is needed, only the ability to install a browser plug-in. If there are any web companies dealing in personal information for which this wasn’t a all-hands-on-deck security wake-up, they are grossly negligent. Indeed, other sites like Gmail work with https all-the-time. But still, in 2011, Yahoo Mail doesn’t. I have a soft spot for Yahoo as a former employer, and I want to keep liking them. Too bad they make it so difficult.

The deeper issue at stake is that if this serious of an issue goes unfixed for months, how many lesser issues lurk in the site and have been around for months or years? The issue is trust, my friend, and Yahoo just overdrew their account. I’m leaving.


Q: So what do you want Yahoo to do about this?  A: Well, they should fix their site for their millions of remaining users.

Q: What if they fix it tomorrow? Will you delete this message?  A: No. Since I no longer trust the site, I am leaving, even though it takes time to notify all the people who still send me mail, and no matter what other developments unfold in the meantime. This page will explain my actions.

Q: Do you really want everyone else to leave Yahoo Mail?  A: No, only those who care about their privacy.

Q: What’s your new email address?  A: I have a couple, but <my first name> @ <this domain> is a good general-purpose one.

I will continue to update this page as more information becomes available. -m

Thursday, September 2nd, 2010

Is XForms really MVC?

This epic posting on MVC helped me better understand the pattern, and all the variants that have flowed outward from the original design. One interesting observation is that the earlier designs used Views primarily as output-only, and Controllers primarily as input-only, and as a consequence the Controller was the one true path for getting data into the Model.

But with browser forms, input and output are tightly intermingled. The View takes care of input and output. Something else has primary responsibility for mediating the data flow to and from the model–and that something has been called a Presenter. This yields the MVP pattern.

The terminology gets confusing quickly, but roughly

XForms Instance == MVP Model

XForms Model == MVP Presenter

XForms User Interface == MVP View

It’s not wrong to associate XForms with MVC–the term has become so blurry that it’s easy to lump variants like MVP into the same bucket. But to the extent that it makes sense to talk about more specific patterns, maybe we should be calling the XForms design pattern MVP instead of MVC. Comments? Criticism? Fire away below. -m

Wednesday, July 7th, 2010

Grokking Selenium

As the world of web apps gets more framework-y, I need to get up to speed on contemporary automation testing tools. One of the most popular ones right now is the open source Selenium project. From the look of it, that project is going through an awkward adolescent phase. For example:

  • Selenium IDE lets you record tests in a number of languages, but only HTML ones can be played back. For someone using only Selenium IDE, it’s a confusing array of choices for no apparent reason.
  • Selenium RC has bindings for lots of different languages but not for the HTML tests that are most useful in Selenium IDE. (Why not include the ability to simply play through an entire recorded script in one call, instead of fine grained commands like selenium.key_press(input_id, 110), etc.?)
  • The list of projects prominently mentions Selenium Core (a JavaScript implementation), but when you click through to the documentation, it’s not mentioned. Elsewhere on the site it’s spoken of in deprecating terms.
  • If you look at the developer wiki, all the recent attention is on Web Drivers, a new architecture for remote-controlling browsers, but those aren’t mentioned in the docs (yet) either.

So yeah, right now it’s awkward and confusing. The underlying architecture of the project is undergoing a tectonic shift, something that would never see public light of day in a proprietary project. In the end it will come out leaner and meaner. What the project needs in the short term is more help from fresh outsiders who can visualize the desirable end state and help the ramped and productive developers on the project get there.

By the way, if this kind of problem seems interesting to you, let me know. We’re hiring. If you have any tips for getting up to speed in Selenium, comment below.


Saturday, June 12th, 2010

Command Lines on the frontier of user interface

This came from a comment on the prior post, and it’s worth a shout of its own. Don Norman on the importance of command lines, including the ubiquitous search box, in modern UI. -m

Wednesday, June 9th, 2010

“Google syntax” for semantic queries?

Thought experiment: are there any commonly-expressed semantic queries–the kind of queries you’d run over a triple store, or perhaps a SearchMonkey-annotated web site–expressible in common type-in-a-searchbox query grammar?

As a refresher, here’s some things that Google and other search engines can handle. The square brackets represent the search box into which the queries are typed, not part of the queries themselves.


[term -butnotthis]

[term1 OR term2]

[“phrase term”]

[tem1 OR term2 -“but not this” filetype:html]

So what kind of semantic queries would be usefully expressed in a similar way, avoiding SPARQL and the like? For example, maybe [by:”Micah Dubinko”] could map to a document containing a triple like <this document> <dc:author> “Micah Dubinko”. What other kinds of graph queries are interesting, common, and simple to express like this? Comments welcome.


Friday, May 14th, 2010

Geek Thoughts: verbing facebook

Facebook (v): to deliberately create an impenetrable computer user interface for purposes of manipulating users.

More collected Geek Thoughts at

Wednesday, May 5th, 2010

Geek Thoughts: no-fly lists and CAP Theorem

According to this article, a recent terror suspect almost got on a plane despite being recently added to the no-fly list. Why is it so difficult to administer a no-fly list? The CAP Theorem has answers. (Disclaimer: as always, this blog is apolitical–this isn’t about whether no-fly lists are a good idea or not, only a matter of technical interest)

Without stretching the imagination too much, one can think of a no-fly list as a distributed database. The list apparently changes frequently, and it needs to be accessible from thousands of airport gates and reservation desks. Thus CAP Theorem applies. In a nutshell, that theorem states that of Consistency, Availability, and Partition-tolerance, you can only pick, at most, two. Hit the link above for a much better, more complete description.

If there was one centralized list, the system would be Consistent and Available, but every time a name needed to be checked it would require an immediate network round-trip–should the connection to that central list go down, no further checks would be possible–no Partition tolerance.

Of course, the airline could set a policy that if said network connection goes down, no passengers at all would be able to get on planes. This would be a case of lack of Availability.

Or, the complete list could be periodically copied to each location that needs it. This provides good Availability and Partition tolerance, but fails Consistency, since it’s possible to miss out on late-breaking updates. Apparently, something like this is what happened.

More collected Geek Thoughts at

Friday, December 11th, 2009

Tinderbox 5 is out

At first glance, this seems to be the Snow Leopard of Tinderbox releases–lots of behind-the-scenes technology updates and largely the same core features. If you’re looking for a way to get more organized, it’s worth a look. Link. -m

Sunday, November 29th, 2009

The Model Endpoint Template (MET) organizational pattern for XRX apps

One of the lead bullets describing why XForms is cool always mentions that it is based on a Model View Controller framework. When building a full XRX app, though, MVC might not be the best choice to organize things overall. Why not?

Consider a typical XRX app, like MarkLogic Application Builder. (You can download a your copy of MarkLogic, including Application Builder, under the community license at the developer site.) For each page, the cycle goes like this:

  1. The browser requests a particular page, say the one that lets you configure sorting options in the app you’re building
  2. The page loads, including client-side XForms via JavaScript
  3. XForms requests the project state as XML from a designated endpoint; this becomes the XForms Instance Data
  4. Stuff happens on the page that changes the client-side state
  5. Just before leaving the page, XML representing the updated state is HTTP PUT back to the endpoint

The benefit of this approach is that you are dealing with XML all the way through, no impedance mismatches like you might find on an app that awkwardly transitions from (say) relational data to Java objects to urlencoded name/value pairs embedded in HTML syntax.

So why not do this in straight MVC? Honestly, MVC isn’t a bad choice, but it can get unwieldy. If an endpoint consists of a separate model+view+controller files, and each individual page consists of separate model+view+controller files, it adds up to a lot of stuff to keep track of. In truly huge apps, this much attention to organization might be worth it, but most apps aren’t that big. Thus the MET pattern.

Model: It still makes sense to keep the code that deals with particular models (closely aligned with Schemas) as a separate thing. All of Application Builder, for example, has only one model.

Endpoint: The job of an endpoint is to GET and PUT (and possibly POST and DELETE) XML, or other equivalent resource bundles depending on how many media types you want to deal with. It combines an aspect of controllers by being activated by a particular URL and views by providing the data in a consistent format.

Template: Since XForms documents already contain MVC mechanics, it not a high-payoff situation to further use MVC to construct the XForms and XHTML wrapper themselves. The important stuff happens within XForms, and then you need various templating mechanisms for example to provide consistent headers, footers, and other pieces across multiple pages. For this, an ordinary templating mechanism suffices. I can imagine dynamic assembly scenarios where this wouldn’t be the case, but again, many apps don’t need this kind of flexibility, and the complexity that comes along with it.

What about separation of concerns? Oh yeah, what about it? :-) Technically both Endpoints and Templates violate classical SOC. In an XRX app, this typically doesn’t lead to the kinds of spaghetti situations that it might otherwise. Endpoints are self contained, and can focus on doing just one thing well; with limited scope comes limited ability to get into trouble. For those times when you need to dig into the XQuery code of an endpoint, it’s actually helpful to see both the controller and view pieces laid out in one file.

As for Templates, simplicity wins. With the specifics of models and endpoints peeled away, the remaining challenge in developing individual pages is getting the XForms right, and again, it’s helpful to minimize the numbers of files one XForms page are split across. YAGNI applies to what’s left, at least in the stuff I’ve built.

So, I’ve been careful in the title to call this an “organizational pattern”, not a “design pattern” or an (ugh) “architectural pattern”. Nothing too profound here. I’d be happy to start seeing XRX apps laid out with directory names like “models”, “endpoints”, and “templates”.

What do you think? Comments welcome.


Saturday, September 26th, 2009

Geek Thoughts: stability theory

My personal stability theory, as it applies to software engineering: in a multilayered software architecture, the likelihood layer N works well can be expressed as a probability (less than 1 in practice) relative to the lower level layer N-1. For example, if you attempt to write a mission critical Tcl app on a flaky Tcl interpreter, you’re in for some long nights. Via multiplication, a corollary is that the more layers a system has, the less likely it is to work well. (As an aside, I’m not arguing that all software architectures should have fewer layers–other forces outside the scope of this article work against systems with too few layers.)

Joel said something similar lately in the article The Duct Tape Programmer. There is a strong tendency for many coders to over-engineer a system, building towering heights of abstraction. In contrast, a Duct Tape Programmer gets the job done by making something ugly (and with fewer layers) but at least it works. So far this is a fit with what stability theory predicts.

But then he speaks out against unit testing, referring to it in similar terms to the extravagant tower. Quoting JWZ: “If there’s no unit test the customer isn’t going to complain about that.” Here stability theory makes a different prediction. Particularly in the lower levels of the system, flakiness is disastrous. You have to be sure that your foundation is stable before building upon it, or you’re in for keyboard-on-forehead-induced head trauma. This is true no matter how tight the deadlines are or how much pressure is on. In fact, when you don’t have time for a write-over, its even more important to get it right the first time.

The top accomplishment for a coder is shipping software. Duct Tape Programmers make this happen by avoiding needless complexity, which is a great principle to live by. I’m reminded of what Brian Kernighan is attributed as saying:

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.

Debugging, or more generally making software that works well all the way to the user-facing layer, is hard. Anything that provides fundamental assertions about the stability of your foundation is a useful tool, so don’t slack off on the unit testing.

What about you? Have you found stability theory to be supported by the facts? Comment below.


Wednesday, April 22nd, 2009

TextWrangler and special characters

Hey readers, all seven of you, can you help me out?

I’m perhaps finally switching to a Mac-native text editor, TextWrangler, or if I really like it, BBEdit. Within that app, what’s the easiest way to enter unusual characters not found on a keyboard, say š (Latin s with háček) or ḫ (h-breve below)? In jEdit, one can set up longer strings that get automatically converted into harder-to-type ones. What’s the equivalent in TextWrangler or BBEdit? -m

Sunday, March 8th, 2009

Wolfram Alpha

The remarkable (and prolific) Stephen Wolfram has an idea called Wolfram Alpha. People used to assume the “Star Trek” model of computers:

that one would be able to ask a computer any factual question, and have it compute the answer.

Which has proved to be quite distant from reality. Instead

But armed with Mathematica and NKS [A New Kind of Science] I realized there’s another way: explicitly implement methods and models, as algorithms, and explicitly curate all data so that it is immediately computable.

It’s not easy to do this. Every different kind of method and model—and data—has its own special features and character. But with a mixture of Mathematica and NKS automation, and a lot of human experts, I’m happy to say that we’ve gotten a very long way.

I’m still a SearchMonkey guy at heart, so I wonder how much Wofram’s team is familiar with existing Semantic Web research and practice–because at a high level this seems very much like RDF with suitable queries thereupon. If that’s a good characterization, that’s A Good Thing, since practical application has been one of SemWeb’s weak spots.


Wednesday, January 7th, 2009

On porting WebPath to Python 3k

I’ve started looking into porting the WebPath code (and eventually XForms Validator) over to Python 3. The first step is external libraries, of which there is only one. WebPath uses the module from PLY. I had got it into my head that Python 2.x and 3.x were thoroughly incompatible, but leave it to the remarkable David Beazley to blow that assumption out of the water: the latest version of from SVN works in both 2.x and 3.x.

From there the included 2to3 tool was easy enough to run. (Relatively more difficult was getting 2.6 and 3.0 versions of Python frameworks installed on Mac, but even that wasn’t too bad.) The tool made some moderate changes, and I can run the unit tests, and a few even pass!

The primary remaining problem stems from code where the documentation is a little unclear, and my inexperience is severe. The part of the code in that reads nasty, grotty HTML via Tidy and produces a clean DOM throws an exception every time. Seems to be a mismatch between String and Byte (encoded string) types, but manifested as a failed XML parse. Sans exception handling, the code looks like:

    page = urllib.request.urlopen(fullurl)
    markup =
    dom = xml.dom.minidom.parseString(markup)

urlopen() returns a file-like object, but the docs didn’t seem clear on whether it’s like a file opened in byte or string mode. In any case, I’m almost certainly doing it wrong. Suggestions?


Tuesday, December 30th, 2008

RDFa parser in XQuery now open source

After a delay, the code to my RDFa parser in XQuery is now available under an Apache license. Go get it. This is some of the earliest XQuery code I ever wrote, so go easy on me. It follows the earlier work on a functional definition of RDFa. And feel free to send in patches. -m

Monday, December 8th, 2008

XML 2008 non-liveblog: Content Authoring Schemas

I was on the panel with Bob DuCharme, Frank Miller, and Evan Lenz discussing content authoring, from DITA to DocBook with some WordML sprinkled in for good measure. It was a good discussion, nothing earth-shaking. This session was laptopless, so I don’t have any significant notes. -m

Friday, December 5th, 2008

Mystery Python Theatre 3K

The long-awaited Python 3.0 is out. It fixes almost every annoyance I have with the language, particularly around Unicode handling, which is important in the kinds of projects I work on.

Now, to revisit some of my Open Source projects… -m

Thursday, October 30th, 2008

XiX (XForms in XQuery)

I’m pondering implementing the computational parts of the XForms Model in XQuery. Doing so in a largely functional environment poses some challenges, though. Has anybody tackled this before? How about in any functional language, including ML, Haskell, Scheme, XSLT, or careful Python?

I borrowed the book Purely Functional Data Structures from a friend–this looks to be a good start. What else is out there? Comment below. -m

Monday, October 20th, 2008

Running jEdit on Mac Java 1.6

I haven’t seen this anywhere else: jEdit doesn’t start up under the recent Mac Java 1.6. It bounces in the dock a few times then goes away.

The solution: manually run the main jar with java -jar path-to/jedit.jar, which will work. Go to the plugin manager and delete the MacOSX plugin. Java integration is good enough in 1.6 that this really isn’t needed anyway. Quit jEdit and now it will start up fine the usual way. -m

Monday, October 13th, 2008

Software narratives: write better software by watching movies

Without any exception I can think of: every top-notch software developer I know is also a skilled technical writer. Technical writing requires skill in choosing words, constructing sentences and paragraphs, and putting together the pieces in the right order to most effectively present the material.

In contrast, narrative writing requires an eye towards the bigger picture, an overall story arc. To put it another way, beginnings, middles, and ends. Hollywood screenwriters have got this down to a science, dividing screenplays into three acts. Next time you visit the movies, look for the parts and how the connect.

Act I, comprising about 1/4 of the whole work, introduces the characters and situation. Between Act I and Act II a key even happens to propel the story forward. Neo swallows the pill. Luke Skywalker finds his Aunt and Uncle killed. In Act II, comprising about 1/2 of the story, the “real story” begins. Another key moment happens to introduce the final Act III, which culminates during the final 1/4 of the story. Three acts: beginning, middle, and end. Other aspects of fiction writing, say characterization, are relatively less important in technical narratives.

A great introduction to these concepts is Syd Field’s Screenplay, to give one a broader view on what story is really all about, and why some stories move people more than others. Many of the concepts apply equally to software narratives. And like I wrote about earlier, such narratives are a powerful (if underused) tool in software development. -m

Friday, October 10th, 2008

More mobile XForms goodness

I haven’t tried this, but these guys claim to have a solution where

The form definitions are saved and exchanged as XForms, and the data as XForm[s] models. The data can be exchanged over http (if the phone users can afford GPRS and have a data connection) or over compressed SMS messages.

Sounds like they have the right idea… -m

Wednesday, October 1st, 2008

Evernote: the final piece of “it’s the data, stupid” clicks into place

Evernote now has import/export (in an XML format), meaning it now passes the generation test for data availability and lock-in-avoidance, as I wrote about some years ago. There’s a server API, as well as client-side scripting. I need to look into the details more, but as a start it looks like a home run. -m

Update: looking at the actual export XML, I’m disappointed. Each note is CDATA-escaped XML? Why???

Thursday, September 25th, 2008

The power of narrative in software development

I’m working on a piece of software that, while not the answer to world peace, is still pretty neat and approaches a specific problem in a fresh way. The project is at the stage where it needs to get unveiled to early adopters in the target audience. So how does one introduce possibly unfamiliar concepts in the form of a new API?

The approach we ended up using for the initial documentation is essentially a narrative–telling a story. Narrative fills the gap between use case and solution in an engaging way. People are naturally inclined to listen to stories, and to expect certain story structures, such as having a beginning, middle, and end with suitable transitions. Thus, if the listener senses a gap in the story, it’s easy for them to speak up. When the story works, people find it easier to map their personal story on to the narrative, leading to better absorption of new concepts, and a more positive impression of the software.

And it’s working. So far we’ve gotten far more useful feedback than we would have otherwise. Even before showing others, the exercise of writing the narrative has exposed gaps and flaws in our thinking, leading to a better, more cohesive design.

If you think back about how you learned about, say, object oriented programming, or event-driven programming, likely there was a story or detailed use case involved that helped you get on board with a new way of thinking. Software + story: It’s a powerful combination, I recommend it.

BTW, my team is hiring full-time positions. Especially if you’ve got XML skills, you could be part of this team. Send me email if interested. -m

Wednesday, September 17th, 2008

The case for native higher-order functions in XQuery

The XQuery Working Group is debating the need for higher-order functions in the language. I’m working on honing my description of why this is an important feature. Does this work? What would work better?

Imagine you are writing a smallish widget app, in an environment without a standard library. When you need to sort your widgets, you’d write a simple function with a signature like sort(sequence-of-widgets). That’s great.

Now imagine you find your app to be steadily growing. An accumulation of smaller one-off solutions won’t work anymore, you need a general solution. What you’ll end up with is something like qsort in C, which takes a pointer to a comparator function. By providing different comparators, you can sort anything any way you like, all through only a single sort function. C and C++ have something like this, as do PHP, Python, Java, JavaScript, and even assembly language. XSLT has it, as proven by Dimitre.

XQuery doesn’t. It should, because people are now using it for more than short queries. People are writing programs in it. -m

P. S. Comment please.