• Selenium IDE lets you record tests in a number of languages, but only HTML ones can be played back. For someone using only Selenium IDE, it’s a confusing array of choices for no apparent reason.
• Selenium RC has bindings for lots of different languages but not for the HTML tests that are most useful in Selenium IDE. (Why not include the ability to simply play through an entire recorded script in one call, instead of fine grained commands like selenium.key_press(input_id, 110), etc.?)
• The list of projects prominently mentions Selenium Core (a JavaScript implementation), but when you click through to the documentation, it’s not mentioned. Elsewhere on the site it’s spoken of in deprecating terms.
• If you look at the developer wiki, all the recent attention is on Web Drivers, a new architecture for remote-controlling browsers, but those aren’t mentioned in the docs (yet) either.

So yeah, right now it’s awkward and confusing. The underlying architecture of the project is undergoing a tectonic shift, something that would never see public light of day in a proprietary project. In the end it will come out leaner and meaner. What the project needs in the short term is more help from fresh outsiders who can visualize the desirable end state and help the ramped and productive developers on the project get there.

By the way, if this kind of problem seems interesting to you, let me know. We’re hiring. If you have any tips for getting up to speed in Selenium, comment below.

-m

As a refresher, here’s some things that Google and other search engines can handle. The square brackets represent the search box into which the queries are typed, not part of the queries themselves.

[term]

[term -butnotthis]

[term1 OR term2]

["phrase term"]

[tem1 OR term2 -"but not this" site:dubinko.info filetype:html]

So what kind of semantic queries would be usefully expressed in a similar way, avoiding SPARQL and the like? For example, maybe [by:"Micah Dubinko"] could map to a document containing a triple like <this document> <dc:author> “Micah Dubinko”. What other kinds of graph queries are interesting, common, and simple to express like this? Comments welcome.

-m

More collected Geek Thoughts at http://geekthoughts.info.

Without stretching the imagination too much, one can think of a no-fly list as a distributed database. The list apparently changes frequently, and it needs to be accessible from thousands of airport gates and reservation desks. Thus CAP Theorem applies. In a nutshell, that theorem states that of Consistency, Availability, and Partition-tolerance, you can only pick, at most, two. Hit the link above for a much better, more complete description.

If there was one centralized list, the system would be Consistent and Available, but every time a name needed to be checked it would require an immediate network round-trip–should the connection to that central list go down, no further checks would be possible–no Partition tolerance.

Of course, the airline could set a policy that if said network connection goes down, no passengers at all would be able to get on planes. This would be a case of lack of Availability.

Or, the complete list could be periodically copied to each location that needs it. This provides good Availability and Partition tolerance, but fails Consistency, since it’s possible to miss out on late-breaking updates. Apparently, something like this is what happened.

More collected Geek Thoughts at http://geekthoughts.info.

Consider a typical XRX app, like MarkLogic Application Builder. (You can download a your copy of MarkLogic, including Application Builder, under the community license at the developer site.) For each page, the cycle goes like this:

1. The browser requests a particular page, say the one that lets you configure sorting options in the app you’re building
2. The page loads, including client-side XForms via JavaScript
3. XForms requests the project state as XML from a designated endpoint; this becomes the XForms Instance Data
4. Stuff happens on the page that changes the client-side state
5. Just before leaving the page, XML representing the updated state is HTTP PUT back to the endpoint

The benefit of this approach is that you are dealing with XML all the way through, no impedance mismatches like you might find on an app that awkwardly transitions from (say) relational data to Java objects to urlencoded name/value pairs embedded in HTML syntax.

So why not do this in straight MVC? Honestly, MVC isn’t a bad choice, but it can get unwieldy. If an endpoint consists of a separate model+view+controller files, and each individual page consists of separate model+view+controller files, it adds up to a lot of stuff to keep track of. In truly huge apps, this much attention to organization might be worth it, but most apps aren’t that big. Thus the MET pattern.

Model: It still makes sense to keep the code that deals with particular models (closely aligned with Schemas) as a separate thing. All of Application Builder, for example, has only one model.

Endpoint: The job of an endpoint is to GET and PUT (and possibly POST and DELETE) XML, or other equivalent resource bundles depending on how many media types you want to deal with. It combines an aspect of controllers by being activated by a particular URL and views by providing the data in a consistent format.

Template: Since XForms documents already contain MVC mechanics, it not a high-payoff situation to further use MVC to construct the XForms and XHTML wrapper themselves. The important stuff happens within XForms, and then you need various templating mechanisms for example to provide consistent headers, footers, and other pieces across multiple pages. For this, an ordinary templating mechanism suffices. I can imagine dynamic assembly scenarios where this wouldn’t be the case, but again, many apps don’t need this kind of flexibility, and the complexity that comes along with it.

What about separation of concerns? Oh yeah, what about it? :-) Technically both Endpoints and Templates violate classical SOC. In an XRX app, this typically doesn’t lead to the kinds of spaghetti situations that it might otherwise. Endpoints are self contained, and can focus on doing just one thing well; with limited scope comes limited ability to get into trouble. For those times when you need to dig into the XQuery code of an endpoint, it’s actually helpful to see both the controller and view pieces laid out in one file.

As for Templates, simplicity wins. With the specifics of models and endpoints peeled away, the remaining challenge in developing individual pages is getting the XForms right, and again, it’s helpful to minimize the numbers of files one XForms page are split across. YAGNI applies to what’s left, at least in the stuff I’ve built.

So, I’ve been careful in the title to call this an “organizational pattern”, not a “design pattern” or an (ugh) “architectural pattern”. Nothing too profound here. I’d be happy to start seeing XRX apps laid out with directory names like “models”, “endpoints”, and “templates”.

What do you think? Comments welcome.

-m

Joel said something similar lately in the article The Duct Tape Programmer. There is a strong tendency for many coders to over-engineer a system, building towering heights of abstraction. In contrast, a Duct Tape Programmer gets the job done by making something ugly (and with fewer layers) but at least it works. So far this is a fit with what stability theory predicts.

But then he speaks out against unit testing, referring to it in similar terms to the extravagant tower. Quoting JWZ: “If there’s no unit test the customer isn’t going to complain about that.” Here stability theory makes a different prediction. Particularly in the lower levels of the system, flakiness is disastrous. You have to be sure that your foundation is stable before building upon it, or you’re in for keyboard-on-forehead-induced head trauma. This is true no matter how tight the deadlines are or how much pressure is on. In fact, when you don’t have time for a write-over, its even more important to get it right the first time.

The top accomplishment for a coder is shipping software. Duct Tape Programmers make this happen by avoiding needless complexity, which is a great principle to live by. I’m reminded of what Brian Kernighan is attributed as saying:

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.

Debugging, or more generally making software that works well all the way to the user-facing layer, is hard. Anything that provides fundamental assertions about the stability of your foundation is a useful tool, so don’t slack off on the unit testing.

What about you? Have you found stability theory to be supported by the facts? Comment below.

-m

I’m perhaps finally switching to a Mac-native text editor, TextWrangler, or if I really like it, BBEdit. Within that app, what’s the easiest way to enter unusual characters not found on a keyboard, say š (Latin s with háček) or ḫ (h-breve below)? In jEdit, one can set up longer strings that get automatically converted into harder-to-type ones. What’s the equivalent in TextWrangler or BBEdit? -m

that one would be able to ask a computer any factual question, and have it compute the answer.

Which has proved to be quite distant from reality. Instead

But armed with Mathematica and NKS [A New Kind of Science] I realized there’s another way: explicitly implement methods and models, as algorithms, and explicitly curate all data so that it is immediately computable.

It’s not easy to do this. Every different kind of method and model—and data—has its own special features and character. But with a mixture of Mathematica and NKS automation, and a lot of human experts, I’m happy to say that we’ve gotten a very long way.

I’m still a SearchMonkey guy at heart, so I wonder how much Wofram’s team is familiar with existing Semantic Web research and practice–because at a high level this seems very much like RDF with suitable queries thereupon. If that’s a good characterization, that’s A Good Thing, since practical application has been one of SemWeb’s weak spots.

-m

From there the included 2to3 tool was easy enough to run. (Relatively more difficult was getting 2.6 and 3.0 versions of Python frameworks installed on Mac, but even that wasn’t too bad.) The tool made some moderate changes, and I can run the unit tests, and a few even pass!

The primary remaining problem stems from code where the documentation is a little unclear, and my inexperience is severe. The part of the code in platonicweb.py that reads nasty, grotty HTML via Tidy and produces a clean DOM throws an exception every time. Seems to be a mismatch between String and Byte (encoded string) types, but manifested as a failed XML parse. Sans exception handling, the code looks like:

    page = urllib.request.urlopen(fullurl)
    markup = page.read()
    dom = xml.dom.minidom.parseString(markup)

urlopen() returns a file-like object, but the docs didn’t seem clear on whether it’s like a file opened in byte or string mode. In any case, I’m almost certainly doing it wrong. Suggestions?

-m

Now, to revisit some of my Open Source projects… -m

I borrowed the book Purely Functional Data Structures from a friend–this looks to be a good start. What else is out there? Comment below. -m