If you have webdev skillz, you might be interested in the SearchMonkey launch party on May 15. Good food, good drink, good coding. Space is limited, but I have a few invites to share. Comment here or contact me offline if interested. -m

I haven’t mentioned it yet, but SearchMonkey (now an official name, not just a project name) is in external limited beta. Keep an eye on ysearchblog, lots more technical content is on the way. -m

I’m not involved in the the corporate wrangling about Microsoft and Yahoo! talks. Which leaves me relatively free to comment on it. [Disclosure: I am, not too surprisingly, a Yahoo! shareholder.]

Lots of things have been happening lately. A deadline of, well, today. Talks of Google adsense trials. And all kinds of merger speculation involving Rupert Murdoch in some fashion, or else AOL.

But I haven’t seen anyone point out this connection: Google owns 5% of AOL, having invested a billion bucks and taken over search there a couple of years ago. So if Yahoo! and AOL merged, there would already be a Google advertising connection in place. Running pre-trials now is just due dilligence on something that might happen anyway.

Having both an in-house advertising network and an outsourced one has some advantages too, namely in the form of “knobs” that can be adjusted to tune margins as conditions warrant. And maintaining the in-house system keeps Google honest and makes sure that relatively good deals can be negotiated in the future.

Lots of pundits talk about regulatory scrutiny, but honestly, it’s been years since any antitrust machinery in this country has been effective. And the recent spectrum auctions showcased Google’s skill at turning regulatory tables in their favor. If it came down to it, the smart people on both sides of the table shouldn’t have a problem crafting an agreement in a way that meets muster, even in the stricter EU.

Summary: based solely on public reports, it seems like the AOL connection might be a credible threat to Microsoft’s appetite. The ball is firmly in Steve’s court now. We’ll see what he does.

So today Yahoo! announced a major facet of what I’ve been working on lately: making the web more meaningful. Lots of fantastic coverage, including TechCrunch and ReadWriteWeb (and others, please link in the comments), and supportive responses and blog posts across the board. It’s been a while since I’ve felt this good about being a Yahoo.

So what exactly is it?

A few months ago I went through the pages on this very blog and added hAtom markup. As a result of this change…well, nothing happened. I had a good experience learning about exactly what is involved in retrofitting an existing site with microformats, but I didn’t get any tangible benefit. With the “SearchMonkey” platform, any site using microformats, or RDFa or eRDF, is exposed to developers who can enhance search results. An enhanced result won’t directly make my my site rank higher in search, it it most certainly make it prone to more clicks, and ultimately more readership, more inlinks, and better organic ranking.

How about some questions and answers:

Q: Is this Tim Berners-Lee’s vision of the Semantic Web finally getting fulfilled?

A: No.

Q: Does this presuppose everybody rushing to change their sites to include microformats, RDF, etc?

A: No. After all, there is a developer platform. Naturally, developers will have an easier time with sites that use official and community standards for structuring data, but there is no obligation for any site to make changes in order to participate and benefit.

Q: Why would a site want to expose all its precious data in an easily-extractable way?

A: Because within a healthy ecosystem it results in a measurable increase in traffic and customer satisfaction. Data on the public web is already extractable, given enough eyeballs. An openness strategy pays off (of which SearchMonkey is an existence proof).

Q: What about metacrap? We can never trust sites to provide honest metadata.

A: The system does have significant spam deterrents built in, of which I won’t say more. But perhaps more importantly, the plugin nature of the platform uses the power of the community to shape itself. A spammy plugin won’t get installed by users. A site that mixes in fraudulent RDFa metadata with real content will get exposed as fraudulent, and users will abandon ship.

Q: Didn’t ask.com prove that having a better user interface doesn’t help gain search market share?

A: Perhaps. But this isn’t about user interface–it’s about data (which enables a much better interface.)

Q: Won’t (Google|Microsoft|some startup) just immediately clone this idea and take advantage of all the new metadata out there?

A: I’m sure these guys will have some kind of response, and it’s true that a rising tide lifts all boats. But I don’t see anyone else cloning this exactly. The way it’s implemented has a distinctly Yahoo! appeal to it. Nobody has cloned Yahoo! Answers yet, either. In some ways, this is a return to roots, since Yahoo! started off as a human-guided directory. SearchMonkey is similar, except a much broader group of people can now participate. And there are some specific human, technical and financial reasons why as well, but I suggest inviting me out for beers if you want specifics. :-)

Disclaimer: as always, I’m not speaking for my employer. See the standard disclaimer. -m

Update: more Q and A

Q: How is SearchMonkey related to the recently announced Yahoo! Microsearch?

A: In brief, Microsearch is a research project (and a very cool one) with far-reaching goals, while SearchMonkey is targeted as imminently shipping software. I frequently talk to and compare notes with Peter Mika, the lead researcher for Microsearch.

Somehow I missed this posting and the underlying news that a Y Research project has a nice public demo of semantic search, driven by RDF, RDFa, and microformats. Still a rough sketch of a full solution, with multiple-second access times. But I particularly like the query for renaissance faire. -m

As spotted on TechCrunch, full article. This is a game-changer folks. Check out the comments attached to the article. -m

It’s been an exhausting past couple of weeks, but life goes on. WebPath made front page at next.yahoo. I’m starting to get feedback from developers who are actually using it, filing bugs, suggesting features, and it’s gratifying. The community is still building up. Won’t you join too? -m

WebPath, my experimental XPath 2.0 engine in Python is now an open source project with a liberal BSD license. I originally developed this during a Yahoo! Hack Day, and now I get to announce it during another Hack Day. Seems appropriate.

The focus of WebPath was rapid development and providing an experimental platform. There remains tons of potential work left to do on it…watch this space for continued discussion. I’d like to call out special thanks to the Yahoo! management for supporting me on this, and to Douglas Crockford for turning me on to Top Down Operator Precedence parsers. Have a look at the code. You might be pleasantly surprised at how small and simple a basic XPath 2 engine can be. So, who’s up for some XPath hacking?

Code download. (Coming to SourceForge with CVS, etc., in however many days it takes them to approve a new project) I hope this inspires more developers to work on similar projects, or better yet, on this one! -m

Take a look at this URL, and the page behind it. This is a list of all the Flickr photos with the tag “xmlns:dc=http://purl.org/dc/elements/1.1/“. Although these have been around for a while, I hadn’t been aware of this kind of tagging until recently.

Why “xml” in the namespace declaration? This doesn’t have much to do with XML. How many tags are there in the world that start with “dc:” and are not referring to Dublin Core? At least the tag declaring the namespace provides a good hook for finding things with machine tags. It’s only a small step up to RDFa from here, which is good! -m

Admittedly, their marketing folks wouldn’t describe it that way, but essentially that’s what was announced today. (documentation in PDF format, closely related to what-used-to-be Konfabulator tech; here’s the interesting part in HTML) The press release talks about reaching “billions” of mobile consumers; even if you don’t put too much emphasis on press releases (you shouldn’t) it’s still talking about serious use of and commitment to XForms technology.

Shameless plug: Isn’t it time to refresh your memory, or even find out for the first time about XForms? There is this excellent book available in printed format from Amazon, as well as online for free under an open content license. If you guys express enough interest, good things might even happen, like a refresh to the content. Let’s make it happen.

From a consumer standpoint, this feels like a welcome play against Android, too. Yahoo! looks like it’s placing a bet on working with more devices while making development easier at the same time. I’ll bet an Android port will be available, at least in beta, before the end of the year.

Disclaimer: I have been out of Yahoo! mobile for several months now, and can’t claim any credit for or inside knowledge of these developments. -m

P. S. Don’t forget the book.

Thanks to all the folks who showed interest in this little XPath puzzler published here a few weeks ago. Some asked to see the dataset, but I’m not able to release it at this time (but ask me again in 3 months).

Turns out it was a combination of two bugs, one mine, one somebody else’s. Careful observers noted that I wasn’t using any namespace prefixes in the XPath, and since I did specify that it was XPath 1.0, that technically rules out XHTML as the source language. Like nearly all XML I work with these days, the first thing I do is strip off the namespaces to make it easier to work with. Bug #1 was that in a few cases, the namespaces didn’t get stripped.

Bug #2 was in the XPath engine itself. Which one? Uh, whatever one ships with the “XPath” plugin for JEdit. It’s hard to tell directly, but I think it might be an older version of Xalan-J. In the case of the expression //meta, it properly located only those elements part of no namespace. But in the case of //meta/@property, it was including all the nodes that would have been selected by //*[local-name(.)='meta']/@property. Hence, a larger number of returned nodes.

Confusing? You bet!  -m

P.S. WebPath would not have this problem, since in the default mode it matches local-names only to begin with.

OK, the majority of the buzz came from my talk, where I strongly encouraged folks to take a look at Hadoop. This article seems to be saying much the same things. If you’re curious about the future of distributed computation and storage, it’s worth a look. -m

I didn’t get to do much for Yahoo Hack Day, but I did get to help a coworker a teeny bit with an implementation of Y! Search for social web sites, including Facebook. There could be some interesting repercussions from that, so I won’t say more now. But what did surprise me is how many Yahoos are active on Facebook.

Myself–I’m still a Facebook curmudgeon. But mostly I simply haven’t had the time to check it out, or figure out the value proposition of accepting an invitation. -m

As widely reported by now, the final schedule for XML 2007 this December in Boston is up. All I have to add is the suggestion of careful attention to the Tuesday program at 4:00. :) If you can’t wait, some technical details are forthcoming in this space. That is all. -m

A strange note at the top of the web site that appeared in the last couple of days:

Yahoo! apologizes deeply, but we will be closing down the Podcasts site on Oct. 31, 2007

It’s underlined to look like a link, but in fact is just styled that way. Some further observations and questions:

• The writing style is out of sync with the rest of the site, clearly done by someone else, likely not a marketing person nor designer
• View-source shows the markup was jammed in there without regard to the CSS/structure on the rest of the site (look for style=”color:#fff;text-decoration:underline;padding:6px;margin:4px;”)
• Is this a case of the original folks who wrote the site having moved on, and the new owners find it easier to shut down?
• Is this a part of “finding focus in hollywood”? (linked earlier)
• Is it a good idea for Yahoo! Podcasts to exist? If there’s even a Y! site for Wii, why not podcasts?
• Is this a case of not being able to automate things down enough?
• Why doesn’t Yahoo! produce any podcasts?

-m

(Note: I wasn’t involved in Y! Podcasts nor in the apparent decision to shut it down)

Today is my 2nd anniversary at Yahoo!. Looking back, it’s been a great time. Since I don’t know how long ago, I’ve fantasized about being involved in research. Check. Since sitting across from the mobile guys for 5 years in W3C meetings, I’ve fantasized about working in mobile. Check. And since I wrote Web search, without the web (demo), I’ve fantasized about working on web-scale search.

Check.

What will the next two years bring? I don’t know, but I’m certain they will be even better than the previous two. -m

Still more mobile news. Yahoo! Go is shipping. No alpha, beta, gamma, etc.–the real deal. Give it a whirl. If your phone, like mine, can’t handle the awesomeness, you can visit the slick web-only version at m.yahoo.com. -m

Have a look at these new Samsung phones, especially on page 2. Still limited to Asian territories, but this sort of thing has to be coming to the US and Europe as well… -m

The big buzz today is news that Terry Semel has stepped down as CEO of Yahoo, and Jerry Yang has stepped up. Believe it or not, Y is the first place I’ve worked at that’s large enough that I’m not on a first name basis with the CEO, so this kind of thing is much less personal. I can’t remember even shaking his hand.

In reflection, Terry’s management style seems to be based largely on the Dale Carnegie How-to-win-friends-and-influence-people style that’s been influential in my life and career. I have more respect for him than for most of the armchair CEO’s out there. I never understood some of the venom hurled at Semel, but then again I’m happy to see the change–I think Yahoo needs a little less Hollywood right now.
In short, there’s lots of change happening at Yahoo, both at the management level and in my personal role to play. But each change so far has been good, and the changes are adding up to something even bigger. After some recent personal tough times, I’m glad I decided to stay with Yahoo. What’s even better, there will probably be no more Tom Cruise visits. :-) -m

The approximately seven readers of this blog have probably already heard this, but just in case: I have a new role at Yahoo!–working on next generation search.

Lots of details are still falling into place. For now I describe it: “Imagining, specifying, prototyping, developing, and evangelizing next-generation web search experiences leveraging the full and unique capabilities available within Yahoo!”

In many ways, this is a logical stepping stone after oneSearch, and I’ll be dealing with lowercase semantic web issues more now. Expect the focus of this blog to shift accordingly (though I’m still interested in mobile and will make note of important happenings.)

Search On! -m

I thought this article was interesting in overall tone and a specific quote:

Modifying the software for each phone’s display is a matter of brute-force labor. There’s no intellectual way around it. Yahoo! is one of the few companies that’s been able to pull this off, but only because they have an army of Ph.D. hackers working for them.

Thanks! The primary design for the content adaptor was done by one non-Ph.D.–me–with plenty of help from the resident “phone whisperer” and a talented team of fellow non-Ph.Ds. It’s not a matter of “brute force” at all. The only way to solve the problem with finite resources is to understand developers, understand the problem space, and be smart about drawing a line between the two (and being flexible enough to handle the inevitable unknown).

One thing is certain: the industry is changing fast. A mobile app working great today will look dodgy in a year, and be obsolete in two years. It’s not clear if this will stabilize at some point, or keep shifting..

But I’m curious about what the rest of you think. Is mobile the next big thing, or a huge sand trap? Comment below. -m

Here’s a NYT article on Yahoo! Mobile, including a picture of the “warroom” where I spend a few hours on a typical day. The sign on the back wall says “platform team”, and on top of that “Maru” in Kanji and roman characters. -m

Here’s a great comparison site. Try out some searches you might run from your phone and let me know: which one did you prefer? Why? -m

Today Yahoo! launched oneSearch on their other front page, m.yahoo.com. OneSearch has been available for a while, but only from within Yahoo! Go. Now it’s available to millions of mobile devices equipped with a data connection and XHTML browser.

The basic premise behind oneSearch is to replace the tri-modal search box, where you have to say whether you are searching the web, local, or images, with a single all-knowing search box. Available context information, such as your zip code, is used to guide the search. Internally, the application is smart about figuring out what kind of things you might be looking for. For example, someone searching for “pizza” in a mobile context is probably more interested in a list of restaurants (with reviews) than in a list of hyperlinks. Behind the simplicity of a single search box, there is a great deal of work going on to make your life easier.
Ever since Yahoo! Go betas (and gammas) started coming out, folks have been asking me how else they could get access to this application. Now it’s easy.

Not too long ago, the front page relaunched simultaneously in 19 countries. The new design was simple, and based on a new platform called Sushi, as mentioned in published sources. OneSearch shows off the power of this approach, even though this launch didn’t cover 19 countries…yet. (Getting access to local data for movies, restaurants, sporting events, and so on is no small feat.)

As I said before, this is only a small part of an overall strategy that has been years in the making. Much more to come. Watch this space. -m

Spotted under the headline Windows Live Search for Mobile Goes Final, Still Great (like they were expecting it to suddenly plummet in quality?) on Gizmodo. It’s a 114k jar file that runs on my SLVR, where Yahoo! Go isn’t yet available yet, so points for that. Search suggestions show as you type, hugely useful on a klunky 9-key entry situation. They use an interesting UI to hold search results, densely packed–6 down the screen–with a status bar on top, and each search result marquee-scrolling back-and-forth as needed. A detail page can zap you in to map mode or set up a call.

My standard test search–a little offbeat but still plausible–for mead near Sunnyvale produced disappointing results. The meadery within walking distance didn’t show, and of the top 6, two were duplicates. Scrolling down to the 10th result, though, did show an interesting, useful result, albeit 60.15 miles away: Knowne World Meads. I wanted to visit the web site, but here lies another problem: there’s no web integration. None of the search results include a URL or clickable link.

For all the hassle, I’ll stick with Opera Mini and my favorite search engine, thank you. -m

Check it out. -m

A few more tidbits on the Softbank Mobile turnaround, for which helped architect the mobile platform.

SoftBank phones have a “Y!”-button which links to Yahoo!-keitai. Yahoo-Keitai! offers a list of official sites, new services (e.g. a new communicator service), and also access to free mobile internet sites through the YAHOO directory, as well as access to YAHOO services, such as YAHOO-auctions.

-m

Congrats to Opera Mini on its first anniversary. I just installed it on my new SLVR, and the download is an astounding 98k. Why can’t more software be this lean? And yes, Y! search came as the default. -m

My second career as a hand model has been exposed, by a blog post from the guys behind Yahoo! Messenger. BTW, I have even more stickers now, including one of those Flickr stickrs seen in the shot. -m

P.S. The photo is credited to Yodel Anecdotal, which is another blog. Did anyone catch when that first showed up? URL anyone?

(Press release) Starting today, Y! is the exclusive search partner for Opera Mini across more than 100 countries. The release also names “oneSearch”, going live later in Q1–definitely something to keep an eye on. -m