Building and Operating a Large-scale Computing Infrastructure

Last night, Google Kirkland hosted a Tech Talk for Cornell CS alumni.

Building and Operating a Large-scale Computing Infrastructure

Google is quite a bit more than a typical Internet company in that we spend a significant effort architecting our computing infrastructure for maximum efficiency. In this talk we will discuss some of the challenges involved in handling failures and optimizing performance for large-scale services, and how dealing with such issues requires a broad range of expertise and the development of new, sophisticated tools.

About the Speaker:
Reza Behforooz is a Software Engineer at Google. He’s passionate about building large distributed systems and working on consumer facing communication products in an attempt to make the world a smaller place. Reza holds a BS from Cornell and a MS from Stanford in Computer Science. He occasionally blogs on http://www.rezab.com.

[ Aside: AC, who also graduated from my alma mater, is also Canadian, used to work in the same building I do now (but now works on the same floor as I used to work in Messenger), and works on AdCenter, wanted to carpool so I gave him a ride.  And we were wearing the *exact* same CK shirt.  What are the chances of that?!  I thought it was a pretty darn unique shirt as well.  ]

Anyway, the talk was largely a very high level overview of some of the challenges and solutions to building highly distributed services.  I didn’t really learn anything from it, but it did strike me, having intimate knowledge of how Live Messenger and Live Search work, that they both tackle the same issues (duh) and both came up with very similar solutions to the problem.  It’s like being presented with conquering the problem of flight.  Insects, birds, and mammals (eg. bats) all solve the problem using wings, but they all evolved differently and use slightly different mechanics.  [Note: I have absolutely no idea if the original group of people working on MSN Search studied how Goog purportedly built their system at the time.]  The problems?  Building fault tolerant systems on faulty hardware, monitoring health of the system, collecting diagnostic data, etc etc.

Other interesting observations: a large number of the Goog employees at the Kirkland office are Microsoft defectors (I know one or two myself).  I have nothing against these people, but it’s interesting how they like to talk about how much better it is to work at Goog than MS.  I’ll cede to the fact that not all MS employees are happy (Mini), but I think, with such a large company, things are mostly isolated to pockets of discontent (or pockets of content, depending on how you look at it).  It’ll be interesting to see what the Goog brain drain is like in the upcoming year.

Lots of colours in a Goog office: a yellow wall, red exercise balls, etc.  A massage room.  It’s been a while since I’ve seen a bunch of filing cabinets in the hallway..  Kind of cramped offices.  I’m happy with having an office all to myself with my Aeron chair.  🙂

It was fun seeing some alumni: one guy who I worked with in our Security class (but I couldn’t remember his name) now works at Amazon; some TAs, etc.

Cool evening in the dragon’s den. 😉

Writing about Writer

[ Yes, this post is written with Windows Live Writer 1.0.109. ]

So everybody seems to be buzzing about the newly released Windows Live Writer Beta.  I had stumbled across the team’s Sharepoint site a month or so ago (don’t recall how; I do remember the message, "THIS IS INTERNAL ONLY!! DO NOT BLOG ABOUT WRITER!!" in big red font).  I perused through their FAQ and schedule and found it to be fairly interesting.  With all the hype surrounding AJAX-y websites, it’s easy to forget that desktop apps are still much more functional and responsive than the best of the web apps.  The only problem with desktop apps this day and age is that they don’t roam your data and settings.  So the ‘Post Draft to Weblog’ feature seems very handy, even if it is a manual process.

I wasn’t going to download it (like I need another little app that does some form of word processing).  Even when I was browsing their team site I was skeptical about the niche that they had set out to fill.  Blogging sites all have their own rich edit blogging control, all with different features and capabilities.  It’s not that I don’t see the need for an app like Writer, it’s just that, in a way, I feel there’s not much to do: text box + some font choices + interop with every blogging tool known.  Maybe throw in a little spell-check.  I don’t know of any app that does this, and maybe there’s a reason for that.  Anyway, I wasn’t going to download it.  Notepad works just fine for me.  *But* after seeing so many blogs I read mention it, I decided to give it a whirl.

Pros:
– I love how it downloads the layout/format for Windows Live Spaces.  ‘Web Preview’ is very cool.
– That’s got to be the least invasive prompt for a Passport Windows Live ID that I’ve ever seen.  Nice!  Smart move trying to get people to sign up with Windows Live Spaces who aren’t already hooked in.

Cons:
– wow, it’s already 4.75 MB.
– (minor bug) MSI wraps long lines while installing so that line appears hidden under the progress bar.
– it’s a little feature-less, but it is a beta after all (whatever that means).  At least it’s stable. 🙂  And responsive.  (Although that 68 MB it’s using on my machine is dwarfing Messenger–currently at 37 MB–and that’s quite an achievement.)
– need spell-check on the fly.  A spell-check button is so 90s.  I want my red squigglies!  (Sorry, Blogger.)  I guess the trade-off in not loading MS Word here is that although you don’t get all those fancy features, it’s not as slow/memory intensive as Word (not to mention the licensing issues).  (Speaking of spell-check, it has never ceased to amaze me how many spell-checkers incorporated into blogs will catch some form of the word ‘blog’ as a spelling mistake.)
– the blue-ish UI is ok at first, but I really could do without it.  I can’t believe I’m suggesting this, but I’m sure a lot of people would appreciate skins.
– the ‘Category’ picker isn’t exactly easily discoverable.  And I can’t seem to create a new one.  And if the category name’s too long it just cuts it off.
– It’s yet another app.  I need a one-stop-shop app.

Hmm, lots of ‘cons’.  I just like to nitpick.  It’s actually a pretty nifty little app.  And it hasn’t crashed on me yet!  I might keep it around for a bit.  I’ll try using it with Blogger next.

It Needed To Be Said

It Needed To Be Said, courtesy of Channel 9.

Pew on Blogging

Yet another Pew Internet Report, this one on bloggers! Bloggers: A portrait of the internet’s new storytellers. Actually I find these reports quite interesting. Even though some of their findings are sometimes well, duh! moments, it’s always neat to see numbers as evidence. This one is a sample of 233 bloggers. Of interest:

Most bloggers say they cover a lot of different topics, but when asked to choose one main topic, 37% of bloggers cite “my life and experiences” as a primary topic of their blog.

Naturally. Unless it’s your job, maintaining a blog devoted solely to a single topic/theme without going off on tangents is a lot of work.

The most distinguishing characteristic of bloggers is their youth. More than half (54%) of bloggers are under the age of 30. Like the internet population in general, however, bloggers are evenly divided between men and women, and more than half live in the suburbs. Another third live in urban areas and a scant 13% live in rural regions.

Diversity is nice.

• 55% of bloggers blog under a pseudonym, and 46% blog under their own name.
• 84% of bloggers describe their blog as either a “hobby” or just “something I do, but not something I spend a lot of time on.”
• 59% of bloggers spend just one or two hours per week tending their blog. One in ten bloggers spend ten or more hours per week on their blog.
• 52% of bloggers say they blog mostly for themselves, not for an audience. About one-third of bloggers (32%) say they blog mostly for their audience.

I pulled these from the summary section, but the rest of the report is worth skimming through as well. Oh, and page 22 has the list of popular blog tools. MSN Spaces is not on that list. Another surprising thing is:

RSS does not have a strong presence yet, even within the blogosphere. Only 18% of bloggers in our survey say they offered an RSS feed of their blog. Nearly 6 in 10 (59%) say they do not have an RSS feed for their blog content, and close to another quarter (23%) say they do not know if they had a feed, or did not answer the question. It is worth noting that bloggers are not behind the curve when it comes to this new technology.

Essentially the report says this: although a great deal of bloggers are heavy internet users, not all of them are. Got that?

Related:
Who Are All These Bloggers?, Slate
Pew Blogger Study, Guy Kawasaki

53,651 2.0 M6

Yes, I missed the "meme" about the 53,651 bubble. Like Scoble, I’ve been back in Toronto for the past week (and this coming one) for a family emergency, so I’ve been a little unplugged from my day-to-day life. What’s funny is that this Techcrunch "chasm" is so true. The whole idea behind Live.com (at least one reason I had heard) was to create something for the early-adopters to get excited about. "Get them and the rest will follow"–something we were trying to learn from Google. And while working on a part of Windows Live and constantly trying to catch up on the new stuff that’s coming out, you tend to lose perspective of the average Joe.

I’ve been trying to get my sister, an avid MSN Messenger user, to upgrade to Windows Live Messenger for the longest time. She would never do it (remember the "ugly orange"?). She would even avoid the Mail Beta (with "What’s wrong with Hotmail?"). Now here’s someone that’s fairly technologically comfortable: on MSN practically 24/7, with her Xanga and Facebook community (sorry, MySpace isn’t popular here yet); comfortable with installing programs (ie. has been virus-plagued in the past); uses Sony’s Imagestation to share pictures; uses DC++ to download stuff in university; has a GMail account; and yet with all that, will stay away from a new Microsoft-backed venture (unless I really force it onto her). Forget about all those Techcrunch’ed offerings out there. (Then again, I’ve never really been a beta user at heart either. I like my things to work. That means I don’t like it when my apps crash or take 110% of my CPU [ahem Msgr].)

Then there are my parents. I upgraded my Dad from classic Hotmail to Windows Live Mail this past week. The reason? He didn’t like how Hotmail didn’t display the year; so when he sorted by date, the mail ordering got screwed up. Such a simple reason! (Actually, he had that problem and I upgraded his account because I wanted to and it just so happened it’s fixed in the Mail beta.) I showed him the contacts and the calendar and how if he had it here he could get it anywhere there was an internet connection (he used to use Outlook with Exchange and then just plain Outlook when he changed jobs; essentially only to keep track of his e-mail and contacts–which he tried to export from Outlook 2003 and is now having trouble importing into Outlook 2002 because formats are incompatible… UGH! .. And.. I’m sidetracking again). Anyway, I think he likes it. (Hmm.. although I don’t recall he ever explicitly saying so.)

I’ve never even tried to explain Live.com to my parents; trying to get my sister excited about it was hard enough (pretty much a flop). So it’s interesting–how does one get the general populace to start using their software? How does one get past the initial 50K early adopters? My sis will do join the crowd once enough of her friends/network are on it (like Facebook, etc–get the kids jazzed about it), but for people that grew up with slide-rules and double-click on everything? Fat chance. You need to get the Geek Squad to the rescue! (omg, she’s got a badge on her belt 😐 )

Some randomness: I read in the paper this morning: High hopes for next-gen Net where they talk a little about "Web 2.0" and highlight a particular venture started out of Toronto: EndlessEurope.com. Basically what these guys are doing is taking reality tv and mixing it with the Internet. Having extreme disdain for reality TV, I don’t think I would ever watch, but they’re taking what’s been successful and mixing it with what the ‘net has to offer. Sounds like it has the potential to actually do well. (I used to read a lot of those "Choose your own adventure …" books as a kid. Visiting all those places in Europe sounds like it would be amazing.)

Anyhow, I am loving the new update (M6) to Windows Live Mail! The mouse-over with the top menu; the disappearance of the skyscraper ad.. awesome! Seems much faster to me too. Now if they could only get rid of the ad at the top 😉 I wonder how many users are on the Mail Beta (and percentage of the Hotmail user base)?

(Other things: clicking on the calendar takes you to Windows Live Calendar; click around a bit (I tried to remove a shared calendar from an MSN group) and got a weird calendar UI with no way of going back to Mail. Also, what’s with that bottom left box (where the Mail/Contacts/Calendar/Tasks used to be?) It doesn’t seem to be serving any function.)

Lists 2.0

Techcrunch has a post about online "To Do" list offerings: Do More: Online To Do Lists Compared.

I use lists as much as (or more so) than the next guy, but I’ve never used any of these new "Web 2.0" lists. In fact, I use a novelty item to record my lists: a pen and paper. Well, actually, that’s not entirely true. My transient lists are written down on paper (like weekend to-do lists and the like), but my more permanent lists, say a list of TV episodes I haven’t watched/downloaded (or have, as the case may be), are stored in a text file with notepad. It isn’t completely ideal for roaming (paper’s only great if you remember to take it with you). Sometimes for more "urgent" material that I’d like to take around with me, I’ll use Outlook and its ‘Notes’ or ‘Tasks’ features.

So I’m slightly amazed that companies can build a business around the idea of something as simple as a list. Now I haven’t looked at any of these offerings (although I signed up for Ta-da List on advice of Techcrunch. But this is something I could envision a ‘list’ company doing:

• easy to create/rename/remove/reorder lists/list items
• add RSS feeds (a requirement these days) [Ta-da list’s RSS feed is sorta strange.] Mobile aware?
• share your lists (some read ACLs or other); maybe even shared editing of lists
• integration with calendar ("must do by this date", etc)

(There’s probably other things I’m missing.) The craze to webify everything of late that Outlook/Office had already quite seamlessly integrated (e-mail, calendar, lists, word procesing) on a client-side app is a little puzzling. Which leads to the aging debate of whether or not AJAX-enabled apps will render client side apps irrelevant. Eh. Some other time.

Back to lists. So what are each of these list companies doing that distinguishes them from each other? Do they have some technology that others can’t (easily) replicate? If you examine the points above they boil down to the same tenents that are needed for any service these days:

• ease of use
• standards to export/import data
• sharing
• integration

These themes are manifesting themselves in every single new offering coming out (or at least, attempt to; not everyone gets it right). It’s very cookie-cutter.

Oh, and I just realized I forgot something major. You need a way of monetizing your user base. That’s sort of important. I guess you display ads–it seems to be the buzzword of late. (Someone’s really gotta find some other model–or maybe this is really what works best? Hell, one of my managers mentioned that they now even have free flights with advertising which apparently is working really well for them.) OK, I’m going off on a tangent again.

So which of these list apps are going to sizzle? Which are going to be bought out by a big company? (Would a big company really buy any of these–what do these little guys have that M/G/Y couldn’t whip-up in-house?  Disclaimer here: I didn’t really try out any of these offerings.) I remember there used to be a crappy to-do list gadget up on Live.com in the beginning. Someone should just fix it up and render all these small companies irrelevant 🙂

Let’s just say I’m not at all impressed with Ta-da list.  I do like how it’s fast.  But it doesn’t really do anything.  I guess that could be why it’s fast.  It’s so fast! 😐  Anyway, I need to go find a scrap piece of paper and a pen. Pen and paper are great for quickly scratching something down off the top of your head–the only flaw is that you need to find a pen that works. :p  Let me know when someone builds something really compelling.

Tech Forrest

Not too long ago, we had an analyst from Forrester, Charlene Li, come to campus and brief us on one of our major competitors. I’d never seen an analyst briefing before so it was quite interesting, to say the least. There were a couple insights in her talk too that I found gave me a fresh perspective on the competition. Here was a person whose sole purpose was to research tech companies and to try to figure out what they were doing and where they were heading, not to mention where the industry is heading as a whole. How cool is that?

Anyhow, there was one thing in particular she mentioned that got to me. She said that (and I am completely paraphrasing here–the usual disclaimer applies) she is always surprised how MS employees are to some extent clueless with what happens outside of the company, considering that we are at the forefront of the industry as a whole, being a major player and all. She asked how many people read Techcrunch (there weren’t many), and if we didn’t, well.. we should.

So I’ve been reading Techcrunch for three or four weeks now, and I am amazed by it (amazed may be too strong a word). Working for MS as an intern, and then coming straight here after college, you tend to think that MS is the be-all and end-all. You know of the existence of start-ups, but it’s just this vague notion of filthy developers coding into the night, high on caffeine and sugar–with no real impact. Techcrunch is constantly profiling these start-ups with cool new ideas. These companies either fade away from lack of funding, or get acquired by MS/Y!/G/eBay. I had no idea these companies really existed. I definitely recommend it if you’re in the tech industry.  Very cool.

RSS Hoopla

So about a year ago Start.com launched (v2). I loved the customization concept–in a web browser of all things! It was only then did I start playing with the whole RSS stuff. I had heard of RSS (and Atom feeds via Google/Blogger) but had only a vague notion of what they were. Indeed, it turned out that I wasn’t the only one.

Anyhow, in the past year, my list of RSS feeds that I read has (slowly) grown, encompassing friends’ Spaces, Xanga, Blogger feeds, news feeds, etc etc. And even with the launch of Live.com, I’ve stuck with Start.com (for some reason I just like it better). I loved how my Start.com page could roam.

Earlier this year, I realized that using Start.com just wasn’t going to scale for me. After adding 20-30 feeds, my Start.com page was beginning to look like a haphazard kludge of links. It was nice to be able to glance at the page to see if there was anything new I hadn’t read, but now I had to scroll through the entire page.. up/down/up/down. So I decided to give a client-side RSS reader a try. And what better client to try than Dare‘s baby RSS Bandit?

And it’s not half bad. It’s not too slow (even on my seven year old laptop) and accomplishes its job. (Feeds that include comments–very cool.) Things I don’t like include:

• The web browser in the app doesn’t understand the ‘back’ button on my mouse.
• The web browser in the app doesn’t understand javascript.
• I can’t turn off the dang pop-up bubble that says "6 feed(s) with 549 new message(s)".
• It’d be great to have a little X on the top right of each new feed so I can ‘clear’ them as I read them.
• The whole notion of only showing ‘unread’ feeds when clicking the feed name is a little strange. It took me a while to realize that I had read all the entries in a feed (which is why it was completely blank).
• I don’t like how it collapses all my folders when I restart it.
• I’d like to be able to reorder feed categories and folders (and creating a new one seems to put it on the top of the list.
• It took me forever to find the "Import" button, and by then I had already added each of my feeds by hand (but that’s probably just me). :S
• What’s up with the goofy little icon?  And I don’t get that bag jumping up and down when new feeds arrive at all.  (I guess it’s a bag of cash or something.)

But overall, I’m quite satisfied. I get my influx of things to read (although I can’t stand those sites that only publish a couple sentences so that you have to click through for the entire article). Even then, I’m convinced that there’s got to be a better way to organize all these feeds. I still find it very busy. I’m not sure how to describe what I want–but I’m quite sure this isn’t it.  How are my parents ever going to figure out RSS?  It’s still not usable for the common person.  (Then again, my parents figured out PSMs and winks and nudges and custom emoticons in Messenger, so you never know…)

In a semi-related note, Ray Ozzie (CTO of MS), announced at ETech today the concept of the Live Clipboard. It’s pretty cool. Should be interesting to see how quickly this is picked up.