I’ve been playing a lot with Comet lately. It started with Notify.io, in which I decided to prove that HTTP streaming was a simpler alternative to XMPP in getting messages to the desktop. That went quite well, but it was easy because it wasn’t all that different from a socket connection. Then I built a yet-to-be-announced site that uses real-time updates, and I was forced to deal with Comet to the browser. That’s a bit more complicated.

Actually, I’m arguably a veteran of Comet in the browser. I was doing it before it had a name, all the way back in 2005. A friend and I were using it (without knowing that it was terribly novel) to build a real-time strategy game in the browser called AjaxWar. I haven’t really done a lot with it since, but I was hoping after almost 5 years there would be all kinds of advances in libraries and tricks that would make it super easy.

That was not really the case.

There are things, but not easy things. The Bayeux protocol? I guess all that would be easy to do if there were a lightweight Javascript library for it. But there isn’t really. There’s a jQuery one, but it’s completely undocumented. Plus I was just sitting there thinking, do I need all this? Handshaking? Message envelopes?

I was also hoping for actual persistent connections (that’s what we did in AjaxWar), but it turns out the standard today is long-polling. This is a semi-persistent connection that drops after every message and then reconnects and waits for the next message. I also wanted JSONP and cross-domain support. So I ended up using the dynamic script tag technique:

<script type="text/javascript">
	$(document).ready(function(){ waitForMsg(); });

	function waitForMsg() {
		$('body').append('\<script type="text/javascript" src="http://mycometserver.com/channel?callback=gotMsg">\<\/script>');
	}

	function gotMsg(msg) {
		// Do something with it
		waitForMsg();
	}
</script>

It’s fairly elegant in its simplicity and cross-browser support. But it has some weird side-effects that I’m not sure if any of the fancier systems got around. For one, it keeps the browser loading. While it’s waiting for messages, the browser says that page is still loading. I also ran into some issues where if I included, say, a Google Calendar widget on the page, it might not decide to keep the connection open (or even start it). I ended up putting a delay on the first call to waitForMsg() until after the calendar widget was likely loaded.

So it’s a bit brittle. You don’t know if it stops working. Therefore you can’t do retries. And you never know if you happen to miss a message between connections (unlikely, but something to worry about). Plus I think if you hit Escape it also kills it.

But this worked well enough for my projects. I knew that if I found a better way, I’d switch over to it, but it was good enough.

…heh…

Then today I decided to solve the problem right once and for all with a project called CometCatchr.

CometCatchr a lightweight Flash component to be used by Javascript that gives you a persistent connection for Comet streams.

I know a very small number of people won’t agree with my approach using Flash, but I tend to be pragmatic. I trust Flash is generally available and the benefits completely outweigh everything else to me. It’s also not new. Even the Bayeux protocol includes Flash as supported connection type. Still, I couldn’t find any simple Flash component that gave me what I wanted.

CometCatchr gives me Javascript callbacks on messages, maintains a persistent connection across messages, retries on lost connections, works in all browsers, supports (participating) cross-domain message sources, and just freaking works.

It was a drop-in replacement to my previous technique that worked right out of the box:

<script type="text/javascript">
	function gotMsg(msg) {
		// Do something with it
	}
</script>
<embed type="application/x-shockwave-flash" width="0" height="0" src="CometCatchr.swf?url=http://mycometserver.com/channel&callback=gotMsg"></embed>

It simplified my code, not just client-side, but now I don’t have to support JSONP callbacks on the server. CometCatchr also parses the JSON messages before passing the callback, so that’s taken care of too. I realize it seems less than ideal to couple this with JSON payloads, but I didn’t say it wasn’t an opinionated component.

In fact, it’s very opinionated. It likes single-line JSON messages sent via HTTP using chunked transfer encoding. That’s because that’s how I do Comet streams. Actually, that’s how Twitter does them, too. I’m quite alright with such constraints, but if you want to make changes, it’s MIT licensed and super simple to hack on.

So for the time being, I’ve more or less solved simple Comet to the browser, particularly for me. It also might be worth knowing that another motivation for building this component is that I intend to solve Comet and real-time stuff in the browser entirely… but uhh, yeah. Stay tuned.

Update: I’ve basically stopped development and support of Yapper because Notify.io is the right way to solve this problem. However, this post describes that problem and is still worth the read. ;)

Yapper, a Jabber/XMPP interface for Growl

Today I released Yapper, a full featured Jabber/XMPP interface for Growl. It was based on the simple Twisted script I wrote while building my notification email to Growl piping. Although it’s limited to OS X users, I still think this is a big deal. Let me explain.

If you haven’t heard of Growl, it’s a global notification system that lets applications notify you of events in a consistent, customizable way. It’s been around for a while and is integrated or has plugins for lots of popular apps like iTunes, Adium, and Tweetie. It has bindings for a bunch of languages and also has a command line tool that you shell scripters can use to pipe notifications into. For example, I used this recently for my reminder system.

I think Growl is great particularly because it very nicely solves passive real-time notifications. That is, you want to be notified, but you don’t want it to interrupt you by requiring action. This is why I hate email notifications.

However, the major problem with Growl is that for the most part, it’s limited to local notifications. Sure, you can get notifications from “out there,” like Twitter or IRC or Gmail … but it requires you to have a local application that pushes them into Growl. One for each, in fact.

It turns out, I don’t actually care about most local notifications. Telling me a file is done downloading or what song is playing in iTunes is not terribly notification worthy. If I’m sitting there to get the notification, I probably already know. The most useful notifications are of events important to me that I’m nowhere near … events from things out on the Internet.

So Growl needs a network interface. Oh, wait! It has one! Well, what’s the problem? Two things: it’s a non-standard protocol, and it requires a direct connection. That not only raises the bar for things “out there” to notify you with Growl, but it literally makes it impossible if you’re constantly changing IPs, sitting behind a firewall or NAT, etc.

Let’s see… who’s solved this problem already? Right, IM! XMPP, the open standard for IM and real-time message passing (also known as Jabber), seems to have everything we need. It’s a popular protocol, fairly accessible from any language, and doesn’t require a direct point to point connection. So it’s a perfect transport for network Growl notifications!

Since I’m not one to wait around for people to implement what I think they should implement, I solved this problem by making Yapper. Yapper is a lightweight Jabber client made specifically to receive Growl notifications. It starts when your machine starts and sits in the background just waiting for whatever you have sent to it so it can pop it up with Growl.

You can now easily have Growl notifications sent to you from anywhere. Websites have less of a reason not to directly notify you with Growl for events that are important to you. More importantly for me, I can more easily set up Growl notifications for whatever I want. With the rise of webhooks, so can you. Anyway, to me this generally makes Growl much more useful.

Give Yapper a look. It’s on GitHub under MIT license. It’s the first release, but it should work fine. If not, drop off an issue and it’ll likely be taken care of.

The other day I realized I hate notification emails. In particular I’m talking about SVN commit notifications. I have them set up to skip my inbox and filter into a label on GMail, but this doesn’t really help. They don’t pollute my inbox, but I still have to go out of the way to read them. Or at the very least, mark them as read, which is still quite annoying!

Growl is supposed to help solve this problem. In fact, when I was retooling the notification system at work, I would have loved to integrate Growl … except that Growl’s network interface requires a direct connection.

Taking a step back, it’s not just work notifications. A lot of us get a lot of notification emails about a lot of things. Most of them are poorly designed, for example, without important details of the notice in the subject. Plus, as email they require disrupting action. Not just to open and read if you have to, but just to archive and/or mark as read.

It’s great to have an archive in email, but email is just not ideal for notifications. If I’m around to receive them, I should get the important information passively and not have to do anything else until I need to reference it later, if ever. That means Growl.

Getting Notification Emails to Growl

I thought to myself, “I should be able to rig this up. And it should be easy.” And if the infrastructure and plumbing I’ve wanted was in place, it would be easy. So I decided to lay it down and make it happen. Here’s what I needed:

  • An email to webhook bridge, like my old Mailhook service.
  • An XMPP to Growl bridge. We all want this anyway, right?
  • An HTTP to XMPP bridge. Why? You’ll see.
  • Glue code that can run in the cloud. This is what Scriptlets is for.

I’ve since turned off Mailhook, but I’ve been planning to roll its code into my still early Protocol Droid project. This is an HTTP to anything bridge written in Python with Twisted that makes heavy use of webhooks. I’d plan to roll the HTTP to XMPP bridge in there as well. For now, the XMPP to Growl bridge would be separate.

Having all that, here was the pipeline I envisioned:

  • A notification email would come into Gmail and get filtered
  • It would forward to an address that points to Protocol Droid
  • Protocol Droid would post the email to a script on Scriptlets
  • The script would use Protocol Droid again to send an XMPP message
  • The message would be sent to a JID for my laptop if online
  • The XMPP to Growl bridge on my laptop would receive and notify me

Building the Infrastructure

I decided to build all this last Friday night. I got caffeinated and went to work, starting with Protocol Droid. I hadn’t touched it since I was trying to build an HTTP to SMTP module for it. It wasn’t part of this project, but I decided to finish it off. You know, to get warmed up and have an early win.

That happened pretty quickly, so I was excited get started on the next task: an email to webhook bridge. This was a little different. So far everything in Protocol Droid was for outgoing connections. This would be the first listening module. The idea was that it would be an SMTP server that would parse incoming email and post it to a callback URL registered for a recipient domain. I’d written several of these before for mailhook.org, so I ported the code to Twisted for Protocol Droid. Another pretty easy win. I used PostBin to debug this, obviously.

At that point, I decided to take a break and watch an episode of Pushing Daisies. I still haven’t seen all of the second season!

Then came the hard part. Well, I thought it would be easy. The HTTP to XMPP bridge. It turns out the XMPP support in Twisted isn’t very well documented and slightly underdeveloped. I had to build a couple of prototypes to get the hang of it. Even then, building this module and the XMPP to Growl daemon took about 7 hours! Compared to the 3 hours spent doing the last two modules. XMPP is definitely a bit hard to really get into.

By 8am I had all the pieces working. I thought about going the last mile and getting it all online and set up the pipeline … but I decided I deserved some sleep. I’d do it when I wake up.

Getting the Pipeline in Place

When I woke up on Saturday, I was still burnt out from coding. I spent the day hanging out with friends. Then later I invited some more friends over for hacking at my place. There I spent more time than I wanted getting my code to run on my server. Apparently OpenSSL for Python wasn’t installed and the Google Talk servers required TLS. It took a while to figure this out since no error was raised and Google would just drop my connection.

I finally got all the infrastructure I built online and tested. Finally, I could work on the actual pipeline. It was supposed to be easy from that point on. And it was! Except for one bit …

I got almost all the pieces connected and it all came down to the glue code on Scriptlets. The script would parse the email however I like and set up the message that would end up on my laptop via Growl. It turns out Scriptlets is a bit of a pain to use. No edit, no debugging, crappy error messages. Since I built it as a proof of concept, I hadn’t done much development on it, let alone with it. Anyway, I learned a lot and have some things to fix. But! I was still able to get everything working!

I had my friend Adam send the first test email that would go through the entire pipeline I outlined above. He sent it from his phone, just to add a little to the magic. Low and behold, it worked perfectly, quite quickly. It took 4 seconds from him sending to the notice appearing … but about 3 of those seconds were the time it took for the phone to send the email. Nevertheless, brilliant!

Next Steps

I’ve since found out about Wokkel, which might simplify some of my XMPP code. I have some other enhancements to make to each piece to make them more general. And I mentioned some of the things I need to add to Scriptlets.

Functionally, the pipeline is pretty good. The only addition I can think of would require an HTTP to IMAP bridge in Protocol Droid (something I’ve already prototyped). I’d use this to mark the notification email as unread if it wasn’t able to deliver the IM. But for now, I think I’m done on this project. And you can all take advantage of the infrastructure I built out for rigging up your own cool hacks.

Most of the code this weekend went into these two projects, so check them out!