Yapper, a Jabber/XMPP interface for Growl

Update: I’ve basically stopped development and support of Yapper because Notify.io is the right way to solve this problem. However, this post describes that problem and is still worth the read. ;)

Today I released Yapper, a full featured Jabber/XMPP interface for Growl. It was based on the simple Twisted script I wrote while building my notification email to Growl piping. Although it’s limited to OS X users, I still think this is a big deal. Let me explain.

If you haven’t heard of Growl, it’s a global notification system that lets applications notify you of events in a consistent, customizable way. It’s been around for a while and is integrated or has plugins for lots of popular apps like iTunes, Adium, and Tweetie. It has bindings for a bunch of languages and also has a command line tool that you shell scripters can use to pipe notifications into. For example, I used this recently for my reminder system.

I think Growl is great particularly because it very nicely solves passive real-time notifications. That is, you want to be notified, but you don’t want it to interrupt you by requiring action. This is why I hate email notifications.

However, the major problem with Growl is that for the most part, it’s limited to local notifications. Sure, you can get notifications from “out there,” like Twitter or IRC or Gmail … but it requires you to have a local application that pushes them into Growl. One for each, in fact.

It turns out, I don’t actually care about most local notifications. Telling me a file is done downloading or what song is playing in iTunes is not terribly notification worthy. If I’m sitting there to get the notification, I probably already know. The most useful notifications are of events important to me that I’m nowhere near … events from things out on the Internet.

So Growl needs a network interface. Oh, wait! It has one! Well, what’s the problem? Two things: it’s a non-standard protocol, and it requires a direct connection. That not only raises the bar for things “out there” to notify you with Growl, but it literally makes it impossible if you’re constantly changing IPs, sitting behind a firewall or NAT, etc.

Let’s see… who’s solved this problem already? Right, IM! XMPP, the open standard for IM and real-time message passing (also known as Jabber), seems to have everything we need. It’s a popular protocol, fairly accessible from any language, and doesn’t require a direct point to point connection. So it’s a perfect transport for network Growl notifications!

Since I’m not one to wait around for people to implement what I think they should implement, I solved this problem by making Yapper. Yapper is a lightweight Jabber client made specifically to receive Growl notifications. It starts when your machine starts and sits in the background just waiting for whatever you have sent to it so it can pop it up with Growl.

You can now easily have Growl notifications sent to you from anywhere. Websites have less of a reason not to directly notify you with Growl for events that are important to you. More importantly for me, I can more easily set up Growl notifications for whatever I want. With the rise of webhooks, so can you. Anyway, to me this generally makes Growl much more useful.

Give Yapper a look. It’s on GitHub under MIT license. It’s the first release, but it should work fine. If not, drop off an issue and it’ll likely be taken care of.

Why minimalist software wins at workflow

Recently I’ve been evaluating software to help support agile/scrum development on our team, and ideally to roll into our NASA Code product for others to use. We’re already married to Trac, so we’ve been playing with Agilo and are looking at some of the other agile plugins for Trac. Unfortunately they’re all so heavyweight, despite some that claim not to be.

I came back to a realization I’m sure a lot of us have had: most software sucks. Especially software that’s intended to augment some real-life process. When asking Timothy Fitz (my insight into the amazing continuous improvement processes used at IMVU) about recommendations on agile tools, he said: “a board and post-its (seriously)”

This is part of the reason most enterprise software sucks so terribly. Enterprise is about lots of real-life process and workflow, and given that process augmentation software even in small doses generally sucks, large amounts of it will suck exponentially.

A lot of us have learned that less software is more effective. One major attraction of Trac was their goal of staying out of the way through minimalism. The trick with minimalism, in general, is knowing what’s actually important—the essence of the message or design. This is a big part of my design process. Asking, “How can I fold these requirements into fewer features and UI?” instead of directly implementing a feature for every requirement.

The other thing about minimalism is that, like abstraction (another form of compression), everything you leave in the design makes such a huge difference. In programming, when you make abstractions, you’re deciding what you can assume. This means abstractions can go in different directions depending on the assumption requirements of what’s going to use the abstraction. The risk with minimalist software is that a simple design choice can drastically change the direction of the abstraction and make or break whether the software fits your needs.

Luckily, minimalism buys you a sort of abstraction that can enable projection. By this I mean that users can project their actual process and workflow onto the software. If it doesn’t have features that impose a particular process, users are free to do what works for them. This is why wikis are so powerful.

Coming back to Timothy’s “a board and post-its” remark, why do you even need software? If you can do it without software, why would you want to bring software in to slow things down?

Software does have a couple strengths. First, it encodes process in way that means you can automate parts of it. Nobody has to worry about manually typesetting when using a word processor. Second, it persists and organizes information that would normally be lost in handwritten notes, or worse, somebody’s head. The trick is getting these advantages without getting in the way.

A naive approach to software design is thinking that perfectly modeling a system, such as your development process, is the way to good software. I used to think this. It sounds great because then you can programmatically reason about every aspect of the system. But in the real-world, no two systems are exactly alike. In fact, a given system can change quite a bit in its lifetime. So there’s really no point to modeling systems with that kind of precision.

However, I’m seeing a lot of this in agile/scrum software. Requirements have stories, stories have tasks, organized into iterations and releases. CRUD for every noun mentioned in scrum. This on top of abstractions in a direction different than we need. Numbers where it doesn’t really matter. Nice pie chart breakdowns we’ll rarely use. Top it off with horrible UI, since with all these features there isn’t time to make them easy to use.

Honestly, Pivotal Tracker seems to have the best abstraction of agile. It folds requirements, stories and tasks into just stories. It automatically and dynamically creates iterations and calculates velocity. It keeps you on a single page, keeping you focused on what’s important.

Unfortunately, we can’t use Pivotal Tracker since we’d need it on our servers and the licensing they offered doesn’t scale if we want to essentially give it away as part of NASA Code. It’s likely I’ll want to just start nudging Trac in the right direction using Pivotal Tracker as a model reference, pulling together code from Agilo and other plugins. If there’s one thing that complements minimalist design, it’s an extension architecture, and Trac has an excellent plugin system.

Anyway, as far as augmenting process and workflow, I’ve always liked the idea of starting with a wiki and lazily formalizing the process into custom software as needed. As long as you can keep it under control, mind your requirement abstractions, and avoid writing too much software.

Fun with Internet Plumbing

The other day I realized I hate notification emails. In particular I’m talking about SVN commit notifications. I have them set up to skip my inbox and filter into a label on GMail, but this doesn’t really help. They don’t pollute my inbox, but I still have to go out of the way to read them. Or at the very least, mark them as read, which is still quite annoying!

Growl is supposed to help solve this problem. In fact, when I was retooling the notification system at work, I would have loved to integrate Growl … except that Growl’s network interface requires a direct connection.

Taking a step back, it’s not just work notifications. A lot of us get a lot of notification emails about a lot of things. Most of them are poorly designed, for example, without important details of the notice in the subject. Plus, as email they require disrupting action. Not just to open and read if you have to, but just to archive and/or mark as read.

It’s great to have an archive in email, but email is just not ideal for notifications. If I’m around to receive them, I should get the important information passively and not have to do anything else until I need to reference it later, if ever. That means Growl.

Getting Notification Emails to Growl

I thought to myself, “I should be able to rig this up. And it should be easy.” And if the infrastructure and plumbing I’ve wanted was in place, it would be easy. So I decided to lay it down and make it happen. Here’s what I needed:

• An email to webhook bridge, like my old Mailhook service.
• An XMPP to Growl bridge. We all want this anyway, right?
• An HTTP to XMPP bridge. Why? You’ll see.
• Glue code that can run in the cloud. This is what Scriptlets is for.

I’ve since turned off Mailhook, but I’ve been planning to roll its code into my still early Protocol Droid project. This is an HTTP to anything bridge written in Python with Twisted that makes heavy use of webhooks. I’d plan to roll the HTTP to XMPP bridge in there as well. For now, the XMPP to Growl bridge would be separate.

Having all that, here was the pipeline I envisioned:

• A notification email would come into Gmail and get filtered
• It would forward to an address that points to Protocol Droid
• Protocol Droid would post the email to a script on Scriptlets
• The script would use Protocol Droid again to send an XMPP message
• The message would be sent to a JID for my laptop if online
• The XMPP to Growl bridge on my laptop would receive and notify me

Building the Infrastructure

I decided to build all this last Friday night. I got caffeinated and went to work, starting with Protocol Droid. I hadn’t touched it since I was trying to build an HTTP to SMTP module for it. It wasn’t part of this project, but I decided to finish it off. You know, to get warmed up and have an early win.

That happened pretty quickly, so I was excited get started on the next task: an email to webhook bridge. This was a little different. So far everything in Protocol Droid was for outgoing connections. This would be the first listening module. The idea was that it would be an SMTP server that would parse incoming email and post it to a callback URL registered for a recipient domain. I’d written several of these before for mailhook.org, so I ported the code to Twisted for Protocol Droid. Another pretty easy win. I used PostBin to debug this, obviously.

At that point, I decided to take a break and watch an episode of Pushing Daisies. I still haven’t seen all of the second season!

Then came the hard part. Well, I thought it would be easy. The HTTP to XMPP bridge. It turns out the XMPP support in Twisted isn’t very well documented and slightly underdeveloped. I had to build a couple of prototypes to get the hang of it. Even then, building this module and the XMPP to Growl daemon took about 7 hours! Compared to the 3 hours spent doing the last two modules. XMPP is definitely a bit hard to really get into.

By 8am I had all the pieces working. I thought about going the last mile and getting it all online and set up the pipeline … but I decided I deserved some sleep. I’d do it when I wake up.

Getting the Pipeline in Place

When I woke up on Saturday, I was still burnt out from coding. I spent the day hanging out with friends. Then later I invited some more friends over for hacking at my place. There I spent more time than I wanted getting my code to run on my server. Apparently OpenSSL for Python wasn’t installed and the Google Talk servers required TLS. It took a while to figure this out since no error was raised and Google would just drop my connection.

I finally got all the infrastructure I built online and tested. Finally, I could work on the actual pipeline. It was supposed to be easy from that point on. And it was! Except for one bit …

I got almost all the pieces connected and it all came down to the glue code on Scriptlets. The script would parse the email however I like and set up the message that would end up on my laptop via Growl. It turns out Scriptlets is a bit of a pain to use. No edit, no debugging, crappy error messages. Since I built it as a proof of concept, I hadn’t done much development on it, let alone with it. Anyway, I learned a lot and have some things to fix. But! I was still able to get everything working!

I had my friend Adam send the first test email that would go through the entire pipeline I outlined above. He sent it from his phone, just to add a little to the magic. Low and behold, it worked perfectly, quite quickly. It took 4 seconds from him sending to the notice appearing … but about 3 of those seconds were the time it took for the phone to send the email. Nevertheless, brilliant!

Next Steps

I’ve since found out about Wokkel, which might simplify some of my XMPP code. I have some other enhancements to make to each piece to make them more general. And I mentioned some of the things I need to add to Scriptlets.

Functionally, the pipeline is pretty good. The only addition I can think of would require an HTTP to IMAP bridge in Protocol Droid (something I’ve already prototyped). I’d use this to mark the notification email as unread if it wasn’t able to deliver the IM. But for now, I think I’m done on this project. And you can all take advantage of the infrastructure I built out for rigging up your own cool hacks.

Most of the code this weekend went into these two projects, so check them out!

• Protocol Droid
• XMPP to Growl daemon

Oh no! Hackers!

That’s right. Hide your floppies and cover your Ethernet ports. Virus-laden hackers are coming to take over your computer, steal your passwords, and do terrible things! Like hack your MySpace! Oh noes!

I guess I’m getting over the fact that we probably won’t be able to undo the damage done with the public’s perception of what a “hacker” is. Perhaps, though, we can overload it to the point of ambiguity, so there’s at least some question of context whether it’s the good kind or the bad kind.

The problem is that positive connotations aren’t enough. The people that would venture to understand “hackers” the slightest bit more than what they hear in the headlines are going to pretty quickly find the “good hackers” … they’re white hats, right?

The other day I was questioned by somebody (that should know better) if I hacked somebody’s website that was recently compromised. Seriously? My response was “I barely have enough time to do important things, let alone something that would be a waste of my time.”

I self-identify as a hacker. I invite my friends over to “hack.” I started a party for “hackers and thinkers” (a very intentional choice of words). I’m co-founding a community center called Hacker Dojo. What are we doing at all these functions? Building and learning.

We use the term perhaps too liberally, but always implying tribute to the true hackers that, as Steven Levy put it, “regard computing as the most important thing in the world.”

These people push the envelope of what’s possible through hands-on exploration, driven by relentless curiosity and a desire to challenge the status quo. Steve Wozniak, Lee Felsenstein, Linus Torvalds, Tim Berners-Lee, John Carmack …

Hell, there’s something big behind this idea, why stop at computing? Buckminster Fuller, Nikola Tesla, Richard Feynman, Alfred Kinsey, Ben Franklin …

Perhaps we’re generalizing too far. Perhaps we’re rendering “hacker” meaningless. Or are we giving it more meaning? Getting down to its essence. I wouldn’t be defending this idea so strongly if I didn’t think it had some great significance to humanity.

What upsets me is that many who would identify as hackers in this sense seem to be afraid to claim it. Most likely in fear of confusing the layman that has the media’s myopic view of hackers. You’ve never heard of the canonical conference for real hackers. No, it’s not DefCon. (Just get up and leave.) It’s a conference called Hackers. You’ve never heard of it because they keep it secret!

This conference was started to gather everybody together that was mentioned in Steven Levy’s book Hackers: Heroes of the Computer Revolution (one of the last good publication on hackers, and it came out in 1984!). This conference holds all the values of true hackerism and it’s been happening for 25 years. But they won’t promote it! It goes by a fake name and even has all mentions of “hacker” on their website replaced with an image so it won’t be indexed. Seriously??

Trying to supplant public perception of hackers by just saying they’re something different and providing a better name nobody uses (“crackers”) is not going to work. It hasn’t worked. They need something to replace those visions of crackers with. We need tangible examples and stories. We need heroes. Heroes willing to wear the title.

Luckily, we have a new generation of hackers. One that has started a global movement called hackerspaces, probably one of the biggest things for hackers in years. Our local hackerscene fostered by Silicon Valley culture and events like SuperHappyDevHouse have led to a hackerspace we hope will have a big impact. One that proudly wears the name hacker: Hacker Dojo.

Twitter Hooks

When I first gave a public talk on webhooks in 2007, the guys at Twitter were the first ones to come up to me saying they wanted to implement them. Unfortunately it never happened, and this was well before they were as popular as they are now. It wouldn’t scale, they said.

As an alternative, just a few months ago, they released a stream API using long-running HTTP. It not only included a firehose stream, but a stream that allowed you to pull updates from selected users. I knew immediately from this I could build a webhook transformer, effectively giving Twitter webhooks. Unfortunately, the stream for selected users was behind a EULA that prevented that use of their API. Something about not redistributing data.

Then a few weeks ago, they added a selective stream that was publicly available. I had already designed the system and started writing some code before when I found out about the EULA. It took me a while to find some time, but last night and today I did it. I finished a proof of concept service called Twitter Hooks.

It’s very simple and is mostly about the plumbing behind it. You sign in with your Twitter account using OAuth, then you just give it a callback URL. Behind the scenes, it has a stream consumer listening for updates from the users that have a callback URL. When it gets an update from you, it posts it to your callback URL.

Obviously, there is only so much you can do with your own tweets. The next step is to listen to updates from your friends and post them to your callback URL. After that, using some trickery not involving the stream API, it will post direct messages and new followers to your URL. Throw in keyword tracking, and you have hooks for almost every significant event in Twitter. John Herren would be proud.

The problem is… Twitter is still too limiting with their stream API. As it stands, the non-EULA stream only allows following 200 users. That means Twitter Hooks can only support 200 users. It also means there’s no way we can roll out the friend update hook because some people have over 200 friends by themselves. Until that’s resolved, Twitter Hooks will remain a proof of concept demo.

But I have to say, having this is pretty sweet. The stream API is about as real-time as you can get. While experimenting, I was getting updates from the stream quicker than the page reload from posting a new tweet. This is, of course, then posted to your URL immediately for you to do whatever you want with. Here’s a tweet of mine posted to PostBin:

If you want to play with Twitter Hooks, it’s currently limited to users I authorize since it can only have 200 users and it’s terribly early/rough. Just @ me on Twitter and I can get you set up.

Reminders with Quicksilver and Growl

For my todo system, I’ve been using Quicksilver with Remember the Milk for a while. It’s great for being able to throw things into my todo list at anytime (as long as I’m at my laptop). Especially with its natural language date parsing: “buy some cheerios tomorrow” or “see brothers bloom tuesday.” However, it’s not a very good reminder system.

Occasionally I’d use it with a specific time (“dentist appointment tomorrow 9am”), but for the most part, Remember the Milk todos are not about micro time management within the day. Especially since they don’t have a good notification method. So for reminders, I had to build my own system. Here’s what I wanted:

• Simple text scheduling from Quicksilver
• Growl for the reminder
• Natural scheduling (“go home at 7pm”, “eat lunch in 5m”)

Since most Unix systems (including OS X) have the `at` command for scheduling commands with a decent syntax, and Growl has the `growlnotify` command, I decided to start at the command line with a shell script.

#!/bin/bash
growlnotify=/usr/local/bin/growlnotify
normal=$(echo $1 | sed -e 's/ in / at /')
what=${normal%% at*}
when=${normal##*at }
when=$(echo $when | sed -e 's/^\([0-9]*\)m$/now + \1 minutes/')
when=$(echo $when | sed -e 's/^\([0-9]*\)h$/now + \1 hours/')
result=$(echo "$growlnotify -s -m '$what' > /dev/null 2>&1" | at $when 2>&1)
if [ $? -eq 0 ]; then
    result=${result##*at }
    result="Scheduled for $result"
fi
$growlnotify -m "$result"

This script takes a single argument string in the format I described above, splitting $what and $when with either “at” or “in”. Then I expand relative time (“10m” or “2h”) into the format `at` wants (“now + 10 minutes” — which I didn’t want to have to type). It schedules the command to use `growlnotify` with $what (as a sticky notification, so it won’t go away until I click it), and then immediately tells you when it was scheduled for if successful, or it tries to show you the error if not.

So that works fine. Now Quicksilver. You can write new actions in AppleScript, but I didn’t want to try and figure out how to write the above in AppleScript. It was hard enough figuring it out in bash. So the AppleScript action is just going to call out to that shell script.

using terms from application "Quicksilver"
	on process text str
		do shell script "/Users/Jeff/.scripts/growlat.sh \"" & str & "\" > /dev/null 2>&1"
		set selection of application "Quicksilver" to str
	end process text
end using terms from

Apparently AppleScript files are not plain text, so you have to write it using Script Editor. Make sure the path to growlat.sh points to the shell script above. Then save this as Remind.scpt in your user’s “~/Library/Application Support/Quicksilver/Actions” directory. If it doesn’t exist, just make the directory. Restart Quicksilver and you should now have a Remind action! It’ll look like this: