Twitter as a Stockbroker. With Webhooks.

Webhooks. User-defined callbacks on the web. Yes. YES! Wait, what does that even mean? That stupid website doesn’t seem to explain them at all!

Let’s explore this through an analogy. This analogy uses stockbrokers! Yay! (?)

Consider, before the web, before email, how people would interact with their stockbroker: the telephone. Whenever you wanted to manage your investments and find out what’s really going on in the market, you would call your stockbroker. It’s similar to how you might use your browser to visit Twitter to update your status and find out what’s going on with your friends, right? This is the basis of the analogy. Bear with me.

Imagine in this story your phone is a browser, a phone call is a web request, and the broker is a web application, like Twitter.

It’s 1989 and you like to think of yourself as a big time stock trader with an impressive portfolio. Good for you. You got to this point by staying in close touch with your stockbroker. You’d call her (!) to find out what was happening in the market, discuss whether buying certain stock was a good idea, and maybe put in an order. You were able to get information out of the system and put orders into the system by picking up your phone and calling your broker.

The problem is that you can only react to events in the market as quickly as you get the information. This means you’d have to call your broker quite a bit to stay on top of a fast moving, highly volatile market. A programmer might try to automate this with a script. You take a similar approach: you hire an assistant to deal with your stockbroker. Perhaps this is unheard of in reality, but stay with me.

Now an assistant is not unlike a computer program in theory. You can give them instructions and based on certain conditions and input, they’ll do as you say. For example, maybe you instruct your assistant to buy a certain type of stock after it behaves a certain way, and if there’s not enough money to buy the amount that you told them, they can transfer the money from your savings account after a quick confirmation phone call with you. In today’s world, that sort of thing is not terribly out of the question to automate with a script using the web APIs and programmatic infrastructure available to us now.

Again, in order to react to events in the market in a timely manner, it requires lots of phone calls to your stockbroker. Sure, your assistant can handle it, but it’s a lot of work and wastes a lot of time. In fact, it wastes you money because you pay for your assistant by the hour! If you were a programmer, your script would have to constantly poll the broker API requiring a touchy cron setup or a long running process that gets more inefficient the closer to real-time you want it to be. Not to mention it’s just more work than you should have to deal with. If only there was some way for your assistant or script to be notified when things happen so they could simply react.

One day, your stockbroker says they’re providing a new service they describe as callbacks. You give them a phone number, and they’ll call it whenever something interesting happens regarding stocks relevant to you. Wow! Well, you could give them your number, but there are two problems. First, it’s 1989 and you don’t happen to have a cell phone. You can’t always be reached. Second, even if you could, you’d still want your assistant to handle the events because they have been instructed with how to deal with them and will do a lot of the legwork for you. Luckily, your assistant has a cell phone and is always available, so you give the broker their number.

Now your assistant is working very efficiently making you lots of money. Their job is so much easier because they don’t have to do anything until they get a call from either you or, more importantly, the broker. They can finally react to events as they happen, without a lot of nonsense trying to stay on top of things. What’s more is they will automatically take care of whatever situations you’ve told them.

What a cool feature, this callbacks thing. Thanks, broker!

If you haven’t guessed, the callback feature in this story is the exact mechanics of webhooks. Webhooks would allow you to tell the web apps that you use to “callback” scripts you have online. These scripts will deal with whatever events that web app produces. These callbacks use the same protocol you use to talk to them and that you both know how to use: web requests. Putting your script online at a URL is analogous to your assistant that has a cell phone: it’s always available for direct connection at a persistent “phone number,” unlike you, with dynamic IPs and NATs and turning off your computer. And just like an assistant, the script can do a lot of interesting things for you that you wouldn’t necessarily want to do, most of which is a sort glue work of making different systems work together given some logic. For example, transferring money from your savings account to the broker account when there isn’t enough.

A concrete example of a story made possible from webhooks that might be a useful scenario for many of you involves Twitter. Let’s say Twitter supported webhook callbacks for when somebody follows you. Right now you get an email, and from there you can decide what to do manually: follow them back, block them, or do nothing. I used to go out of my way to block users that I knew were spam bots, but now there’s so many it’s not worth the time. And of course I also generally follow back people that I actually know. If Twitter would simply call a script of mine whenever somebody followed me passing along the user ID, I could very easily run this logic in a PHP script or a simple App Engine app. Or perhaps I’d use Scriptlets (ahem, which was made exactly for these kinds of web scripts). It would work like this:

First, use the Twitter API to look up the user from the ID, and grab their name. Then use the Facebook API to check if that name shows up in my list of friends on Facebook. If so, use the Twitter API to follow them back. Otherwise, if they’re following over 1000 users and that number is more than twice the number that’s following them (which is roughly the heuristic I use manually), use the Twitter API to block them. All automatic.

If you could do this, not only would you have glued together Facebook and Twitter in an interesting and useful way, but you’ve sort of extended the net functionality Twitter provides you. You could imagine perhaps someday Twitter releasing features that would do exactly what you just did. But they won’t. In fact, they’ll never release features that are so specific to you (who says my spam algorithm is the algorithm everybody should use?). This is how webhooks make the web more about you. You can start extending web applications and gluing them together just the way you want.

This is a win-win for web application users and developers. Users get more functionality. Developers can implement less.

Coming back to the stockbroker analogy, there is a type of order called a limit order where you say to buy or sell when it’s at a certain price. With webhooks (or the broker “callback” service), this is merely a convenience because you could easily set that up outside of their system. Speaking of that example, when I mentioned your assistant transferring money between accounts, that reminds me, you know that feature banks have called overdraft protection? If banks had webhooks, all banks could have overdraft protection. Wow, right? That would have been nice for me when I had Washington Mutual, a bank known for its lack of overdraft protection.

What else could you do in a world with webhooks? Basically everything would have a common event-driven infrastructure, allowing you, with just a little bit of scripting glue, to accomplish so much more and make the systems you use better, and more personal.

Prepare for posts; I’m speaking at SXSW

Last week I got an email that said my SXSW speaking proposal was accepted. Strangely, my joy was coupled with a bit of disappointed. After the PanelPicker closed, I felt like I didn’t market my talk well enough. That combined with my OSCON proposal being denied, I felt there was no chance I was going to speak at SXSW. However, the more I thought about it, the more I liked the sound of it. I’ve had such a busy year going to conferences, speaking, writing, and building stuff. You all realize this webhooks stuff doesn’t pay, right? I was actually looking forward to not having a huge stressful talk to worry about.

But now I do! Lucky for you, I’ve decided to make it a really great talk. I’ve started work on it immediately and will be developing it probably up until March.

What is the talk? It’s not another “WebHooks are the Future” talk. Well it is, but in disguise. I’ve decided to focus on context. Like I mentioned before, all that vision stuff usually at the end of my talks will come front and center. The talk is called: How WebHooks Will Make Us All Programmers.

In order to describe this, I’ll need to first explain webhooks to those that are unfamiliar. I feel this is getting easier and easier as I gain experience and more tangible examples pop up, like PubSubHubbub and TweetHook. However, I’ll have to take it up a notch further because I won’t have that much time to talk about them. Before I can explain how webhooks will make us all programmers, I also have to share why this is worth doing at all. I have to explain why we should all be programmers.

Yay! It’s a sort of philosophy of computer science. I mean, that’s the stuff I really love talking about, so why don’t I go all out? Well I will.

But I have a lot of work in front of me. I need to finish building more examples. I’ve finally gotten to a point and built enough infrastructure where I can really start showing the power of webhooks through examples. I need to develop a super concise explanation of webhooks, probably something visual and animated — a mini presentation within a presentation. I also need to start practicing speech again. The last time I went without practice it was a disaster, despite getting my point across.

Part of my preparation will involve lots of writing and sharing on this blog. I need to prototype and validate individual ideas before I bring them all together in an epic talk. That’s the only way to make it as awesome as it should be. It also means there will be lots of follow up material for attendees.

So at the very least, my goal is to post something relating to the ideas in this talk once a month. And your feedback will be much appreciated.

Getting webhook callbacks when people click-through links

So I recently put this blog’s feed through FeedBurner. If I recall, I think I wanted to get subscriber statistics, as well as to try out their integration of PubSubHubbub publishing. But since I didn’t think I would get new subscribers often and I’m too lazy to check stats, I really wanted a way to be notified when people subscribed. Since the means of subscribing would be a link, it hit me: click-through webhooks.

I wanted a link wrapper that would trigger a URL callback (aka a webhook) when people clicked through the link. Similar to a URL shortener, you’d just give it a URL and it would give you another URL, but instead of being shorter, it would be tied to a callback that would run as it redirected the user to the original URL. What could you do with this, you ask? It’s a webhook; you can do anything with this event.

Coming back to my original use case, I’d probably tie it to a script that sent me an IM that somebody subscribed. Or better yet, with something like Yapper, I could get a Growl notification. Sweet, right?

I knew it would be trivial to build, and even though it was a feature that I really wanted, it was still just a feature. I thought, it really doesn’t need its own app, does it? So I went to my URL shortener of choice, tr.im, and suggested it as a feature. I also happened to mention to all my friends they should vote it up. Well, it ended up being the 6th most requested feature because of that. I thought for sure they’d implement it. I mean it was so much easier than the others, and I gave them code to help do it and everything.

A couple weeks went by and nothing happened. I got no response. I emailed them and got nothing back. They didn’t even mark it as “under review.” It may have had to do with the fact they were then deciding to shut down … which I hear they’ve changed their mind about. Nevertheless, it still hadn’t happened. And I really wanted to be able to do this notification on subscription thing!

Along came my WebHooks and PubSubHubbub meetup where I wanted to demo this bit of plumbing. Of course, I had no way to do it because nobody had implemented it. So I figured I’d just build it that day before the meetup. Twenty minutes later, I had it live: ClickHooks.

Okay, so that part’s done. How do I get it to IM me? Well, Scriptlets was made to write the glue code to use with these kinds of webhooks. And my project Protocol Droid was made to make it easy to use other protocols from HTTP. So I just threw this little baby on Scriptlets and I was done:

import urllib

payload = {
 'username': 'demo@progrium.com',
 'password': 'secret',
 'to': 'progrium@gmail.com',
 'body': "You got a new subscriber on blogrium!"
}

fetch("http://pdroid.progrium.com/xmpp:talk.google.com/send", urllib.urlencode(payload), "POST")

Yes, that pdroid endpoint is live. No, I don’t recommend using it for production. Stand up your own Protocol Droid gateway. ;)

Anyway, simple enough to understand though, right? It should be. And it was simple to get set up. That’s the whole point of a world of webhooks: you can easily do so much cool stuff if this simple infrastructure is there.

As an aside, later I realized I could generalize my IM scriptlet to use a GET param for the body of the message, allowing me to make a personal messaging micro-webservice: http://www.scriptlets.org/abcdef?body=Hello, world! The data in this URL could then be hidden behind a URL shortener, giving me a regular looking URL that I could then give to people that would IM me the message when they clicked it. Kind of silly, but I almost set up a link that would IM me “At the door!” that people could click from their phone browsers as a doorbell for the WebHooks meetup.

Ah, fun with infrastructure.

Fun with Internet Plumbing

The other day I realized I hate notification emails. In particular I’m talking about SVN commit notifications. I have them set up to skip my inbox and filter into a label on GMail, but this doesn’t really help. They don’t pollute my inbox, but I still have to go out of the way to read them. Or at the very least, mark them as read, which is still quite annoying!

Growl is supposed to help solve this problem. In fact, when I was retooling the notification system at work, I would have loved to integrate Growl … except that Growl’s network interface requires a direct connection.

Taking a step back, it’s not just work notifications. A lot of us get a lot of notification emails about a lot of things. Most of them are poorly designed, for example, without important details of the notice in the subject. Plus, as email they require disrupting action. Not just to open and read if you have to, but just to archive and/or mark as read.

It’s great to have an archive in email, but email is just not ideal for notifications. If I’m around to receive them, I should get the important information passively and not have to do anything else until I need to reference it later, if ever. That means Growl.

Getting Notification Emails to Growl

I thought to myself, “I should be able to rig this up. And it should be easy.” And if the infrastructure and plumbing I’ve wanted was in place, it would be easy. So I decided to lay it down and make it happen. Here’s what I needed:

• An email to webhook bridge, like my old Mailhook service.
• An XMPP to Growl bridge. We all want this anyway, right?
• An HTTP to XMPP bridge. Why? You’ll see.
• Glue code that can run in the cloud. This is what Scriptlets is for.

I’ve since turned off Mailhook, but I’ve been planning to roll its code into my still early Protocol Droid project. This is an HTTP to anything bridge written in Python with Twisted that makes heavy use of webhooks. I’d plan to roll the HTTP to XMPP bridge in there as well. For now, the XMPP to Growl bridge would be separate.

Having all that, here was the pipeline I envisioned:

• A notification email would come into Gmail and get filtered
• It would forward to an address that points to Protocol Droid
• Protocol Droid would post the email to a script on Scriptlets
• The script would use Protocol Droid again to send an XMPP message
• The message would be sent to a JID for my laptop if online
• The XMPP to Growl bridge on my laptop would receive and notify me

Building the Infrastructure

I decided to build all this last Friday night. I got caffeinated and went to work, starting with Protocol Droid. I hadn’t touched it since I was trying to build an HTTP to SMTP module for it. It wasn’t part of this project, but I decided to finish it off. You know, to get warmed up and have an early win.

That happened pretty quickly, so I was excited get started on the next task: an email to webhook bridge. This was a little different. So far everything in Protocol Droid was for outgoing connections. This would be the first listening module. The idea was that it would be an SMTP server that would parse incoming email and post it to a callback URL registered for a recipient domain. I’d written several of these before for mailhook.org, so I ported the code to Twisted for Protocol Droid. Another pretty easy win. I used PostBin to debug this, obviously.

At that point, I decided to take a break and watch an episode of Pushing Daisies. I still haven’t seen all of the second season!

Then came the hard part. Well, I thought it would be easy. The HTTP to XMPP bridge. It turns out the XMPP support in Twisted isn’t very well documented and slightly underdeveloped. I had to build a couple of prototypes to get the hang of it. Even then, building this module and the XMPP to Growl daemon took about 7 hours! Compared to the 3 hours spent doing the last two modules. XMPP is definitely a bit hard to really get into.

By 8am I had all the pieces working. I thought about going the last mile and getting it all online and set up the pipeline … but I decided I deserved some sleep. I’d do it when I wake up.

Getting the Pipeline in Place

When I woke up on Saturday, I was still burnt out from coding. I spent the day hanging out with friends. Then later I invited some more friends over for hacking at my place. There I spent more time than I wanted getting my code to run on my server. Apparently OpenSSL for Python wasn’t installed and the Google Talk servers required TLS. It took a while to figure this out since no error was raised and Google would just drop my connection.

I finally got all the infrastructure I built online and tested. Finally, I could work on the actual pipeline. It was supposed to be easy from that point on. And it was! Except for one bit …

I got almost all the pieces connected and it all came down to the glue code on Scriptlets. The script would parse the email however I like and set up the message that would end up on my laptop via Growl. It turns out Scriptlets is a bit of a pain to use. No edit, no debugging, crappy error messages. Since I built it as a proof of concept, I hadn’t done much development on it, let alone with it. Anyway, I learned a lot and have some things to fix. But! I was still able to get everything working!

I had my friend Adam send the first test email that would go through the entire pipeline I outlined above. He sent it from his phone, just to add a little to the magic. Low and behold, it worked perfectly, quite quickly. It took 4 seconds from him sending to the notice appearing … but about 3 of those seconds were the time it took for the phone to send the email. Nevertheless, brilliant!

Next Steps

I’ve since found out about Wokkel, which might simplify some of my XMPP code. I have some other enhancements to make to each piece to make them more general. And I mentioned some of the things I need to add to Scriptlets.

Functionally, the pipeline is pretty good. The only addition I can think of would require an HTTP to IMAP bridge in Protocol Droid (something I’ve already prototyped). I’d use this to mark the notification email as unread if it wasn’t able to deliver the IM. But for now, I think I’m done on this project. And you can all take advantage of the infrastructure I built out for rigging up your own cool hacks.

Most of the code this weekend went into these two projects, so check them out!

• Protocol Droid
• XMPP to Growl daemon

Twitter Hooks

When I first gave a public talk on webhooks in 2007, the guys at Twitter were the first ones to come up to me saying they wanted to implement them. Unfortunately it never happened, and this was well before they were as popular as they are now. It wouldn’t scale, they said.

As an alternative, just a few months ago, they released a stream API using long-running HTTP. It not only included a firehose stream, but a stream that allowed you to pull updates from selected users. I knew immediately from this I could build a webhook transformer, effectively giving Twitter webhooks. Unfortunately, the stream for selected users was behind a EULA that prevented that use of their API. Something about not redistributing data.

Then a few weeks ago, they added a selective stream that was publicly available. I had already designed the system and started writing some code before when I found out about the EULA. It took me a while to find some time, but last night and today I did it. I finished a proof of concept service called Twitter Hooks.

It’s very simple and is mostly about the plumbing behind it. You sign in with your Twitter account using OAuth, then you just give it a callback URL. Behind the scenes, it has a stream consumer listening for updates from the users that have a callback URL. When it gets an update from you, it posts it to your callback URL.

Obviously, there is only so much you can do with your own tweets. The next step is to listen to updates from your friends and post them to your callback URL. After that, using some trickery not involving the stream API, it will post direct messages and new followers to your URL. Throw in keyword tracking, and you have hooks for almost every significant event in Twitter. John Herren would be proud.

The problem is… Twitter is still too limiting with their stream API. As it stands, the non-EULA stream only allows following 200 users. That means Twitter Hooks can only support 200 users. It also means there’s no way we can roll out the friend update hook because some people have over 200 friends by themselves. Until that’s resolved, Twitter Hooks will remain a proof of concept demo.

But I have to say, having this is pretty sweet. The stream API is about as real-time as you can get. While experimenting, I was getting updates from the stream quicker than the page reload from posting a new tweet. This is, of course, then posted to your URL immediately for you to do whatever you want with. Here’s a tweet of mine posted to PostBin:

If you want to play with Twitter Hooks, it’s currently limited to users I authorize since it can only have 200 users and it’s terribly early/rough. Just @ me on Twitter and I can get you set up.