Twitter as a Stockbroker. With Webhooks.

Webhooks. User-defined callbacks on the web. Yes. YES! Wait, what does that even mean? That stupid website doesn’t seem to explain them at all!

Let’s explore this through an analogy. This analogy uses stockbrokers! Yay! (?)

Consider, before the web, before email, how people would interact with their stockbroker: the telephone. Whenever you wanted to manage your investments and find out what’s really going on in the market, you would call your stockbroker. It’s similar to how you might use your browser to visit Twitter to update your status and find out what’s going on with your friends, right? This is the basis of the analogy. Bear with me.

Imagine in this story your phone is a browser, a phone call is a web request, and the broker is a web application, like Twitter.

It’s 1989 and you like to think of yourself as a big time stock trader with an impressive portfolio. Good for you. You got to this point by staying in close touch with your stockbroker. You’d call her (!) to find out what was happening in the market, discuss whether buying certain stock was a good idea, and maybe put in an order. You were able to get information out of the system and put orders into the system by picking up your phone and calling your broker.

The problem is that you can only react to events in the market as quickly as you get the information. This means you’d have to call your broker quite a bit to stay on top of a fast moving, highly volatile market. A programmer might try to automate this with a script. You take a similar approach: you hire an assistant to deal with your stockbroker. Perhaps this is unheard of in reality, but stay with me.

Now an assistant is not unlike a computer program in theory. You can give them instructions and based on certain conditions and input, they’ll do as you say. For example, maybe you instruct your assistant to buy a certain type of stock after it behaves a certain way, and if there’s not enough money to buy the amount that you told them, they can transfer the money from your savings account after a quick confirmation phone call with you. In today’s world, that sort of thing is not terribly out of the question to automate with a script using the web APIs and programmatic infrastructure available to us now.

Again, in order to react to events in the market in a timely manner, it requires lots of phone calls to your stockbroker. Sure, your assistant can handle it, but it’s a lot of work and wastes a lot of time. In fact, it wastes you money because you pay for your assistant by the hour! If you were a programmer, your script would have to constantly poll the broker API requiring a touchy cron setup or a long running process that gets more inefficient the closer to real-time you want it to be. Not to mention it’s just more work than you should have to deal with. If only there was some way for your assistant or script to be notified when things happen so they could simply react.

One day, your stockbroker says they’re providing a new service they describe as callbacks. You give them a phone number, and they’ll call it whenever something interesting happens regarding stocks relevant to you. Wow! Well, you could give them your number, but there are two problems. First, it’s 1989 and you don’t happen to have a cell phone. You can’t always be reached. Second, even if you could, you’d still want your assistant to handle the events because they have been instructed with how to deal with them and will do a lot of the legwork for you. Luckily, your assistant has a cell phone and is always available, so you give the broker their number.

Now your assistant is working very efficiently making you lots of money. Their job is so much easier because they don’t have to do anything until they get a call from either you or, more importantly, the broker. They can finally react to events as they happen, without a lot of nonsense trying to stay on top of things. What’s more is they will automatically take care of whatever situations you’ve told them.

What a cool feature, this callbacks thing. Thanks, broker!

If you haven’t guessed, the callback feature in this story is the exact mechanics of webhooks. Webhooks would allow you to tell the web apps that you use to “callback” scripts you have online. These scripts will deal with whatever events that web app produces. These callbacks use the same protocol you use to talk to them and that you both know how to use: web requests. Putting your script online at a URL is analogous to your assistant that has a cell phone: it’s always available for direct connection at a persistent “phone number,” unlike you, with dynamic IPs and NATs and turning off your computer. And just like an assistant, the script can do a lot of interesting things for you that you wouldn’t necessarily want to do, most of which is a sort glue work of making different systems work together given some logic. For example, transferring money from your savings account to the broker account when there isn’t enough.

A concrete example of a story made possible from webhooks that might be a useful scenario for many of you involves Twitter. Let’s say Twitter supported webhook callbacks for when somebody follows you. Right now you get an email, and from there you can decide what to do manually: follow them back, block them, or do nothing. I used to go out of my way to block users that I knew were spam bots, but now there’s so many it’s not worth the time. And of course I also generally follow back people that I actually know. If Twitter would simply call a script of mine whenever somebody followed me passing along the user ID, I could very easily run this logic in a PHP script or a simple App Engine app. Or perhaps I’d use Scriptlets (ahem, which was made exactly for these kinds of web scripts). It would work like this:

First, use the Twitter API to look up the user from the ID, and grab their name. Then use the Facebook API to check if that name shows up in my list of friends on Facebook. If so, use the Twitter API to follow them back. Otherwise, if they’re following over 1000 users and that number is more than twice the number that’s following them (which is roughly the heuristic I use manually), use the Twitter API to block them. All automatic.

If you could do this, not only would you have glued together Facebook and Twitter in an interesting and useful way, but you’ve sort of extended the net functionality Twitter provides you. You could imagine perhaps someday Twitter releasing features that would do exactly what you just did. But they won’t. In fact, they’ll never release features that are so specific to you (who says my spam algorithm is the algorithm everybody should use?). This is how webhooks make the web more about you. You can start extending web applications and gluing them together just the way you want.

This is a win-win for web application users and developers. Users get more functionality. Developers can implement less.

Coming back to the stockbroker analogy, there is a type of order called a limit order where you say to buy or sell when it’s at a certain price. With webhooks (or the broker “callback” service), this is merely a convenience because you could easily set that up outside of their system. Speaking of that example, when I mentioned your assistant transferring money between accounts, that reminds me, you know that feature banks have called overdraft protection? If banks had webhooks, all banks could have overdraft protection. Wow, right? That would have been nice for me when I had Washington Mutual, a bank known for its lack of overdraft protection.

What else could you do in a world with webhooks? Basically everything would have a common event-driven infrastructure, allowing you, with just a little bit of scripting glue, to accomplish so much more and make the systems you use better, and more personal.

Web notifications just got real with Notify.io

The “real-time web” is a popular topic right now. My WebHooks initiative is both riding on this success and helping make it a reality. One sector of this trend is about notifications. Real-time notifications to you about events you care about.

For a long time we’ve had helper apps like the Google Notifier and more recently the Facebook Desktop Notifications app that bring events from the web to your desktop. Twitter has created a whole ecosystem of clients that not only let you actively check Twitter, but passively get updates from Twitter.

Simultaneously, we’ve had a bunch of systems like Growl emerge that give you a consistent, well-designed and customizable system for local applications to give you notifications. While your IM client is in the background, it can tell you somebody IM’d you and what they said in an unobtrusive way. It integrates with email applications to tell you of new emails. It gives any application developer a nice way to present notifications to the user in a way that’s in their control.

Some of the apps that bring web applications to your desktop like Tweetie, Google Notifier, etc will integrate with Growl (which has a counterpart on pretty much every platform, including the iPhone). The problem is that you only get these notifications when the desktop apps are running, despite the fact web apps are always running. And yes, you have to have an app running for each web application you use.

And that’s only if they built a desktop app and you were convinced to download it. Most web applications are never be able to notify you with any means other than email. But as I’ve argued before, notifications don’t belong in your inbox!

Another minor point is that all these apps use polling to get updates. In some cases this doesn’t matter, but as data starts moving in real-time, this batches your notifications into bursts that you may not be able to parse all at once. I use Tweetie to get Growl notifications from Twitter at the moment, and if a lot of people are updating, I get a huge screen of updates that I don’t have time to read before they disappear. It becomes useless.

A while back I attempted to make an app called Yapper that lets anybody send real-time notifications to your desktop via XMPP. It was an experiment, and ultimately not the answer. It was only part of the solution.

But today I’m announcing the full solution: a free, public, open-source web service called Notify.io (Notify-I-O).

Notify.io integrates with Growl and other local notifiers (as well as email, Jabber, Twitter, and webhooks) and provides a dead-simple API for any web developer to send real-time notifications to their users.

You can think of Notify.io as a web-level Growl system. It empowers users with a consistent, controllable way to get notifications, and it provides developers with a simple, consistent way for sending those notifications.

Notify.io is an open platform for notifications. It’s still in a pre-alpha state, but it already has several useful notification sources. Last Thursday I built Feed Notifier, which uses PubSubHubbub to give you real-time desktop notifications of Atom and RSS feed updates.

At SHDH 35 last Saturday, Abi Raja built a Facebook notification adapter for Notify.io that’s yet to be released. And there a couple more in the pipeline (by me and others) to show the power of Notify.io.

Again, it’s pre-alpha, so before I talk much more about it, I should probably finish more of it. I just wanted to make sure I blogged about it in somewhat of a timely fashion. I seem to have a backlog of blog posts about apps I’ve built recently. However, Notify.io is a pretty significant one. Feel free to check it out, just remember that despite its looks, it’s nowhere near finished — but it does work.

Need icons for filetypes? Hotlink from stdicon.com

This is about ten days old, but still worth mentioning. I was working on PostBin trying to support file uploads. I realized I really wanted to have icons for the files to help differentiate them from other POST params and make it feel more polished. Of course, this would mean I’d have to find icons for the popular file/mime types … and if I were going to do that, I might as well build a service … but I didn’t want to build a service.

I turned to Twitter. In that past, I’d mentioned wanting something and somebody actually built it (using my tools no less). I figured this was a bit more work than last time, but it couldn’t be that much more. So I gave it a shot and tweeted it. Next thing I know, Paul Tarjan is on it. Some hours later: stdicon.com

I’m credited at the bottom, but really, I just had the idea and came up with the domain. Paul wrote all the code and even collected all the icons (and manually uploaded them to the app). We discussed implementation details over IM, but that was it. Pretty rad!

The idea is that given a file extension (“txt”, “gif”, etc) or a mimetype (“text/plain”, “application/zip”, etc) you can get a resizable icon by putting together a simple URL, like http://www.stdicon.com/mp3?size=16 or from a particular icon set http://www.stdicon.com/neu/html?size=64. Here are some examples from various sets:

Nuvola set:
Apache set:
Crystal set:

And that’s it! Paul wrote a post about it. Somebody requested a simple API for just doing the mimetype to file extension conversion, so Paul added it. Strangely, the same day, MIME API was released. That’s fine. Just means stdicon can focus on icons.

Anyway, I pulled this same stunt just the other day, using Twitter to get more cool infrastructure built, but I’ll have to write about it later.

Twitter Hooks

When I first gave a public talk on webhooks in 2007, the guys at Twitter were the first ones to come up to me saying they wanted to implement them. Unfortunately it never happened, and this was well before they were as popular as they are now. It wouldn’t scale, they said.

As an alternative, just a few months ago, they released a stream API using long-running HTTP. It not only included a firehose stream, but a stream that allowed you to pull updates from selected users. I knew immediately from this I could build a webhook transformer, effectively giving Twitter webhooks. Unfortunately, the stream for selected users was behind a EULA that prevented that use of their API. Something about not redistributing data.

Then a few weeks ago, they added a selective stream that was publicly available. I had already designed the system and started writing some code before when I found out about the EULA. It took me a while to find some time, but last night and today I did it. I finished a proof of concept service called Twitter Hooks.

It’s very simple and is mostly about the plumbing behind it. You sign in with your Twitter account using OAuth, then you just give it a callback URL. Behind the scenes, it has a stream consumer listening for updates from the users that have a callback URL. When it gets an update from you, it posts it to your callback URL.

Obviously, there is only so much you can do with your own tweets. The next step is to listen to updates from your friends and post them to your callback URL. After that, using some trickery not involving the stream API, it will post direct messages and new followers to your URL. Throw in keyword tracking, and you have hooks for almost every significant event in Twitter. John Herren would be proud.

The problem is… Twitter is still too limiting with their stream API. As it stands, the non-EULA stream only allows following 200 users. That means Twitter Hooks can only support 200 users. It also means there’s no way we can roll out the friend update hook because some people have over 200 friends by themselves. Until that’s resolved, Twitter Hooks will remain a proof of concept demo.

But I have to say, having this is pretty sweet. The stream API is about as real-time as you can get. While experimenting, I was getting updates from the stream quicker than the page reload from posting a new tweet. This is, of course, then posted to your URL immediately for you to do whatever you want with. Here’s a tweet of mine posted to PostBin:

If you want to play with Twitter Hooks, it’s currently limited to users I authorize since it can only have 200 users and it’s terribly early/rough. Just @ me on Twitter and I can get you set up.