I’ve long argued that failure is the only way a person can learn. This has been a very reasonating bit of wisdom for me, despite the fact people have often argued against it with rather sound logic. “Yes, you can learn from failure, but you can also learn from success!” Only after reading Jason Fried’s blog post from nearly one year ago to the day titled, Learning from failure is overrated have I realized exactly what’s wrong. We’re talking about different kinds of failure.

Learning happens when you correct the mismatch of an expected outcome from the actual outcome. This is where the idea of learning from failure actually comes from. Failure represents a mistake in judgment, a disparity between expectation and reality. Therefore, if your expectation is validated by success and there is no disparity, then you didn’t actually learn anything — you already knew.

However, validation of an expected but unsure success is obviously learning because it corrects the expectation of doubt you had. Furthermore, analysis of an unexpected success can result in learning from correcting the assumptions that led to expecting failure. A successful outcome, but a failure to expect it.

This is where the confusion comes in. Unfortunately it’s an issue of semantics. The meaning of the word failure in the context of “learning from failure” is this failure to know the outcome. In this way, it’s true: you can only learn from failure. However, it is only this specific instance of expectation-failure this applies to so absolutely.

Separately although related (which I think adds to the confusion), despite outcome-failures not necessarily teaching you what will work, they tend to be the strongest lessons experienced. I mostly attribute this to the greater level of disparity and correction made to your mental model that comes from expecting a failing outcome to succeed.

In any case, you learn from experience. There is no way to acquire knowledge without experience. In fact, that’s an entirely different argument. You can only acquire knowledge through experience. Not reading, not getting advice. But I’ll save that one for later.

Webhooks. User-defined callbacks on the web. Yes. YES! Wait, what does that even mean? That stupid website doesn’t seem to explain them at all!

Let’s explore this through an analogy. This analogy uses stockbrokers! Yay! (?)

Consider, before the web, before email, how people would interact with their stockbroker: the telephone. Whenever you wanted to manage your investments and find out what’s really going on in the market, you would call your stockbroker. It’s similar to how you might use your browser to visit Twitter to update your status and find out what’s going on with your friends, right? This is the basis of the analogy. Bear with me.

Imagine in this story your phone is a browser, a phone call is a web request, and the broker is a web application, like Twitter.

It’s 1989 and you like to think of yourself as a big time stock trader with an impressive portfolio. Good for you. You got to this point by staying in close touch with your stockbroker. You’d call her (!) to find out what was happening in the market, discuss whether buying certain stock was a good idea, and maybe put in an order. You were able to get information out of the system and put orders into the system by picking up your phone and calling your broker.

The problem is that you can only react to events in the market as quickly as you get the information. This means you’d have to call your broker quite a bit to stay on top of a fast moving, highly volatile market. A programmer might try to automate this with a script. You take a similar approach: you hire an assistant to deal with your stockbroker. Perhaps this is unheard of in reality, but stay with me.

Now an assistant is not unlike a computer program in theory. You can give them instructions and based on certain conditions and input, they’ll do as you say. For example, maybe you instruct your assistant to buy a certain type of stock after it behaves a certain way, and if there’s not enough money to buy the amount that you told them, they can transfer the money from your savings account after a quick confirmation phone call with you. In today’s world, that sort of thing is not terribly out of the question to automate with a script using the web APIs and programmatic infrastructure available to us now.

Again, in order to react to events in the market in a timely manner, it requires lots of phone calls to your stockbroker. Sure, your assistant can handle it, but it’s a lot of work and wastes a lot of time. In fact, it wastes you money because you pay for your assistant by the hour! If you were a programmer, your script would have to constantly poll the broker API requiring a touchy cron setup or a long running process that gets more inefficient the closer to real-time you want it to be. Not to mention it’s just more work than you should have to deal with. If only there was some way for your assistant or script to be notified when things happen so they could simply react.

One day, your stockbroker says they’re providing a new service they describe as callbacks. You give them a phone number, and they’ll call it whenever something interesting happens regarding stocks relevant to you. Wow! Well, you could give them your number, but there are two problems. First, it’s 1989 and you don’t happen to have a cell phone. You can’t always be reached. Second, even if you could, you’d still want your assistant to handle the events because they have been instructed with how to deal with them and will do a lot of the legwork for you. Luckily, your assistant has a cell phone and is always available, so you give the broker their number.

Now your assistant is working very efficiently making you lots of money. Their job is so much easier because they don’t have to do anything until they get a call from either you or, more importantly, the broker. They can finally react to events as they happen, without a lot of nonsense trying to stay on top of things. What’s more is they will automatically take care of whatever situations you’ve told them.

What a cool feature, this callbacks thing. Thanks, broker!

If you haven’t guessed, the callback feature in this story is the exact mechanics of webhooks. Webhooks would allow you to tell the web apps that you use to “callback” scripts you have online. These scripts will deal with whatever events that web app produces. These callbacks use the same protocol you use to talk to them and that you both know how to use: web requests. Putting your script online at a URL is analogous to your assistant that has a cell phone: it’s always available for direct connection at a persistent “phone number,” unlike you, with dynamic IPs and NATs and turning off your computer. And just like an assistant, the script can do a lot of interesting things for you that you wouldn’t necessarily want to do, most of which is a sort glue work of making different systems work together given some logic. For example, transferring money from your savings account to the broker account when there isn’t enough.

A concrete example of a story made possible from webhooks that might be a useful scenario for many of you involves Twitter. Let’s say Twitter supported webhook callbacks for when somebody follows you. Right now you get an email, and from there you can decide what to do manually: follow them back, block them, or do nothing. I used to go out of my way to block users that I knew were spam bots, but now there’s so many it’s not worth the time. And of course I also generally follow back people that I actually know. If Twitter would simply call a script of mine whenever somebody followed me passing along the user ID, I could very easily run this logic in a PHP script or a simple App Engine app. Or perhaps I’d use Scriptlets (ahem, which was made exactly for these kinds of web scripts). It would work like this:

First, use the Twitter API to look up the user from the ID, and grab their name. Then use the Facebook API to check if that name shows up in my list of friends on Facebook. If so, use the Twitter API to follow them back. Otherwise, if they’re following over 1000 users and that number is more than twice the number that’s following them (which is roughly the heuristic I use manually), use the Twitter API to block them. All automatic.

If you could do this, not only would you have glued together Facebook and Twitter in an interesting and useful way, but you’ve sort of extended the net functionality Twitter provides you. You could imagine perhaps someday Twitter releasing features that would do exactly what you just did. But they won’t. In fact, they’ll never release features that are so specific to you (who says my spam algorithm is the algorithm everybody should use?). This is how webhooks make the web more about you. You can start extending web applications and gluing them together just the way you want.

This is a win-win for web application users and developers. Users get more functionality. Developers can implement less.

Coming back to the stockbroker analogy, there is a type of order called a limit order where you say to buy or sell when it’s at a certain price. With webhooks (or the broker “callback” service), this is merely a convenience because you could easily set that up outside of their system. Speaking of that example, when I mentioned your assistant transferring money between accounts, that reminds me, you know that feature banks have called overdraft protection? If banks had webhooks, all banks could have overdraft protection. Wow, right? That would have been nice for me when I had Washington Mutual, a bank known for its lack of overdraft protection.

What else could you do in a world with webhooks? Basically everything would have a common event-driven infrastructure, allowing you, with just a little bit of scripting glue, to accomplish so much more and make the systems you use better, and more personal.

Two weeks ago it was being taped for a story on Fox News. One month ago it received a congratulatory visit from the mayor of Mountain View. Two months ago it was featured on the front page of the Mercury News. And four months ago was when we signed the lease for it. Hacker Dojo is officially on its way to something big.

Hacker Dojo, aspiring to be a global hub of innovation, is based on a simple idea: to provide a community center for hackers, thinkers and technologists to meet, discuss, learn and create.

It’s a non-profit, volunteer-run, member-supported operation that simply provides a space for hackers to do their thing 24/7, whether it’s working on side projects, organizing a meetup, building the next startup, giving a class, working remotely, or just hanging out with diverse yet like-minded people. The thing they have in common is the hacker spirit, a force that drives ordinary but curious people to create what can end up being extraordinary, such as the personal computer, the first video game system, or the technology powering the Internet.

A dojo is considered a place to do and train one’s craft with others. It’s most commonly used in the context of Japanese martial arts or other physical training. But the word “dojo” is simply defined as “place of the way.” In the case of Hacker Dojo, that way is the way of the hacker.

The hacker is not what most people think. Although 99% of the 100+ members of Hacker Dojo are capable of doing the things most people think “hackers” do, I’m 100% positive none of us have the intention of doing those things maliciously if at all. In fact, security has almost nothing to do with the way of the hacker. We define a hacker something like this:

A hacker is expert in their field, whether hobby or professional, that pushes the envelope of what’s possible through hands-on exploration, driven by relentless curiosity and a desire to challenge the status quo.

Steven Levy, author of Hackers: Heroes of the Computer Revolution, describes computer hackers as people that “regard computing as the most important thing in the world.” It’s about passion. It’s about something most of us can’t even describe. It inexplicably compels us to explore technology. To build things. To learn things. To be like the heroes in our field and achieve the remarkable, which is something I think everybody can relate to.

In fact, there is no reason why anybody shouldn’t aspire to be like a hacker, an innovator. This is what Hacker Dojo is about. We want to foster hacker culture so it can grow, develop and spread. We want to make its values explicit, and be the “place of the way” of the hacker.

If you still don’t understand the hacker, or want to learn more, take a look at the Hacker ethic article on Wikipedia. There’s also an excellent, short documentary available to watch online that shows you the energy and excitement of true hackers called Hackers: Wizards of the Electronic Age.

If you want to get involved in Hacker Dojo, come stop by or visit the website! We hold many events at the Dojo, but it’s also generally open for anybody to come and hack. If you’re too far to come to us, you may find another hackerspace nearby. For example, in SF there is the excellent Noisebridge hackerspace.

When you’re this obsessed with technology, so much that most “normal” people have no idea what you’re talking about, it helps to have a place where people don’t think you’re crazy. People have moved to Mountain View to be closer to Hacker Dojo. If you already live nearby, you should definitely take advantage of this place of the way of the hacker.

The “real-time web” is a popular topic right now. My WebHooks initiative is both riding on this success and helping make it a reality. One sector of this trend is about notifications. Real-time notifications to you about events you care about.

For a long time we’ve had helper apps like the Google Notifier and more recently the Facebook Desktop Notifications app that bring events from the web to your desktop. Twitter has created a whole ecosystem of clients that not only let you actively check Twitter, but passively get updates from Twitter.

Simultaneously, we’ve had a bunch of systems like Growl emerge that give you a consistent, well-designed and customizable system for local applications to give you notifications. While your IM client is in the background, it can tell you somebody IM’d you and what they said in an unobtrusive way. It integrates with email applications to tell you of new emails. It gives any application developer a nice way to present notifications to the user in a way that’s in their control.

Some of the apps that bring web applications to your desktop like Tweetie, Google Notifier, etc will integrate with Growl (which has a counterpart on pretty much every platform, including the iPhone). The problem is that you only get these notifications when the desktop apps are running, despite the fact web apps are always running. And yes, you have to have an app running for each web application you use.

And that’s only if they built a desktop app and you were convinced to download it. Most web applications are never be able to notify you with any means other than email. But as I’ve argued before, notifications don’t belong in your inbox!

Another minor point is that all these apps use polling to get updates. In some cases this doesn’t matter, but as data starts moving in real-time, this batches your notifications into bursts that you may not be able to parse all at once. I use Tweetie to get Growl notifications from Twitter at the moment, and if a lot of people are updating, I get a huge screen of updates that I don’t have time to read before they disappear. It becomes useless.

A while back I attempted to make an app called Yapper that lets anybody send real-time notifications to your desktop via XMPP. It was an experiment, and ultimately not the answer. It was only part of the solution.

But today I’m announcing the full solution: a free, public, open-source web service called Notify.io (Notify-I-O).

Notify.io integrates with Growl and other local notifiers (as well as email, Jabber, Twitter, and webhooks) and provides a dead-simple API for any web developer to send real-time notifications to their users.

You can think of Notify.io as a web-level Growl system. It empowers users with a consistent, controllable way to get notifications, and it provides developers with a simple, consistent way for sending those notifications.

Notify.io is an open platform for notifications. It’s still in a pre-alpha state, but it already has several useful notification sources. Last Thursday I built Feed Notifier, which uses PubSubHubbub to give you real-time desktop notifications of Atom and RSS feed updates.

At SHDH 35 last Saturday, Abi Raja built a Facebook notification adapter for Notify.io that’s yet to be released. And there a couple more in the pipeline (by me and others) to show the power of Notify.io.

Again, it’s pre-alpha, so before I talk much more about it, I should probably finish more of it. I just wanted to make sure I blogged about it in somewhat of a timely fashion. I seem to have a backlog of blog posts about apps I’ve built recently. However, Notify.io is a pretty significant one. Feel free to check it out, just remember that despite its looks, it’s nowhere near finished — but it does work.

“Efficiency is doing things right; effectiveness is doing the right things.” –Peter Drucker

When people, usually analytical people, want to improve a situation, they tend to optimize efficiency: achieve maximum output for input. “Let’s reduce waste! Let’s simplify! Let’s make things smoother! Let’s try and get more out of the system!” I suppose the obsession with efficiency is explained in the Drucker quote: that efficiency is “doing things right.” Who wouldn’t want to do things right?

The problem with efficiency is that it has nothing to do with whether or not what you are currently doing is the right thing to do. Whereas effectiveness is about achieving the right result, or being on the right path.

Too many people assume a system is on the right path. If there is a problem, they address it by smoothing things out and making the process more efficient without questioning the larger system they were produced in. But if the system is going in the wrong direction, that’s only going to make the real problem worse. The push for more standardized testing in public education comes to mind.

What’s really important is effectiveness. In the end, it doesn’t matter if your business is spending the least amount possible or your computer program running as fast as possible or your lifestyle entirely streamlined. If it’s effective, it achieves the desired goal. Your business is producing value, your program is functionally useful, your lifestyle is making you happy. Effectiveness is qualitative. Efficiency is quantitative, which is why I think it’s so big with analytical people. In fact, intelligent people in general.

If you think about it, intelligence, especially knowledge, is mostly concerned with efficiency. It’s more about how to solve problems, less so with what problems to solve. Knowledge is a tool. It’s neutral. To what ends do you actually use it for? That requires values and intention–the realm of wisdom. A wise person tends to be an effective person.

When approaching a problem, wisdom and pragmatism must frame intelligence. Before you start thinking about efficiency, you should step back and think about effectiveness. In computer engineering this idea spread with Donald Knuth’s quote “premature optimization is the root of all evil.” His argument being that 97% of efficiency optimizations are unnecessary to achieve functionality, at which point you can determine which optimizations will be the most effective improvements of efficiency.

In a way, it’s a look before you leap argument. Don’t get me wrong with all this. Efficiency is terribly valuable and can improve a situation, but only if you’re on the right path. Just because a system is currently working or was previously working doesn’t mean it should be, or will in the future. You should always consider effectiveness before efficiency, even in “working” systems. Here’s why:

Effectiveness opens the door for efficiency, but efficiency can change the requirements for effectiveness.

Quantitative improvements can qualitatively change the situation if taken far enough. The game can change. For example, you can become so efficient at producing cars that production isn’t the problem anymore. Then it’s a question of variety, like choice of color. “You can have any color as long as that color is black,” Ford said, and soon after lost the lead in car manufacturing. Perhaps when business slowed, they tried to make their sales and marketing or administrative organization more efficient. They didn’t re-asses whether the thing they were doing so right (building cars so efficiently)… was the right thing. The effective thing.

Efficiency is important, but powerless without effectiveness. Always keep an eye on effectiveness.

Public Open Source Services

October 29, 2009

Last night I went off and put up a wiki about an idea I’ve been thinking about for a while: public open source services or POSS. Think: public services or utilities on the web run as open source.

Unlike open source software, web services aren’t just source code. They’re source code that runs. They have to be maintained in order to keep running, and the resources they consume have to be paid for. This is why most web services are built using a business as the vehicle. This effectively constrains what you can build by framing it as something that needs to turn a profit or support you to work on it. But does it need to be that way? Can web services be built in a way that make it self-sufficient? Not needing some ambivalent leader to take responsibility for it?

I originally blogged about it in February 2007, 6 months after I first wrote about webhooks. Unfortunately my old blog isn’t online right now. Back then, I was trying to solve the administrative problem. How do you maintain the servers in an open source way? My idea then, was to build a self-managing system using something like cfengine or Puppet, where the recipes and configurations are kept with the publicly available source code. As new configurations are checked in, the server(s) adopt the new directives and continue to self-manage.

The practicality of such a setup is a little far fetched, but seemed pretty feasible for smaller projects. However, since the release of Google App Engine, this concern for simple web applications has disappeared. Google just automates the system administration, and scaling! This means to run the app, you just have to write the code and hit deploy. That’s a huge step! Administration concerns? Pretty much solved.

The next thing is the financial concern. How do you pay for it? Or rather, how does it pay for itself? This took longer to figure out, but here we are. From the wiki essay:

You use the same Google Merchant account that App Engine debits as the one that accepts donations. This way no bank account is involved. Then you track the money that goes into the account (using the Google Merchant IPN equivalent). Then you look at your usage stats from the App Engine panel and predicate future usage trends. Then calculate the cost per month. Then divide the cash in the account by that and you have how long the service will run. You make this visible on all pages (at the bottom, say) that this service will run for X months, “Pay now to keep it running.” You accept any amount, but you are completely clear about what the costs are. And this is all automated.

Take the humans out of the loop! (That’s a WarGames reference)

Then you rely on the same sort of community approach of open source to contribute to the application. Like a few members of the project community are given certain rights, some will be given permission to deploy the app from time to time for updating the running service.

If the service isn’t useful, nobody uses it, it’s not paid for, it disappears. If it is useful, people will pay for it to keep it running. They are assured they are paying operating costs, which are significantly lower than most because it doesn’t include paying for human resources! Volunteers might need to meddle with settings, but otherwise, the coders are in control and the community accepts or denies changes made by whoever wants them.

So if this is interesting, read the full essay I wrote up on the wiki. It’s been my intention to prototype and validate this model with many of my projects.

Meet me at SXSW 2010 (http://sxsw.com) Last week I got an email that said my SXSW speaking proposal was accepted. Strangely, my joy was coupled with a bit of disappointed. After the PanelPicker closed, I felt like I didn’t market my talk well enough. That combined with my OSCON proposal being denied, I felt there was no chance I was going to speak at SXSW. However, the more I thought about it, the more I liked the sound of it. I’ve had such a busy year going to conferences, speaking, writing, and building stuff. You all realize this webhooks stuff doesn’t pay, right? I was actually looking forward to not having a huge stressful talk to worry about.

But now I do! Lucky for you, I’ve decided to make it a really great talk. I’ve started work on it immediately and will be developing it probably up until March.

What is the talk? It’s not another “WebHooks are the Future” talk. Well it is, but in disguise. I’ve decided to focus on context. Like I mentioned before, all that vision stuff usually at the end of my talks will come front and center. The talk is called: How WebHooks Will Make Us All Programmers.

In order to describe this, I’ll need to first explain webhooks to those that are unfamiliar. I feel this is getting easier and easier as I gain experience and more tangible examples pop up, like PubSubHubbub and TweetHook. However, I’ll have to take it up a notch further because I won’t have that much time to talk about them. Before I can explain how webhooks will make us all programmers, I also have to share why this is worth doing at all. I have to explain why we should all be programmers.

Yay! It’s a sort of philosophy of computer science. I mean, that’s the stuff I really love talking about, so why don’t I go all out? Well I will.

But I have a lot of work in front of me. I need to finish building more examples. I’ve finally gotten to a point and built enough infrastructure where I can really start showing the power of webhooks through examples. I need to develop a super concise explanation of webhooks, probably something visual and animated — a mini presentation within a presentation. I also need to start practicing speech again. The last time I went without practice it was a disaster, despite getting my point across.

Part of my preparation will involve lots of writing and sharing on this blog. I need to prototype and validate individual ideas before I bring them all together in an epic talk. That’s the only way to make it as awesome as it should be. It also means there will be lots of follow up material for attendees.

So at the very least, my goal is to post something relating to the ideas in this talk once a month. And your feedback will be much appreciated.