Tuesday, July 15, 2014

The Saga Of The Stank

My hand is over my mouth, my eyes are watering, and I'm trying to hold my breath, but every few seconds, my throat tries to turn itself inside out. Most smells, after a little while, fade into the background -- you become desensitized and can no longer detect them. But not this smell. You remember the sloth scene from Se7en? The one where the cops, handkerchiefs over their mouths, walk into a room that has hundreds of Little Tree car air fresheners hanging from the ceiling and find a guy who has been chained to a bed for a year, living in his own waste while his body rotted away? It was like that. Minus the air fresheners.


A week ago, Molly and I had entered our apartment for the first time after a month and a half of traveling. We had been to the Philippines, Thailand, Boston, and were finally home, after a long flight. It was nearly midnight and all I wanted to do was bury my face in a pillow, cocoon myself in a blanket, and breath in my bed, which smells like sleep. Instead, I found myself breathing in noxious fumes. Which smell like noxious fumes.

Molly's first guess was that we had forgotten to take out the trash, but when we checked, it was empty. I looked for anything rotting in the sink, but it was spotless. Molly cracked open the fridge -- also empty. However, as she closed the refrigerator door, it pushed some air out of the fridge and in our direction. A second later, a wave of warm, putrid shit smelling foulness washed over us.

"Oh dear, we've got to deal with this", I said. Or maybe it was something more along the lines of "Oh Jesus fuck what the shit is that???"

I readied myself and peeled the fridge door open again and peered around. We had eaten or thrown out all the food before leaving on vacation, so the shelves were empty. All that remained were a few crumbs and stains. And yet, a cloud of corruption was flooding out of the fridge -- I swear I could see green stink lines.

"We've got to clean this up", I said. Molly suggests it was more like "Oh come on! Why the... what is the... what the fuck is that goddamn smell???" It was go time.

Approximate re-enactment of me approaching the fridge
Molly and I began pulling out the shelves and drawers and washing them. The fumes of -- what was it? Rotten eggs and a dead skunk floating in a bath of spoiled milk? -- continued to waft through the apartment. We were finally on the last shelf when I randomly decided to peer into the freezer. Although the fridge was empty, we had left the freezer running to preserve frozen goods and various condiments. I peered around and saw a package of ravioli, a bag of Ikea meatballs, a bottle of maple syrup, some pink liquid, and then, wait a second, on the door... Oh god, what the hell was that??

I slammed the freezer door shut.

The creature I thought I had seen on my freezer door
My brain was still processing what I had seen. Unfortunately, shutting the freezer door had pushed out a new cloud of noxious gas that nearly knocked me unconscious. As I came to, I began to wonder: why did the freezer smell worse than the fridge? Why was there an Alien facehugger on my freezer door? But most of all, I couldn't get out of my mind the pink liquid in the freezer.

Wait a minute, why was there liquid in my freezer?

I took a peek at the controls in the fridge:


Pop quiz: what happens if you leave the freezer setting at "4", but turn the refrigerator setting to "off"? Did you say that both the freezer and refrigerator shut off? Because that's exactly what happened. For the entire time we were traveling. That's 1.5 months.

For six weeks, some monstrosity had been stewing, growing, incubating in my freezer. We had been scrubbing the refrigerator shelves, but all this time, the real evil lay quietly above. But what was it? What fiend lay in waiting?


I had to know. But I didn't want to know. Molly and I stared at each other. Our imaginations danced. Within the freezer, there lurked unknown dangers, and I felt naked. Actually, I was mostly naked, because I had stripped down after noticing that the putrid smell was getting in my clothes. I searched the house for some protection and found... a single glove. Not zero, not two, but one glove. Seriously, I don't understand why you would have just a single rubber glove, but there you have it. It was purple.

I also found a bottle of Lysol, but it was almost empty. So there I was, wearing a pair of boxers and a single rubber glove, holding a bottle of Lysol, heading into the den of a dragon that could peel the armor off an Abrams tank with a single breath.

I closed my eyes and pulled at the freezer door. As it swung open, I caught sight of a pint of Ben and Jerry's ice cream that had transformed into a Xenomorph egg. No, that's not right. The ice cream had spawned a huge blob of mold that tore through the lid and crawled, stretched, and oozed its fuzzy blue green body up the freezer door. I brought the lysol up, took aim... but I was too late. As the door swung a bit further, the full power of the smell hit me.

Later, when I told this story to my sister, she asked why I didn't take a picture of the mold. The answer is that this was not mold. This was biological warfare, a weapon of mass destruction, the poison gas from the Hunger Games that causes your skin to burst into boils on contact. As the toxic cloud washed over me, it was as if millions of voices suddenly cried out in terror... and then farted. A smell so foul that we gave it a name: it became known as The Stank. This was no time for selfies.

My senses returned to me and I went to war. With one hand, I shoveled all the food left in the freezer into the trash. With the other, I began spraying with lysol. And I kept spraying. And spraying. And spraying. And spraying some more.

A reenactment of our battle against The Stank.
Finally, the bottle was empty. Now, the hard part. I had to wipe it all down. That meant reaching my hand - the one gloved hand - into The Stank. This had all the appeal of reaching straight into Satan's asshole.

I don't really remember what happened next. Somehow, hours later, I was huddled in corner of the shower, knees pressed to my chest. No amount of soap could wash out The Stank.

But our battle was not finished. The next day, the freezer still smelled horrible. We washed everything with soap and water. Then we tried baking soda and water. Then vinegar and water. Then we realized it isn't a great idea to mix baking soda and vinegar, unless, of course, you're trying to create a volcano for your 4th grade science fair.

The next day, the smell was still there. We went out and bought an anti-mold spray that contained bleach. We filled the fridge and freezer with the stuff and let the chlorine gas waft through it. It didn't help.

Then we filled the fridge with smell absorbers: baking soda, charcoal, coffee grinds, oatmeal. We packed plates, bowls, and cookie trays full of the stuff and arranged it in the fridge. It looked like an altar of offerings to a vengeful god. It was our tribute to The Stank. It was not enough.


Worse yet, The Stank followed us. A smell that powerful doesn't just go away when you leave the house. You begin to imagine The Stank everywhere you go. The funky cheese at the fancy restaurant reminds you of The Stank; the bathroom at the library smells of Stank; does that guy on the treadmill have terrible body odor or is that The Stank too? Wait, do I smell like The Stank? Have I become The Stank?

We spent a full week trying every remedy imaginable. We took out shelves, we unscrewed panels, we scrubbed, wiped, sprayed, brushed, and polished every surface. Every visible inch of the fridge became a shiny, unnatural white. But despite it all, The Stank remained.

Finally, today, we ordered a new fridge. The Stank won.



So, kids, here's the moral of the story:
  1. Fuck the environment. We tried to be green -- to save a little energy by shutting off the fridge during our trip. In return, we got The Stank. In fact, once you factor in all the chemical cleaning products, the rolls of paper towels, the years of our lives lost breathing in spores and cleaning agents, and the need to buy and ship a new fridge, I'm not entirely convinced our conservation efforts paid off.
  2. Fuck bad design. If your fridge has separate control knobs for the fridge and freezer, perhaps they should really be separate? Learn design, learn the principle of least surprise, and don't make me think.



Thursday, May 29, 2014

Must-See Tech Talks for Every Programmer

I love a good tech talk. I like to watch them, I like to give them, and now that I have lots of free time on my hands, I've put together a list of the must-see talks for every programmer. In this list, I've avoided language or library specific talks and instead focused on high-level, general topics that apply to everyone:
  1. The future of technology
  2. User interface
  3. Programming language design
  4. Software engineering
  5. Computers and learning
  6. A career in programming
  7. Computer gaming
  8. Fun talks
If I missed a talk you love, leave a comment. Happy watching!



When thinking about the future, you can't do better than Alan Kay. In The Future Doesn't Have To Be Incremental, Kay describes how Xerox PARC was able to develop so many new technologies in such a short time, including the personal computer, bitmap displays, GUI, desktop publishing, word processing, laser printing, Ethernet, and object oriented programming. The key was a culture focused on invention - that is, fundamentally new research - instead of incremental innovation. Invention requires a significantly higher investment of money, much longer time frames, and a different approach to problem solving ("wouldn't it be ridiculous if in 30 years we didn't have...").

Other essential talks on the future of technology:
  • The Mother of All Demos by Douglas Engelbart. A demonstration of hypertext, graphics, video conferencing, the mouse, word processing, and much more - all in 1968! This is the kind of leap frog invention Kay is referring to.
  • The Future of Programming by Bret Victor. A brilliant talk where Bret Victor takes us back in time and reminds us that "the most dangerous thought that you can have as a creative person is to think that you know what you're doing. Because once you think you know what you're doing you stop looking around for other ways of doing things and you stop being able to see other ways of doing things. You become blind."
  • The Computer Revolution Hasn't Happened Yet by Alan Kay. "I made up the term object-oriented and I can tell you I didn't have C++ in mind."


Inventing on Principle by Bret Victor

Bret Victor's talk Inventing on Principle will make all of your programming languages and tools feel obsolete. He presents a new way to write code: a user interface that makes the computer do the tedious work so that you can experiment with and react to your code instead of trying to simulate it in your head. This not only makes it much easier to learn programming, it fundamentally changes how we go about solving problems.

More UI goodness:


Simple Made Easy by Rich Hickey

Rich Hickey, the creator of Clojure, has the ability to make you see basic concepts in computer science in a whole new light. Complexity is one of these basic concepts and Simple Made Easy defines some of the best tools - the best language - to reason about it.

A few other talks to add to your playlist:
  • Are We There Yet? by Rich Hickey. Another foundational talk by Hickey that will force you to reconsider state, time, identity, values, and types.
  • Growing a Language by Guy Steele. One of the most clever presentations I've ever seen on programming language design. Give it about 10 minutes - the payoff is amazing.
  • The Science of Insecurity by Meredith Patterson. Why current systems and protocols are inherently insecure and how to fix that in the future.



Greg Wilson will force you to look closely at how you make decisions in software engineering: should you use Java? Ruby? Play Framework? Rails? TDD? Agile? Code reviews? Most of your answers are probably based solely on opinions, memes, trends, and anecdotes. What We Actually Know About Software Development, and Why We Believe It's True is an important call to change our practices and move to a world of evidence-based software engineering.

Other great talks:
  • Real Software Engineering by Glenn Vanderburg. Software engineering as it's taught in universities simply doesn't work. It doesn't produce software systems of high quality, and it doesn't produce them for low cost. Sometimes, even when practiced rigorously, it doesn't produce systems at all.
  • Hammock Driven Development by Rich Hickey. "Most of the biggest problems in software are problems of misconception." To solve hard problems, step away from the computer, take some time to think, and write it down.
  • The Language of the System by Rich Hickey. We focus extensively on the perfect programming language to build a single system, but what about languages for how multiple systems communicate with each other?

Daphne Koller and Andrew Ng are trying to reinvent education. They created Coursera, which offers real university classes, online and for free, to anyone in the world. What we're learning from online education talks about some of the techniques they are using to run classes of hundreds of thousands of students - including video, discussion forums, interactive UI's, automatic grading, and peer grading - and how this experiment is giving us unprecedented insight into how humans learn.

A few other powerful talks on education:



You and Your Research by Richard Hamming

You and Your Research is the blueprint for a successful career in any discipline, not just research; in fact, the talk has the nickname "You and Your Career". In this lecture, Richard Hamming shares his observations on "why do so few scientists make significant contributions and so many are forgotten in the long run?'' Some of the key ideas include courage, luck, drive ("knowledge and productivity are like compound interest"), a focus on important problems ("If you do not work on an important problem, it’s unlikely you’ll do important work"), open doors, selling the work ("I suggest that when you open a journal, as you turn the pages, you ask why you read some articles and not others"), and much more. This should be required viewing for every high school student.

More talks on how to succeed in the programming industry:
  • The Myth of the Genius Programmer by Brian Fitzpatrick and Ben Collins-Sussman. This talk isn't about geniuses or 10x programmers, but rather, about building a culture that avoids elitism and provides support for personal growth, collaboration, and ideas.
  • Programming Well With Others: Social Skills for Geeks by Brian Fitzpatrick and Ben Collins-Sussman. The Fitz and Ben duo are back to remind you that you need to learn more than just programming languages, compilers, debuggers, and algorithms to be a successful software engineer
  • JavaScript Masterclass by Angelina Fabbro. Don't let the title fool you: this talk is a tutorial for how intermediate developers can become experts in any topic, not just JavaScript.




By the time kids graduate high school, they have spent 10,000 hours playing games: this is roughly equivalent to the time they spend in school (if they had perfect attendance!) and the amount of time it takes to become an expert. What are they learning during those 10,000 hours? Jane McGonigal will show you how this time and learning can be used to make a better world.

More gaming goodness:
  • The game that can give you 10 extra years of life by Jane McGonigal. A powerful talk by McGonigal about how games can improve our lives and how they helped her recover from suicidal depression following a severe concussion.
  • Human Computation by Luis von Ahn. We can use human brain power and gaming to solve problems that are difficult for computers, such as image recognition, translation, and "common sense".
  • Design Outside the Box by Jesse Schell. What happens when games invade every aspect of our lives.



Wat by Gary Bernhardt

Now, it's time for some fun. Gary Bernhardt's talk Wat is 5 minutes of pure awesome.

A few others to brighten your day:


Monday, May 19, 2014

Don't learn to code. Learn to think.

It seems like everyone is trying to learn to code: Code.org has celebrities like Bill Gates, Mark Zuckerberg, and Chris Bosh telling you anyone can code; CoderDojo's are springing up all over the country; the UK has made it part of their official curriculum for all grade school kids.

I think this is slightly misguided. Don't get me wrong - I do think the world would be better off if everyone had some familiarity with coding - but coding itself should not be the goal. Computers and programming are just tools. They are a means to an end.

The real goal should be to teach people a new way to think. In other words, we should be trying to teach computer science and not just coding. In this blog post, I'll explain the difference between the two, and why focusing on the right one is critical for the movement to succeed.

If you prefer a video explanation, I highly recommend Simon Peyton Jones' wonderful TED talk Teaching Creative Computer Science, which was the inspiration for this post:


Still here for the written version? Great. Let's get started by asking a key question: why should you care about coding or computer science at all?

To answer that, we'll take a walk.

Welcome to the real world
You're probably reading this blog post in Chrome or Firefox, running on Windows or OS X, on a laptop or desktop. I'm guessing you also spent some time today reading email, checking your friends' Facebook statuses, or watching a video on Youtube. Much of your life is on computers these days: your medical records are in a database; you resume is on LinkedIn; you use Google and Facebook to market your products; you use Amazon to buy them; you file your taxes online; you manage your bank account on a website; perhaps you even dabble in digital currencies.

Now, look up from your computer: in your pocket, purse, or on a desk nearby, you may have a smartphone. It's loaded with a GPS, camera, touch screen, and tons of apps. If you're in your living room, you might also have an LCD TV hooked up to digital cable, a DVR, DVD player, Apple TV, XBox, or PlayStation. The movies, music, and games you may play on those devices are packed full of computer graphics and digital audio processing.

Let's head outside. Did you walk past your car? Modern cars are designed using software, built in a factory full of robots, and stuffed full of computers. If you drive your car, you might use Google Maps to find your way around, Yelp to find a place to eat, or TripAdvisor to find a place to stay. Now, look up: somewhere above you, a plane will pass by that is controlled by auto pilot, has in-flight Wifi and entertainment systems, and is constantly communicating with other planes, traffic controllers, and its manufacturer. Somewhere above that, satellites and space stations are orbiting the earth, taking pictures, measuring the weather, and routing phone calls.

Software is eating the world. But this is only the beginning. Before you know it, you'll be wearing technology, locking your doors with computers, using robots to deliver goods and clean your house, building your own electronics, running your own manufacturing plant, living in virtual reality, traveling in self driving cars, and flying to space.

The matrix is everywhere
Absolutely all of the technology I just described is powered by software. In every aspect of your life, you are surrounded by code. And the amount of code is only going to increase in the future.

Now, just because a technology is ubiquitous doesn't mean you have to study it in school. For example, we all fly in airplanes, but getting your pilot's license is not part of the K-12 curriculum.

However, the tools you need to understand how to think about flying are part of the curriculum:
  1. Physics and math help you understand gravity, forces, pressure, velocity, friction, and lift.
  2. Biology teaches you what happens to the human body at high altitudes, with limited oxygen, and extreme cold.
  3. History explains how the airplane was developed, how it evolved, and its role in travel, commerce, and warfare.
By the time you graduate high school, you have an idea of what a plane is, how it works, and how to use it safely. General purpose classes like physics, math, biology, and history teach you how to think about a wide variety of topics, including airplanes; this is in contrast to a class that teaches you how to use a tool, such as how to fly one specific type of airplane.

For the same reason, we should focus on teaching computer science and not just coding: the former is a general purpose way of thinking, whereas the latter is a specific tool. Let's look closer at computer science to get a better understanding of the distinction.

What is computer science?
Computer science is the study of computation: that is, how to represent and process information. Here are just a few of the concepts you might study:
  1. Problem solving: you'll learn algorithms - that is, general strategies, such as divide and conquer, recursion, heuristics, greedy search, and randomized algorithms - that help you model, decompose, and solve any kind of problem.
  2. Logic: you will start to use precise and formal methods of thinking, including abstraction, boolean logic, number theory, and set theory, so you can solve problems in an air tight manner.
  3. Data: you will touch information theory and start asking questions like what is information? How do you represent it? How do you model the real world?
  4. Systems: how do you design and build complex systems that satisfy a set of requirements and constraints? Systems engineering is an essential topic in almost every business.
  5. Thinking: one of the best ways to understand the human mind is to try to replicate it. Topics like artificial intelligence, machine learning, computer vision, and natural language processing are at the forefront of not only computer science, but also biology, psychology, philosophy, and mathematics.
Note that the above list doesn't really mention coding or programming, because they are just tools that can perform computation: they are not, in and of themselves, computer science.
Computer science is no more about computers than astronomy is about telescopes, biology about microscopes, or chemistry about beakers and test tubes. Science is not about tools. - Michael Fellows and Ian Parberry
It turns out there is another tool that we rely on for computation even more: the brain! The goal of computer science is to teach your brain new, general purpose, and widely applicable ways to think. As technology becomes more and more ubiquitous, this new way of thinking will become just as important as physics, math, biology, and history.

All that said, thinking alone is not enough: we need to know how to apply it. In physics, you do experiments with scales, prisms, and magnets; in biology, you might use test tubes, plants, and petri dishes; in computer science, you learn programming.

What is programming?
Programming, or writing code, is how you instruct a computer to perform some operation. If you've never written code before, you're probably used to interacting with a computer by clicking on things in an existing app. Under the hood, this app consists of code that tells the computer how to display the application, where to store or retrieve data, and how to react to your clicks.

All of programming is based on the principles of computer science we discussed above. It is remarkable that the same set of concepts - logic, algorithms, data, systems engineering - can be used to build everything from the web browser you're using to read this post to the autopilot software on an airplane. Although programming involves lots of math and structure, it is also a remarkably creative exercise: you think products into existence, one line of code at a time.

Learning programming as part of a computer science education brings about a number of benefits:
  1. DIY: if you can code, you can build things for yourself. You can start simple: create a script to rename a bunch of travel photos or an Excel formula to help calculate your taxes. Then, get fancier: create a website for your portfolio; create a mobile app for your company; build a game to play with your friends.
  2. Troubleshooting: once you've built a few apps yourself, figuring out other apps is easier. Once you stop fearing the computer - the unknown - you will become a master of tech support. As technology touches every part of your life, knowing how to navigate it will become as important as knowing how to read.
  3. Career: the goal of learning computer science is not to become a professional programmer. We all study math, physics, and chemistry in school, but we don't all become professional mathematicians, physicists, and chemists. However, if you do have a passion for it, you'll find that software engineering is one of the highest rated, highest paid, and fastest growing jobs out there.
Putting it all together
Let's recap:
  1. Computer science is a new way of thinking. The concepts in it are useful for every single person in a technology-filled world.
  2. Programming is an essential part of learning computer science by applying the new way of thinking. However, by itself, programming is not nearly as general purpose.
Confusing these two concepts is causing problems for the learn-to-code movement. Slate published an article called Maybe Not Everybody Should Learn to Code; the Atlantic wrote Should Journalism Schools Require Reporters to 'Learn Code'? No; Jeff Atwood wrote Please Don't Learn To Code, where he asks a question that neatly summarizes the confusion:
"How [would] Michael Bloomberg be better at his day to day job of leading the largest city in the USA if he woke up one morning as a crack Java coder?"
This is, of course, the wrong question. It is the result of public campaigns that suggest that learning to code, as opposed to learning to think, is the end goal. If even Jeff Atwood, an experienced and respected programmer, is fooled by this distinction, then the average person has no chance of getting it right. The question we should be asking is:
Would Bloomberg - or anyone else - be better at their job if they improved their ability to think by learning new problem solving strategies and developing a better grasp of logic?
I think the answer here is obvious. As the world fills up with more and more technology, I think the answer becomes even more obvious. This is why we need to focus on teaching computer science and not just coding.

How to get started
The good news is that you don't need to wait for Code.org to get this message - you can start learning computer science right now! In fact, it's one of the easiest topics to learn, as all you need is a computer and an Internet connection, and if you're reading this post, you probably have both.

Here are some great resources to get you going:

University courses
  1. Coursera
  2. Udacity
  3. MIT OpenCourseWare
  4. Stanford Engineering Everywhere
  5. Academic Earth
Online tutorials
Communities and clubs
  1. CoderDojo
  2. Girls Who Code
  3. Girl Develop It
  4. Code.org
  5. Coding meetup Groups

Monday, May 5, 2014

You are what you document


Hey, grab a seat - we need to talk about documentation. Now, I know what you're thinking: documentation is tedious, a chore, an afterthought, a redundant source of information given your beautiful, self-documenting code. It's just like a good diet and exercise - you'll do it when you have the time!

Well, this blog post is an intervention. You're hurting others and you're hurting yourself. You poured countless hours into a project, but your co-workers won't use it. You tried to run it in production, but the OPs team won't support it. You put the project on Github, but the fools on Hacker News just don't see the brilliance of what you've done.

The number one cause of startup failure is not the product, but the distribution: it doesn't matter how good the product is if no one uses it. With software, the documentation is the distribution: it doesn't matter how good the code is if no one uses it. If it isn't documented, it doesn't exist.


Think of this blog post as documentation for your documentation. By "documentation", I don't just mean a written manual, but all the pieces that go into making your software learnable: the coding practices, tutorials, white papers, marketing, the community, and the user experience.

I'll be discussing three types of documentation:
  1. Written documentation: READMEs, tutorials, reference guides, white papers.
  2. Code documentation: API docs, comments, example code, the type system.
  3. Community documentation: blog posts, Q&A sites, talks, meetup groups.
Each type of documentation solves a different problem, so most projects should include some mix of all three types. I've tried to include links to open source projects that best demonstrate each of the different types of documentation. If you know of other great examples or other types of documentation that I've missed, please leave a comment.

1. Written documentation

Let's start with what people typically think of when they hear the word "documentation": READMEs, tutorials, reference guides, etc.

1a. The README

Every project should have a README: it is the single most important document in your codebase. The README is typically your first contact with a new user, so your goal is to introduce them to the project as quickly as possible, convince them why it's worth learning more, and give them pointers on how to get started and where to get more info.

A typical README should have the following information:
  1. Description: short "sales pitch". Tell the reader why they should keep reading.
  2. Quick examples: short code snippets or screenshots to support the description.
  3. Quick start: how to get going, install instructions, and more examples.
  4. Further documentation: links to the full docs and more info.
  5. Project organization: who are the authors, how to contribute, how to file bugs.
  6. Legal notices: license, copyright, and any other legal details.
Here are some examples of great README's:
  1. Twitter Bootstrap
  2. guard
  3. Ace
  4. jekyll
  5. hogan.js
  6. ember.js
I usually practice Readme Driven Development, writing the README before writing any code. This forces me to be clear on exactly what I'm trying to build, helps me prioritize the work (anything in the "sales pitch" is a must-have), and provides a great sanity check on what the basic user experience looks like (the quick example and quick start sections are essential). See the original Readme Driven Development post and The Most Important Code Isn't Code for more info.

1b. Tutorials, walkthroughs, and guides

The README gets the user in the door; the tutorial shows them how to walk around. The goal is to guide a new user through example use cases that highlight the idiomatic patterns, the best practices, and the unique features of the project. Use the tutorial to have a dialogue with the user, walking them through the typical development flow step by step and introducing the key ideas. You don't have to cover every single topic and you don't have to go too in-depth: instead, at each step of the tutorial, provide links to where the user can find more info.

For small, simple projects, you may be able to squeeze a tutorial into the README itself, but most projects will want to use a wiki, a blog post, a standalone webpage, slide deck, or even a recorded video. Here are some great examples:
  1. Ruby on Rails Guides
  2. Django Tutorial
  3. Dropwizard Getting Started
  4. Intro to Play Framework for Java
  5. Twilio quick start tutorials
The gold standard, however, is the interactive tutorial. Most developers learn best by doing, so a step-by-step guide that lets the developer participate is the ultimate learning tool. Here are a few great examples:
  1. A Tour of Go
  2. Scala Tutorials
  3. Typesafe Activator
  4. Try Redis and Redis commands
  5. Try Git
  6. Codecademy
Creating your own interactive tutorial is not easy, but it dramatically lowers the bar for trying and learning about your project. Here are some (language/framework specific) tools you may find helpful: io.livecode.ch, IPython Notebook, java-repl, Pamflet, Typesafe Activator, repl.it, Ace Editor, CodeMirror, Cloud9 IDE, jsfiddle, Codecademy, and codepen.    

1c. Reference documentation

Ok, your new user got their foot in the door with the README and they took a few steps by following the tutorial; now, the user actually knows enough to start asking questions. This is where the reference documentation comes into play: the goal is to give users a way to find the specific information they need. In this part of the documentation, you can cover all the major topics in depth, but make sure to organize the information in a way that is easy to search and navigate.

Here are some great examples of reference documentation:
  1. Stripe docs
  2. Django documentation
  3. Dropwizard user manual
  4. Codahale metrics
  5. SQLite documents
For large projects, the amount of reference documentation can be pretty large. How do you keep it up to date? One technique is to include references to real code: that is, instead of typing code snippets directly into your docs, build a system to dynamically include them from a real repository.

For example, consider this entry in the Play Framework async docs:


This documentation is generated from markdown files using the play-doc project. For example, here is the Markdown for the "Returning futures" section:

Notice that the code snippet is not in the Markdown. Instead, there is just the line @[async-result](code/ScalaAsync.scala), which is a reference to ScalaAsync.scala in Play's git repo, where the relevant code is demarcated using special comments:

Since this file is compiled and tested, developers have to update it whenever they make changes to the framework - otherwise, the build fails. Moreover, as the comments identify the section of code as "used in the documentation", there is a good chance the developers will remember to update the relevant part of the documentation as well.
1d. Project websites

Standalone project websites are a great example of documentation as marketing: you can give your project its own home, with a custom look and feel, and content that is linkable, tweetable, and indexable.

Here are a few great examples:
  1. Bootstrap
  2. jekyll
  3. Yeoman
  4. Ember
  5. Foundation
The easiest way to create a website for your project is with Github Pages: create a repo on Github, put a few static HTML files in it (possibly using jekyll), git push, and you have your own landing page on the github.io domain.

1e. White papers and books


If you want to make a project look legit, a white paper, and especially a book, is the way to go. White papers are a great way to explain the background for the project: why it was built, the requirements, the approach, and the results. Books, of course, can contain the material in all the sections above: a quick intro, a tutorial, a reference guide, and more. Books are a sign that your project has "made it": there is enough interest in it that a publisher is willing to put money into printing the book and programmers are willing to put money into buying the book.

Some great examples:
  1. Bitcoin: a peer-to-peer electronic cash system
  2. Ethereum white paper
  3. Kafka: a distributed messaging system for log processing
  4. C Programming Language
  5. Effective Java

2. Code documentation

We now understand the role of written documentation: the README gets your foot in the door; the tutorial shows you how to walk around; the reference guide is a map. But to truly understand how a piece of software works, you have to learn to read the source. As the author of a project, it is your job to make the code as easy to understand as possible: programs must be written for people to read, and only incidentally for machines to execute.

However, the code cannot be the only documentation for a project. You can no more learn how to use a complicated piece of software by reading the source than you can learn to drive a car by taking apart the engine.


As we'll discuss below, code structure, comments, API docs, design patterns, and test cases all contain critical information for learning how to use a project, but remember that they are not a replacement for written documentation.

2a. Naming, design patterns, and the type system

There is no such thing as "self documenting" code, but there are ways to make the code easier or harder to understand. One of the first aspects of code readability is naming: every piece of software defines its own mini language or DSL that consists of class names, package names, method names, and variable names. When a developer uses your code, they are really learning a new language, so choose the words in it wisely! However, since naming is one of the two hardest problems in computer science, I recommend getting yourself a copy of Code Complete, which dedicates quite a few pages to this topic:


Design patterns are another tool for communicating the intent of your code. You have to be careful not to overuse them (see Rethinking Design Patterns), but having a shared vocabulary of terms like singleton, factory, decorator, and iterator can be useful in setting expectations and making the naming problem a little easier. The classic book in on this topic is Design Patterns: Elements of Reusable Object-Oriented Software, aka "The Gang of Four":


Finally, the type system in statically typed languages can be another powerful source of information. A type system can reduce not only the number of tests you write (by catching a certain class of errors automatically), but also the amount of documentation you have to write. For example, when calling a function in a dynamically typed language, there is no way to know the types of parameters to pass in unless the author of the function manually documented it; in a statically typed language, the types are known automatically, especially with a good IDE.


Of course, not all type systems are equal, and you have to use them correctly (e.g. avoid stringly typed programming) to see the benefits. For examples of powerful type systems, check out (in increasing order of power and crazy) Scala, Haskell, and Idris.

2b. API docs and literate programming

API docs are documentation for each class, function, and variable in your code. They are a fine-grained form of documentation that lets you learn about the inputs and outputs of each function, the preconditions and postconditions, and, perhaps most importantly, why a certain piece of code exists and behaves the way it does.

Many programming languages have tools to generate API docs. For example, Java comes with JavaDoc, which lets you add specially formatted comments to the code:

You can then run a command line utility that generates a webpage for each class with the JavaDoc comment formatted as HTML:


Good IDEs can show API docs automatically for any part of the code:


Some frameworks have special handling for API docs as well. For example, rest.li automatically extracts the documentation from your REST service and exposes it in a web UI. You can use this UI to browse all the RESTful services available, see what resources they expose, what methods and parameters they support, and even make REST calls straight from your browser:


Here are a few nice examples of API docs:
  1. Java API docs
  2. Scala API docs
  3. Stripe API docs
  4. Twilio API docs
  5. Github API docs
  6. rest.li API docs
Literate programming goes even further: the idea is that program logic should be described first in natural language; the code comes second, interspersed amongst the English description where convenient. Instead of organizing programs in a way that's easy for compilers to process (ie, rigid file, folder, and package structure), literate programs should be organized in a way that makes it easier for humans to understand, such as an essay format.

I think literate programming is a great concept, but I'm not aware of any mainstream languages that support it fully. The closest I've seen are projects that use tools like docco, which lets you generate an HTML page that shows your comments intermingled with the code, and feels like a halfway point between API docs and literate programming. Here's an example from Literate CoffeeScript:

There are flavors of docco tailored for specific languages, such as rocco (Ruby), Pycco (Python), Gocco (Go), and shocco (POSIX shell). There is also an extension of docco called Groc, which adds support for a searchable table of contents, handles hierarchies of files and folders, and integrates with Github Pages..
2c. Comments

When used correctly, comments are another important source of information: whereas the code tells you how, comments tell you why. The trick is finding the right balance. Code without any comments can't explain why the program is being written, the rationale for choosing this or that method, or the reasons certain alternative approaches were taken; code with too many comments can often be a sign that the code itself is unclear and instead of fixing the code, the comments are being used as a crutch.

In short: always use comments in moderation and always to explain why.

For the "best" examples of comments, I point you to a hilarious StackOverflow thread: What is the best comment in source code you have ever encountered?

2d. Example code and test code

No matter how good your docs are, you can't force developers to RTFM. Some developers prefer to learn by example - which is a polite way of saying that they like to copy and paste.


Getting the example code right is critical to the success of a project, as many developers will blindly copy and paste it. Your goal is to make as many clean, idiomatic examples available as possible. You may also want to invest extra time with the first few teams that adopt your project to help them write clean code: their projects may become the models for everyone else, so make sure it's a model that's worth following!

Here are some projects with great example code:
  1. Twilio HowTo's and Example Code
  2. Twitter bootstrap examples
  3. Typesafe Activator templates
  4. async.js
  5. Firebase examples
Automated tests are a special case of example code. Tests can be useful as documentation in that they show the expected behavior of the code for a variety of use cases. BDD style unit tests, such as Specs2 and RSpec, even encourage writing test cases as a formal specifications. However, in practice, test code can get tangled up with mock objects, test frameworks, and corner cases, all of which can be a source of confusion if you try to rely on it too heavily as a form of documentation.

Projects with great test code:
  1. SQLite
  2. Apache Lucene
  3. backbone.js
  4. Chromium
  5. jQuery

3. Community documentation

We've talked about written documentation and code documentation; the final piece of the puzzle comes from the people involved with the project and the tools they use.

3a. Project management tools


Most teams use bug tracking software (e.g. JIRA, bugzilla, github issues) and/or project management software (e.g. Basecamp, Asana, Trello). These systems contain a lot of information about the project: what you worked on before, what you're working on now, what you'll work on in the future, bugs found, bugs fixed, and so on.

A few examples:
  1. Play Framework Github Issues
  2. Mozilla Bugzilla
  3. Firefox Roadmap Wiki
  4. Chromium Issues
It's hard to imagine how a TPS report can be useful as documentation, but very often, the discussions over a tricky bug or the requirements gathering before starting a new project contain critical information not available anywhere else. It's not uncommon to come across a bug report or an old wiki page while searching for information about a project, especially if it's an open source project that makes all of this information publicly available.

3b. Mailing lists and Q&A boards


Discussions from Q&A sites like StackOverflow and mailing lists like google groups also come up frequently in search results. Even the best documentation will not be able to answer everything, so cultivating community websites can be a critical part of making software learnable. Over time, these may become some of the most important parts of your project's documentation, as they inherently deal with issues where many developers got stuck.

A few examples:
  1. Play Framework Google Group
  2. Android StackOverflow Tag
  3. Ruby on Rails StackOverflow Tag
This is one area where open source projects shine: being able to instantly find answers by using google is a huge win. That said, for internal/proprietary projects, I encourage you to setup internal mailing lists, maintain an FAQ, and/or install an internal StackOverflow-style Q&A site within your company.

3c. Blog posts, talks, meetup groups


For popular open source projects, some of the best documentation comes in the form of content contributed by the community. For example, blog posts and talks from end users are a valuable source of information, revealing what's really working and what isn't; they are also great marketing, as it makes it clear other people are using project. Even blog posts that completely trash the project can be useful - think of it as a free design review!

If your project is open source, growing a community around it can have a huge pay off. A small investment in "marketing" your project - via good documentation, custom project pages, giving talks, and setting up meetup groups - can yield huge returns in the form of free labor, cleaner code, and better branding.

There are countless great blog posts and talks, so here are a few unbiased, randomly selected links that you should definitely check out:
  1. The Ultimate Guide to Getting Started with the Play Framework
  2. Composable and Streamable Play Apps
  3. The Play Framework at LinkedIn
  4. Play Framework: Async I/O with Java and Scala
  5. Bitcoin by Analogy

Further reading

If you've made it this far, you should now know how, and why, to document your code. I hope you join me in building software that is easier to use and learn.

If you're hungry for more info, I recommend the following resources:

  1. Writing Great Documentation
  2. The Most Important Code Isn't Code
  3. Teach, Don't Tell
  4. Designing Great API Docs
  5. No docs == no product
  6. Pointers to useful, well-written, and otherwise beautiful documentation
  7. If It Isn't Documented, It Doesn't Exist
  8. A beginners guide to writing documentation
  9. Tips for Writing Good Documentation

Thursday, April 24, 2014

Bitcoin by analogy



Hackers are the animals that can detect a storm coming or an earthquake. They just know, even though they don’t know why, and there are two big things hackers are excited about now and can’t articulate why – Bitcoin and 3D printing.

Paul Graham, Wall Street Journal
Bitcoin is the first purely digital, decentralized money. It has been on my mind a lot lately and apparently, I'm not the only one. Paul Graham called it a paradigm shift; Marc Andreessen believes Bitcoin is as big of a technological breakthrough as PCs and the Internet; Ben Bernanke said virtual currencies may hold long-term promise; Chris Dixon is investing millions in it; Google is interested in Bitcoin; Apple is afraid of it. In short, Bitcoin is something you should be paying attention to.

Using Bitcoin is easy: to pay with Bitcoin, you just install an app called a Bitcoin wallet on your computer or mobile device and click some buttons in the app to send money to other Bitcoin users over the Internet:


To receive money, you just give people your bitcoin address, or create a QR code. For example, here's mine:


Understanding what's happening under the hood is a bit trickier. I read a number of articles, but still couldn't wrap my head around many of the details. Therefore, in the spirit of "the best way to learn is to teach", I forced myself to write this blog post as a learning tool.

The post is broken up into a series of questions about Bitcoin:
  1. Validity: is Bitcoin really money?
  2. Decentralized money: how can you have a currency that no one controls?
  3. Decentralized mint: how is money created in Bitcoin?
  4. Decentralized transactions: how do you transfer money in Bitcoin?
  5. Decentralized ledgers: how are transactions recorded in Bitcoin?
  6. Further reading: where can I find more info?
Also, in the spirit of "if you can't explain it simply, you don't understand it well enough", I've tried to make the key Bitcoin concepts accessible to audiences without a programming background. Most sections in this post start with a simple analogy for Bitcoin that involves no tech whatsoever before diving into the tech details.

Of course, I'm a Bitcoin novice myself, so if after reading this you're still confused, or I've made any errors or omissions, please leave a comment!

Validity


Before discussing how Bitcoin works, let's answer a common question: how can Bitcoin be considered money? It was created by a bunch of computer programmers with no government backing; it's purely digital, so it exists only on computers and has no "intrinsic value"; it might not even fit the standard definition of money because it's not a stable store of value and rarely used as a unit of account.

Despite all that, Bitcoin is still used as a medium of exchange: thousands of merchants are willing to accept Bitcoin in trade for real goods or services. Why? Because these merchants see Bitcoin as an effective medium of exchange and they believe that other merchants will feel the same way in the future. Or, to put it another way:
It’s not as much that the Bitcoin currency has some arbitrary value and then people are trading with it; it’s more that people can trade with Bitcoin (anywhere, everywhere, with no fraud and no or very low fees) and as a result it has value.

Marc Andreessen, Why Bitcoin Matters
Decentralized money (analogy)

Yap is a nation in the South Pacific that uses a unique form of money called Rai stones, which are circular disks carved out of limestone, with a hole in the middle, that can be up to 12 feet in diameter and weigh up to 8,800 lbs!


Trading stones of this size is difficult: no one wants to cart around a 4 ton stone every time they make a purchase. As a result, the Yapese came up with a clever solution: they decided to determine ownership by verbal agreement. Whenever there was a trade, the parties involved would communicate to the rest of the tribe the amount of stone that had been exchanged. The stones wouldn't actually move from one house to another, but the knowledge of who owned what was memorized and handed down through oral history.


At this point, I must apologize to historians: in the rest of the post, I'm going to completely reinvent Yap history so I can use it as an effective analogy for understanding Bitcoin.

As a first step, imagine that a tribe on Yap struggled to accurately track Rai ownership purely through memory and oral history. After many arguments and fights over who owns what, they decided they were going to write down every transaction in a book called a ledger, which would be considered the source of truth.

The chief appointed one of the tribesmen as the bookkeeper. If Alice wanted to pay Bob 10 lbs of Rai, Alice would go to the bookkeeper's house and announce the transaction. The bookkeeper would go through the ledger, make sure Alice actually owned 10 lbs of Rai, and if she did, add the new transaction to the book.

Everything worked well for a while, but gradually, problems appeared: the bookkeeper started charging transaction fees; trade would sometimes halt entirely because the bookkeeper was on vacation or sick;  pressured by the chief, the bookkeeper would charge very high fees or completely ban certain transactions, especially with other tribes; sometimes, after a dispute, the bookkeeper would seize someone's assets entirely. Eventually, the bookkeeper became one of the most rich, powerful, and controversial figures in society: despite rumors of corruption, fraud, and favoritsm, who would dare question the person who controls all the money?

10 families in the tribe, upset with the bookkeeper's behavior, decided to find a new way to manage their money. Since a single person cannot be trusted to maintain the ledger, these families had a radical idea: every family would maintain its own ledger!


For example, if Alice wanted to to pay Bob 10 lbs of Rai, Alice would go to the center of town and announce the new transaction to all the other families. Each family would then check their own ledger, make sure Alice really had 10 lbs of Rai, and if she did, add the new transaction to their ledger. Since each family now kept a ledger, no one family had more power than any other!

Decentralized money (tech details)

All modern currencies, such as the US Dollar and Euro, are centralized: that is, they are controlled by a small number of institutions, such as governments, banks, and credit card companies. For example, if Alice wants to transfer $10 to Bob, she'll login to her bank's website, wait for the bank to verify her identity, send the transfer information to the bank's servers, wait a few days for the bank to clear the transaction, at which point the bank will update her balance, and send the info over to Bob's bank. In short, even though it's Alice's money, the entire process is controlled by the bank's rules and procedures:


This centralized approach has the same problems as the Yap bookkeeper: even though it's your money, a small group of institutions controls almost everything about it, including who can spend money, when they can spend it, what they can spend it on, where they can spend it, what fees and taxes are imposed, and so on.

Bitcoin offers an alternative without all of these limitations: a decentralized currency. As in the Yap analogy, Bitcoin uses a distributed ledger approach. Of course, instead of 10 families, Bitcoin consists of thousands of computers, each of which maintains its own ledger; and instead of someone yelling from the town center, these computers communicate with each other by sending messages over the Internet.

For example, if Alice was transferring 10 Bitcoins to Bob, she'd click some buttons in her Bitcoin wallet, and it would broadcast this transaction to all other Bitcoin users:


This collection of computers is known as the Bitcoin network and anyone can join it by installing the Bitcoin software, which will automatically download the global ledger onto your computer. This allows you to see every transaction that has ever happened in Bitcoin's history! You can even browse them online on sites like blockchain.info and blockexplorer.com. This is a very different model of privacy than you get with banks, so you may want to check out Bitcoin privacy and Is Bitcoin anonymous for more info.

When each of these computers - I'll refer to them as "nodes" to indicate that they are running Bitcoin software that automatically handles all of these tech details - receives Alice's message, it'll check its ledger to make sure Alice owns 10 Bitcoins, and if she does, add the new entry to the ledger.

Decentralized mint (analogy)


In a decentralized system, there is no mint to create money. So how do more Rai stones enter circulation? The answer is mining!

There is no limestone on the Yap islands themselves, so the villagers have to sail to distant islands to find it. The limestone is scarce and randomly distributed, so finding it can take a long time and is mostly a matter of luck. If Alice mines some new stone, she can get all the villagers to recognize that she now owns more Rai by displaying the new stone in front of her home and announcing it to the other villagers. All villagers will enter this in their ledgers as a transaction where Alice is the recipient of some amount of Rai stone.

Decentralized mint (tech details)



How do Bitcoins get created? The answer here is also mining!

However, Bitcoin mining does not involve any pick axes: it's purely digital. Any Bitcoin node that solves a computationally expensive math problem gets Bitcoin as a reward! Much like the stone mining, these math problems can take a long time to solve, and which node solves it first is mostly a matter of luck. However, once some lucky node finds a solution, it will broadcast it out to the rest of the Bitcoin network, and all other nodes will record in their ledgers that the lucky node has earned some new Bitcoin.

The problems to solve involve cryptographic hash functions; the math behind them is beyond the scope of this post, but I'll give a brief introduction.

If you pass a string of text through a cryptographic hash function, it will convert it to a different string of text, called the "digest", in a totally unpredictable manner:

cryptographic-function("Hello World") = 124610xktj1l32kxjcj24j1
Given the slightest change to the text, such as adding an exclamation point, you get a totally different digest as output:

cryptographic-function("Hello World!") = 444lkxckl15lsck51lk234
The most important feature of cryptographic hash functions is that they are "one way": for a given string T, you'll always get back the same digest D; however, given an arbitrary digest D, there is no way to figure out what was the original text T. In other words, there is no way to "reverse" the hash function.

In Bitcoin mining, you pass two pieces of data into the SHA-256 cryptographic hash function:
  1. Information about a block, B: we'll discuss the details of this later
  2. A random guess, R
sha-256(B, R) = digest
The goal is to find the right R so that you get back a digest that starts with some required number of leading zeroes:

sha-256(B, R) = 00000000000001234sdfxxc1414...
Since cryptographic hash functions are "one way", there is no way to know what value(s) of R will produce a digest that starts with zeroes. All you can do is repeatedly guess random values of R until you accidentally stumble across one that works. Since SHA-256 has 2256 possible outputs, as the number of required leading zeroes goes up, the odds of any one guess being right becomes extremely small.

In fact, the problem is intentionally designed - and occasionally recalibrated (see Blockchain stats) - to take a very long time: a single computer will have to guess non-stop, on average, for several years to find the right value of R. However, with all the nodes on the Bitcoin network guessing, the average time to find the right value of R is roughly 10 minutes.

Spending so much CPU time and energy on useless calculations may seem wasteful, but as we'll see later, the fact that the calculations are expensive is essential in establishing a consistent timeline.

A few interesting notes on Bitcoin mining:
  1. There is pre-determined, fixed supply of Bitcoin: every year, the number of Bitcoins that can be mined will drop by half and the total supply will max out at 21 million.
  2. After all Bitcoins have been mined, the reward for mining will switch to small fees on each Bitcoin transaction. These are expected to be significantly smaller than bank or credit card fees.
  3. The number of nodes participating in Bitcoin mining today means it is not practical or cost effective to try to do mining on your home computer. Instead, the recommendation is to join a mining pool and even invest in dedicated hardware (see bitcoinmining.com for more info).
  4. The fixed supply of Bitcoin is not a problem as you can pay with tiny fractions of a Bitcoin, all the way down to 0.00000001 BTC, known as a Satoshi.
Decentralized transactions (analogy)


Each of the 10 families maintains a ledger by hand that lists every transaction that has ever happened. Each transaction consists of 3 sections:
  1. From: the name of the person sending money.
  2. To: the name of the recipient of the money.
  3. Amount: how many pounds of Rai stone to transfer.
To find out Alice's balance, you start at the beginning of the book and go through every transaction, adding up transactions where she received money, and subtracting transactions where she sent money. When Alice announces a new transaction, before adding it to the book, you can use this balance calculation to make sure she has enough money for the transaction.

Decentralized transactions (tech details)

The Yap transactions contained three pieces of data: From, To, and Amount. With Bitcoin, the To and From fields are tricky: we can't just look to the town center to see who is announcing a new transaction and in a decentralized world, there is no central system to store usernames and passwords.

To manage and verify identity in a decentralized fashion, Bitcoin uses public key cryptography. Public key cryptography involves lots of math that is beyond the scope of this post, so this section is just a brief intro to make it clear how it applies to Bitcon.


In this form of cryptography, there are two keys, or long strings of letters and numbers, that are mathematically linked:
  1. Public key: a public identifier that can be freely shared with others.
  2. Private key: a secret or password that must never be shared with anyone.
When you install a Bitcoin wallet on your computer, it will automatically generate a public and private key pair for you. You can freely share your public key: in fact, the public key is your identity or address in Bitcoin.

Public/private keys can be used for several tasks, but the main one we care about is "authenticity": that is, we can use them to mathematically verify that a message really came from the person we expect and that the message contents have not been modified along the way.

Every time Alice sends a message, she can pass the contents of the message, along with her private key, through a sign function:

sign("Hello World", Alice's private key) = n67n54n6l10xf15
The value she gets back, called a digital signature, is a completely unrecognizable string of letters and numbers. Without Alice's private key, there is no way to guess what that string should be. Moreover, if you change the input text at all, even by a single character, such as adding an exclamation point to "Hello World", the signature will change in totally unpredictable ways:

sign("Hello World!", Alice's private key) = vk34jxl140501025
This is similar to the cryptographic hash functions we saw earlier, except that the mathematical link between the private and public keys means Bob can use Alice's public key to verify the signature:

verify("n67n54n6l10xf15", "Hello World", Alice's public key) = valid or invalid
If the signature is "valid", then Bob can be confident that it was really Alice who sent the message and that the message is exactly as she originally created it.

We now know enough to take a look at a typical Bitcoin transaction. For example, if Alice was sending 10 Bitcoins to Bob, the message might look something like this:

Signature:mn546yhg (signed with Alice's private key)

Inputs:nhn3891a (transaction where Alice got 7 BTC)
vc4232v32 (transaction where Alice got 3 BTC)

Outputs:To: 60sdfs951sdfxo66 (Bob's public key)
Amount: 10 Bitcoins

The message consists of 3 sections:
  1. Signature: Alice includes a digital signature with her messages so that other Bitcoin nodes can verify the message really came from her.
  2. Inputs: this is a list of the signatures of transactions already in the ledger where Alice was the recipient of Bitcoins. In other words, these are the funds Alice is using in this transaction, a total of 10 Bitcoins.
  3. Outputs: this is a list of how the funds in the inputs should be distributed. To keep calculations simple, you are required to redistribute all the funds in the inputs. You can include more than one recipient in the ouputs section - including yourself, if you need change. In this case, Alice is sending 10 Bitcoin to a single recipient, Bob, identified by his public key.
Since each transaction references a previous transaction in its "inputs" section, it is possible to follow the graph of transactions all the way back to the beginning of Bitcoin. This is the mechanism for checking the ownership of bitcoins!

For example, to calculate a user's balance, we use an approach very similar to the Yap ledger: you go through every transaction, add up the ones where the user was a recipient, and subtract the ones where they were a sender. To check that a new message is valid, such as Alice's transfer to Bob, you can check that the inputs refer to valid transactions already in the ledger where Alice was the recipient.
Decentralized ledgers (analogy)

Over time, the number of families using the distributed ledger system grew from 10 to 30. The homes for the new families were spread out, so instead of one town center, there were now 3 town centers, one for every 10 houses. Therefore, when Alice wanted to transfer 10 lbs of Rai to Bob, she had to announce the transaction 3 times.


Eventually, the villagers noticed that this led to a problem: the order of transactions was not consistent across all ledgers! This could lead to problems. For example, Alice wants to transfer 10 lbs of Rai to Bob. She announces it at village center #1 and heads off to announce it at the other village centers. Bob, who lives near village center #1, hears Alice's announcement and, excited to finally have some money, immediately starts his own transaction to transfer 10lbs of Rai to Carole. Bob announces his transaction at village center #1, where all the families now have the follow transaction order:
  1. Alice -> Bob, 10 lbs Rai
  2. Bob -> Carole, 10 lbs Rai
Bob then heads off to village center #2 to make the same announcement. However, it turns out that Alice went to village center #3 first and hasn't made it to #2 yet. As a result, the families in village center #2 have the following transaction order:
  1. Bob -> Carole, 10 lbs Rai
  2. Alice -> Bob, 10 lbs Rai
The ledgers in different villages centers are out of sync! Moreover, since Bob had no money before Alice's transaction, the families in village center #2 reject his transaction, even though the families in village center #1 accepted it!

We need a way to establish a consistent order of transactions in all ledgers in a decentralized way. For example, simply letting a single family choose the order wouldn't work since that family could use that power to their advantage (e.g. by constantly delaying a rival family's transactions). This is a hard problem, but the ingenious villagers once again came up with a clever solution: use random chance!

Here's the idea: when a new transaction comes in, the first step is to add it to an "unverified" transactions list. For example, when Alice announces her transfer of 10 lbs of Rai to Bob, the ledger and unverified lists would look like this:


Every time a family mines new Rai stone, to get all the families to recognize the new stone in their ledgers, this family must pick one transaction to move from the unverified list to the ledger.

This mechanic accomplishes several goals at once:
  1. Since limestone is scarce and randomly distributed, it's a matter of luck which family will get to "verify" the next transaction, so no family can control transaction order to their advantage.
  2. Every family now has a strong incentive to participate in maintaining ledgers: it's the only way that their newly mined limestone will be allowed to enter circulation!
  3. Since new limestone is randomly distributed and takes a long time to find, the odds of two families overlapping in finding new stone are essentially zero*. This give us a consistent ordering for transactions: in the case above, either all village centers accept Bob's transfer to Carole or all of them reject it.
* To be honest, point #3 is not entirely accurate: an overlap would be rare, but could still happen. Unfortunately, this is one place where I can't find a simple analogy that captures the math Bitcoin uses to handle these occasional overlaps. Nevertheless, the main point still holds: a random, unpredictable process means no family will be able to control the order of multiple transactions in a row.

Decentralized ledgers (tech details)

The naive strategy of immediately adding new transactions to the ledger didn't work for the Yap when they got too big; since Bitcoin is several orders of magnitude bigger (thousands of nodes, multiple transactions per second), the strategy definitely doesn't work for Bitcoin. So how does Bitcoin establish a consistent timeline amongst distributed ledgers?

It turns out that this is a general problem in distributed systems research known as the Byzantine Generals Problem. As we'll discuss next, Bitcoin offers the first practical (read: probabilistic) solution to this problem - see Bitcoin & the Byzantine Generals Problem for more info.

The ledger in Bitcoin is called the block chain. Here's a rough idea of what a block chain might look like:


The block chain consists of a series of blocks (3 are shown above), where each block contains:
  1. Transactions: transactions or messages sent between users.
  2. Proof of work: this is the digest from Bitcoin mining!
  3. Previous reference: a reference to the digest of the previous block.
Notice how each block has a reference to the previous block: this chain of references is what defines the timeline in the Bitcoin network. The transactions in a single block happened "at the same time" (there must be no dependencies between them); the transactions in previous blocks happened earlier. This is different than the Yap ledger, where order is implicit from the order the transactions are written in the ledger.

You can follow the "previous references" from block to block, all the way back to the very first block, known as the genesis block. The genesis block was part of the seed data when Bitcoin was first created; how do all other blocks get added in a consistent order across all computers? Just as we discussed in the Yap analogy section, when new messages arrive at a Bitcoin node, they initially go into an "unverified" bucket:


Any node in the Bitcoin network can put several unverified transactions into a block and send it out to the rest of the network as the proposed next block in the chain. The catch is that the proposed block must include a "proof of work", which is the solution to a computationally expensive math problem involving cryptographic hash functions. Sound familiar? That's right, this is Bitcoin mining!

Just like the Yap families propose the next transaction when they mine new Rai stones, it is the Bitcoin miners who propose new blocks for the block chain when they mine new Bitcoin. Here are the rules: take all the text from several unverified transactions T, plus the digest of the most recent block in the ledger D, plus a random guess R, and do the following SHA-256 calculation:

sha-256(T, D, R) = digest
The miners keep guessing different values of R until they find a digest with the required number of leading zeroes. The first miner to find it gets a reward of Bitcoin: to receive it, the miner must send out the new block, which includes the digest as the "proof of work", to all other Bitcoin nodes. Assuming the new block is valid, it becomes a part of the block chain:


In the example above, block 54 is now part of the block chain - that is, the Bitcoin timeline - and all the transactions in it, including Alice's, are considered "verified".

What if multiple nodes come up with a proof of work at the same time? This is a rare occurrence, but if it happens, the network will temporarily have multiple possible paths in the block chain:


The solution is simple: Bitcoin nodes always accept the longest available chain.

In the example above, some parts of the network will be mining a new block that has 56 as its previous reference, and others will be mining a new block with 57 as its previous reference. Eventually, someone will complete a proof of work on one of these paths, making it the longest one. For example, if the first proof of work calculation to finish was for a new block on the block 57 path, the network would switch to this path, and the transactions in block 56 would get put back into the unverified bucket:


Of course, it's possible that two blocks, one on each path, will be found simultaneously again, but a) this is even more unlikely and b) it just means that the block chain stays diverged for a little while longer while we wait for yet another block to be found. Eventually, some path will end up longer, and the network will converge on it.

Since nodes always accept the longest path, couldn't an attacker create their own block chain with lots of fraudulent transactions and get the whole network to adopt it, so long as it was longer? For example, if Mallory managed to generate blocks 59, 60, and 61 while the network was still working on 57 and 58, then Mallory's fraudulent blocks would be accepted and all the others would be dropped:


Attacks of this sort are very unlikely to succeed because Mallory is in a race against the entire Bitcoin network to generate those blocks. This is why the proof of work calculation is intentionally designed to be very expensive. Mallory would have to control more computing power than all other nodes on the Bitcoin network, combined, to have a viable chance of winning this race for a single block, let alone 3.

This is the real reason Bitcoins are offered as a reward to miners: they are an incentive to make the network as big and active as possible, so an attacker has no chance of being faster than the rest of the network. In fact, most attackers will find it more profitable to use their computing resources to mine bitcoins legitimately instead of taking the risk of trying to add fraudulent blocks to the block chain.

The odds of an attacker succeeding get even smaller as time goes on. For example, if you want to tamper with a transaction in block 51, you'll have to recalculate the proof of work for that block; since the proof of work calculation for block 52 includes the proof of work from block 51, you now also have to recalculate the proof of work for block 52 as well; and then block 53; and in the meantime, you're racing the rest of the network, which has since added blocks 54 and 55, so your chain is not the longest!

The implication here is that older blocks - those further back in the chain - are more "secure" than newer ones.

In fact, if Bob is a merchant, he can tune the level of risk he's willing to tolerate by deciding how many blocks must elapse before he considers a transaction verified. If it's a tiny transaction, a single block (~10 minutes) may be enough; if it's a large transaction, waiting for 6 blocks (~1 hour) may be more advisable. For a merchant, waiting 1 hour to avoid fraud may still be better than the situation today with credit cards, where a chargeback may appear a month after the transaction.

An important disclaimer: while Bitcoin's design seems sound from a security standpoint, it's still susceptible to fraud as a result of user error, just like any other system. The difference with Bitcoin is that it's a decentralized system: when something goes wrong, there is no one you can call for help.

For example, if you accidentally send Bitcoin to the wrong address, there is no way to get that money back. If you fall for a phishing attack, there is no fraud department to report it to. If you lose your private key (e.g. in a hard drive crash), you lose access to any Bitcoins associated with it, and there is nothing anyone can do to help you get them back. In fact, if a private key is lost, those Bitcoins are no longer accessible to anyone on the Bitcoin network; they are gone, effectively out of circulation, forever!

Further reading

In the spirit of giving credit where it's due, these are the resources that helped me put this post together, and may help you learn more: