Shit recruiters say

Just today, I got my 1,000th recruiter email in the last ~4 years. I know this is the number because I'm slightly obsessed with organization and have added a specific label to each and every recruiter email I've gotten in my gmail account. Let's see: 365 days per year, minus 104 weekend days and, say, 10 vacation days, gives us roughly 250 working days per year.

That means that, on average, 1 recruiter emails me every single business day ^[1].

There are a few take aways from this:

1. Being a software engineer is awesome ^[2]. We don't apply to jobs, they apply to us. 
2. If you are a recruiter, you're up against some serious competition.
3. I've seen recruiter emails from every tech company you can think of. Some are comically bad.

In this blog post, I'm going to share snippets from a few of the bad ones. Enjoy!



[1] This doesn't even include the sketchy recruiters who track down my phone number and call me. If you do this, everything you say is shit.

[2] I suspect that if kids had any idea how amazing it is to be a programmer in Silicon Valley, all of the problems US schools have with math and science would vanish overnight.

---

Subject: Follow up

Hi Yevgeniy,

I’m not assuming you’re looking for a new roll right now but I wanted to follow up and gather your feedback.

---

Wrong! I could totally go for some sushi.

---

Subject: Opportunities at XXX

Hi Yevgeniy,

[...]

We are challenged every day, moving very quickly, and recruit only rock stars.

---

Man, I really wish I could join your band.

---

Subject: Quick question (Developer opportunities)

Yevgeniy,

Good day. Hope all is well.

I am reaching out to you because of the recent severe reorg and layoffs at your company I read about.

---

Wow, that escalated quickly.

---

Subject: Hello from XXX

Hello <name>,

My name is XXX -- I'm the Director, Engineering of the Growth Team at XXX.

---

Yes, this email literally said <name>. Umm, no thanks <company>.

---

Subject: Full time Scala opportunity

Hi Yevgeniy,

Currently, we are looking for an, "Engineer" to help deliver a range of new products. High-performance, high-scalability, high-traffic, heavy Scala, etc. Any interest in having a casual chat about this role...a few minutes...you have an interesting background?

---

This email from a, "recruiter", has, an interesting... way, with words?

---

Subject: principal java software engineer


Hi Yevgeniy,

happy valentine's day!
hope this new year has provided time for some fun activities.
our family started off the new year attending a terrific concert a couple of weeks ago. the performance reinforced the experience of music as a true combination of four creative forces (composer, conductor, musician and audience) in communicating ideas, thoughts & feeling. we were fortunate to hear pieces from Wagner, Beethoven, Mendelssohn, Leonard Bernstein to Holst. an awesome introduction to the symphony for kids!

if you might have time for a quick chat, i'd like to share a principal-level swe opportunity:

---

Worst valentines day card ever.

---

Subject: looking for a great technical dev manager!


Hi Yevgeniy,

I think this role may not be exactly up your ally, but I am hoping you can recommend some great folks for us!

---

I don't want anything up my ally, thank you.

---

Subject: Getting back in touch. Kleiner Perkins and Sequoia portfolio companies.

The portfolio company that I'm most excited about is financed by Greylock and Reid Hoffman who founded PayPal and LinkedIn, which as you know is one of the high-profile IPOs of 2011.

---

Ah, yes, Link-ed-in... I may have heard of it.

---

Subject: Stealth startup

[..]
The service will be disruptive and impact billions of technology users.

---

Billions!

Some ground rules for debate

For some reason, you've gotten into a debate with me about philosophy, software, psychology, history, life, the universe, and everything. Awesome!

Before we go any further, let's lay down a few ground rules to help ensure that we don't waste each other's time.

Rule 1: cogito ergo sum

If we debate long enough, it's possible that we'll become unhinged from reality and start to question everything. Perhaps the world is not what it seems. Maybe our senses are lying to us. What if this is all a dream? What if you're a figment of my imagination or I'm a figment of yours?

If we get to this point, we need to stop immediately, and back up.

All arguments boil down to just a single truth:

I think, therefore I am. - René Descartes

Unfortunately, this is not a particularly useful result. If we're going to have a debate, we need more than that.

I am, therefore I think

Even though I can't prove it, I'm going to assume that there is some sort of real world out there; that there isn't a robot deity keeping my brain trapped in a computer simulation; that my senses aren't lying to me all the time. I'm going to assume that the world is, more or less, what it seems, and our role is merely to do our best to adapt to it.

If we can't agree to these axioms, then we shouldn't waste any more time debating. If "I think, therefore I am" is the only thing you're willing to believe, then we won't make it very far anyway. After all, if everything is a lie, what could we possibly gain from a debate?

Rule 2: theories and models

Although I'll rely on some axioms as the basis for my reality, most of my claims will come from theories. I put this word in italics because it needs some explanation.

Outside of a scientific context, the word theory often means that something is uncertain or unproven. For example, people like to criticize the theory of evolution by saying "it's only a theory." 

This is an unfortunate mix-up because, in science, theory means something different. A theory is an explanation based on observation and experimentation that can be used to describe and predict something. 

In other words, a theory is a tool that you use. It is a model of reality that can be used to make predictions; whether or not the theory corresponds to some underlying truth about that reality is (almost) irrelevant.

In fact, I'll go one step further: 

All models are wrong, but some are useful. - George E.P. Box

Yep, all theories are wrong. And that's OK. For example, we know that the theory of gravity is wrong (see: general relativity); we know that Newton's laws of motion are wrong (see: special relativity); in fact, most of classical physics is wrong (see: quantum mechanics); there is a whole wiki page of superseded scientific theories. Nevertheless, all of these theories have been and continue to be incredibly useful, providing the essential tools for building airplanes, bridges, computers, spaceships, and understanding the universe.

The goal of a theory isn't to be right; after all, you can never really be right, as per rule 1. The goal of a theory is to be useful. The important question is not "is this theory correct" but rather, "does this theory let us make predictions about the world better than other theories"?

For example, the theory of evolution by natural selection lets us make some good predictions about how plant and animal species came to be and what will happen to them in the future. It may be only a theory, but it's a very useful one. On the other hand, intelligent design doesn't match our observations, isn't useful for making predictions, and therefore not a theory.

So, when you and I debate, it's not worth debating axioms, so all we can really do is debate theories. And we won't debate them based on the merit of whether they are true or not, but whether they are more useful than other theories.

Rule 3: the most important question

One final step before we can have a debate: you need to ask yourself a question. It's arguably the most important question in all of science, reason, and debate:

What would convince you that you're wrong?

Be honest. What facts, evidence, or events would convince you that your current stance in the debate is wrong? This question is the very basis of the scientific method.

If there is not a single thing in the world that could convince you that you're wrong, then your stance isn't a theory, but an article of faith. And faith cannot be debated.

What would convince an evolutionary scientist that the theory of evolution is totally wrong? That is, what would show that the theory of evolution is not a useful model? I can think of a few examples: discovering evidence that the earth was just a few thousand years old; observing a new species magically materialize with no connection to any other species; finding evidence that the entire fossil record is an elaborate hoax by the liberal media.

What would convince someone who believes in intelligent design that they are wrong? As far as I know, nothing. Apparently, numerous observations of natural selection in progress, mounds of data showing the earth is over 4 billion years old, an extensive fossil record, and countless examples of unintelligent design are not enough. This is yet another reason intelligent design is not a scientific theory and should not be taught as one in schools.

As long as a debate is about faith instead of theory, it is largely pointless, since there is nothing that can change either party's mind. This is why it's so painful to debate topics like abortion, conspiracy theories, politics, and religion.

Therefore, I propose that we only have a debate if both of us can identify, at least to ourselves, that there is at least one piece of evidence that could convince us that we're wrong.

Alright, let's do this

Made it this far?

Good.

Because I'm ready to put on a clinic.

What my grandfather taught me about happiness

Boris K.
1919-2013

This is the eulogy I gave for my grandfather on March 14, 2013. He was 93 years old. 

I want to share with you something I learned from my grandfather. It may sound a bit odd at a time like this, but I want to tell you guys what I learned from him about happiness.

Here’s the thing: my grandfather had a tough life. He went through war, communism, poverty, emigration, and somehow, he came out the other end happy, kind, and loving.

His happiness showed through in every conversation.

For example, he spent over 20 years in the Red Army, and not by choice; his original plan had been to study history in college, but when WWII broke out, he was drafted. But if you asked him about it, the story he would tell would not be about the misery of military life, but rather about the life-long friends he made in the army and the discipline and strength he learned.

He fought in some of the biggest battles of World War II, including Stalingrad and Kursk, and was wounded several times. But if you ask him about that, he’d mostly smile as he told you about the pretty nurses he met while recovering from his wounds. He’d also tell you that he was wounded in the leg... though the photographs clearly show a bandage on his head... He’d just laugh about that too.

After fighting the Nazis for years across all of Europe, he ended up being part of the offensive that took Berlin. There is even a photo of him in front of the fallen Reichstag. Ask him about that and he'd tell you how much he enjoyed living in Berlin. With a German family. And how hospitable and friendly they were.

He lived through some of the toughest years of Communism in the Soviet Union, sometimes at near poverty levels in horrible communal apartments. But if you ask him about that, he'd tell you about his circle of friends, he’d go on about how Riga was beautiful, and he’d tell you about going to theater every night. 

I found his attitude astonishing, but wonderful. Life was never easy for my grandfather, but he never complained. Whereas I freak out when my iPhone loses signal for a minute and I can't check my email; and then freak out again because I can't use twitter to complain about the signal loss.

So what was his secret?  

I think my grandpa realized, perhaps subconsciously, that you can’t just be happy. You can be tall or you can be fat or you can be strong, but happiness is different.

Happiness is not something you have, it’s something you do.

My grandfather was happy because he always focused on the things he loved. That’s what he did. And when you do what you love, you do it well, and you’ll be happy.

For example, in the army, he was a medic. I cannot imagine the daily horrors and stress of a job like that. But my grandpa was able to focus on the parts he loved: learning medicine, helping people, and working with a team of doctors he admired. My grandpa became an excellent medic. 

Later in life, in his 40’s, he went to law school and began studying law. That’s not an easy career change at that age, and not an easy job at any age. But he never complained; in fact, he’d always light up when talking about being an attorney, about the amazing people he got to work with, and the perks of the job - you could tell he loved it. It’s no surprise he spent the next 30 years as a successful attorney.

Now, this isn’t just blind optimism; there is science behind it. There is research that indicates that if you just change your body language - force a smile on your face, assume an open, comfortable posture - after a few minutes, your cortisol levels drop, testosterone levels go up, and you feel better. You feel happier.

It’s also possible that how you react to events - in fact, the very words you use to think about those events - has a dramatic affect on your mood. When something bad happens, some people react by starting to curse, yell, frown, shake their hands, complain... Other people just look around, nod, and say “that’s... mildly inconvenient”. And just like that, the entire problem doesn’t seem so bad.

It’s counter-intuitive, but the way you think about things and your verbal and body language are not just a reflection of your emotions, but an active cause of those emotions. It’s a two way street.

By choosing to focus on the things you love, you aren’t just pretending to be happy, you really are happy. My grandpa’s happy memories of living in communist Riga or even post-war Berlin weren’t delusions: they were genuine happiness.

Happiness is not something you have, it’s something you do.

This doesn’t mean my grandfather totally blocked out all the bad memories. He wasn’t in denial. Years ago, I was watching Saving Private Ryan, a violent WWII movie, and my grandfather, who never watched any American movies, happened to walk by the room and watched for a minute. After a drawn out battle scene, he said “I’ve seen this before... it’s not that great”, shrugged, and moved on.

He remembered the war, both the good and the bad. But his reaction was of the “that’s... mildly inconvenient” variety, which made the bad, even the awful, tolerable. And then, of course, he’d be able to focus on the good. In fact, he was proud of what he did in the war - we all were. V day was one of the most important days of the year for him and always called for a celebration.

My grandpa had other passions that made him happy. He loved to go on walks. Every day, even multiple times per day, rain or shine, he’d go out on a walk. Walks made him happy, or maybe being happy made him walk, but either way, he got good at it. He even walked the dogs of family friends, partially because he loved dogs and partially because dogs were the only ones that could keep up with him on a walk.

He also loved a good conversation and became quite good at that too. Every family dinner would start with the a toast from my grandfather: he’d stand up, raise his glass, and inspire everyone with a few well thought out sentences. Sometimes he’d throw in a joke or even a poem. And he could talk to anyone. Language barriers didn’t matter: Russian, Ukranian, German, Yiddish, even learning a little English at an old age. The language of kindness and a smile is universal.

And of course, he loved his family. I think my mom turned out pretty well, as did the grandkids, so I guess he did a good job there too. I like to think we made him happy.

So, that’s the secret. Happiness is not something you have, it’s something you do. No matter what is happening in your life, actively focus on what you love and you’ll be happy. I think on a day like today, this is more important than ever.

So, as you remember my grandpa, do what he would have: remember the things that are good. Remember the stories he told you, the laughs you shared, and the amazing life he led. Better yet, share those stories with someone you love, while out on a nice, long walk.

MySQL error: last packet sent to the server was XXX ms ago

I just spent a few weeks battling a strange, infrequent, hard-to-reproduce error when using JDBC to talk to MySQL. After about a dozen experiments, I think I've finally found a solution and I've decided to capture the details here, since my online searches didn't turn up this particular answer anywhere else.

tldr: If you see a "last packet sent to the server was XXX ms ago" error, you may want to upgrade your version of the mysql-connector-java library.

The Symptoms

I had a simple Java app in production that was using JDBC to talk to a MySQL DB. Everything was running great: DB calls were taking 2 ms on average and 8 ms in the 99th percentile. However, once every 4-8 hours, a strange error would pop up that looked something like this:


I can understand the occasionally slow query, but 28800126 ms? EOFException? What's going on here?

Lots of ineffective options

As usual, I turned to a programmer's two best friends: Google and StackOverflow. I quickly found my way to the MySQL docs and found out that MySQL has two timeout settings that will close a connection if it is idle for too long: interactive_timeout and wait_timeout. The default value for these two settings is 28800000 ms or 8 hours.

The general advice online was to make sure that your connection management library was sending periodic "keep-alive" queries to prevent connections from going idle. I was using BoneCP, so I tried everything I could to make it behave properly, including a few configuration tweaks as well as workarounds for a connection leak bug and a releaseHelperThreads bug. Nothing worked.

Eventually, I swapped out BoneCP entirely for a different connection management library. Nevertheless, after a few hours, the dreaded "last packet sent to the server was XXX ms ago" error would pop up on the production box.

The solution at last

For a while, I was at a loss. I couldn't see how two entirely different DB connection management libraries could have the same bug. I began digging for what the two had in common and realized that, under the hood, both would be using the same JDBC driver. For MySQL, this is Connector/J.

It's at this point that I noticed that I was, for some reason, using Connector/J version 3.1.12, which is quite old. In fact, it is officially obsolete and only compatible with MySQL 5.0 and below. This is unfortunate, as I was using MySQL 5.5 in production.

I figured it was a long shot that this was the cause of the errors I was seeing, but I figured that using the "recommended" connector version was a good idea anyway. I updated from mysql-connector-java version 3.1.12 to version 5.1.22.

And just like that, all the errors were gone.

The final word

So, there you have it.  If you see a "last packet sent to the server was XXX ms ago", it's likely one of two things:

1. Your DB connection management library is leaving idle connections open too long
2. You're hitting an incompatibility bug between the Connector/J version and the MySQL DB version

Seven Languages in Seven Weeks: Erlang, Day 2

After learning some basic Erlang syntax on Day 1, I take on the second Erlang chapter, which introduces some more interesting concepts.

Erlang, Day 2: Thoughts

I'm finding it very easy to dive into Erlang. After going through the Prolog and Scala chapters of this book, as well as making heavy use of Scala at work, the functional constructs used in Erlang feel natural. I've grown very fond of pattern matching in the last few months and have found it to be a very powerful tool for expressing complex concepts in a very concise and readable manner. Erlang's heavy reliance on pattern matching makes me happy.

However, the syntax does feel slightly clunky: I constantly forget to end lines with dots and separating clauses of control structures with semi-colons gets annoying. I suspect this is something you get used to. Moreover, the end result, at least in the dead-simple code snippets I've looked at so far, is pleasantly readable.

Erlang, Day 2: Problems

List lookup

Consider a list of keyword-value tuples, such as [{erlang, "a functional language"}, {ruby, "an OO language"}]. Write a function that accepts the list and a keyword and returns the associated value for the keyword.

My implementation:


Shopping list price

Consider a shopping list that looks like [{item, quantity, price}, ...]. Write a list comprehension that builds a list of items of the form [{item, total_price}, ...] where total_price is quantity times the price.

My implementation:

Sample usage:


Tic-tac-toe

Write a program that reads a tic-tac-toe board presented as a list or a tuple of size nine. Return the winner (x or o) if a winner has been determined, cat if there are no more possible moves, or no_winner if no player has won yet.

My implementation:

Sample usage:

My first tic-tac-toe solution was a bit more complex, using recursion to scan all rows, columns and diagonals. However, I found that for a 3x3 board, the simple pattern matching approach, while somewhat verbose, was much easier to read.

Seven Languages in Seven Weeks: Erlang, Day 1

After a long hiatus, I'm finally back to working my way through Seven Languages in Seven Weeks. After finishing up Scala, I'm now on the 5th language, Erlang, though it has taken me quite a bit longer than 5 weeks to get here.

Erlang, Day 1: Thoughts

The first Erlang chapter is just a gentle introduction to the language, so I haven't formed much of an impression of it yet. So far, it looks like a dynamically typed functional programming language with Prolog syntax and pattern matching. Of course, I mostly know of Erlang for its concurrency story, so I'm excited to experiment with that in later chapters.

Erlang, Day 1: Problems

Write a function that uses recursion to return the number of words in a string

Write a function that uses recursion to count to ten

Write a function that uses matching to selectively print "success" or "error: message" given input of the form {error, Message} or success

On to day 2

Check out Erlang, Day 2, for more functional programming goodness.

Seven Languages in Seven Weeks: Scala, Day 3

After some functional programming on day two, it's time for the third and final day of Scala in Seven Languages in Seven Weeks.

Scala, Day 3: Thoughts

After two lengthy chapters on the object oriented and functional programming syntax/options in Scala, the third day rushes through some of the most intriguing features, including pattern matching and concurrency via actors. I would have preferred to spend a bit more time on these complicated topics.

In fact, I had the same complaint on Day 3 of IO, where we also blasted through a discussion of concurrency in just a few pages. I respect the difficulties of plowing through seven different languages in a single book and don't expect deep discussions of any one of them, but I think the book would've been stronger if it had a greater bias towards the more advanced "day 3" topics of each language instead of basic syntax discussions in day 1 or 2.

Scala, Day 3: Problems

Extend the "sizer" application to count and size links

Take the sizer application (code, output) and add a message to count the number of links on a page. Follow these links and calculate their size as well, so you get the total size for each page.

The code:

The output:

This problem was a great way to experiment with actors in Scala. The sequential solution is self explanatory, so here's an outline of the concurrent one:

1. The caller creates B Base Actors, one for each of the B base URLs.
2. The caller then calls receive to await a message from each Base Actor.
3. Each Base Actor concurrently loads its base URL, finds the links on the page, and creates L Link Actors, one for each of the L links on the page.
4. The Base Actor then calls receive to await a message from each of its Link Actors.
5. Each Link Actor concurrently loads the page for its given link and sends a message to its parent Base Actor with the size of that page.
6. When the Base Actor has received a message form each of his Link Actors, it sends a message to the caller with the total size.
7. When the caller has received B messages from the Base Actors, we are done.

The sequential code takes nearly 20 seconds to run while the concurrent code takes less than 3 seconds, a 7x improvement. The concurrent code is definitely more complicated, but not unreasonably so. Even though it was my first time using Scala actors, the code took less than 30 minutes to get working, much of it spent learning about the self keyword. In fact, I find it very easy to reason about Scala's actors, which is not something I can say for Java's synchronized keyword, Executors, Runnable, and various other concurrency constructs. 

Wrapping up Scala

I'm a bit torn when it comes to Scala. Most of the time, I saw it as a vastly improved version of Java. The support for closures, functional programming, pattern matching, and actors all seem like genuinely useful tools that would dramatically improve productivity, code readability, correctness, and expressiveness. I've already used Scala in a few of my projects to build some features that would've been nearly impossible or incredibly ugly in Java. 

However, even in my limited exposure to Scala, I've already come across a number of hiccups. As I mentioned on day 1, the API docs are not helpful and look like they are written for academics. The IDE support is vastly inferior compared to Java. I've now tried both Eclipse and IntelliJ, and each one has significant problems: e.g. missing compile errors on some code; finding errors on other code that's actually valid; broken/missing auto complete; issues with the refactor/rename functionality; poor support for running scripts. The compiler is slow. The type hierarchies are complicated. Type inference doesn't always work as well as you'd hope.  

However, there is one issue that worries me above all else: feature overload. It seems like Scala is trying to be all things to all people. It's object oriented; it's functional; it has type inference; it has lots of syntactic sugar; it has actors; it's compatible with Java; it has first class support for XML; they are even trying to add macros. While all of these features could lead to an incredibly powerful language, they could also lead to one that's incredibly complicated and difficult to use.

User experience counts. Not only for products, but for programming languages too.

How many features can you pile into one language before it becomes too cumbersome? How much syntax sugar do you need to understand to be able to read or write code? How many paradigms and mental models do I need to juggle to follow along? Do so many options make a language more flexible or less? Will there be such a thing as "idiomatic Scala" or will it be a free-for-all? Is it better to have a dozen ways to do something or one "proper" and well known way?

I don't know the answers to these questions, but I suspect they'll have a large impact on Scala's adoption. In the meantime, I'll keep hacking away at it to see what I can learn.

On to Erlang

Read on to learn about the next language in the book, Erlang.