Using Amazon Auto Scaling with stateful applications

You’ve heard this before. The team has been working on this service and a couple months later traffic is picking up. Pretty awesome you think, customers are loving this feature! Hold on, now you hear finance people screaming at the Amazon bill. The application is considerably resource intensive. You got two options: 1) find the bottleneck and optimize, 2) limit cost of running the service. Let’s focus on the latter.

Welcome to Amazon Auto Scaling. If the fundamental premise of the cloud is “use only the resources you need for as long as you need”, then add “dynamic scaling based on traffic”.  Bottom line: you save money when traffic is low. Enterprise SaaS is a great use case since customers are using your product during typical business hours, resulting in very low traffic at night. So what does it take to make the switch?

Stateless VS stateful application

Ideally you want to be dealing with a stateless application, where terminating one node won’t produce side effects on user experience. Typical stateless apps include frontend web servers or any app that doesn’t rely on keeping session state in memory.

Unfortunately not all software is created equal. Our use case is a video recorder for web based meetings. While the presenter is discussing slides, the recorder is watching the presentation unfolding real time.
Try to terminate one instance with active sessions and you’re impacting user experience. But there is a solution to which we’ll come back shortly.


One thing with dynamically terminating instances is that you can’t rely on SSH access any longer:

  • Logs need to be forwarded to a remote host using Elasticsearch, Splunk or similar.
  • Provisioning an instance is done in an automated fashion. We use Chef and Terraform.
  • Not directly related to Auto Scaling here but proper deployment tooling is also a requirement. We use Jenkins pipelines and Chef.

Architecting the app around Auto Scaling

Our video recorder app went through a few changes. First you need to expose a health check endpoint that returns HTTP 200 if the app is in a good state. Amazon is continuously polling it and will replace unhealthy instances.

Then you need the app to properly report your chosen autoscaling metric to CloudWatch. For instance our metric is “number of running jobs”. A java thread can handle the task or even a cron job.

One more thing about stateful applications: we want to make sure we don’t disrupt running jobs during a scale down event. Amazon conveniently provides Lifecycle Hooks which allows to perform a custom action before terminating the instance. For instance: a decrease in traffic triggers a scale down event. The oldest instance (by creation time) is picked and moves to Termination:Wait state. Amazon sends a notification using SNS to check if the instance is ready to be terminated. The instance gets the notification and will give the green light for termination if there is no job running, otherwise it’ll respond with a heartbeat to keep waiting until all jobs are done. Which means your app needs to have a thread listening to SNS notifications.
Interestingly lifecycle hooks cannot be set up on Amazon web console, you'll have to use the CLI.

Amazon SDKs make it pretty straightforward to implement the above two items.

My books of 2012

In no particular order: 

Learning Android

Very practical guide to dive into the workings of a microblogging app.

Making Things Happen

Priorities get things done. Plus Scott is such a great writer.

Zen and the Art of Motorcycle Maintenance

The real purpose of scientific method is to make sure Nature hasn’t misled you into thinking you know something that you actually don’t.

Ignore Everybody

If you have the creative bug, it isn’t ever going to go away. You should just get used to the idea of dealing with it.

The Miracle of Mindfullness


Every day we are engaged in a miracle which we don't even recognize: a blue sky, white clouds, green leaves, the black, curious eyes of a child-our own two eyes. All is a miracle.

The Creative Habit

Best book on creativity I have read. Dive into your creative DNA and get to work.

What Technology Wants

Kevin's point is that innovations tend to grow organically no matter what. When someone invents something new on the planet, he will be part of this group of people who worked on the exact same thing. The concept of ownership for innovation is becoming increasingly harder to pinpoint. As technology matures its own needs are growing and need to be satisfied. The computing infrastructure that runs Google needs a ton of energy before it can even return a single search result. Important book.

Free Ride

I was hoping to learn about ideas that combine the culture business with Internet/piracy. I did not. That's just a rant.

10 Days to Faster Reading

Reading faster is seriously underrated. Why not dedicate some time to the most important skill ever? Every student should read this.

Business Model Generation

If you don't know where to start with business plans, this is a great reference. That will get you going with coming up with models and ideation. Great pairing with Steve Blank's Lean Launchpad class.

Delivering Happiness

Investing in company culture does scale your business. Tony researched the science of happiness and turned out to combine his findings with running a business. It works.

The Referral Engine

The fact that the author is good at referrals got this book in my hands. But the editing is so poor. Skip this one!

The Flinch

A necessary reminder to get you standing up against the lizard brain. Doing the uncomfortable is key.

The Invention of Air

This book is about the fascinating biography of Joseph Priestley combined with the case on how ideas spread and evolve. Plus Steven is super thoughtful.

The Google Resume

People at the top share many similarities on their resume.

The Design of Everyday Things

Read it and start looking weird at poorly designed door handles.

Moonwalking with Einstein

The art of remembering things has everything to do with physical spaces.

The Art of War

A classic, but what else? Still wondering.

The Happiness Hypothesis

Very interesting book on the science of happiness. What are the tangible factors that make up how happy you are?

Our Posthuman Future

Important topic but poor editing.

The Dip

If you're not going to be number one you might as well quit now. A little book with big impact.

The Big Short

I got the audio book.

The 4 Essentials of Entrepreneurial Thinking

This book is good at explaining the thinking part. Don't forget the doing.

The Shallows

Another important book about the interruption economy.

Getting to Yes

Go find out your BATNA before your next negotiation.

The Case for Marriage


How the Stock Market Works


Just Kids

Patti Smith's autobiography. Her music will resonate differently to you.

What to expect at Startup Weekend?

Startup Weekend is a learning experience about entrepreneurship. You’re building a company from scratch. I just finished my second Startup Weekend San Francisco and it was awesome.

On Friday it starts with a bit of networking to get to know each other and see who would be a good fit to work with. Then it’s time for pitchfire: anyone can pitch an idea and has one minute to do so. 39 ideas got pitched that weekend, wow! People vote for their favorite ideas and 13 teams were formed. What follows is a crazy mix of business model creation, customer validation, programming and connecting pieces together.

You got a team, great. Now you start making hypotheses about the business model. You go out of the building and talk to people to validate these hypotheses.

The weekend culminates with presentations in front of judges. Below are judging criterias and what you need to focus on in my experience:

- customer validation (50%): are you solving a problem that people have?
- business model (25%): how are you going to make revenue?
- execution (25%): what did you build?

Customer validation is so crucial since time is so short. You don’t have time to mess around and need to figure out what should be built. You should have in your presentation metrics of your customer validation. Videos of users talking about your product are even better. Startup Weekends are not hackathons. No one cares about your technology unless it solves a real world problem.

I’m glad of being part of the GoVelo team. We had a great time and our unique combination of skills made a difference. We are working on the problem of safe peer-to-peer bike sharing. Safe is about providing an Arduino based electronic U-lock to bike owners. Peer-to-peer bike sharing is about creating the platform that allows the community to flourish: iPhone and web apps.

You should go to a Startup Weekend if you want super creativity, to know your limits or be surrounded by driven people. You should go if you want to make something happen.


The Checklist Manifesto

After reading this book you will never get into something without doing your checklist first. Atul Gawande is challenging what we think of the reliability of experts with the question: what do we do when expertise is not enough?


He demonstrates how good the impact of the checklist is through carefully chosen stories. It's about surgeons, airline pilots, engineers, investors and the degree of complexity they face to do their job.


Here is my curated version of the book:



The problem of extreme complexity


« Here, then, is the fundamental puzzle of modern medical care: you have a desperately sick patient and in order to have a chance of saving him you have to get the knowledge right and then you have to make sure that the 178 daily tasks that follow are done correctly—despite some monitor's alarm going off for God knows what reason, despite the patient in the next bed crashing, despite a nurse poking his head around the curtain to ask whether someone could help “get this lady's chest open”.

There is complexity upon complexity. And even specialization has begun to seem inadequate. So what do you do?”


“There are degrees of complexity, though, and medicine and other fields like it have grown so far beyond the usual kind that avoiding daily mistakes is proving impossible even for our most superspecialized.”



The checklist


“ Substantial parts of what software designers, financial managers, fire-fighters, police officers, lawyers, and most certainly clinicians do are now too complex for them to carry out reliably from memory alone.”


The end of the master builder


“Four generations after the first aviation checklists went into use, a lesson is emerging: checklists seem able to defend anyone, even the experienced, against failure in many more tasks than we realized. They provide a kind of cognitive net. They catch mental flaws inherent in all of us—flaws of memory and attention and thoroughness.”


There is three different kinds of problems in the world: the simple, the complicated and the complex.


“Simple problems are ones like baking a cake from a mix. There is a recipe. Following the recipe brings a high likelihood of success.”


“Complicated problems are ones like sending a rocket to the moon. They can sometimes be broken down into a series of problems. But there is no straightforward recipe. Success frequently requires multiple people, often multiple teams, and specialized expertise.”


“ Complex problems are ones like raising a child. Once you learn how to send a rocket to the moon, you can repeat the process with other rockets and perfect it. But not so with raising a child. Every child is unique. Although raising one child may provide experience, it does not guarantee success with the next child.”



The checklist factory


“Good checklists are precise. They are efficient, to the point, and easy to use even in the most difficult situations. They do not try to spell out everything—a checklist cannot fly a plane. Instead, they provide reminders of only the most critical and important steps—the ones that even the highly skilled professionals using them could miss. Good checklists are, above all, practical.”


“Pilots nonetheless turn to their checklists for two reasons. First, they are trained to do so. They learn for the beginning of flight school that their memory and judgment are unreliable and that lives depend on their recognizing that fact. Second, the checklists have proved their worth—they work. Aviation checklists are by no means perfect. Some have been found confusing or unclear of flawed. Nonetheless, they have earned pilots' faith.”



The test


“In London, during a knee replacement by an orthopedic surgeon who was one of the toughest critics, the checklist brought the team to recognize, before incision and the point of no return, that the knee prosthesis on hand was the wrong size for the patient—and that the right size was not available in the hospital. The surgeon became an instant checklist proponent.”


“More than 250 staff members—surgeons, anesthesiologists, nurses, and others—filled out an anonymous survey after three months of using the checklist. […] Then we asked the staff one more question. “If you were having an operation, would you want the checklist to be used?”

A full 93 percent said yes.”



The hero in the age of checklists


“ Discipline is hard—harder than trustworthiness and skill and perhaps even than selflessness. We are by nature flawed and inconstant creatures. We can't even keep from snacking between meals. We are not built for discipline. We are built for novelty and excitement, not for careful attention to detail. Discipline is something we have to work at.”



Since then I started thinking in terms of checklists for every task I'm working on. Checklists are big deal. How did the checklist impacted your daily work?

First Quote Of 2011

When a fine idea is compressed into a definite metre, the very same thought comes hurtling at one like a missile launched from a fully extented arm.

―Seneca, Letters from a Stoic

Peace Is Every Step

I went through the whole piece of Peace Is Every Step. This thin book brings a lot of valuable insights from Buddhist teachings into our information-overloaded world. It's that simple and deeply refreshing.

You've got a selection of my best picks:
  • The starting point is mindfulness. And mindfulness is about being fully immersed in the present moment. One way of getting into that state is mindful breathing. Sounds obvious? Just try it and notice how it feels.
  • Our society has materialized happiness with things. They expect you to get them so you'll feel happy somewhere in the future. You actually don't have to. Once you're right in the present moment, you have peace and joy right now. Living in the present moment is taking good care of the future.
  • Aimlessness: The basic condition for being happy is our consciousness of being happy. If we're not aware that we're happy, we're not really happy.
A good example is this: When we have a toothache, we know that not having a toothache is a wonderful thing. But when we don't have a toothache, we're still not happy. A non-toothache is very pleasant though.
  • What's Not Wrong?: We often ask "What's wrong?". Doing so, that makes we think of things that we're guilty of. Negative thinking often comes up. We would be much happier if we tried to stay on track with the positive things around us. We should learn to ask, "What's not wrong?" and be in touch with that.
  • There is a big emphasis on interbeing.
When you're driving your car, you become one with the car. You think you control the car while the car is changing yourself. Once in the car, you expect to arrive pretty quickly. So you're stuck in traffic, stress comes up and you have lost control over your expectation. We should drive our cars with mindfulness so we stay happy as the traveler who walks without thinking to arrive.

Create Stories Using The Social Web With Storify

I first met Storify Cofounder Xavier Damman at a conference in San Francisco one year ago. He was working on a product called Publitweet back then. He also built in 24 hours ListiMonkey right at the time when Twitter launched the list feature. He was very enthusiastic and ready to take the next step.

We're now living in the connected age of social media and we try to catch up with never-ending feeds. Whatsoever on Twitter or Facebook. It's been kind of cool in 2010. We werescrolling on the Twitter feed looking for interesting stuff to learn about.

But another truth lies ahead: it's getting more and more overwhelming to keep up with what's happening with your friends and Twitter followers. People increasingly spend quite a good deal of their life online. People are sharing more and more stuff. People get excited about telling their friend about that awesome video. It all comes down to bring so much noise.

We need filters to get the most relevant information from the noisy social medias. Storify is kind of big deal on the web curation scene. It's about finding the meaning in the noise and turn that meaning into a format that people can actually understand. That format is a story. And people read stories. The stories you create with Storify are made of tweets, status updates, youtube videos and pretty much every other social web service.

I think it's very promising. To me it'll be a lot more handy to tell the story of what happened in a concert while putting together real-time pieces of the show.

Have a look at this recent interview of Storify Cofounders by Robert Scoble:

And here is what a Storify looks like:

Brain Rules

Brain Rules is one of those books that really makes you feel smarter. Who wouldn't want to know how your brain is running?

I loved this book. John Medina did a great work in putting his tremendous knowledge into twelves small chapters. The book is packed up with many stories, each one illustrating one brain rule at a time. I learned a handful of tricks for dealing with school, work and home life.

Here are some of the notes that I put in my Moleskine:
  • Most of the events that predict whether something learned also will be remembered occur in the first few seconds of learning.
  • Many of the same mechanisms that cause you to run from a predator are also used when you're having sex.
  • Every brain is wired differently. Even though the current system is founded on expectations that certain learning goals should be achieved by a certain age. Students of the same age show a great deal of intellectual variability.
  • One NASA study showed that a 26-minute nap improved a pilot's performance by more than 34%. The afternoon nap is a biological need.
  • If information is presented orally, people remember about 10%, tested 72 hours after exposure. That figure goes up to 65% if you add a picture.
I first heard about this book in Scott Berkun's Confession of a Public Speaker and I'm sure he did benefit a lot from it too.

Attention Is The New Black

Music bands are doomed. The Internet era has completely shifted the distribution platform. It used to be the record shops. Today it's all about streaming monthly subscription. There is one good news and one bad news to that.

Good news is that the customer accesses to pretty much any track on powerful streaming platforms such as Spotify.

But bad news is every band or artist has now the same visibility on the distribution platform. I mean the size of the record shop is limited by the space in which a limited number of records can be on sale. And therefore a selection is made up. The customer easily finds what he's looking for. The artist can somehow size up his audience.

Although with the streaming platforms there is no limit. In a couple of minutes the customer can switch between as many as ten records. Because aside from the TOP 50 songs, there is no selection. The customer is lost with the ten million tracks choice. So he's trying out a bit of everything and is frustrated of leaving some unheard.

If the physical record get purchased, a meaningful connection is established with the artist as the customer will dedicate time and attention to listen to the record. Because that's a physical thing that you can hold with both hands. While the digital record is manipulated from your fingertip for a few seconds. The perceived value obviously is here close to nothing.

For an artist to get valuable audience, customer's attention needs to be focused on an experience for some time in the proper context. So that's a lasting moment.

This is going to be pretty tough for artists as the attention span is getting smaller and smaller. Even though there is a solution. They've got either to be really good in social medias or to make really great music. They can be either a member of the tribe or they can set the tone for what everyone is talking about within the tribe.