One of my favorite interview question that I ask to candidates is “How do you scale?”. I don’t give much of a context. I don’t provide more information on topic and at the same time i don’t define what i expect as an outcome. If we are going to have an interview in the future, please do not think that the answer is here, because there is not one and you, as a candidate, should always be able to answer this question with your own thoughts.
Scaling is a long thought process. We have have terms for companies: start up, scale up and then enterprise. In technical world, this is also the same. Your organisation grows, so does your code base / server count / developer count etc. etc. They are all connected. What I see as a misconception in scaling is, people always talk about spending more money.
Do you need to get stuff delivered faster? Hire more developers. Do you need to acquire more customers? Spend more money on marketing. Do you need to close more sales? Hire more sales rep. The list can go on.
Most of the time, what is overlooked is process improvements. I know, the current meta is all about people over processes. Yet, I tend to disagree. I am not saying neither processes nor people are not important. It all depends on the context. As human beings, we all need some kind of a guidance. We tend to become blind to problems that are there. Sometimes, we don’t like to admit that we are not efficient, because it hurts our own pride.
Then how do you scale? First of all, one thing that needs to be identified is what is going wrong? For engineering, most of the time it turns out to be communication. Literally, nothing else. In most of my blogs, you will see that I connect things to communication in the end. But that is the unfortunate truth. At the same time, scaling is mostly what management has to think about.
Let’s say you are seeing a rapid growth in customer acquisition. Your query per second is going wild, your concurrent user count is top of the roof. Now forget about the communication, you need to support that traffic. Definitely add more servers to the farm, because otherwise you might be late. The term for this is horizontal scaling. You basically add more machines and voila, you are done. But there is an aftermath. If you continue growing, there is a fine line between too many servers vs. too much money / maintenance cost. Because you are just not adding new servers, you are also adding more stuff to deal with growing number of servers.
In macroeconomics, there is short term and long term investment. In the first phase, you know you will eventually hit a level where you have to add new servers. This is your short term investment. You want to justify further purchases. Now as soon as you hit the level, your long term investment starts. The decision you have to make here is:
a) Continue adding more servers
b) Find places to improve performance of your application
Can you do both simultaneously? Definitely. Just make sure, you are keeping an eye on other areas as well and not constantly add things do deal with.
In my opinion, the same applies to all departments. Once you start throwing money away, problems are solved for a while. Then they come back bigger. To prevent this, while spending money for a short breath, start looking areas for improvements. It is not for saving money, it is for sustaining the growth and being prepared for what is to come next.