System Design – Part 1

Prologue

Not long back, popularity of Yahoo Messenger was at its peak. Google mail was in diapers, to be subscribed only through invitation. A time when TheFaceBook was to see light of the day and WhatsApp was never heard.
Late in night, when I have nothing much to do, often I contemplate on the Tech advancement I have witnessed ever since I knew what software was. Please don’t take off your seat just yet, I am not gonna take you back to medieval age when computers sat only on the desks of rich people.

It was 2006 when I got my first PC. 256 MB DDR1 RAM, 20 GB HDD, single core processor. It’s fascinating to see users and how they use devices & services have come a long way. SaaS (Software as a Service) is on the top of mind. The world is going paperless and companies which are scared of moving to cloud may fray. With overwhelming 24 billions devices connected to internet today, the emphasis on Scalability has become more than it ever was. The amount of data we produce is enthralling and mind-boggling – that’s 2.5 quintillion bytes of data per day. 90% of all the data till date was generated over the last two years. It is everywhere – AI, Machine Learning, self driving cars, home automation system. It’s the future.

Data never sleeps

If we talk about numbers, we have 1.5 billion monthly WhatsApp users sending 65 billion messages and making 2 billion minutes of call every day, 73k Google searches & 78k Youtube video views per second, 175 billion Pinterest pins. Every 60 seconds on Facebook: 510,000 comments are posted, 293,000 statuses are updated, and 136,000 photos are uploaded. Imagine the storage and computational power sitting on the servers to process and store the humongous amount of data.
Compare it with what we had 10 years ago. With advancement in technology, IoT, cheaper network bandwidth and cloud storage, more people connecting to the internet, the data is bound to grow exponentially.

Growing data - Scalability
Data growth

Consumers, today, don’t care which Operating System they use, which device they hold in their hands and how much power those devices are packed with. They just want Software and services to work like a charm.

Every company, if they are to take the world with the storm (read Google Search, WhatsApp or MS Office) they are building, has to tackle two things eventually – Scalability and Cross Platform strategy.
I would scope this blog to Scalability and leave the latter for some other day.

So what’s Scalability?

You are the founder of a finance company, often dealing with clients (read it people). Clients come to you for your hassle free, time bounded services. They are happy because you promise them to get the work done in 3 working days. That’s how you make money.
For one year, you are pretty much happy. Happy clients, happy you. No big deal. More people hear about your services through word of mouth. And soon enough, you are flooded with clients from all over the world. You compromise sleeps and start working overtime. This works well for another five months before you realize that you can’t work alone. You hire more people, as beefy and efficient as you. Because businesses run on trust. You can’t just say that you don’t have time. People stop coming when you no longer provide what you promise.
Scalability is when you can serve the requests in the reasonable time window by hiring more resources. Bigger hard drives, more RAM, powerful processor (more cores). That’s vertical scaling. But there is a limit on how much resources you can add to a machine. Costs go up and there comes a point when buying a new machine is cheaper and more feasible. That’s horizontal scaling.

In the next part, I would touch upon Databases, Caching, Message Queue, Sharding and few other topics before delving into System Design problem space. Stay tuned!

Resources

Copyright Β© 2019, Aashish Barnwal, Β All rights reserved.

5 Comments, RSS

  1. Anand April 13, 2019 @ 9:51 pm

    Good write up.
    Very helpful.
    Looking forward to next part soon.

  2. Venki April 13, 2019 @ 10:35 pm

    We are just the beginning of data explosion. Think why would a company like Google buys Nest for approximately ~ $3B? Is it the technology, certainly not.

    It is good attempt that you are writing on System Design, take some real examples and keep in mind that ‘a design is complete when nothing to take out from the design’. Write precise and to the point thought worthy articles. It would require lot of study, but don’t fail without attempt.

    Looking forward for next series of articles. I would recommend to post them on Hacker News after first few posts.

  3. Aashish Barnwal April 13, 2019 @ 11:07 pm

    Had mentioned Nest acquisition by Google in one of my other posts. Data is the most valuable resource today (I removed this point while fine tuning the post :)). That was the driving factor of Nest and WhatsApp acquisitions.
    Absolutely agreed! System Design is vast and it needs a lot of study, research and experience to write it right. It’s an attempt to pen down my thoughts and improvise as I go on.
    Thanks for your thoughts πŸ™‚

  4. Manisha Sinha April 15, 2019 @ 1:11 pm

    Amazing write up Ashish.
    Waiting for the next part πŸ™‚

Leave a Reply

%d bloggers like this:
Bitnami