A data-intensive application is typically build from standard building blocks that provide commonly needed functionality. For example, many applications need:

• Databases: Sore data so that they, or another application, can find it again later
• Caches: Remember the result of an expensive operation, to speed up reads.
• Search Indexes: Allow users to search data by keyword of filter it in various ways.
• Stream Processing: Send a message to another process, to be handled asynchronously.
• Batch Processing: Periodically crunch a large amount of accumulated data.

While building an app, we need to figure out which tools and which approaches are the most appropriate for the task at hand.

Why lump everything together under an umbrella term like data systems?

• Many new tools for data storage and processing have emerged in recent years. They are optimized for a variety of different use cases, and they no longer neatly fit into traditional categories. For example,
  • There are datastores that are also used as message queues (Redis) and there are message queues with database-like durability guarantees (Kafka). The boundaries between these tools are becoming blurred.
• Increasingly many aps now have such demanding or wide-ranging requirements that a single tool can no longer meet all of its data processing and storage needs. Instead, the work is broken down into tasks that are handled by special-purpose tools. For example,
  • If you have an application-managed caching layer (using Memcached), or a full-text search server (such as Elasticsearch or Solr) separate from your main DB, it is normally the application code's responsibility to keep those caches and indexes in sync with the main DB.

Factors

There are many factors that may influence the design of a data system. These factors depend very much on the situation. Here we focus on three concerns that are important in most software systems.

• Reliability: System should continue to work correctly (performing correct function at desired level of performance) even in the face of adversity (faults).
• Scalability: There should be reasonable way of dealing with system growth (in data volume, traffic volume, or complexity).
• Maintainability: Multiple people (engineering and operations) should be able to work productively on a system.

Typical expectations for a system to be reliable:

• Performing functions that the user expects
• Tolerating user making mistakes or using the app in unexpected ways.
• Good enough performance for the required use case, under the expected load and data volume.
• Preventing any unauthorized access and abuse.

Faults

• The things that can go wrong are called faults, and systems that anticipate faults and can cope with them are called fault-tolerant or resilient.
• A Fault is usually defined as one component of the system deviating from its spec, whereas a failure is when the system as a whole stops providing the required service to the user. It is impossible to reduce the probability of a fault to zero; therefore it is usually best to design fault-tolerance mechanism that prevents faults from causing failures.
• Increase rate of faults by triggering them deliberately - for example, by randomly killing individual process without warning. Many critical bugs are actually due to poor error handling
• By deliberately inducing faults, you ensure that the fault-tolerance machinery is continually exercised and tested, which can increase ur confidence that faults will be handled correctly when they occur naturally. eg. the Netflix Chaos Monkey.
• There are cases where prevention is better than cure (eg because no cure exists). This is the case with security matters, eg. if an attacker has compromised a system and gained access to sensitive data.
• Faults can be of 3 types
• Hardware Faults
• Software Errors
• Human Errors

Hardware Faults

• HDDs are reported as having a MTTF (Mean Time To Failure) of about 10-15 yrs. On a storage cluster with 10000 disks, we should expect on average one disk to die per day.
• Oure first response is usually to add redundancy to the individual hardware components in order to reduce the failure rate of the system.
• As data volumes and app's computing demands have increased, more apps have begun using larger numbers of machines, which proportionally increases the rate of hardware faults.
• Moreover some cloud platforms (eg AWS) make VM instances unavailable without warning, as they are designed to prioritize flexibility and elasticity over single-machine reliability.
• A single-server system requires planned downtime if you need to reboot the machine (eg. to apply OS security patches), whereas a system that can tolerate machine failure can be patched one node at a time, without downtime of the entire system, known as a rolling upgrade

Software Errors

• Software Errors are harder to anticipate, and bcoz they are correlated across nodes (unlike hardware faults), they tend to cause many more system failures.
• There is now quick solution to this. But a lot of small things can help
• carefully thinking about assumptions and interactions in the system; thorough testing; process isolation; allowing processes to crash and restart; measuring, monitoring and analyzing system behavior in prod.
• eg. in a message queue, the number of incoming messages equals the number of outgoing messages, it can constantly check itself while it is running and raise an alert if a discrepancy is found

Human Errors

There can be several approaches to make the system reliable, in spite of human errors.

• Design system in a way that minimizes opportunities for error. eg. well-designed abstractions, APIs and admin interfaces make it easy to do "the right thing" and discourages "the wrong thing".
• However if the interfaces are too restrictive people will work around them, negating their benefit, so this is a tricky balance to get right.
• Decouple the places where people make the most mistakes from the places where they can cause failures. eg. providing sandbox environment where people can explore and experiment safely using real data without affecting real users.
• Test thoroughly at all levels, from unit tests to whole-system integration tests and manual testing.
• Allow quick and easy recovery from human errors, to minimize the impact in case of failure. eg. roll backs, and incremental roll outs.
• Set up detailed and clear monitoring, such as performance metrics and error rates. This can show us early warning signals.
• Implement good management practices and training.

One study of large internet services found that configuration errors by operators were the leading cause of outages, whereas hardware faults (servers or network) played a role in only 10-25% of outages.

• Scalability is used to describe a system's ability to cope with increased load.
• Discussing scalability means considering questions like "If the system grows is a particular way, what are our options for coping with the growth?" and "How can we add computing resources to handle the additional load?"

Load

• Load can be describe with a few numbers which we call load parameters. The best choice of parameters depends on the architecture of your system. eg requests per second to a server, ratio of reads to writes in a db, number of simultaneously active users in a chatroom, hit rate on a cache.

Twitter case study

Assuming only 2 features: posting tweets and viewing the timeline of people you follow. With 12k writes per second, Twitter's scaling challenge is not primarily due to tweet volume, but due to fan-out - each user follows many people, and each user is followed by many people. there are broadly 2 ways of implementing these 2 operations:

1. Posting a tweet simply inserter new tweet into db. Reading the timeline requires finding all the people a user follows, retrieving each of their tweets, and merging them into a sorted list. In a relational db, it would look something like,

   
       SELECT tweets.*, users.* FROM tweets
         JOIN users ON tweets.sender_id = users.id
         JOIN follows ON follows.followee_id = users.id
         WHERE follows.follower_id = CURRENT_USER;
               

2. Maintain a cache for each users' home timeline - like a mailbox of tweets for each recipient user. When a user posts a tweet, look up all the people who follow that user, and insert the new tweet into each of their home timeline caches.
• Request to read the home timeline is then cheap, bcoz its result has been computed ahead of time.

Conclusion

• The first version of Twitter used approach 1, but the systems struggled to keep up with the load of home timeline queries, so the company switched to approach 2. This works better bcoz the average rate of published tweets is almost 2 orders of magnitude lower than the rate of home timeline reads.
• However, the downside of approach 2 is that posting a tweet now requires a lot of extra work. eg. for a person with 30M followers, a single tweet may result in over 30M writes. Doing this in a timely manner (Twitter tries to deliver tweets to followers within 5 secs) is a significant challenge
• In case of Twitter, distribution of followers per user is a key load param for discussing scalability, since it determines the fan-out load.
• Now Twitter is moving to a hybrid of both approaches. Most users' tweets continue to be fanned out to home timelines as soon as they are posted, but a small number of users (with very large number of followers) are excepted from this fan-out. These tweets are fetched separately and merged with that users' home timeline when it is read.

Describing Performance

• Think of response time as a distribution of values that can be measured, instead of a single number.
• Most requests are reasonably fast, but there are occasional outliers.
• Hence, percentile is a better metric than arithmetic mean.
• Queueing delays often account for a large part of the response time at high percentiles. As a server can process a small number of things in parallel (limited, for example by its number of CPUs).
• So, it is important to measure response time on the client side.

Scaling: Coping with Load

• Types of Scaling
• Vertical Scaling (aka Scaling up): moving to a more powerful machine.
• Horizontal Scaling (or Scaling out): Distributing the load across multiple smaller machines.
• Distributing load across multiple machines is also known as a shared-nothing architecture.
• Ways of Scaling
• Elastic Scaling can automatically add computing resources when they detect a load increase.
• Manual Scaling relies on a human intervention to analyze the capacity and deciding to add more machine to the systems.
• An elastic system can be useful if load is highly unpredictable, but manual scaling is simpler and may have fewer operational surprises.

3 design principles to make systems easy to maintain.

• Operability: Make it easy for operations teams to keep the system running smoothly.
• Simplicity: Make it easy for new engineers to understand the system, by removing as much complexity as possible.
• Evolvability: Make it easy for engineers to make changes to the system in the future, adapting for unanticipated use cases as requirement changes. aka extensibility, modifiability, or plasticity.

Operability

"Good operations can often work around the limitations of bad (or incomplete) software, but good software cannot run reliably with bad operations."

Operations team is typically responsible for following tasks:

• Monitoring health of system, and quickly restoring service if it goes into a bad state.
• Tracking down the cause of problems, such as system failures or degraded performance.
• Keeping software and platforms up to date, including security patches.
• Keeping tabs on how different systems affect each other, so that a problematic change can be avoided before it causes damage.
• Anticipating future problems and solving them before they occur (eg capacity planning).
• Establishing good practices and tools for deployment, configuration management, and more.
• Performing complex maintenance tasks. eg. moving app from one platform to another.
• Maintaining the security of the system as configuration changes are made.
• Defining processes that make operations predictable and help keep the production environment stable.
• Preserving the org's knowledge about the system, even as individuals come and go.

Data Systems can do various things to make routine tasks for Good operability easy:

• Providing visibility into the runtime behavior and internals of the system, with good monitoring.
• Providing good support for automation and integration with standard tools.
• Avoiding dependency on individual machines (allowing machines to be take down for maintenance while the system as a whole continues running uninterrupted)
• Providing good documentation and an easy-to-understand operational model.
• Providing good default behavior, but also giving administrators the freedom to override defaults when needed
• Self-healing when appropriate, but also giving admins manual control over the system state when needed.
• Exhibiting predictable behavior, minimizing surprises.

Simplicity

• There are various possible symptoms of complexity: explosion of the state space, tight coupling of modules, tangled dependencies, inconsistent naming and terminology, hacks aimed at solving performance problems, special-casing to work around issues elsewhere and many more.
• making a system simpler does not necessarily mean reducing its functionality; it can also mean removing accidental complexity.
• Moseley and Marks define complexity as accidental if it is not inherent in the problem that the software solves (as seen by the users) but arises only from the implementation.
• One of the best tool for removing accidental complexity is abstraction. A good abstraction can hide a great deal of implementation detail behind a clean, simple-to-understand facade.

Evolvability

• In terms of organizational processes, Agile working patterns provide a framework for adapting to change.