24 Sep

SRE Risks and Error Budgets

How the SRE discipline reduces stress on speed / stability between product teams and system operators by quantifying risk and activating error budgets. Striving for 100% availability of service is not simply impossible, it is unnecessary. Maximum stability limits the speed at which new features can be provided to users. Extreme availability yields diminishing as the user experience is controlled by less reliable components like cellular networks or WiFi. We want to reduce the risk of system failure, but we must accept the risk of providing new products and features.

How the SRE discipline reduces stress on speed / stability between product teams and system operators by quantifying risk and activating error budgets. Striving for 100% availability of service is not simply impossible, it is unnecessary. Maximum stability limits the speed at which new features can be provided to users. Extreme availability yields diminishing as the user experience is controlled by less reliable components like cellular networks or WiFi. We want to reduce the risk of system failure, but we must accept the risk of providing new products and features. In the SRE discipline, error budgets are the quantitative measurements that record how much service risk is prepared to endure. Error budgets are a by-product of agreed SLOs (service-level goals) between product owners and systems engineers. Risk and error budgets are directly related to many DevOps principles. Error budgets clearly define "accidents" by quantifying accidents and risk. Error budgets also enforce that "change needs to be gradual," because non-gradual changes can quickly break the SLO and prevent further development in the quarter. This is why we say that SRE class implements DevOps. Embrace a risk You can expect Google to try to build 100% reliable services - services that never fail. Apparently, a certain point has passed, however, increasing reliability for service (and its users) and not better! Extreme reliability costs: Maximum stability limits how nelicited press coverage.

Site Reliability Engineer Manager SRE

In the SRE discipline, error budgets are the quantitative measurements that record how much service risk is prepared to endure. Error budgets are a by-product of agreed SLOs (service-level goals) between product owners and systems engineers. Risk and error budgets are directly related to many DevOps principles. Error budgets clearly define “accidents” by quantifying accidents and risk. Error budgets also enforce that “change needs to be gradual,” because non-gradual changes can quickly break the SLO and prevent further development in the quarter. This is why we say that SRE class implements DevOps.

Embrace a risk

You can expect Google to try to build 100% reliable services – services that never fail. Apparently, a certain point has passed, however, increasing reliability for service (and its users) and not better! Extreme reliability costs: Maximum stability limits how new features can be developed quickly and how quickly products can be delivered to users, dramatically increasing their cost, which in turn reduces the number of features a team can afford. Furthermore, users generally do not notice the difference between high reliability and extreme service reliability, because the user experience is dominated by less reliable components such as the cellular network or device they work with. Simply put, a 99% reliable smartphone user cannot understand the difference between 99.99% and 99.999% service reliability! With this in mind, rather than simply maximizing operating time, site reliability engineering seeks to balance the risk of unavailability for rapid innovation and efficient service operations so that overall user happiness – with features, service and performance – is optimized.

Risk Management
Unreliable systems can quickly erode user confidence, so we want to reduce the chance of system failure. However, experience shows that when we build systems, the cost does not increase linearly as reliability additions – incremental improvement in reliability may cost 100 times more than the previous increment. The cost has two dimensions:

Cost of resources for unnecessary computers / computers
The cost of unnecessary equipment, for example, allows us to take offline systems for ongoing or unexpected maintenance, or provides us with space to store parity code blocks that provide minimal data compliance guarantee.

The opportunity cost
The cost that an organization incurs when it allocates engineering resources to build systems or features that reduce risk rather than features directly visible to or usable by end users. These engineers are no longer working on new features and products for end users.

At SRE we manage service reliability to a large extent by risk management. We realize the risk as a continuum. We give equal importance to understanding how Google can be more trusted and identify the appropriate level of tolerance for the services we run. This allows us to perform cost / benefit analysis to determine, for example, where in the (non-linear) risk sequence we should place search, ads, Gmail, or images. Our goal is to explicitly align the risk of taking a given service with the risk the business is willing to bear. We strive to make service reliable enough, but not more reliable than it should be. That is, when we set a 99.99% availability target, we want to exceed it, but not much: it will waste opportunities to add features to the system, clear technical debt or reduce its operating costs. In a sense, we see that the target of availability is minimum and maximum. The main advantage of this framing is that it opens an opportunity for explicit and thoughtful risk-building.Site Reliability Engineer Manager SRE

Service risk measurement
As a standard practice at Google, we usually do our best by identifying an objective metric for representing the property of a system that we want to optimize. By setting a goal, we can assess our current performance and track improvements or degradations over time. Regarding service risk, it is not immediately clear how all potential factors can be reduced to a single value. Service failures can have many potential impacts, including user dissatisfaction, injury, or loss of trust; Direct or indirect loss of income; Influence on brand or reputation; Unsolicited press coverage.

Leave a reply