<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>SreZone Blog</title>
    <description>SREZone Blog</description>
    <link>https://srezone.com/blog/</link>
    <atom:link href="https://srezone.com/blog/feed.xml" rel="self" type="application/rss+xml"/>
    <pubDate>Sat, 08 Aug 2026 02:25:39 +0000</pubDate>
    <lastBuildDate>Sat, 08 Aug 2026 02:25:39 +0000</lastBuildDate>
    <generator>Jekyll v3.10.0</generator>
    
      <item>
        <title>Incident Response Management</title>
        <description>Incident Response Management (IRM) is a non-negotiable for all critical production systems. The goal of IRM is to &lt;strong&gt;provide high service reliability&lt;/strong&gt; through rapid response to production incidents, without burning out humans through alert fatigue and unsustainable on-call burden.

&lt;h2 id=&quot;why-consider-incident-response-management&quot;&gt;Why consider Incident Response Management?&lt;/h2&gt;

You can bake as many resilience practices into a system as you want, but &lt;strong&gt;outages are inevitable in complex systems&lt;/strong&gt;. When incidents occur it’s &lt;strong&gt;crucial that the incidents are responded to with a structured, controlled manner&lt;/strong&gt;.

Without a structured IRM process, chaos ensues: downtime is extended longer than necessary and mistakes are likely to be made. After all, even the best engineers make mistakes while under stress. Therefore, it’s vital that proper preparation, response, and learning occurs throughout the incident response lifecycle.

This article focuses on the concepts of building an effective IRM process that helps cultivate calm, fast, and safe recoveries during incidents.

&lt;h2 id=&quot;how-to-implement-incident-response-management&quot;&gt;How to implement Incident Response Management&lt;/h2&gt;

Simply creating an “on-call rotation” isn’t enough; creating a successful IRM framework takes effort.

This section describes the IRM cycle: starting with tools, onboarding, handling and incident, and ending with postmortems.

&lt;h3 id=&quot;tools-needed&quot;&gt;Tools needed&lt;/h3&gt;

&lt;strong&gt;Establish tooling before defining process.&lt;/strong&gt; Without proper tooling, incident response cannot occur effectively.

The following tools are critical for successful incident response:

&lt;ul&gt;
  &lt;li&gt;Incident management platform (e.g.: PagerDuty, Opsgenie) - plans on-call schedules and pages the on-call responder during incidents.
    &lt;ul&gt;
      &lt;li&gt;Alerts must sent in a &lt;em&gt;timely&lt;/em&gt; manner.&lt;/li&gt;
      &lt;li&gt;Alerts must be &lt;em&gt;actionable&lt;/em&gt;.&lt;/li&gt;
      &lt;li&gt;In most cases, alerts should be triggered by user-facing symptoms based on SLO error-budget concerns.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Documentation management system (e.g.: Confluence) - create, save, and collaborate on incident information and postmortems.
    &lt;ul&gt;
      &lt;li&gt;IRM-related documents act as &lt;strong&gt;transparent, centralized, single sources of truth&lt;/strong&gt;.&lt;/li&gt;
      &lt;li&gt;Templates for IRM documents should be available for fast and consistent documenting.
        &lt;ul&gt;
          &lt;li&gt;An example incident response template may contain sections for incident summary, timeline, hypotheses, mitigation steps, and ownership.&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Communication platforms:
    &lt;ul&gt;
      &lt;li&gt;Dedicated chatrooms (e.g.: Slack, IRC, MSTeams) for async communication.&lt;/li&gt;
      &lt;li&gt;Conference calls for real-time coordination and decision making.&lt;/li&gt;
      &lt;li&gt;Email or social media for communicating with stakeholders and users.&lt;/li&gt;
      &lt;li&gt;Optionally, health/status pages.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;(Goes without saying) an observability platform with proper telemetry for diagnosing the incident.&lt;/li&gt;
&lt;/ul&gt;

To simplify IRM tooling, all-in-one solutions such as &lt;a href=&quot;https://grafana.com/docs/grafana-cloud/alerting-and-irm/&quot;&gt;Grafana IRM&lt;/a&gt; exist to provide all of the tools necessary for incident response out-of-the-box.

Additionally, it may be worth setting up a secondary monitoring platform – a monitor for your monitor, ideally deployed in separate regions. A potential pitfall of having a single monitoring instance is if the single monitoring instance fails, you may not know about the failure immediately.

&lt;h3 id=&quot;irm-onboarding&quot;&gt;IRM Onboarding&lt;/h3&gt;

Once you have the required IRM tools available, your services can start being onboarded to your IRM system with the following steps:

&lt;h4 id=&quot;1-identify-critical-services&quot;&gt;1. Identify critical services&lt;/h4&gt;

Depending on organization size, you may have few to many services. Get the most incident response value ASAP by identifying and focusing on onboarding your mission critical services to IRM first.

&lt;h5 id=&quot;define-slos&quot;&gt;Define SLOs&lt;/h5&gt;

Without an SLO, it’s near impossible to define what good[-enough] reliability is for each service. Establishing clear targets for service reliability and performance will ensure alignment between your service and customer needs, and may ultimately influence SLAs.

&lt;h4 id=&quot;2-define-an-alert-policy&quot;&gt;2. Define an alert policy&lt;/h4&gt;

Alert policies are jointly determined by SRE teams and business stakeholders. A few recommended alert guidelines are as follows:

&lt;ul&gt;
  &lt;li&gt;Alerts should be SLO-based.
    &lt;ul&gt;
      &lt;li&gt;Alerts based on signals other than SLOs are &lt;a href=&quot;/blog/2024/06/16/beyond-anomaly-detection/&quot;&gt;likely to lead to noise&lt;/a&gt;.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Alerts must be worthy of waking up a human.
    &lt;ul&gt;
      &lt;li&gt;If a human doesn’t need to act immediately or if the alert is not actionable, do NOT alert the person on-call; create a ticket instead.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Alerts must be related to user-facing symptoms.
    &lt;ul&gt;
      &lt;li&gt;Exception: infrastructure-based alerts such as disk space nearing exhaustion, resource constraints preventing a cluster from scaling, high amounts of pod evictions, etc. are OK to alert on if you’re responsible for infrastructure.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Alerts must be sent timely manner.&lt;/li&gt;
  &lt;li&gt;Alerts must be responded to in a timely manner.
    &lt;ul&gt;
      &lt;li&gt;Example response time agreements may specify that on-call must respond to incidents within:
        &lt;ul&gt;
          &lt;li&gt;5 minutes for incidents on services that impact revenue.&lt;/li&gt;
          &lt;li&gt;30 minutes for degraded but not catastrophic incidents.&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h4 id=&quot;3-accept-a-service-into-on-call&quot;&gt;3. Accept a service into on-call&lt;/h4&gt;

&lt;strong&gt;Note:&lt;/strong&gt; “Accepting” a service into on-call is primarily applicable if a dedicated SRE team will be on-call for an application team’s service.

Each critical system going to production should follow some sort of “production readiness check” before going live. For example:

&lt;ul&gt;
  &lt;li&gt;Ensure proper observability that &lt;a href=&quot;/blog/2023/08/12/vendorLockIn/#1-telemetry-generation&quot;&gt;emits, processes, and stores&lt;/a&gt; &lt;a href=&quot;/blog/2023/07/07/signals/&quot;&gt;telemetry signals&lt;/a&gt;.&lt;/li&gt;
  &lt;li&gt;Potential failure points are known, documented, and resilience practices are put into place.
    &lt;ul&gt;
      &lt;li&gt;Use forms of stress testing such as load testing and chaos engineering as tools to identify gaps.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Signoff/approval from a senior Site Reliability Engineer.&lt;/strong&gt;&lt;/li&gt;
  &lt;li&gt;Any items mentioned in &lt;a href=&quot;https://sre.google/sre-book/evolving-sre-engagement-model/#:~:text=The%20PRR%20Model&quot;&gt;Google’s SRE book’s Production Readiness Review (PRR)&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;em&gt;Note that if an application supported by a dedicated SRE team consistently fails to meet it’s agreed-upon SLO or becomes unsustainable for the SRE team to continue supporting, “handing the pager back” to the application team is an option until appropriate reliability fixes have been implemented to resolve the issue(s).&lt;/em&gt;

&lt;h4 id=&quot;4-create-runbooks&quot;&gt;4. Create runbooks&lt;/h4&gt;

A runbook is traditionally a detailed, step-by-step document that guides engineers through common tasks during incident response. Most of the runbook tasks should be automated when possible, so that the responder can focus on solving the issue rather than performing tedious tasks.

With automation in place, runbooks should be slimmed down to information that will get the incident responder on the right track for diagnosing and solving the incident as soon as possible. Some resources you’d find in a runbook could be links to useful telemetry, dashboards, documentation, and escalation contacts.

&lt;h4 id=&quot;5-be-proactive&quot;&gt;5. Be proactive&lt;/h4&gt;

If you’re lucky, the systems you’re on-call for will have very few incidents. The caveat to not having many incidents is that &lt;strong&gt;incident response skills atrophy if not exercised regularly&lt;/strong&gt;, leading to unprepared response when a real incident occurs.

Incident response should be second nature. Introducing incident response drills by purposely injecting failure into systems or hosting SRE-involved activities such as the &lt;a href=&quot;https://sre.google/sre-book/postmortem-culture/#wheel-of-misfortune&quot;&gt;Wheel of Misfortune&lt;/a&gt;. The cadence that these drills are performed can be set by your organization, and should be performed monthly to quarterly depending on incident frequency and team size.

&lt;h3 id=&quot;handling-an-incident&quot;&gt;Handling an incident&lt;/h3&gt;

&lt;h4 id=&quot;severity-levels&quot;&gt;Severity levels&lt;/h4&gt;

Before explaining how incidents should handled, it’s important that your organization establishes incident severity levels to help determine:
&lt;ul&gt;
  &lt;li&gt;Who should be paged.&lt;/li&gt;
  &lt;li&gt;Response time expectations.&lt;/li&gt;
  &lt;li&gt;Escalation paths.&lt;/li&gt;
  &lt;li&gt;Communication requirements.&lt;/li&gt;
&lt;/ul&gt;

A common severity classification framework is as follows:

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;P0/SEV1 (Critical)&lt;/strong&gt; - complete service outage or severe degradation affecting all or most users. Immediate response required.
    &lt;ul&gt;
      &lt;li&gt;Examples: entire application down, data loss occurring, security breach.&lt;/li&gt;
      &lt;li&gt;Response: page entire on-call team, establish incident command, update status page.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;P1/SEV2 (High)&lt;/strong&gt; - significant feature degradation or partial outage affecting a subset of users. Urgent response required.
    &lt;ul&gt;
      &lt;li&gt;Examples: critical feature broken, performance severely degraded, authentication failing intermittently.&lt;/li&gt;
      &lt;li&gt;Response: page primary on-call, consider incident commander for coordination.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;P2/SEV3 (Medium)&lt;/strong&gt; - minor feature issues or performance degradation with workarounds available. Timely response required.
    &lt;ul&gt;
      &lt;li&gt;Examples: non-critical feature broken, elevated error rates within SLO threshold.&lt;/li&gt;
      &lt;li&gt;Response: page primary on-call, can typically be handled during business hours.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;P3/SEV4 (Low)&lt;/strong&gt; - minor issues with minimal user impact. Can be addressed during normal work hours.
    &lt;ul&gt;
      &lt;li&gt;Examples: cosmetic bugs, minor inconveniences.&lt;/li&gt;
      &lt;li&gt;Response: create ticket, no page required.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;strong&gt;Your organization’s severity levels should align with your SLOs and business needs.&lt;/strong&gt;

&lt;h4 id=&quot;people&quot;&gt;People&lt;/h4&gt;

Though machines and AI are becoming more useful, humans are still at the core of IRM. &lt;a href=&quot;https://sre.google/resources/practices-and-processes/incident-management-guide/&quot;&gt;Google’s Incident Management Guide&lt;/a&gt; describes the following common people-roles in incident response:

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Incident Commander (IC)&lt;/strong&gt; - coordinates the overall incident response.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Operations Lead (OL)&lt;/strong&gt; - focuses on mitigating the issue, minimizing user impact, and resolving the problem.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Communication Lead (CL)&lt;/strong&gt; - provides regular updates to stakeholders and acts as a point of contact for incoming communications.
    &lt;ul&gt;
      &lt;li&gt;May also be responsible for updating incident docs and external customers.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

Additionally, a planner role may be introduced to assist in the tracking of work, filing bugs, managing shifts/handoffs (for incidents that span many hours or days), and coordinating long-term tasks.

&lt;strong&gt;The incident response roles in your organization may deviate from the roles defined above depending on your organization’s size and IRM configuration.&lt;/strong&gt;

&lt;h4 id=&quot;the-3-cs&quot;&gt;“The 3 C’s”&lt;/h4&gt;

It’s worth noting that the &lt;a href=&quot;https://nctr.pmel.noaa.gov/education/IOTWS/ICS/ICS_Factsheet_Feb07.pdf&quot;&gt;Incident Command System (ICS) standard used by the US government&lt;/a&gt; to respond to emergencies has proven successful and reproducible enough to be used in software incident response; even Google’s incident response system is based on ICS. ICS responds to emergencies using the “three C’s”:

&lt;ul&gt;
  &lt;li&gt;Coordinate - organize the response effort, assign roles, manage tasks, and keep everyone aligned.&lt;/li&gt;
  &lt;li&gt;Communicate - ensure information flows to the right people by providing clear, timely updates to incident responders, stakeholders, and users.&lt;/li&gt;
  &lt;li&gt;Control - maintain focus and overall contol of the incident by preventing chaos.
    &lt;ul&gt;
      &lt;li&gt;Some examples of chaos may be confused ownership of roles and taking premature actions.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

No matter the roles and size of your incident response team, the 3 C’s ought to be followed to help ensure smooth incident response.

&lt;h4 id=&quot;incident-response-workflow&quot;&gt;Incident response workflow&lt;/h4&gt;

When an incident occurs, a typical response workflow is:

&lt;ol&gt;
  &lt;li&gt;&lt;strong&gt;Acknowledge the alert&lt;/strong&gt; - confirm receipt of the alert through your incident management platform to stop further escalation.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Assess severity&lt;/strong&gt; - quickly determine the incident severity level using your organization’s severity classification framework. This determines response urgency and who else needs to be involved.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Establish communication channels&lt;/strong&gt; - set up dedicated incident communication:
    &lt;ul&gt;
      &lt;li&gt;Create an incident-specific chat channel.&lt;/li&gt;
      &lt;li&gt;Start a conference bridge if needed for real-time coordination.&lt;/li&gt;
      &lt;li&gt;Update user-facing status page if it exists.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Assign roles&lt;/strong&gt; - for higher severity incidents, explicitly assign incident response roles (Incident Commander, Operations Lead, Communication Lead).&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Begin investigation&lt;/strong&gt; - use observability tools to:
    &lt;ul&gt;
      &lt;li&gt;Identify affected components and scope of impact.&lt;/li&gt;
      &lt;li&gt;Review recent changes (deployments, config changes, infrastructure updates).&lt;/li&gt;
      &lt;li&gt;Formulate and test hypotheses about the root cause.&lt;/li&gt;
      &lt;li&gt;Document findings in the incident document.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Implement mitigation&lt;/strong&gt; - take action to restore service:
    &lt;ul&gt;
      &lt;li&gt;Stop the bleeding; prioritize restoring service over finding root cause.&lt;/li&gt;
      &lt;li&gt;Consider safe, reversible actions first (e.g., rollback, failover, scaling).&lt;/li&gt;
      &lt;li&gt;&lt;strong&gt;Communicate planned mitigation steps before executing.&lt;/strong&gt;&lt;/li&gt;
      &lt;li&gt;Verify each mitigation action’s effect before proceeding.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;(Optional) &lt;strong&gt;escalate the incident&lt;/strong&gt; if:
    &lt;ul&gt;
      &lt;li&gt;Impact is worse than initially assessed.&lt;/li&gt;
      &lt;li&gt;Mitigation efforts are unsuccessful after reasonable attempts.&lt;/li&gt;
      &lt;li&gt;Additional expertise or authority is needed.&lt;/li&gt;
      &lt;li&gt;Incident spans multiple teams or systems.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Verify resolution&lt;/strong&gt; - confirm the incident is resolved:
    &lt;ul&gt;
      &lt;li&gt;Check telemetry to verify that the symptoms have cleared.
        &lt;ul&gt;
          &lt;li&gt;Monitor error rates, metrics, traces, and logs.&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
      &lt;li&gt;If applicable, verify with affected users and/or stakeholders.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Hand off or close&lt;/strong&gt; - properly close out the incident:
    &lt;ul&gt;
      &lt;li&gt;Update status page and stakeholders.&lt;/li&gt;
      &lt;li&gt;Mark incident as resolved in incident management platform.&lt;/li&gt;
      &lt;li&gt;If further investigation is necessary, hand off to appropriate team.&lt;/li&gt;
      &lt;li&gt;Schedule postmortem review.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ol&gt;

&lt;h3 id=&quot;postmortems&quot;&gt;Postmortems&lt;/h3&gt;

A postmortem is a &lt;strong&gt;blameless&lt;/strong&gt; analysis of an incident, with a focus on learning and improving system reliability.

After each incident is resolved, there’s still work to be done; for each incident, it’s crucial to understand:

&lt;ul&gt;
  &lt;li&gt;What incident occured.&lt;/li&gt;
  &lt;li&gt;Why the incident occurred.&lt;/li&gt;
  &lt;li&gt;The impact the incident had.&lt;/li&gt;
  &lt;li&gt;How the incident was solved.&lt;/li&gt;
  &lt;li&gt;How the incident (and similar incidents) can be prevented in the future.&lt;/li&gt;
&lt;/ul&gt;

&lt;em&gt;Note that postmortems do not need to be “perfect”; postmortems should have sufficient details to be useful, while not taking too much time to write. Timeboxing may be an appropriate time-management technique to use when writing postmortems.&lt;/em&gt;

Good postmortems are:

&lt;ul&gt;
  &lt;li&gt;Written immediately after the incident was resolved, so that details can be captured accurately with fresh minds.&lt;/li&gt;
  &lt;li&gt;Distributed and transparently available inside of the organization, so that other teams can learn from the incident.
    &lt;ul&gt;
      &lt;li&gt;Sensitive user data MUST be redacted from the postmortem.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Reviewed and collaborated on by the appropriate parties.
    &lt;ul&gt;
      &lt;li&gt;Shared ownership and joint postmortem reviews help foster a successful IRM culture.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Blameless (see following section).&lt;/li&gt;
  &lt;li&gt;Followed up on (see following section).&lt;/li&gt;
&lt;/ul&gt;

&lt;em&gt;Note: Google’s SRE book has an &lt;a href=&quot;https://sre.google/sre-book/example-postmortem/&quot;&gt;Example Postmortem&lt;/a&gt; in it’s appendix.&lt;/em&gt;

&lt;h4 id=&quot;blameless&quot;&gt;Blameless&lt;/h4&gt;

Attaching personal blame to incidents creates a threat to psychological safety, encourages engineers to hide issues instead of address issues, and stunts learning.

Postmortems &lt;strong&gt;must&lt;/strong&gt; be blameless. Postmortems are meant to fix systems, not people. If a person “caused” an incident, the true cause is within the system/process. Acting blamelessly encourages the hardening of systems and processes to prevent similar incidents from happening in the first place.

&lt;h4 id=&quot;follow-ups&quot;&gt;Follow-ups&lt;/h4&gt;

After your organization gathers a few postmortems, it’s useful to look through your collection of postmortems to identify patterns. You may find recurring patterns such as monitoring gaps, issues with automation, or insufficient guardrails. Once common patterns are identified, solutions to these common failure patterns can be prioritized with appropriate effort (process hardening, automation, etc.) to improve the system’s reliability.

The investment in following up on postmortems is crucial for continuous improvement; no postmortem should go unreviewed.

&lt;h2 id=&quot;final-thoughts&quot;&gt;Final thoughts&lt;/h2&gt;

A well-defined IRM framework is a requirement when supporting critical production systems. Mutual agreement between application engineers, SREs, and business stakeholders must occur to foster a successful IRM culture and process. IRM is one component of Site Reliability Engineering that ultimately contributes to business continuity and satisfied customers through high service reliability.

&lt;ul id=&quot;markdown-toc&quot;&gt;
  &lt;li&gt;&lt;a href=&quot;#why-consider-incident-response-management&quot; id=&quot;markdown-toc-why-consider-incident-response-management&quot;&gt;Why consider Incident Response Management?&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#how-to-implement-incident-response-management&quot; id=&quot;markdown-toc-how-to-implement-incident-response-management&quot;&gt;How to implement Incident Response Management&lt;/a&gt;    &lt;ul&gt;
      &lt;li&gt;&lt;a href=&quot;#tools-needed&quot; id=&quot;markdown-toc-tools-needed&quot;&gt;Tools needed&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#irm-onboarding&quot; id=&quot;markdown-toc-irm-onboarding&quot;&gt;IRM Onboarding&lt;/a&gt;        &lt;ul&gt;
          &lt;li&gt;&lt;a href=&quot;#1-identify-critical-services&quot; id=&quot;markdown-toc-1-identify-critical-services&quot;&gt;1. Identify critical services&lt;/a&gt;            &lt;ul&gt;
              &lt;li&gt;&lt;a href=&quot;#define-slos&quot; id=&quot;markdown-toc-define-slos&quot;&gt;Define SLOs&lt;/a&gt;&lt;/li&gt;
            &lt;/ul&gt;
          &lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#2-define-an-alert-policy&quot; id=&quot;markdown-toc-2-define-an-alert-policy&quot;&gt;2. Define an alert policy&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#3-accept-a-service-into-on-call&quot; id=&quot;markdown-toc-3-accept-a-service-into-on-call&quot;&gt;3. Accept a service into on-call&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#4-create-runbooks&quot; id=&quot;markdown-toc-4-create-runbooks&quot;&gt;4. Create runbooks&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#5-be-proactive&quot; id=&quot;markdown-toc-5-be-proactive&quot;&gt;5. Be proactive&lt;/a&gt;&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#handling-an-incident&quot; id=&quot;markdown-toc-handling-an-incident&quot;&gt;Handling an incident&lt;/a&gt;        &lt;ul&gt;
          &lt;li&gt;&lt;a href=&quot;#severity-levels&quot; id=&quot;markdown-toc-severity-levels&quot;&gt;Severity levels&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#people&quot; id=&quot;markdown-toc-people&quot;&gt;People&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#the-3-cs&quot; id=&quot;markdown-toc-the-3-cs&quot;&gt;“The 3 C’s”&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#incident-response-workflow&quot; id=&quot;markdown-toc-incident-response-workflow&quot;&gt;Incident response workflow&lt;/a&gt;&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#postmortems&quot; id=&quot;markdown-toc-postmortems&quot;&gt;Postmortems&lt;/a&gt;        &lt;ul&gt;
          &lt;li&gt;&lt;a href=&quot;#blameless&quot; id=&quot;markdown-toc-blameless&quot;&gt;Blameless&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#follow-ups&quot; id=&quot;markdown-toc-follow-ups&quot;&gt;Follow-ups&lt;/a&gt;&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#final-thoughts&quot; id=&quot;markdown-toc-final-thoughts&quot;&gt;Final thoughts&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</description>
        <pubDate>Fri, 28 Nov 2025 00:00:00 +0000</pubDate>
        <link>https://srezone.com/blog/2025/11/28/incident-response-management/</link>
        <guid isPermaLink="true">https://srezone.com/blog/2025/11/28/incident-response-management/</guid>
        
        <category>Alerting</category>
        
        <category>SLOs</category>
        
        <category>Incidents</category>
        
        <category>Production</category>
        
        <category>Support</category>
        
        <category>Uptime</category>
        
        
        <category>Production-Support</category>
        
      </item>
    
      <item>
        <title>Moving beyond anomaly detection</title>
        <description>This post outlines basic alerting approaches, “anomaly detection”, and why SLOs are necessary for appropriate monitoring and alerting

&lt;h1 id=&quot;basic-alerting-approaches&quot;&gt;Basic alerting approaches&lt;/h1&gt;

Being notified about an issue with your service(s) commonly happened in a couple of ways:

&lt;ol&gt;
  &lt;li&gt;Being notified by the customer(s)
    &lt;ul&gt;
      &lt;li&gt;How embarrassing! This is the worst type of notification – damage has already been done and customers are frustrated. Monitoring and alerting must be more proactively implemented&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Alerting based on log record content
    &lt;ul&gt;
      &lt;li&gt;Simple, but transient errors will introduce alert noise and fatigue. Additionally, clever users can include an alertable keyword such as “ERROR” in their request to trigger a false alert&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Synthetic monitoring
    &lt;ul&gt;
      &lt;li&gt;Sure, automated probes checking your services health can be a comforting thought, but it doesn’t represent true end user experience&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Static metric thresholds
    &lt;ul&gt;
      &lt;li&gt;Noisy and impractical, since compute is naturally bursty/spikey&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Static metric thresholds with a sliding window (i.e.: alert when a threshold is breached for N minutes)
    &lt;ul&gt;
      &lt;li&gt;A bit “better” than a windowless static threshold, but introduces delayed detection and requires additional fine-tuning to prevent over-alerting and under-alerting&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ol&gt;

&lt;em&gt;More detailed ideas about how signals can be used for alerting can be found &lt;a href=&quot;/blog/2023/10/14/monitoring/#signals&quot;&gt;here&lt;/a&gt;&lt;/em&gt;

&lt;h1 id=&quot;introducing-anomaly-detection&quot;&gt;Introducing: anomaly detection&lt;/h1&gt;
Basic alerting approaches have their limitations, so this is a perfect opportunity for observability vendors to upsell. “Add our instrumentation to your apps, we’ll take care of the rest (including alerting)”

But how could a vendor solution automatically detect when your service is not behaving appropriately if it doesn’t even know &lt;em&gt;what&lt;/em&gt; your service’s expected behavior is? Machine learning, AI, and other statistical techniques can &lt;em&gt;help&lt;/em&gt; with identifying abnormal behavior via anomaly detection

Anomaly detection starts by calculating a “performance baseline” based on a systems previous performance data (typically, metrics). As the system continues to emit metrics, these metrics are continuously evaluated against (and incorporated into) the performance baseline in the backend monitoring tool. When the most recent metrics exceed the baseline’s standard deviation threshold (outliers), an alert is triggered

Anomaly detection is offered as a fairly effortless solution for alerting, but it’s important to be aware of its shortcomings

&lt;h2 id=&quot;shortcomings&quot;&gt;Shortcomings&lt;/h2&gt;

&lt;h3 id=&quot;uncommon-events&quot;&gt;Uncommon events&lt;/h3&gt;

High-traffic events such as &lt;a href=&quot;https://grafana.com/docs/grafana-cloud/alerting-and-irm/machine-learning/forecasts/holidays/&quot;&gt;holidays, launches, and special sales&lt;/a&gt; may:
&lt;ul&gt;
  &lt;li&gt;Trigger false-positive alerts due to a [business-expected] fluctuation of performance or traffic&lt;/li&gt;
  &lt;li&gt;Skew your service’s baseline by incorporating these abnormal metrics into its baseline&lt;/li&gt;
&lt;/ul&gt;

A way around this shortcoming is to specify “maintenance windows” in your monitoring tool, which will:
&lt;ul&gt;
  &lt;li&gt;Disable alerts for the specified time period&lt;/li&gt;
  &lt;li&gt;Exclude metrics emitted during the specified time period from the baseline calculation&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;boiling-frog&quot;&gt;Boiling frog&lt;/h3&gt;

If performance &lt;em&gt;slowly&lt;/em&gt; degrades over time, anomaly detections will incorporate these slightly-degraded metrics into the baseline as “normal”. This can lead to the &lt;a href=&quot;https://en.wikipedia.org/wiki/Boiling_frog&quot;&gt;“boiling frog”&lt;/a&gt; problem

&lt;h4 id=&quot;example&quot;&gt;Example&lt;/h4&gt;

The below example outlines how a service latency could &lt;em&gt;double&lt;/em&gt; over time, while sneaking under anomaly detection’s radar

&lt;ol&gt;
  &lt;li&gt;Service latency is initially baselined at 500ms&lt;/li&gt;
  &lt;li&gt;Something related to or within the service changes (e.g.: dependency is added, code is changed, infrastructure allocation decreases, etc.), bumping service latency baseline to 550ms (10% increase, no big deal in most systems)&lt;/li&gt;
  &lt;li&gt;Repeat step 2 a couple of weeks later, causing latency to increase by another 10% (605ms)&lt;/li&gt;
  &lt;li&gt;Repeat steps 2 and 3 (slowly increasing the latency by 10%) ~six more times, and your “normal” baseline latency now surpasses 1sec; roughly double the initial baseline&lt;/li&gt;
&lt;/ol&gt;

While anomaly detection may not pick up on this issue, your users will

&lt;h3 id=&quot;complex&quot;&gt;Complex&lt;/h3&gt;

The standard-deviation-threshold algorithm mentioned earlier is a basic example of anomaly detection and has it’s shortcomings, so why wouldn’t a vendor upsell customers on a more “advanced” (complex) algorithm for anomaly detection?

They do. But there are still issues with these more complex models. How do we know the proper ML algorithm is used? How can we ensure our data is correctly labeled for proper evaluation? Who trained these models, and was the model trained on sufficient, valid data?

A vendor could easily hand-pick data into a demo to show these complex models working and alerting perfectly against &lt;em&gt;their tailored data&lt;/em&gt;, but when your “foreign” metrics are introduced to the model, there’s a high chance that the results won’t be as promising

&lt;em&gt;Disclaimer: I’m far from an ML or AI expert; the above concerns are just some thoughts that come to mind&lt;/em&gt;

&lt;h1 id=&quot;moving-on-slos&quot;&gt;Moving on: SLOs&lt;/h1&gt;

Sure, anomaly detection can provide some initial, low-effort alerts and can also be applied to several problem spaces (e.g.: security and fraud detection), though the best way to gauge and monitor the performance of your systems is through explicitly defined &lt;a href=&quot;https://sre.google/sre-book/service-level-objectives/&quot;&gt;Service Level Objectives (SLOs)&lt;/a&gt;

While SLOs have shortcomings of their own, I can’t think of an alerting approach that’s more robust&lt;del&gt;, simple, and understandable&lt;/del&gt; (okay, there is &lt;em&gt;some&lt;/em&gt; learning curve when first exploring SLOs) than defining an SLO for what “good” looks like, then alerting on the &lt;a href=&quot;https://sre.google/workbook/alerting-on-slos/#6-multiwindow-multi-burn-rate-alerts&quot;&gt;error-budget burn rate&lt;/a&gt; for proactive issue detection. No complex models or algorithms, just simple(-ish), white-box math! As a bonus, you can use SLOs to support your SLAs

So rather than having some “magical” anomaly detection alert you when it &lt;em&gt;thinks&lt;/em&gt; something is wrong, YOU are empowered to (and should!) explicitly define what acceptable behavior is for your systems, through SLOs

&lt;ul id=&quot;markdown-toc&quot;&gt;
  &lt;li&gt;&lt;a href=&quot;#basic-alerting-approaches&quot; id=&quot;markdown-toc-basic-alerting-approaches&quot;&gt;Basic alerting approaches&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#introducing-anomaly-detection&quot; id=&quot;markdown-toc-introducing-anomaly-detection&quot;&gt;Introducing: anomaly detection&lt;/a&gt;    &lt;ul&gt;
      &lt;li&gt;&lt;a href=&quot;#shortcomings&quot; id=&quot;markdown-toc-shortcomings&quot;&gt;Shortcomings&lt;/a&gt;        &lt;ul&gt;
          &lt;li&gt;&lt;a href=&quot;#uncommon-events&quot; id=&quot;markdown-toc-uncommon-events&quot;&gt;Uncommon events&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#boiling-frog&quot; id=&quot;markdown-toc-boiling-frog&quot;&gt;Boiling frog&lt;/a&gt;            &lt;ul&gt;
              &lt;li&gt;&lt;a href=&quot;#example&quot; id=&quot;markdown-toc-example&quot;&gt;Example&lt;/a&gt;&lt;/li&gt;
            &lt;/ul&gt;
          &lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#complex&quot; id=&quot;markdown-toc-complex&quot;&gt;Complex&lt;/a&gt;&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#moving-on-slos&quot; id=&quot;markdown-toc-moving-on-slos&quot;&gt;Moving on: SLOs&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</description>
        <pubDate>Sun, 16 Jun 2024 00:00:00 +0000</pubDate>
        <link>https://srezone.com/blog/2024/06/16/beyond-anomaly-detection/</link>
        <guid isPermaLink="true">https://srezone.com/blog/2024/06/16/beyond-anomaly-detection/</guid>
        
        <category>Monitoring</category>
        
        <category>Alerting</category>
        
        <category>SLOs</category>
        
        <category>Anomaly</category>
        
        <category>Detection</category>
        
        
        <category>Monitoring</category>
        
      </item>
    
      <item>
        <title>AWS re:Invent 2023 - an SREs experience</title>
        <description>I was fortunate enough to be sponsored by my employer to attend AWS re:Invent 2023 (my first conference). This post describes some of the things I learned and thoughts I had from attending the conference

&lt;h1 id=&quot;learnings&quot;&gt;Learnings&lt;/h1&gt;

&lt;h2 id=&quot;resilience&quot;&gt;Resilience&lt;/h2&gt;

&lt;em&gt;“Everything fails all the time” –Werner Vogels, CTO of Amazon&lt;/em&gt;

&lt;h3 id=&quot;failovers&quot;&gt;Failovers&lt;/h3&gt;

Resilience exists on a spectrum: from basic backups (minimally resilient) to active-active deployments (extremely resilient). There are trade-offs for each configuration; if your application requires high resilience (such as multi-region deployments), you must take into account the added complexity, monetary cost, and requirement to &lt;strong&gt;exercise your remediatiation procedure often&lt;/strong&gt;. Two suggestions for exercising remeditation procedures with multi-region deployments (i.e.: failovers) are 1) periodically alternating live traffic between regions, and 2) having preset dates in which you intentionally force a failover to test that the remediation is still working as intended. After all, if a failover region is unhealthy/unusable when it’s needed, what’s the point of having one?

As much as SREs are concerned with high availability, a presenter at a resilience Chalk Talk gave a refreshing statement: “chances are your application does not need multi-region deployments”. The reason being: a single AWS region with multiple AZ’s (availabilty zones) is already pretty resilient. Of course, this statement was followed up by an audience member pointing out the &lt;a href=&quot;https://www.sdxcentral.com/articles/news/aws-us-east-1-region-outage-downed-many-websites-now-partly-resolved-2/2023/06/&quot;&gt;~3-hour-long us-east-1 outage of June 2023&lt;/a&gt;, which the audience and presenters playfully laughed off. This sort of widespread outage is not a common occurrence (and I trust a proper postmortem was conducted to prevent similar issues in the future), but depending on the requirements of your system, you may still want to err on the side of caution

&lt;h3 id=&quot;throttling-timeouts-and-retries&quot;&gt;Throttling, timeouts, and retries&lt;/h3&gt;

When performing throttling, timeouts, or retries: attempt them as early in the call chain as possible. Throttling should occur as close to the client as possible to prevent overwhelming the backend (typically a result of buggy clients or maliciously-intended denial-of-service attacks). Retries should be implemented with backoff-and-jitter to avoid continuously slamming downstream services, which can easily lead to &lt;a href=&quot;https://sre.google/sre-book/addressing-cascading-failures/&quot;&gt;cascading failures&lt;/a&gt;

When working in microservice ecosystems, timeouts should be the highest in duration nearest the caller and lessen in timeout duration as the request goes more downstream. If a downstream component takes its time processing a request (and especially if it performs multiple retries), there’s a chance that by the time said component finally generates a successful response to return, the front-end component has already timed out the clients request. Throwing away the “successful” response in this scenario demonstrates wasted compute resources

Resilience techniques that apply to synchronous systems (as described above) do not always apply 1:1 to asynchronous systems. For example: when dealing with failures and retries from messages on queues, it’s common practice for failed messages to be moved to a DLQ (dead-letter queue) after N [failed] attempts of processing said messages. These DLQ messages can be purged at some point, or if these DLQ messages need to be reprocessed (“redriven”), common practice is to: identify the cause of failure, push code/configs that resolve the failure, then redrive the messages to the original queue for reprocessing. When redriving, it’s extremely important to &lt;strong&gt;ensure that your component performs idempotent operations&lt;/strong&gt; to make certain that data and operations are not duplicated!

&lt;h3 id=&quot;related-aws-services&quot;&gt;Related AWS services&lt;/h3&gt;

AWS offers services (such as &lt;a href=&quot;https://aws.amazon.com/resilience-hub/&quot;&gt;Resilience Hub&lt;/a&gt;) that allow you to set recovery goals (e.g: RTO and RPO) and get automatic recommendations (e.g.: use autoscaling, CW alarms, etc.) to help guide you to meet your resilience goals. Additional tools such as &lt;a href=&quot;https://aws.amazon.com/fis/&quot;&gt;FIS (Fault Injection Service)&lt;/a&gt; and &lt;a href=&quot;https://aws.amazon.com/solutions/implementations/distributed-load-testing-on-aws/&quot;&gt;distributed load testing&lt;/a&gt; exist as well to push your systems to their limits and gain confidence in their robustness. While a lot of these features are nice to have out of the box, it’s important not to overlook open-source tools that accomplish similar

&lt;h3 id=&quot;further-learning-resources&quot;&gt;Further learning resources&lt;/h3&gt;

After a resilience Chalk Talk, I asked one of the presenters if he had any reading recommendations to learn more about resilience best practices. The resources he mentioned were &lt;a href=&quot;https://serverlessland.com/&quot;&gt;ServerlessLand&lt;/a&gt;, &lt;a href=&quot;https://s12d.com/aws-resilience&quot;&gt;Serverless Resilience References&lt;/a&gt;, and “serverlesspresso”

&lt;h2 id=&quot;service-level-objectives&quot;&gt;Service Level Objectives&lt;/h2&gt;

SLOs alone are not enough to define what reliability looks like for an application. Reliability isn’t defined just by engineers, either. The three service-level-*’s along with their (brief) descriptions and corresponding stewards are:

&lt;ul&gt;
  &lt;li&gt;SLAs (Agreements) - the Agreement between business and consumer that the service will offer a certain level of availability. If the service breaches the SLA, repercussions such as credit issuance or refunds to the customer will occur. In some cases, legal action may be taken. SLAs are commonly formulated by sales teams (and likely involve legal teams)&lt;/li&gt;
  &lt;li&gt;SLOs (Objectives) - the Objective/goal for the service to meet; typically a bit tighter than the SLA (e.g.: if an SLA is 99%, the SLO may be 99.9%) to provide a cushion to help protect the SLA. SLOs are commonly designed by product teams and engineers&lt;/li&gt;
  &lt;li&gt;SLIs (Indicators) - the Indicator of how a service is performing, usually calculated as a percentage (typically: &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;sli = goodEventsCount/totalEventsCount&lt;/code&gt;) for evaluation against an SLO. Traditionally, metrics have been used to calculate SLIs, however, SLIs can also be derived from anything with a numerical value (e.g.: counts of certain distributed trace spans). SLIs are most often implemented by engineers&lt;/li&gt;
&lt;/ul&gt;

The more services that have aligning SLOs, the better. If a critical component does not align with its dependents SLOs, the entire system is at risk, since there’s no mutual acknowledgement of expected service level

Note that SLOs can be moving targets. As technical and customer requirements change, so will the SLO goal. If an SLO goal changes, it’s likely that the system will change as well. In business terms: as requirements and SLOs change, the cost of operating the system may be reduced (in the case of loosening the SLO) or increased (in the case of tightening the SLO)

&lt;h3 id=&quot;slos-in-aws&quot;&gt;SLOs in AWS&lt;/h3&gt;

At a Chalk Talk about SLOs, I asked the presenters “what are the next steps/how to get started with SLOs in AWS?”. I was answered with “nothing exists in AWS to create SLOs as of &lt;em&gt;today&lt;/em&gt;, but [strong hint] listen to the keynote on Thursday”. Sure enough, at that keynote, Werner announced CloudWatch “Application Signals”, with one of the features being SLOs!

After re:Invent I hopped into the AWS console to experiment with SLOs in CloudWatch. I was able to create an SLO based on CloudWatch metrics that’d fuel the SLI. Unfortunately, I backed out of my timeboxed exploration shortly after realizing the sandbox account I was in didn’t have any traffic (or at least the metrics I chose didn’t have values to work with) nor components I was familiar with. That’s my bad – if you want more details about SLOs in AWS, check out &lt;a href=&quot;https://aws.amazon.com/blogs/mt/how-to-monitor-application-health-using-slos-with-amazon-cloudwatch-application-signals/&quot;&gt;this AWS blog post&lt;/a&gt;

Out of curiosity, I checked the CloudFormation and AWS Terraform provider docs for an SLO resource, but didn’t find anything. Fair enough; I trust as AWS receives feedback on this SLO feature from its users and the product matures, the API may change a bit (no need to write an IaC component twice)

&lt;h3 id=&quot;forget-the-9s-in-some-cases&quot;&gt;Forget the 9’s (in some cases)!&lt;/h3&gt;

When thinking about SLO goals, most people immediately reach for 9’s (99%, 99.9%, 99.99%, …). For a highly-available service, that might be fine, but not all systems need that level of availability. Maybe 80% is all your users need to be happy. Best to chat with your product team and stakeholders before building an overly reliable system, and remember that the closer to 100% the SLO goal is, the more expensive the system will be

Also, higher SLO goals aren’t always better – it depends on the systems expected outcomes. Take for example, an SLO measuring the correctness of a coin flip application. If the coin lands on heads 99.99% of the time, clearly something is wrong. For this coin flip application, we’d want a correctness SLO with a goal of 50%. The formula for calculating this SLI could be &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;sli = headsResultsCount/totalFlipCount&lt;/code&gt;. Giving the SLO some tolerance such as ±2%, the SLI value would have to be between 48% and 52% to ensure SLO is not breached. A more in-depth article regarding this type of correctness SLO can be found &lt;a href=&quot;https://thenewstack.io/make-sure-your-application-comes-correct-with-correctness-slos/&quot;&gt;here&lt;/a&gt;

&lt;h2 id=&quot;observability&quot;&gt;Observability&lt;/h2&gt;

&lt;h3 id=&quot;x-ray-and-beyond&quot;&gt;X-Ray (and beyond)&lt;/h3&gt;

One of the challenges my team has faced was “how do we [distributed] trace events over EventBridge”? While my team was able to implement a solution by injecting the w3c &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;traceparent&lt;/code&gt; into the event payload (rather than into the headers, since EventBridge dropped any custom headers at the time), the solution felt a bit hacky. I pressed on the presenters at the o11y Chalk Talk about this, and they suggested the use of X-Ray to fulfill the need. No surprise there, and unfortunately I didn’t probe farther on non-X-Ray support for distributed tracing over EventBridge. So for now, we can only hope that AWS will offer better support for non-X-Ray tracing (i.e.: OpenTelemetry) in more of their services in the future

I chatted more with the presenters and I mentioned that “active tracing” is baked into several other AWS services (such as API Gateway). “&lt;em&gt;If&lt;/em&gt; my team uses X-Ray to get end-to-end distributed traces, my team would still like to see the traces in an o11y backend other than X-Ray. Is this possible?” The presenters revealed that [AWS managed] Grafana has a &lt;a href=&quot;https://grafana.com/grafana/plugins/grafana-x-ray-datasource/&quot;&gt;plugin&lt;/a&gt; to display traces from X-Ray. Neat

It also appears that the OTel collector offers an &lt;a href=&quot;https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/f089aa29cae902ff1d81e5cfc9f91ffb45678551/receiver/awsxrayreceiver&quot;&gt;X-Ray receiver&lt;/a&gt;. Since this receiver receives &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;spans emitted by the existing X-Ray SDK&lt;/code&gt;, which seems to only be &lt;a href=&quot;https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-nodejs-configuration.html#xray-sdk-nodejs-configuration-daemon&quot;&gt;configurable in custom code&lt;/a&gt;, I don’t believe it’s possible to configure X-Ray settings for managed services (such as API Gateway or EventBridge) to send X-Ray spans to an X-Ray receiver in your own OTel collector. Would be a cool feature to have in the future, though!

While all AWS services aren’t 100% interoperable with OTel just yet, the Grafana X-Ray plugin and OTel collector X-Ray receiver give me a feeling that things are heading in the right direction for more interoperability in the future

&lt;h3 id=&quot;rum&quot;&gt;RUM&lt;/h3&gt;

Nothing groundbreaking here, just news to me: CloudWatch does offer RUM (Real User Monitoring). As with most browser RUM tools, simply configure some settings then add the provided &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&amp;lt;script&amp;gt;&lt;/code&gt; snippet to your HTML code to enable RUM data to flow in

&lt;em&gt;Before investing fully into CloudWatch RUM, it may be worth exploring &lt;a href=&quot;https://opentelemetry.io/docs/instrumentation/js/getting-started/browser/&quot;&gt;OpenTelemetry’s browser instrumentation&lt;/a&gt;&lt;/em&gt;

On the topic of browser monitoring, though not quite RUM, I learned that CloudWatch also offers “&lt;a href=&quot;https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Synthetics_Canaries.html&quot;&gt;synthetic canaries&lt;/a&gt;” to test your webpages and display diffs between what the synthetic requests captured vs. what was expected

&lt;h3 id=&quot;cross-account-and-inter-region-observability&quot;&gt;Cross-account and inter-region observability&lt;/h3&gt;

I’ve before wondered: how could we get &lt;em&gt;all&lt;/em&gt; of our metrics and logs (from all regions and AWS accounts) in one place, without paying an expensive egress cost? This question resurfaced in my mind as I worked through a CloudWatch workshop and noticed CloudWatch was a regional service, so I raised the question to the facilitator

The answer I got was more around &lt;a href=&quot;https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Unified-Cross-Account.html&quot;&gt;CloudWatch unified/cross-account observability&lt;/a&gt;. There’s allegedly no egress charges for this (for logs and metrics) since the “monitoring” account is simply given IAM permissions to &lt;em&gt;view&lt;/em&gt; telemetry (metrics, logs, traces, etc.) residing in connected AWS accounts. After digging around the AWS docs a bit more, it seems that &lt;a href=&quot;https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Cross-Account-Cross-Region.html&quot;&gt;cross-region&lt;/a&gt; support also exists, but I wasn’t able to find much details around pricing

&lt;h1 id=&quot;the-conference-itself&quot;&gt;The conference itself&lt;/h1&gt;

&lt;h2 id=&quot;wishes&quot;&gt;Wishes&lt;/h2&gt;

The re:Invent [Android] app could’ve been better. Frequently expiring credentials in the app required multiple logins throughout the week of the conference. The app always required internet access to open – if you needed to access your schedule while you didn’t have WiFi or service (e.g.: in underground bus terminals), you were SoL. I heard rumors that AWS contracted this app to be built by a third party, and based on my experience with it, I believe it

Attendees could’ve been less shy. For example, the amount of attendees eating alone at lunch was a bit disheartening. The opportunity to network with other professionals from around the world over lunch is a gift. I sure took the opportunity to meet a new group of people at each meal, connecting with intelligent, ambitious professionals and I am grateful for it

&lt;h2 id=&quot;suggestions-for-future-attendees&quot;&gt;Suggestions for future attendees&lt;/h2&gt;

Reserve your sessions online ASAP! Otherwise you’ll spend a good amount of time waiting in line for the &lt;em&gt;chance&lt;/em&gt; to attend sessions

Sessions are scattered along venues on The Strip. Aim to have most of your sessions at 1 venue per day (or leave at least an hour+ between sessions at different venues) to ensure you’re not rushing between sessions. Bring your issued re:Invent badge everywhere (even afterhours) for access to shortcuts that connect venues

Builder sessions/&lt;a href=&quot;https://workshops.aws/&quot;&gt;workshops&lt;/a&gt; contain a lot of content – likely too much to finish in one sitting. Perform the workshops at home before (or after) the conference (or don’t, and simply skim the workshop content), but either way: &lt;strong&gt;ask questions to the facilitators&lt;/strong&gt;. That’s what you’re paying for. Whether facilitators are principal engineers or technical account managers, they know their AWS products inside and out, and are more than capable of providing answers that may just blow you away

Try to avoid breakout sessions, as they aren’t very engaging and the recordings are uploaded to YouTube for later watching. A lot of the information from breakout sessions can be found in AWS whitepapers, too

With Chalk Talks, stay after the talks to ask additional questions to presenters as well as hear what other attendees have to ask. Also, talk to other attendees! I was fortunate enough to [unknowingly] sit next to and chat with the VP of SRE at a major banking company during a Chalk Talk. Through chatting with him, I learned his company has a 30-40% adoption rate of SRE practices across their services (which I think is impressive)! SRE maturity for their teams is tracked through a scorecard (with criteria such as SLOs, alerts, incident count, and toil) that can be used as a data-driven approach to nudge teams into compliance with SRE best practices

If you haven’t picked up on this already: talk to everyone! Prioritize chatting with participants in sessions you’re attending, whether you’re waiting in a queue or waiting for the presenters to start their talk. Don’t be afraid to sit and chat with new faces at conference-provided meals. Last but not least, if you’re flying into re:Invent, chat with people (who are also likely traveling for the conference) on the plane (I made a great connection doing this)! Optionally, wear some tech swag (tech-branded clothing) to give others a reason to strike up a conversation with you

If you want to eat meals provided by the conference, the earlier you get to a dinning room, the better. Don’t show up any later than 30 minutes before the meal ends

Allocate at least a couple of hours to check out the Expo where hundreds of vendors are set up. Know which vendors are attending, and if you’re lucky enough, you might bump into some of your tech superheroes ;). Also, leave some extra space in your luggage for all the swag you’ll collect

Weeks (or months?) before the conference, search online to find out which re:Invent conference “parties” are happening in the evenings. Most of these are great networking events sponsored by vendors, and spots can fill up pretty quick. Unfortunately for me, the parties I wanted to attend were already booked up by the time I found out about them. Either way, it’s Vegas – there’s always something to do. On the last night of the conference, be sure to attend re:Play for a night of fun, free food, and an open bar

&lt;h1 id=&quot;closing&quot;&gt;Closing&lt;/h1&gt;

re:Invent 2023 was a wonderful experience and it is what you make of it. From my perspective, the conference was roughly 40% networking, 35% learning, and 25% sales. The people you’ll meet and the things you’ll learn are invaluable for any engineers career

&lt;ul id=&quot;markdown-toc&quot;&gt;
  &lt;li&gt;&lt;a href=&quot;#learnings&quot; id=&quot;markdown-toc-learnings&quot;&gt;Learnings&lt;/a&gt;    &lt;ul&gt;
      &lt;li&gt;&lt;a href=&quot;#resilience&quot; id=&quot;markdown-toc-resilience&quot;&gt;Resilience&lt;/a&gt;        &lt;ul&gt;
          &lt;li&gt;&lt;a href=&quot;#failovers&quot; id=&quot;markdown-toc-failovers&quot;&gt;Failovers&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#throttling-timeouts-and-retries&quot; id=&quot;markdown-toc-throttling-timeouts-and-retries&quot;&gt;Throttling, timeouts, and retries&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#related-aws-services&quot; id=&quot;markdown-toc-related-aws-services&quot;&gt;Related AWS services&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#further-learning-resources&quot; id=&quot;markdown-toc-further-learning-resources&quot;&gt;Further learning resources&lt;/a&gt;&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#service-level-objectives&quot; id=&quot;markdown-toc-service-level-objectives&quot;&gt;Service Level Objectives&lt;/a&gt;        &lt;ul&gt;
          &lt;li&gt;&lt;a href=&quot;#slos-in-aws&quot; id=&quot;markdown-toc-slos-in-aws&quot;&gt;SLOs in AWS&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#forget-the-9s-in-some-cases&quot; id=&quot;markdown-toc-forget-the-9s-in-some-cases&quot;&gt;Forget the 9’s (in some cases)!&lt;/a&gt;&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#observability&quot; id=&quot;markdown-toc-observability&quot;&gt;Observability&lt;/a&gt;        &lt;ul&gt;
          &lt;li&gt;&lt;a href=&quot;#x-ray-and-beyond&quot; id=&quot;markdown-toc-x-ray-and-beyond&quot;&gt;X-Ray (and beyond)&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#rum&quot; id=&quot;markdown-toc-rum&quot;&gt;RUM&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#cross-account-and-inter-region-observability&quot; id=&quot;markdown-toc-cross-account-and-inter-region-observability&quot;&gt;Cross-account and inter-region observability&lt;/a&gt;&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#the-conference-itself&quot; id=&quot;markdown-toc-the-conference-itself&quot;&gt;The conference itself&lt;/a&gt;    &lt;ul&gt;
      &lt;li&gt;&lt;a href=&quot;#wishes&quot; id=&quot;markdown-toc-wishes&quot;&gt;Wishes&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#suggestions-for-future-attendees&quot; id=&quot;markdown-toc-suggestions-for-future-attendees&quot;&gt;Suggestions for future attendees&lt;/a&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#closing&quot; id=&quot;markdown-toc-closing&quot;&gt;Closing&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</description>
        <pubDate>Mon, 15 Jan 2024 00:00:00 +0000</pubDate>
        <link>https://srezone.com/blog/2024/01/15/reinvent2023/</link>
        <guid isPermaLink="true">https://srezone.com/blog/2024/01/15/reinvent2023/</guid>
        
        <category>Conference</category>
        
        <category>AWS</category>
        
        <category>re:Invent</category>
        
        <category>Observability</category>
        
        <category>Resilience</category>
        
        <category>SLOs</category>
        
        
        <category>Experiences</category>
        
      </item>
    
      <item>
        <title>Monitoring (and alerting)</title>
        <description>&lt;em&gt;Note: this post focuses primarily on monitoring and alerting; not observability&lt;/em&gt;

&lt;h2 id=&quot;reality-check&quot;&gt;Reality check&lt;/h2&gt;
As brash as it sounds: if you don’t have adequate monitoring and alerting built into your systems (along with proper incident response), then you don’t care about your users

Without proper M+A (Monitoring and Alerting) practices, you risk poor user experiences, which undoubtedly leads to loss of revenue, reputation, and customer trust. Proper M+A (preferably, SLO-driven) along with remediation techniques are crucial to the success of your business

As a rule of thumb: if it’s not monitored, it’s not ready for production. Unfortunately, engineers commonly implement monitoring solely to “check the box”. This can be even worse than not having any monitoring at all – think about the noise generated from false alerts, or flat out spinning your wheels to monitor things that yield no value. It’s vital that your M+A approach is sound and enables you to serve your customers reliably

&lt;h2 id=&quot;how-to-do-it-right&quot;&gt;How to do it right?&lt;/h2&gt;

&lt;h3 id=&quot;an-apm-software-solution&quot;&gt;An APM Software Solution&lt;/h3&gt;

APM (Application Performance Monitoring) solutions come in different shapes and sizes, and at the end of the day aim to provide insights about how your systems are performing. From the data an APM solution captures, to the way it’s built (FOSS, vendor, in-house, etc.), to how it’s deployed (self-managed or as a SaaS), there exist solutions for everyone. Assuming you’re not a M+A company, your best option usually is to purchase an APM solution as a SaaS. A managed SaaS solution is unarguably much less expensive than developing and maintaining an APM solution in-house

Before jumping into buying a SaaS solution that &lt;a href=&quot;/blog/2023/08/12/vendorLockIn/&quot;&gt;falsely promises you the world&lt;/a&gt;, know that there’s no such thing as a tool that offers perfect M+A out-of-the-box (while vendors may try to sell you on this, these tools have zero context about your app or business immediately; you’ll need to invest a bit into tuning the solution before you unlock its full potential). The tool(s) that work best for your organization will take some investigation and configuring, and even then, each solution has its limitations – like everything else in engineering, there are trade-offs to consider

&lt;h4 id=&quot;one-solution-is-not-always-enough&quot;&gt;One solution is not always enough&lt;/h4&gt;

One APM solution may not be sufficient for your orgs needs – you may need to equip your teams with multiple tools to ensure they can achieve operational excellence. Having multiple tools (tool fragmentation) can enable your org with the “best of breed” solutions (keep in mind these tools should work together and correctly correlate your data). For example: you may use ElasticSearch for logs, prometheus for metrics, and a different tool for the alerting. There’s nothing wrong with this, as long as the tools work in harmony

Having several tools in the M+A space adds some overhead, but this may be the appropriate approach based on your orgs needs. As long as you’re investigating tool(s) that fill a need for your engineering teams – whilst erring on the side of caution of vendor lock-in – you should be on the right path

&lt;em&gt;Disclaimer: I’m not saying to grab as many shiny tools as you can from the get-go; starting small with a single solution is usually a good first step, then evolve as necessary&lt;/em&gt;

&lt;h3 id=&quot;culture&quot;&gt;Culture&lt;/h3&gt;

Everyone in your org needs to be on the same page with your M+A and incident response process. From the engineers to upper-level management – on-call or not – the practices must be understood (at different depths, depending on the role) and adopted accordingly. Not having this transparency runs the risk of engineer burnout, underappreciated on-call engineers, excuses, and system downtime

It’s also important that a &lt;strong&gt;blameless&lt;/strong&gt; culture is in place &lt;a href=&quot;https://sre.google/workbook/postmortem-culture/&quot;&gt;to ensure more reliable systems&lt;/a&gt; and to foster a healthy engineering community

&lt;h4 id=&quot;who-builds-it-runs-it&quot;&gt;Who builds it, runs it&lt;/h4&gt;
I’m a firm believer of “you build it, you run it” (basically, if you write the software, you should be responsible for it in production as well). The contrary often results in teams writing and deploying software, then expecting a help-desk/support technician to execute steps from a runbook (more on this later) to solve issues when they occur. This is a &lt;em&gt;terrible&lt;/em&gt; idea!

Firstly, because no matter how well-documented the software and/or runbook is, the software authors likely know how to address its issues best. Having the software author troubleshoot an issue in their application usually results in a lower mean-time to resolution (MTTR) (AKA, happier customers)

Secondly, there’s not much incentive for engineers to write resilient software if they’re not on the hook for its issues in production – not only will there be more bugs, but there likely won’t be sufficient visibility into the system for the poor soul who has to resolve it’s incidents!

&lt;h5 id=&quot;handling-pushback&quot;&gt;Handling pushback&lt;/h5&gt;
As much kicking and screaming engineers may do when hearing about potentially being on-call, the you-build-it-you-run-it methodology is usually the optimal option. If engineers know they’re on the hook when something goes wrong at 3AM, they are more incentivized to write better software, which results in less issues and downtime. To get acceptance of on-call, two things come to mind:
&lt;ol&gt;
  &lt;li&gt;Additional compensation. Afterall, on-call may seem like a nightmare to many – some sort of compensation is only fair (regardless of the profession; from nurses to electricians, any sort of on-call duty that disrupts life should be compensated for). Compensation doesn’t always have to have to be monetary; it can also be rewarded as a form of additional PTO (Paid Time Off, which incidentally, is a great way to recuperate after an on-call shift)&lt;/li&gt;
  &lt;li&gt;Light at the end of the tunnel. On-call may not be forever. If a development team has proven their software reliable and properly observable, the “reward” may be handing off the software to a dedicated SRE or ops team within their org for production management. Of course, once a products reliability starts to slip, production ownership should rightly slide back to the development team&lt;/li&gt;
&lt;/ol&gt;

&lt;h5 id=&quot;handling-more-pushback&quot;&gt;Handling more pushback!&lt;/h5&gt;
Engineers fighting the introduction of an on-call rotation likely won’t be the first occurrence of pushback. Before implementing on-call, practices and concepts must be laid out and agreed upon. The initial discussion of these practices and concepts is highly likely to provoke resistance. People are too often set in their ways, and fearful of change. Most people enjoy the comfort of repeating practices they’ve been familiar with from legacy systems, but remember: we’re in a swim-or-sink industry – tech is an ever-evolving field that we must keep up with

The more people in your organization, the more cultural shift is required to achieve conformity with practices. Whether individuals (or groups) have uncertainty or doubt around evolving practices, they must be shown the way (and ideally, before they and their systems experience pain past their thresholds). Getting buy-in from people is a three-step process: 1) them understanding why change is necessary, 2) shifting from understanding why change is necessary, to believing that change is necessary, and 3) taking the time and effort to adopt the change. To be successful, this process must be as simple as possible, and there’s no better way to do this than by offering easy-to-digest patterns

&lt;h4 id=&quot;patterns&quot;&gt;Patterns&lt;/h4&gt;

No matter the size of your org – and whether you’re running a couple of apps or an enterprise-scale amount of apps – it makes sense to keep your M+A pattern(s) and solution(s) as common as possible (and unique only when necessary). One-off solutions should be avoided, especially as your technical ecosystem grows. If many one-off solutions exist, not only does it cost more to maintain and ensure each solution is working properly, but it also adds decision friction for teams seeking a solution

The familiarity of a common practice instills confidence and makes collaboration easier. Whether a new engineer is joining your team from another team, or you’re looking to assist another team in their troubleshooting, M+A can – and should – be a breeze when following familiar patterns

To achieve commonality, it’s not unusual for a dedicated monitoring team to lay out the tools and patterns for their engineering community to leverage. Be clear where to draw the line draw here, though – these “centralized” monitoring teams may provide guidance and operate the APM platforms, but should &lt;strong&gt;not&lt;/strong&gt; be responsible for instrumenting actual apps; instrumentation is the responsibility of the development teams (the engineers who know their code and business logic best)

If you don’t have a team dedicated to M+A, perhaps assembling a cohort (or “working group”) of production enthusiasts across your company will help lead to a solution. Just know that whichever way you get to a common solution, &lt;strong&gt;the solution must be self-service&lt;/strong&gt;. Otherwise, you’ll end up with frustrated engineers and low levels of adoption

&lt;h3 id=&quot;when&quot;&gt;When?&lt;/h3&gt;

Monitoring should be designed and implemented into each system as early as possible. A car manufacturer wouldn’t wait to add on features such as a speedometer or other gauges until after the car was built and sent to the consumer – so why should your software be any different? If instrumenting your app with monitoring (and observability) from day one of development is not possible – for example, if you’re prototyping and expecting a lot of refactoring – it may make sense to wait a bit before implementing, but do not delay including monitoring any later than your apps release to production. Remember: the longer you wait to implement M+A, the more expensive it will be

At a minimum, SLOs (Service Level Objectives) should be defined as soon as system functionality is determined. SLO &lt;em&gt;definitions&lt;/em&gt; tend to be more of a social agreement than actual implementation. Sometimes SLOs can even influence which technologies to include in your systems stack (e.g.: performance requirements defined in an SLO may help guide which language or components to use in the system) – better to hash these out sooner rather than later. If you’re unsure about SLOs, know they are practices applied by many big tech companies and plenty of &lt;a href=&quot;https://sre.google/resources/practices-and-processes/art-of-slos/&quot;&gt;free resources for creating SLOs&lt;/a&gt; exist

&lt;h3 id=&quot;how&quot;&gt;How?&lt;/h3&gt;

&lt;h4 id=&quot;signals&quot;&gt;Signals&lt;/h4&gt;
&lt;a href=&quot;/blog/2023/07/07/signals/&quot;&gt;Telemetry signals&lt;/a&gt; such as metrics, logs, and traces are useful for monitoring. These signals have the ability to alert engineers when thresholds (preferably, &lt;a href=&quot;https://sre.google/workbook/alerting-on-slos#4-alert-on-burn-rate&quot;&gt;SLO burn rates&lt;/a&gt;) are at risk of being breached. While there are many signals you can monitor and alert on, my recommendations are as follows, in order:

&lt;ul&gt;
  &lt;li&gt;Metrics – metrics come in the form of application metrics (related to your business logic; perhaps even KPIs) and system metrics (related to infrastructure). Your best bet is to capture both, and only alert on what matters most to your end users. &lt;strong&gt;While the nature of computing is often spikey (which can lead to false alarms), aggregating appropriate metrics into SLIs (think of this as a calculation of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;validEvents&lt;/code&gt; divided by &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;totalEvents&lt;/code&gt; over a period of time) for comparison against SLOs is usually the sensible approach to avoid these issues with M+A&lt;/strong&gt;. A few examples:
    &lt;ul&gt;
      &lt;li&gt;Latency – useful; if users experience slowness in your site, they are likely to bounce (leave). A basic way to measure latency as an SLI is &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;fastEnoughRequestsRate = requestsFasterThan100msCount/totalRequestCount&lt;/code&gt; (over a rolling period of time)&lt;/li&gt;
      &lt;li&gt;Availability – useful; if users can’t access your system, they are likely to bounce (leave). A basic way to measure availabilty as an SLI is &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;successRate = successfulRequestCount/totalRequestCount&lt;/code&gt; (over a rolling period of time)&lt;/li&gt;
      &lt;li&gt;CPU utilization – &lt;em&gt;may be&lt;/em&gt; useful, but if you’re at 99% CPU usage and systems are still operating smoothly, why bother alerting?&lt;/li&gt;
      &lt;li&gt;Disk storage – &lt;em&gt;may be&lt;/em&gt; useful, but if you’re in the cloud, using elastic/scalable storage, it’s less useful. Instead, maybe monitor the &lt;em&gt;trend&lt;/em&gt; of how much storage space is utilized (e.g.: disk usage increased by 20gB last night), as well as the prediction of future utilization&lt;/li&gt;
      &lt;li&gt;Other examples can be found in &lt;a href=&quot;https://sre.google/workbook/slo-document/&quot;&gt;Google’s example SLO document&lt;/a&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Traces – could alert on traces, though you’ll likely want to alert on &lt;em&gt;aggregate&lt;/em&gt; traces (e.g.: many traces reporting failures, not just a single trace) to ensure the alert was not triggered by a single, transient failure
    &lt;ul&gt;
      &lt;li&gt;Rather than traces being the &lt;em&gt;why&lt;/em&gt; of an alert, traces may be better utilized to identify &lt;em&gt;who&lt;/em&gt; to alert in a distributed system. For example, if TeamA’s app is facing issues, but the failing spans are stemming from a dependency owned by TeamB, there’s usually not much value in alerting TeamA. Perhaps make TeamA aware via a low-urgency alert, but the high-urgency alert should go directly to TeamB for proper resolution&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Logs – &lt;em&gt;could&lt;/em&gt; alert on logs, but it’s likely to lead to a lot of noise. For this reason, if you do choose to alert on logs, it should be a low urgency alert (not worthy of paging an on-call engineer), such as auto-generating a ticket or sending an email notification
    &lt;ul&gt;
      &lt;li&gt;You &lt;em&gt;could&lt;/em&gt; alert on log levels (e.g.: ERROR, FATAL), but this is less reliable than aggregate metrics, as transient failures tend to happen (i.e.: a dependency goes offline for a few minutes, causing error logs, then comes back online. Probably not a high-priority to alert on since there’s nothing to fix)&lt;/li&gt;
      &lt;li&gt;You &lt;em&gt;could&lt;/em&gt; alert on log string matches such as &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;exception&quot;&lt;/code&gt; or &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;error&quot;&lt;/code&gt;, but the purpose is defeated if your end users enter these trigger terms into an input field for fun, thus triggering a useless alert&lt;/li&gt;
      &lt;li&gt;Depending on the criticality of your application, whenever a never-before-seen error is logged, it may be worth alerting on. This way, the engineering team is made aware of the new failure mode and can adjust their code to handle it appropriately in the future&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

Given the above examples, hopefully you’ve picked up that &lt;strong&gt;alerts must indicate that user experience is in a degraded state, and that the alert is ACTIONABLE by the engineer receiving the alert&lt;/strong&gt;. Afterall, there’s no point in alerting if the end user isn’t experiencing pain, or if a fix is out of reach. An alert should evoke a sense of &lt;em&gt;actionable&lt;/em&gt; urgency. Anything less than urgent should fire a non-disrupting notification such as a log entry, an email, a message to your internal chat room, or an auto-generated ticket

&lt;h4 id=&quot;automate&quot;&gt;Automate!&lt;/h4&gt;

&lt;h5 id=&quot;onboarding&quot;&gt;Onboarding&lt;/h5&gt;
Automation is a huge step towards successful onboarding. The onboarding of a service to a M+A solution should either be entirely self-service, or in a perfect world: self-registering. Skipping automation here (and therefore risking proper onboarding) is the first sign of shortcuts being taken, and hints that future issues are likely to slip through the cracks

&lt;h5 id=&quot;incident-response&quot;&gt;Incident response&lt;/h5&gt;
If you have runbooks that contain manual commands/steps for on-call engineers to execute when things go wrong, get rid of these commands/steps. &lt;strong&gt;If there are any manual commands in your runbook, you’re likely doing something wrong&lt;/strong&gt;; this is a screaming symptom of inadequate automation and opens your system up to human error. Why would you disrupt (sometimes even wake up) your engineers so they can execute commands ad-hoc, when these commands could’ve ran automatically?

Whether a quick-fix solution entails a simple restart of a container or a traffic shift to another region, some sort of self-healing enables a much smoother (perhaps unnoticable) experience for everyone. Automation with self-healing is the only way tight SLAs can be respected – a 99.9% availability allows for 1.44 minutes of downtime a day; that’s barely enough time during an incident to spring to your machine, turn it on, and get to troubleshooting! To relate this concept back to money: if Facebook makes $3,000 per second, every minute of downtime costs them $180,000. Talk about a high stress environment if self-healing is not in place!

&lt;em&gt;Note: while most technical changes can be automated, policies (politics) may block some automation from being feasible in production. For example, the company I currently work for requires a change ticket – along with approval – for *every* modification to production. A huge disservice for everyone from the customers to the engineers to the business. This is something I will raise up in my org soon, as we must evolve this practice to keep our head above water&lt;/em&gt;

&lt;em&gt;Update: above note/concern resolved! The process explained to me was: an initial change ticket to *configure* self-healing requires an approval, but the automated resolution action performed by the system itself once the configuration is in place would *not* require a change ticket and approval to automatically self-heal&lt;/em&gt;

&lt;h3 id=&quot;where&quot;&gt;Where?&lt;/h3&gt;

Always monitor your systems as close to your users as possible. While it may be easy to identify failed requests at the load-balancer, how do you know users were even able to submit a request? Maybe there’s a bug in the client software that makes their ‘Submit’ button hidden or disabled. Perhaps you’re deployed to a single region, resulting in your users on the other side of the globe experiencing higher latencies that make your app less valuable to them. &lt;strong&gt;Many users won’t complain – they’ll simply stop using your application, so monitor proactively!&lt;/strong&gt; You won’t always know your users are experience pain until you monitor as close to your users as possible

Assuming a webapp, the closest place to a user you can monitor is typically in their browser. A couple of things you may want to monitor in a browser application are: DOM load time, user interaction with elements, &lt;strong&gt;stack traces&lt;/strong&gt;, and HTTP response latency. While client-side monitoring opens risks for credibility (and sometimes even loss) of telemetry data, techniques exist (but are not covered in this blog post) to mitigate these concerns

&lt;em&gt;Off-topic of monitoring: to be a bit more proactive, additionally shift left. Perhaps your CICD pipeline can include a post-deployment step in [pre-]production environments that perform performance and E2E testing. This extra step may help catch bugs and performance issues before your users run into them. Afterall, the more changes you make, the more likely things may go wrong&lt;/em&gt;

&lt;h4 id=&quot;less-is-more&quot;&gt;Less is more&lt;/h4&gt;

Adding more software doesn’t necessarily mean you need to add more monitoring. Monitoring does not fix systems, and more of it is not likely to improve anything; it can actually generate unnecessary noise and confusion. While there can be millions of things that can go wrong with within a system (root causes), there are always only a small handful of symptoms that surface (see &lt;a href=&quot;https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/&quot;&gt;The RED Method&lt;/a&gt; for more details). So keep it simple; don’t try to account for everything that can go wrong, and instead focus on monitoring symptoms with SLOs

While searching for a M+A solution, you may be tempted by “AIOps”. I don’t cover this topic in this post as I haven’t had experience with it, and &lt;a href=&quot;https://thenewstack.io/observability-and-the-misleading-promise-of-aiops/&quot;&gt;The Misleading Promise of AIOps&lt;/a&gt; does a lovely job at summing up at why you should be wary of it. To reiterate from the last paragraph: keep it simple

&lt;h2 id=&quot;often-overlooked&quot;&gt;Often overlooked&lt;/h2&gt;
This blog post primarily focused on monitoring metrics from infrastructure (narrowly, compute and storage) and applications. While these metrics are critical to monitor, we can’t forget the backbone holding everything together: networking and security. Without either, nothing is possible, so it makes sense to monitor them as well

&lt;em&gt;Disclaimer: I’m no SME when it comes to security or networking (most people aren’t), so please bear with me and take the following ideas with a grain of salt&lt;/em&gt;

Attackers will almost always find a way to get into systems. Once intruded, logs (e.g.: syslog, sshd, auditd, etc.) may be able to help identify the blast radius. …assuming the logs weren’t maliciously deleted or tampered with, and log forwarding services weren’t disabled (you do have a way to backup your logs, right?) The extremity of software attacks can go beyond our imaginations, so one way to aide in the recovery of malicious attacks on software is to monitor at the hardware level. &lt;a href=&quot;https://en.wikipedia.org/wiki/Network_tap&quot;&gt;Network TAPs&lt;/a&gt; are a hardware solution that capture &lt;em&gt;all&lt;/em&gt; network traffic as it goes over the wire (think of &lt;a href=&quot;https://en.wikipedia.org/wiki/Tee_(command)&quot;&gt;tee (1)&lt;/a&gt;, but for network traffic). For the sake of this blog post, these investigative “monitoring” techniques are simply ideas; to find a strategy that works best for you, consult the proper experts

Performance and reliability considerations are also crucial when monitoring – not only do they affect your end users experience, but they also affect your compute spendings. Are your systems forwarding telemetry data to a backend far away, causing your critical components to slug along while waiting for the telemetry transfer to complete? Don’t be afraid to offload the telemetry forwarding/exporting work to another component. Many patterns exist for this, such as a sidecar’d OpenTelemetry collector that can perform potentially expensive operations such as batching, encrypting, and filtering your telemetry before sending (and also handling retries on transmission failure). An even simpler use-case: a standalone log agent that scrapes logs from disk (or stdout) and forwards them to your logging backend. As long as your engineers can access telemetry data in a timely manner (ideally, no longer than 60 seconds), shedding the responsibility of potentially expensive and error-prone telemetry exports off of your mission-critical apps may be worth investigating

&lt;h2 id=&quot;final-thoughts&quot;&gt;Final thoughts&lt;/h2&gt;

To reiterate the first section of this post: if you don’t have adequate monitoring and alerting in place, you don’t care about the success of your business or users. If you don’t have M+A in place, your service is not ready for production. M+A must be a first class citizen when it comes to building and maintaining reliable services, especially as users become more dependent on (and demanding of!) technology

Many tools exist for M+A purposes; choose your tools well, else suffer the consequences. Remember that any number of tools – no matter how good – are not a magic bullet; you must be aware of the limitations and tradeoffs of the tools you choose, as well as learn and invest in your tools. Also keep in mind that your M+A journey must be mission-driven, not tool-driven

Get creative with your monitoring data. While a good amount of this post focused on using monitoring data to notify a human, it shouldn’t end here. Monitoring data can be used for things such as analytics, or even influencing data-driven events such as self-healing and auto-scaling your compute resources when thresholds reach certain values

If you are unsure of where to start with monitoring: consider business KPIs and user journeys, and leverage SLOs when possible

Lastly, monitoring and alerting is a never-ending evolution. Just as our systems evolve, our M+A practices must evolve with them as well. Sink or swim!

&lt;h3 id=&quot;further-reading&quot;&gt;Further reading&lt;/h3&gt;

This post was inspired by and contain many ideas from the book &lt;a href=&quot;https://www.oreilly.com/library/view/practical-monitoring/9781491957349/&quot;&gt;Practical Monitoring by Mike Julian (2017)&lt;/a&gt;. Although technology has changed vastly since published, the majority of the concepts in the book still hold true; certainly worth a read

&lt;a href=&quot;https://sre.google/sre-book/monitoring-distributed-systems/&quot;&gt;Monitoring Distributed Systems&lt;/a&gt; from Google’s SRE book

&lt;a href=&quot;https://www.reddit.com/r/sre/comments/17g1g9p/comment/k6gznvl&quot;&gt;u/DakkinByte’s feedback on this post&lt;/a&gt; (suggestions incorporated into this post on 2023-Oct-26)

&lt;ul id=&quot;markdown-toc&quot;&gt;
  &lt;li&gt;&lt;a href=&quot;#reality-check&quot; id=&quot;markdown-toc-reality-check&quot;&gt;Reality check&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#how-to-do-it-right&quot; id=&quot;markdown-toc-how-to-do-it-right&quot;&gt;How to do it right?&lt;/a&gt;    &lt;ul&gt;
      &lt;li&gt;&lt;a href=&quot;#an-apm-software-solution&quot; id=&quot;markdown-toc-an-apm-software-solution&quot;&gt;An APM Software Solution&lt;/a&gt;        &lt;ul&gt;
          &lt;li&gt;&lt;a href=&quot;#one-solution-is-not-always-enough&quot; id=&quot;markdown-toc-one-solution-is-not-always-enough&quot;&gt;One solution is not always enough&lt;/a&gt;&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#culture&quot; id=&quot;markdown-toc-culture&quot;&gt;Culture&lt;/a&gt;        &lt;ul&gt;
          &lt;li&gt;&lt;a href=&quot;#who-builds-it-runs-it&quot; id=&quot;markdown-toc-who-builds-it-runs-it&quot;&gt;Who builds it, runs it&lt;/a&gt;            &lt;ul&gt;
              &lt;li&gt;&lt;a href=&quot;#handling-pushback&quot; id=&quot;markdown-toc-handling-pushback&quot;&gt;Handling pushback&lt;/a&gt;&lt;/li&gt;
              &lt;li&gt;&lt;a href=&quot;#handling-more-pushback&quot; id=&quot;markdown-toc-handling-more-pushback&quot;&gt;Handling more pushback!&lt;/a&gt;&lt;/li&gt;
            &lt;/ul&gt;
          &lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#patterns&quot; id=&quot;markdown-toc-patterns&quot;&gt;Patterns&lt;/a&gt;&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#when&quot; id=&quot;markdown-toc-when&quot;&gt;When?&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#how&quot; id=&quot;markdown-toc-how&quot;&gt;How?&lt;/a&gt;        &lt;ul&gt;
          &lt;li&gt;&lt;a href=&quot;#signals&quot; id=&quot;markdown-toc-signals&quot;&gt;Signals&lt;/a&gt;&lt;/li&gt;
          &lt;li&gt;&lt;a href=&quot;#automate&quot; id=&quot;markdown-toc-automate&quot;&gt;Automate!&lt;/a&gt;            &lt;ul&gt;
              &lt;li&gt;&lt;a href=&quot;#onboarding&quot; id=&quot;markdown-toc-onboarding&quot;&gt;Onboarding&lt;/a&gt;&lt;/li&gt;
              &lt;li&gt;&lt;a href=&quot;#incident-response&quot; id=&quot;markdown-toc-incident-response&quot;&gt;Incident response&lt;/a&gt;&lt;/li&gt;
            &lt;/ul&gt;
          &lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#where&quot; id=&quot;markdown-toc-where&quot;&gt;Where?&lt;/a&gt;        &lt;ul&gt;
          &lt;li&gt;&lt;a href=&quot;#less-is-more&quot; id=&quot;markdown-toc-less-is-more&quot;&gt;Less is more&lt;/a&gt;&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#often-overlooked&quot; id=&quot;markdown-toc-often-overlooked&quot;&gt;Often overlooked&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#final-thoughts&quot; id=&quot;markdown-toc-final-thoughts&quot;&gt;Final thoughts&lt;/a&gt;    &lt;ul&gt;
      &lt;li&gt;&lt;a href=&quot;#further-reading&quot; id=&quot;markdown-toc-further-reading&quot;&gt;Further reading&lt;/a&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;
</description>
        <pubDate>Sat, 14 Oct 2023 00:00:00 +0000</pubDate>
        <link>https://srezone.com/blog/2023/10/14/monitoring/</link>
        <guid isPermaLink="true">https://srezone.com/blog/2023/10/14/monitoring/</guid>
        
        <category>Monitoring</category>
        
        <category>Alerting</category>
        
        <category>Tooling</category>
        
        <category>On-call</category>
        
        
        <category>Monitoring</category>
        
      </item>
    
      <item>
        <title>Vendor lock-in: a tale and how OpenTelemetry avoids it</title>
        <description>&lt;details&gt;
  &lt;summary&gt;Expand for a story, or simply continue reading to get to the details of this post&lt;/summary&gt;
  &lt;h2 id=&quot;a-hypothetical-story&quot;&gt;A “Hypothetical” Story&lt;/h2&gt;
  &lt;h3 id=&quot;in-search-of-a-solution&quot;&gt;In search of a solution&lt;/h3&gt;
  Imagine this: you have a problem that needs solving. You scout out several vendors offering similar software solutions. You find a vendor whose offering seems to fit your needs and they promise you the world. You get the purchase approval from your org, then you seal the deal with the vendor. You’re likely in a contract for some time, and if for whatever reason their solution doesn’t work out: you can always switch vendors in the future, right?

  &lt;h3 id=&quot;the-gaps-in-the-solution&quot;&gt;The gaps in the solution&lt;/h3&gt;
  You immediately get some value and excitement from the new solution you purchased, and all is well – until it’s not. You eventually realize there are gaps with the solution. You contact the vendor and three things are likely to happen:
  &lt;ol&gt;
    &lt;li&gt;The vendor knows how to solve your problem and generously assists you with filling the gap&lt;/li&gt;
    &lt;li&gt;The vendor upsells you on something you thought would’ve came included with the initial purchased package&lt;/li&gt;
    &lt;li&gt;The vendor scrambles to shoehorn your request into their feature backlog&lt;/li&gt;
  &lt;/ol&gt;

  Whatever happens, business continues as usual - for both you and the vendor

  &lt;h3 id=&quot;turning-a-blind-eye&quot;&gt;Turning a blind eye&lt;/h3&gt;
  While you happily continue to enjoy your (potentially half-baked) solution, you’ve also unintentionally built a stronger dependency on your vendors solution. OK, no problem, as long as your vendors solution fits your needs and it interfaces with open standards (assuming they exist) - just in case their solution starts to fall short, so you can switch vendors. Note that you may have to research your vendors solution a bit for this information; there’s a good chance open standard interfaces aren’t shouted-from-the-rooftop (or even offered) by your vendor. Afterall, why would a vendor want to equip their customers with the knowledge and tools to easily switch to a competitor?

  Defying existing open standards (or, likely wrapping the open standard in their own proprietary solution) can certainly be part of a vendors business model. But wouldn’t this practice earn them a bad reputation? It could - and should - but it’s hard to point a finger when most vendors are using similar tactics. So you suck it up, continue working with the vendor, and fall a bit deeper into their solution. All is (somewhat) well, until contract renewal time comes

  &lt;h3 id=&quot;the-price-paid&quot;&gt;The price paid&lt;/h3&gt;
  At contract renewal time, the vendor jacks up their price quite a bit - to the point you likely won’t be breaking even with the value their solution provides. “OK, I’ll re-evaluate my options to see if I can get a better price” – but this thought is futile – you’re already deeply locked into your vendors proprietary solution, and the vendor knows this. Good luck not only migrating your data from your current vendors solution to a new one, but also getting your org onboard for a rippling change (think about engineers having to rework every app they’ve already onboarded with the current vendor solution!) In hindsight, what seemed to be a long-lasting vendor-client relationship being built was actually a salesperson blowing smoke up your ass while sinking their claws in deeper the entire time

  …

  OK, that’s all for story time – hopefully it gives some context on why you’d want to avoid vendor lock-in. The remainder of this blog post is primarily written with the context of observability tools in mind, though remember vendor lock-in spans plenty types of technology offerings - for example, cloud platforms (AWS, GCP, Azure). Of course you’ll likely need to choose at least one solution, just remember there’s no need to be on a short leash with any specific vendor. There should always be at least one approach you can execute on to protect yourself from vendor lock-in while still reaping the benefits of a vendor solution - for example, with IaC: why stick with CloudFormation and be bound to AWS when you can use Terraform and flexibly run multi-cloud? Options are available, and it’s your responsibility to be vigilant (to not be blindsighted by a vendors “perfect solution”). Anyhow, let’s get into the observability part of this post…
&lt;/details&gt;

&lt;h2 id=&quot;reality-an-open-standard-for-observability&quot;&gt;Reality: an open standard (for observability)&lt;/h2&gt;
Imagine you need an observability solution for your org. Hopefully you’d avoid the trap mentioned in the above hypothetical story. It’s highly probable that you would not want to build your own observability solution or host a FOSS solution yourself. Using a vendors observability (backend) SaaS is an appropriate approach – assuming you’ve done the due dilligence to not fall into a vendors nasty lock-in trap as outlined above. What enables such avoidance of vendor lock-in with an observability solution is the open standard: OpenTelemetry (OTel). To understand how OTel avoids vendor lock-in, it’s critical to first understand the “pipeline” or lifecycle of telemetry at a high level

&lt;h3 id=&quot;1-telemetry-generation&quot;&gt;1. Telemetry generation&lt;/h3&gt;
To make your systems observable, telemetry must first be generated somewhere within your systems. Your applications or systems must generate &lt;a href=&quot;/blog/2023/07/07/signals/&quot;&gt;telemetry signals&lt;/a&gt;/data (metrics, logs, distributed traces, etc.), then emit this telemetry data to a separate component that will either 1) pre-process the telemetry data or 2) directly store the telemetry data for future retrieval &amp;amp; analysis

&lt;h4 id=&quot;available-options&quot;&gt;Available options&lt;/h4&gt;
&lt;ul&gt;
  &lt;li&gt;Open standard solution: OpenTelemetry API/SDK + OpenTelemetry Agent&lt;/li&gt;
  &lt;li&gt;Vendor solution: vendor-proprietary SDK (&lt;a href=&quot;https://github.com/Dynatrace/OneAgent-SDK-for-NodeJs&quot;&gt;example&lt;/a&gt;) + agent (which is potentially simply a proprietary abstraction wrapping OpenTelemetry)&lt;/li&gt;
  &lt;li&gt;Trade-off between solutions: the open standard solution may be a bit more maintenance, but offers the flexibility needed to fully control your telemetry data. The vendor solution takes away the burden of maintaining an agent, but also limits us greatly in the telemetry we produce by default, often causes us to reinstrument our applications if we choose to switch vendors, and usually limits the forwarding of your telemetry data &lt;em&gt;only&lt;/em&gt; to said vendors service/backend&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;2-telemetry-processing&quot;&gt;2. Telemetry processing&lt;/h3&gt;
Once you have telemetry generated, you may want to pre-process it before persisting it to an observability backend(s) for analysis. Several types of processing can occur, a few being:
&lt;ul&gt;
  &lt;li&gt;Filtering or sampling requests (since it’s not always feasible to to store &lt;em&gt;every&lt;/em&gt; bit of telemetry you generate)&lt;/li&gt;
  &lt;li&gt;Striping out sensitive data from telemetry (e.g.: PII)&lt;/li&gt;
  &lt;li&gt;Enhancing telemetry data (e.g.: adding fields/data that’ll be useful when performing analysis in the backend)&lt;/li&gt;
  &lt;li&gt;Batching telemetry before forwarding it to your observability backend(s) (network optimization)&lt;/li&gt;
&lt;/ul&gt;

&lt;h4 id=&quot;available-options-1&quot;&gt;Available options&lt;/h4&gt;
&lt;ul&gt;
  &lt;li&gt;Open standard solution: OpenTelemetry collector (optional)&lt;/li&gt;
  &lt;li&gt;Vendor solution: vendor-specific DSL used to configure a telemetry processing pipeline (optional)&lt;/li&gt;
  &lt;li&gt;Trade-off between solutions: while you’d likely need to manage and scale your OpenTelemetry collector (I haven’t seen any managed services for the OTel collector as of now), the OTel collector offers many versatile &lt;a href=&quot;https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor&quot;&gt;processors&lt;/a&gt; out-of-the-box and can be extended to use your own custom processing logic. Vendor solutions likely limits the type of processing available and makes the pipeline configuration non-portable (vendor lock-in)&lt;/li&gt;
&lt;/ul&gt;

&lt;em&gt;Disclaimer: I haven’t worked with a vendor telemetry processor before&lt;/em&gt;

&lt;h3 id=&quot;3-telemetry-storage-and-analysis-in-an-observability-backend&quot;&gt;3. Telemetry storage (and analysis) in an observability “backend”&lt;/h3&gt;
Whether your telemetry data is pre-processed or not, it still needs to be stored somewhere before making use of it. Some low-latency observability systems may persist this data in memory for low-latency querying (expensive but fast), but more commonly than not, telemetry is stored in a persistent database for durable access in the future. Regardless of the storage medium, these “backends” also offer a UI that enables engineers to view their telemetry data at an aggregate and granular (per-request) basis

&lt;h4 id=&quot;available-options-2&quot;&gt;Available options&lt;/h4&gt;
&lt;ul&gt;
  &lt;li&gt;Open standard solution: FOSS backends (ideally offering support for OTLP (OpenTelemetry Line Protocol)) such as Zipkin, Jaeger, and Grafana&lt;/li&gt;
  &lt;li&gt;Vendor solution: any vendor solutions. Note that as OpenTelemetry becomes more prevalant, more vendors are offering support for OTLP in addition to their proprietary telemetry protocols&lt;/li&gt;
  &lt;li&gt;Trade-off between solutions: open standard allows customization and extension to support your use case (if needed). Open source solutions likely require you to self-host or pay for a managed instance. Vendor solutions may be self-hosted, but are more often managed as a SaaS. If a vendor solution does not support features you want, you may be waiting a while (sometimes indefinitely) before seeing these features&lt;/li&gt;
&lt;/ul&gt;

&lt;em&gt;NOTE: when choosing a telemetry backend, a couple of major things to look for are (but not limited to) high availability, low latency (&amp;lt;=1 minute from the time of telemetry generation to it being queryable in the backend is usually acceptable), and the ability to query your telemetry data effectively on both the aggregate and per-request level. With any of these features missing, a backend tool can be near useless&lt;/em&gt;

&lt;h3 id=&quot;pulling-it-together&quot;&gt;Pulling it together&lt;/h3&gt;
Now that we have an understanding of the telemetry pipeline, we see where opportunities exist to avoid vendor lock-in. We want to shift non-vendor-specific solutions as left as possible – &lt;strong&gt;if we start off with a vendors proprietary solution for generating telemetry data, we’re likely bound to the vendors pipeline for the rest of the telemetry lifecycle.&lt;/strong&gt; If maintaining an agent/component for generating telemetry seems too daunting, know that plenty of &lt;a href=&quot;https://github.com/open-telemetry/opentelemetry-lambda#extension-layer-language-support&quot;&gt;reproducible agents&lt;/a&gt; exist to make this task almost effortless

Whether you use a telemetry processing stage or not is dependent on your needs – do you need to enrich/mask/transform your telemetry data before storing it? Or perhaps you need to filter and (tail) sample your telemetry to ensure you don’t send overwhelming amounts of telemetry to your backend as your system scales ($$$)!

As long as you have the generation and processing stages of the telemetry pipeline understood, I believe you’ve earned the choice to leverage either an open source or vendor solution as your backend. Having open standard/vendor-agnostic upstream stages in your telemetry pipeline gives you incredible negotiation power and flexbility with less threat of vendor lock-in. Of course, if you choose to change vendors in the future, it will still likely take &lt;em&gt;some&lt;/em&gt; effort to migrate your existing data to your next vendor. But honestly, historical telemetry data may not be as valuable as you think. If you are looking to take a leap to another vendor, you can always update your OpenTelemetry collector (or agent) config to forward telemetry to multiple vendor solutions, making the migration as smooth as possible - all with minor overhead

&lt;h2 id=&quot;moving-forward&quot;&gt;Moving forward&lt;/h2&gt;
Vendor lock-in is never good. Unfortunately, most vendors will push for it (for their own monetary interest) at any given opportunity. While it shouldn’t be this way, it is – it’s a common, current business model in this industry. In my opinion, instead of vendors playing games to trap unknowing customers, vendors should be utilizing their resources to build a solution that is truly superior (in this case: a powerful, reliable, full-featured observability backend.) This sort of no-BS behavior drives competition (and need I mention, innovation?) in the space. The idea of taking an open standard then wrapping it into something proprietary defeats the whole purpose of an open standard and can arguable be considered offensive. Please say “no” to and call out vendors pedaling these shady tactics - it’ll give you the upper hand and may also encourage vendors to shift their strategies so everyone wins. Anyways, the point is: before you even consider using a vendors proclaimed “magic bullet” solution, do your homework and have your escape plan ready.

&lt;ul id=&quot;markdown-toc&quot;&gt;
  &lt;li&gt;&lt;a href=&quot;#a-hypothetical-story&quot; id=&quot;markdown-toc-a-hypothetical-story&quot;&gt;A “Hypothetical” Story&lt;/a&gt;    &lt;ul&gt;
      &lt;li&gt;&lt;a href=&quot;#in-search-of-a-solution&quot; id=&quot;markdown-toc-in-search-of-a-solution&quot;&gt;In search of a solution&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#the-gaps-in-the-solution&quot; id=&quot;markdown-toc-the-gaps-in-the-solution&quot;&gt;The gaps in the solution&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#turning-a-blind-eye&quot; id=&quot;markdown-toc-turning-a-blind-eye&quot;&gt;Turning a blind eye&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#the-price-paid&quot; id=&quot;markdown-toc-the-price-paid&quot;&gt;The price paid&lt;/a&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#reality-an-open-standard-for-observability&quot; id=&quot;markdown-toc-reality-an-open-standard-for-observability&quot;&gt;Reality: an open standard (for observability)&lt;/a&gt;    &lt;ul&gt;
      &lt;li&gt;&lt;a href=&quot;#1-telemetry-generation&quot; id=&quot;markdown-toc-1-telemetry-generation&quot;&gt;1. Telemetry generation&lt;/a&gt;        &lt;ul&gt;
          &lt;li&gt;&lt;a href=&quot;#available-options&quot; id=&quot;markdown-toc-available-options&quot;&gt;Available options&lt;/a&gt;&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#2-telemetry-processing&quot; id=&quot;markdown-toc-2-telemetry-processing&quot;&gt;2. Telemetry processing&lt;/a&gt;        &lt;ul&gt;
          &lt;li&gt;&lt;a href=&quot;#available-options-1&quot; id=&quot;markdown-toc-available-options-1&quot;&gt;Available options&lt;/a&gt;&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#3-telemetry-storage-and-analysis-in-an-observability-backend&quot; id=&quot;markdown-toc-3-telemetry-storage-and-analysis-in-an-observability-backend&quot;&gt;3. Telemetry storage (and analysis) in an observability “backend”&lt;/a&gt;        &lt;ul&gt;
          &lt;li&gt;&lt;a href=&quot;#available-options-2&quot; id=&quot;markdown-toc-available-options-2&quot;&gt;Available options&lt;/a&gt;&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#pulling-it-together&quot; id=&quot;markdown-toc-pulling-it-together&quot;&gt;Pulling it together&lt;/a&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#moving-forward&quot; id=&quot;markdown-toc-moving-forward&quot;&gt;Moving forward&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</description>
        <pubDate>Sat, 12 Aug 2023 00:00:00 +0000</pubDate>
        <link>https://srezone.com/blog/2023/08/12/vendorLockIn/</link>
        <guid isPermaLink="true">https://srezone.com/blog/2023/08/12/vendorLockIn/</guid>
        
        <category>Vendors</category>
        
        <category>Lock-in</category>
        
        <category>Rant</category>
        
        <category>OpenTelemetry</category>
        
        <category>Pipeline</category>
        
        <category>Collector</category>
        
        <category>Money</category>
        
        <category>Politics</category>
        
        
        <category>Tooling</category>
        
      </item>
    
      <item>
        <title>Telemetry Signals</title>
        <description>This blog post outlines the three main telemetry signals as of July 2023 (per the &lt;a href=&quot;https://opentelemetry.io/docs/concepts/signals/&quot;&gt;OpenTelemetry spec&lt;/a&gt;) and how they’re used in harmony to achieve operational excellence (and more!)

&lt;h2 id=&quot;signals&quot;&gt;Signals&lt;/h2&gt;

&lt;h3 id=&quot;metrics&quot;&gt;Metrics&lt;/h3&gt;
Metrics have been traditionally used to monitor infrastructure (how much disk space is remaining? What is my CPU utilization? etc.), and they still are. As we move more towards cloud-native components, we (as application engineers) don’t need to worry as much about these underlying infrastructure metrics because our cloud platform maintains the infrastructure for us (think serverless). As we don’t “own” elastic cloud infrastructure, monitoring/alerting on these metrics is not very actionable as a consumer, and therefore results in noise when things go wrong. This noise often leads to alert fatigue for engineers on your product team (more on this in a future blog post.)

That’s not to say we shouldn’t use metrics at all. Instead of leveraging infrastructure metrics, we should move up a layer and capture metrics at the application level. More specifically, capture &lt;em&gt;application&lt;/em&gt; metrics that are critical to the end users journey and the business. Is the checkout page taking too long to load? Are payments failing to be accepted? Both of those scenarios could negatively affect the business – well, at the core, will affect your customers, who are incidentally the lifeblood of your business.

&lt;!-- Does that mean we should alert each time a request is slow, or for each error a user experiences? Heck no! Doing so would result in alert fatigue and distractions from higher priority initiatives. In a world of modern architectures and distributed systems, things are constantly failing (a lot of them are hidden from us thanks to fault-tolerance). We must accept failure as normal, and be intentional about what we alert on. --&gt;

I’ll ramble on in a separate blog post about metrics, SLOs (Service Level Objectives), and alerting, but the point to take away for now is it’s not feasible to alert on each bad metric emitted. Instead, we must aggregate metrics and calculate if our service is at risk of being unreliable. Once these aggregate metrics (Service Level Indicators/SLIs) show that reliability is in danger (and an actionable solution is feasible), the engineer(s) on-call are notified to restore the service to health so business can continue as usual.

&lt;h3 id=&quot;distributed-traces&quot;&gt;Distributed Traces&lt;/h3&gt;
Once an on-call engineer is notified that something is going wrong, they must identify &lt;em&gt;why and where&lt;/em&gt; the problem is occurring before fully resolving the issue. Depending on the alert message, an engineer is likely to log into their observability backend tool to start “asking questions”, typically through dashboards or saved queries. A chart visual might indicate a spike in latency for certain requests. The engineer might filter for all requests that returned an error. Whatever the error may be, after some investigation, the engineer can see exactly &lt;em&gt;where&lt;/em&gt; in the system something is going wrong, and start to identify attributes in the telemetry data that may reveal the root cause of the issue.&lt;!-- While [most outages are caused by some kind of change](https://sre.google/sre-book/introduction/#:~:text=most%20outages%20are%20caused%20by%20some%20kind%20of%20change)--&gt;

To understand how distributed tracing enables this sort of insight, it’s important to understand the definition and anatomy of a trace. A distributed trace, simply put, is the representation of a request flowing end-to-end (across each component) within (and sometimes across) a system(s). Observability backends commonly display traces in a waterfally view - if you Google image search ‘distributed tracing waterfall’, you’ll see what I mean. Each block in the distributed tracing chart is a span (one unit of work), which contains information about it’s &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;span_id&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;span_parent&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;start_time&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;duration&lt;/code&gt;, and corresponding &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trace_id&lt;/code&gt; at a minimum. (Note, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trace_id&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;span_parent&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trace_state&lt;/code&gt;, and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trace_flags&lt;/code&gt; are considered span “context” and propagated via request headers when crossing boundaries in a system to “stitch” the spans together in the observability backend). Additional values should be included in the span as well, such as data about the emitting component (e.g.: resource name, deployed region, etc.) and the request itself (e.g.: payload, userId, etc.). Attaching useful attributes to spans is what enables us to identify patterns (and anomalies) to better understand erroneous requests.

Implementing distributed tracing is fairly simple. To get started, “auto-instrumentation” &lt;a href=&quot;https://opentelemetry.io/ecosystem/registry/?component=instrumentation&amp;amp;language=js&quot;&gt;automatically instruments popular libraries&lt;/a&gt; (e.g.: http clients, database drivers, etc.) within your application so you can get traces (and other signals) from those libraries with little-to-no work – the application will emit telemetry for that library at runtime without any changes to your application code. This “auto-instrumentation” is often achieved by libraries being “pre-instrumented” with OpenTelemetry or by including a metapackage that performs monkey-patching for the specified library at init time. Auto-instrumentation alone is often not enough, though – it &lt;em&gt;should&lt;/em&gt; be augmented with manual instrumentation in your code to provide a more detailed view of what your system is doing, particularly around custom business logic.

&lt;h3 id=&quot;logs&quot;&gt;Logs&lt;/h3&gt;
When distributed traces don’t provide enough details to fully diagnose an issue - or if distributed traces aren’t present at all - logs are our last resort. Traditionally, sifting through logs required SSHing into a server (sometimes multiple servers) and grepping the log file for issues. To make this process easier, log-forwarding agent software was installed on machines to automatically forward logs to a centralized logging tool (e.g. ELK, Splunk, GrayLog, etc.). When working with distributed and/or redundant systems, it’s important that resource metadata describing the source of each log (where the log came from) is indicated in each log record contained in the centralized logging tool, otherwise it’d be difficult to pinpoint issues back to the affected machine.

In addition to having resource metadata in our logs, it’s important to also include rich context about the request in a consistent, structured format. Why? Because structured data is what machines best understand. Single-lined, unstructured logs are fine for human readability, but as systems scale, this simple approach of a human reading logs line-by-line does not. With the incredible processing power machines offer, it’d be silly not to write structured logs that allow us to leverage machines to effortlessly slice and dice our data to find patterns and anomalies. Proper system insight comes from the combination of human intuition &lt;em&gt;and&lt;/em&gt; the findings generated by powerful machines (big data!)

With systems continuously surpassing record transactional throughput (thousands of transactions per second (TPS) are common, and some systems even have millions of TPS), it’s not always feasible to pinpoint unique issues simply by correlating metrics with logs via timestamp. By leveraging context-rich logging libraries, corresponding &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trace_id&lt;/code&gt;s are often included in generated logs automatically to expose better insight into the systems.

&lt;h2 id=&quot;using-signals-in-harmony&quot;&gt;Using signals in harmony&lt;/h2&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Metrics&lt;/strong&gt; feed SLIs/SLOs, which alert us when our service is unreliable&lt;/li&gt;
  &lt;li&gt;View &lt;strong&gt;distributed traces&lt;/strong&gt; to figure out where in the system things are going wrong, and ideally why they are going wrong&lt;/li&gt;
  &lt;li&gt;Review structured &lt;strong&gt;logs&lt;/strong&gt; to get more details about what’s going on within a component&lt;/li&gt;
  &lt;li&gt;Once the root of the issue is identified, a fix is put into place and deployed to production
    &lt;ul&gt;
      &lt;li&gt;Note: the first “fix” put in place is often a temporary solution to “stop the bleeding”, until a more permanant fix is available for deployment&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

Using telemetry signals in harmony as outlined here enables rapid Mean-Time To Resolution (MTTR), which results in happier customers, and a more successful business

Note that it’s important for telemetry signals to correlate with each other in order to provide an efficient process for the production engineer troubleshooting the system. To achieve this, all telemetry must be stored either A) in a single observability backend, or B) multiple observability backends that play nicely with eachother (enable correlation of data). Before this, the telemetry data should correlate with each other via &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trace_id&lt;/code&gt; – where appropriate, metrics should have ‘exemplars’ containing &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trace_id&lt;/code&gt;, and structured logs should also contain &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trace_id&lt;/code&gt;.

&lt;h2 id=&quot;recommendations&quot;&gt;Recommendations&lt;/h2&gt;
&lt;h3 id=&quot;metrics-1&quot;&gt;Metrics&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;Leverage what you can out of the box where it makes sense (e.g.: API Gateway response codes, latency, etc.)&lt;/li&gt;
  &lt;li&gt;Don’t page on-callers based on raw metrics (things constantly fail in complex systems). Rather, alert on meaningful (and actionable) events, such as SLOs being in danger of being breached (more to come on this in a separate blog post)&lt;/li&gt;
  &lt;li&gt;Emit custom (application) metrics that are meaningful for end-user journeys. Just be cautious of creating metrics with high cardinality dimensions, as this can get expensive&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;distributed-traces-1&quot;&gt;Distributed Traces&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;An absolute must for distributed systems/modern architectures&lt;/li&gt;
  &lt;li&gt;OpenTelemetry is the de facto standard for tracing, and supports most popular languages&lt;/li&gt;
  &lt;li&gt;Leverage auto-instrumentation, and augment with manual instrumentation for a better view of your systems&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;logs-1&quot;&gt;Logs&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;Veer towards including “log” messages as rich events in your distributed tracing spans where it makes sense (when running serverless FaaS, I prefer span events. When running a multi-threaded, long-running application, additional logs seem appropriate to expose behavior that may not tie back to a single request, and likely affects multiple requests). Note that the depth of this suggestion depends on how the end-user will be consuming event information. If your team isn’t super mature with their instrumentation, how about a log wrapper that adds all log records as events to the active span?&lt;/li&gt;
  &lt;li&gt;Include &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;trace_id&lt;/code&gt; as needed (instrumentation should do this automatically)&lt;/li&gt;
  &lt;li&gt;String-concatenated messages aren’t great for machine usability. Rather than having a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;{&quot;message&quot;: &quot;User 32101 has logged in&quot;}&lt;/code&gt;, we should be logging in a high-dimensional way such as &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;{&quot;message&quot;: &quot;User has logged in&quot;, Attributes: { &quot;userId&quot;: 32101 } }&lt;/code&gt;. This makes slicing and dicing our log data much easier&lt;/li&gt;
&lt;/ul&gt;

&lt;strong&gt;No matter the signal you emit, ensure there is no sensitive data such as PII or credentials included – this type of information is an attractive target for attackers&lt;/strong&gt;

&lt;h2 id=&quot;outside-of-operational-excellence&quot;&gt;Outside of operational excellence&lt;/h2&gt;
You’ve probably heard that “data is beautiful”; telemetry data absolutely doesn’t need to be used &lt;em&gt;just&lt;/em&gt; for systems diagnosis. Having telemetry emitted and accessible by anyone on your team (from engineers to product managers to business stakeholders), brings everyone closer to the lens of production. Telemetry data can be used for business intelligence ([How] Are our users using a new feature? Are all of our users having a satisfactory experience using our product? How should we prioritize our work in the future? etc.). Of course, the data points needed to ask some of these questions are business specific - something you wouldn’t get from auto-instrumentation alone ;)

&lt;ul id=&quot;markdown-toc&quot;&gt;
  &lt;li&gt;&lt;a href=&quot;#signals&quot; id=&quot;markdown-toc-signals&quot;&gt;Signals&lt;/a&gt;    &lt;ul&gt;
      &lt;li&gt;&lt;a href=&quot;#metrics&quot; id=&quot;markdown-toc-metrics&quot;&gt;Metrics&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#distributed-traces&quot; id=&quot;markdown-toc-distributed-traces&quot;&gt;Distributed Traces&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#logs&quot; id=&quot;markdown-toc-logs&quot;&gt;Logs&lt;/a&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#using-signals-in-harmony&quot; id=&quot;markdown-toc-using-signals-in-harmony&quot;&gt;Using signals in harmony&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#recommendations&quot; id=&quot;markdown-toc-recommendations&quot;&gt;Recommendations&lt;/a&gt;    &lt;ul&gt;
      &lt;li&gt;&lt;a href=&quot;#metrics-1&quot; id=&quot;markdown-toc-metrics-1&quot;&gt;Metrics&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#distributed-traces-1&quot; id=&quot;markdown-toc-distributed-traces-1&quot;&gt;Distributed Traces&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;#logs-1&quot; id=&quot;markdown-toc-logs-1&quot;&gt;Logs&lt;/a&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#outside-of-operational-excellence&quot; id=&quot;markdown-toc-outside-of-operational-excellence&quot;&gt;Outside of operational excellence&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
        <pubDate>Fri, 07 Jul 2023 00:00:00 +0000</pubDate>
        <link>https://srezone.com/blog/2023/07/07/signals/</link>
        <guid isPermaLink="true">https://srezone.com/blog/2023/07/07/signals/</guid>
        
        <category>Signals</category>
        
        <category>Observability</category>
        
        <category>High-level</category>
        
        
        <category>Observability</category>
        
      </item>
    
      <item>
        <title>First post - Why I&apos;m starting this blog</title>
        <description>&lt;h2 id=&quot;leading-up-to-this&quot;&gt;Leading up to this&lt;/h2&gt;
Being a full-time software engineer since mid-2017, my practices, impact, and mission in the field have evolved (for the better.)

In the early years of my career I was a general software engineer – a jack of all trades, without specialization. In early 2021 I applied for a software engineer position at an insurance company. One thing that made this role different from the others was that it introduced a new concept I hadn’t heard before: SRE (Site Reliability Engineering). While this role wasn’t exactly an SRE position, it aimed to bring SRE culture and practices to the organization… But that’s a story for another time.

&lt;h2 id=&quot;diving-into-observability&quot;&gt;Diving into observability&lt;/h2&gt;
In mid-2022 I heard about OpenTelemetry (OTel). The idea of observability excited me, and even more: OTel was an open (and evolving) standard. I hacked together a quick PoC demonstrating distributed tracing and metrics. It was cool (a term I dislike when it comes to engineering), but I didn’t fully understand how to utilize OTel to reap the vast benefits it offered. There was much more to it. I poked around the internet reading articles, listening to podcasts (commonly, “O11ycast”), and watching videos on the subject.

Gaining confidence and knowledge in the observability space, I was excited to share what I had learned with the company. I started by comparing my learnings against the existing practices at the company, and the differences were astronomical. Though the company operated on fairly legacy software and was just dipping it’s toes into cloud-native/modern architecture, the company was still quite behind on practices when it came to greenfield applications. This seemed like fertile ground to drive change.

&lt;h2 id=&quot;pushing-for-change&quot;&gt;Pushing for change&lt;/h2&gt;
… of course, only after overcoming the hurdles that quickly presented themselves. A couple of hurdles that first come to mind (&lt;em&gt;from my perspective&lt;/em&gt;) were politics, company-wide lack of education (regarding modern observability practices), and the unwillingness to depart from traditional monitoring practices.

Applying traditional practices in modern systems is not scalable. Learning from SRE + observability experts in the industry, then seeing some of the “stategy leads” at this company run the strategy in a seemingly opposite direction at times was disheartening. Not to come off as brash, but way we we operated our “modern” systems was &lt;em&gt;inexcusable&lt;/em&gt;. Seeing some of the practices pushed for at this company made me want to pull my hair out, and once in a while, it still does. I remind myself this sort of evolution is a sociotechnical challenge - in this case: less so of technical, and more so with people &amp;amp; culture. We should’ve been doing much better, and we weren’t.

&lt;h2 id=&quot;wrap-up&quot;&gt;Wrap up&lt;/h2&gt;
So, the primary reason I’ve started to blog is to share knowledge in engineering areas that I’m most passionate about. While most of what I write are opinionated views of my own, I &lt;em&gt;highly&lt;/em&gt; encourage criticism. Everyone needs to be marching to the same beat to get to where we need to be.

The other two reasons for this blog are byproducts of the first:
&lt;ul&gt;
  &lt;li&gt;Networking - building relationships with others who share a similar passion, or are simply looking for ways to incorporate these practices into their own organization&lt;/li&gt;
  &lt;li&gt;Knowledge retention - writing/explaining thoughts helps me remember them better. Also, having an easy way to share and review prior posts helps recall/propagate information in a concise way&lt;/li&gt;
&lt;/ul&gt;

No more settling for mediocracy. This is finally an opportunity to share knowledge, build synergetic relationships in this emerging field of focus, and improve modern software engineering for all.

- Mike

&lt;ul id=&quot;markdown-toc&quot;&gt;
  &lt;li&gt;&lt;a href=&quot;#leading-up-to-this&quot; id=&quot;markdown-toc-leading-up-to-this&quot;&gt;Leading up to this&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#diving-into-observability&quot; id=&quot;markdown-toc-diving-into-observability&quot;&gt;Diving into observability&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#pushing-for-change&quot; id=&quot;markdown-toc-pushing-for-change&quot;&gt;Pushing for change&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;#wrap-up&quot; id=&quot;markdown-toc-wrap-up&quot;&gt;Wrap up&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
        <pubDate>Sun, 02 Jul 2023 00:00:00 +0000</pubDate>
        <link>https://srezone.com/blog/2023/07/02/firstPost/</link>
        <guid isPermaLink="true">https://srezone.com/blog/2023/07/02/firstPost/</guid>
        
        <category>Introduction</category>
        
        <category>Observability</category>
        
        <category>Politics</category>
        
        
        <category>People</category>
        
      </item>
    
  </channel>
</rss>
