Sifflet - Scheduled monitor orchestration disruption – Incident details

eu-west-1-c - Datasource monitoring experiencing partial outage

Scheduled monitor orchestration disruption

Identified
Degraded performance
Started 9 days ago

Affected

Datasource monitoring

Partial outage from 9:00 AM to 12:00 AM

eu-west-1-c - Datasource monitoring

Partial outage from 9:00 AM to 12:00 AM

Updates
  • Identified
    Identified

    We identified the cause of the issue, caused by a failure in the behavior of our orchestration framework that happens in some very specific situations.

    The fix for this issue will be deployed in production on Thursday 26th of February.

    We've also clarified the impact of the incident: only one of the tenants of the cell is impacted, and only 7 scheduled monitors of this tenant are impacted (meaning they are no longer running according to their schedules). The customer associated with this tenant has already been contacted by our team.

  • Investigating
    Investigating

    We’re currently experiencing an issue with the orchestration of some scheduled monitors on the eu-west-1-c cell. Some of them might no longer be running according to their configured schedules (for instance "hourly", "daily" or "weekly").

    Manual runs of monitors through the UI or the API is not impacted and works as expected.

    Our team is currently investigating the issue. We’ll provide updates here as the situation progresses.