Sifflet - Source refresh operations not running for some tenants – Incident details

All systems operational

Source refresh operations not running for some tenants

Resolved
Degraded performance
Started 1 day agoLasted about 3 hours

Affected

Integrations

Degraded performance from 8:45 AM to 11:20 AM

aws-prod-eu-west-1-a - Integrations

Degraded performance from 8:45 AM to 11:20 AM

aws-prod-eu-west-1-b - Integrations

Degraded performance from 8:45 AM to 11:20 AM

aws-prod-eu-west-1-c - Integrations

Degraded performance from 8:45 AM to 11:20 AM

aws-prod-eu-west-1-d - Integrations

Degraded performance from 8:45 AM to 11:20 AM

aws-prod-us-east-1-a - Integrations

Degraded performance from 8:45 AM to 11:20 AM

Updates
  • Resolved
    UTC
    Resolved

    The issue has been fixed on all tenants.
    We are still actively working on preventing any further occurence and improving our detection capabilities.

  • Monitoring
    UTC
    Monitoring

    The mitigation is in progress and should take a few minutes to take effect on all the impacted tenants.

  • Identified
    UTC
    Identified

    The cause of the issue has been identified. It comes from the concurrent occurence of several bugs in the third-party tool we're using in our deployment pipelines.

  • Investigating
    UTC
    Investigating

    Our temporal workers responsible for running the source refresh, as well as some housecleaning tasks, are failing to start on a few of our tenants.
    Both manual and scheduled refresh operations are impacted.
    The user experience on our web application and the data quality monitors are not impacted.