Sifflet - Source refresh operations not running for some tenants – Incident details

Source refresh operations not running for some tenants

ResolvedDegraded performance2 hours 35 minutes

Affected

  • ResolvedUTC

    The issue has been fixed on all tenants.
    We are still actively working on preventing any further occurence and improving our detection capabilities.

  • MonitoringUTC

    The mitigation is in progress and should take a few minutes to take effect on all the impacted tenants.

  • IdentifiedUTC

    The cause of the issue has been identified. It comes from the concurrent occurence of several bugs in the third-party tool we're using in our deployment pipelines.

  • InvestigatingUTC

    Our temporal workers responsible for running the source refresh, as well as some housecleaning tasks, are failing to start on a few of our tenants.
    Both manual and scheduled refresh operations are impacted.
    The user experience on our web application and the data quality monitors are not impacted.