Degraded performance and Snowflake Query Failures
Incident Report for Lightdash
Postmortem

2:11 PM

  • Issue Reported: Users report that dashboards aren’t loading any charts.

3:41 PM

  • Additional Reports: Other users report that app is crashing

3:14 PM - 4:30 PM

  • Observation: High backend latency was detected.
  • Collaboration: Engaged in a chat with other engineers to identify potential causes.
  • Root Cause Analysis:

    • Reviewed changes released that morning.
    • Suspected the Node upgrade might be the cause.
    • Discovered a related GitHub issue with Snowflake and Node 20: GitHub Issue
  • Decision:

    • Immediate rollback.
    • Plan to create a PR to update the snowflake-sdk package.

4:25 PM

  • Rollback: Rolled back from release 0.797.0 to 0.795.0.

4:47 PM

  • Feedback: Users report that app is functioning correctly

5:10 PM

  • Follow-up Action: Began working on the PR to upgrade Snowflake's SDK.
Posted Oct 03, 2023 - 16:01 UTC

Resolved
We've successfully identified the root cause and have taken measures to resolve the issue. We apologize for any inconvenience this may have caused and appreciate your understanding and patience during this time.
Posted Oct 03, 2023 - 15:39 UTC
This incident affected: Lightdash Cloud (Lightdash Cloud (US), Lightdash Cloud (EU)).