TLDR: Limited outage of ‘State Stays’ checks between 11:00 AM - 3:00 PM CST today, 2023-08-29. Issue now resolved.
Summary:
At 2:30 PM CST today, we became aware of an issue affecting a subset of ‘state-stays’ checks. The issue was isolated to one server, which has since been removed from our cluster. Normal operation has been restored.
Impact:
If you had rules set to trigger between 11:00 AM and 3:00 PM CST today, they might have failed to execute ‘state stays’ checks or delayed tasks if they ran on the affected server. Rules triggered after 3:00 PM CST should be working as usual.
Rules that self-reschedule (eg. ‘loops’) may need to be manually restarted.
Note that self-scheduling rules are not officially supported, but I call this out as some community members use this approach. -Josh
Next Steps:
We apologize for any inconvenience this may have caused. We are exploring why the problematic server wasn’t automatically flagged and working on solutions for automated fixes should this issue reoccur.