[RESOLVED] Rule Log "Error enqueuing state-stays"

TLDR: Limited outage of ‘State Stays’ checks between 11:00 AM - 3:00 PM CST today, 2023-08-29. Issue now resolved.

At 2:30 PM CST today, we became aware of an issue affecting a subset of ‘state-stays’ checks. The issue was isolated to one server, which has since been removed from our cluster. Normal operation has been restored.

If you had rules set to trigger between 11:00 AM and 3:00 PM CST today, they might have failed to execute ‘state stays’ checks or delayed tasks if they ran on the affected server. Rules triggered after 3:00 PM CST should be working as usual.

Rules that self-reschedule (eg. ‘loops’) may need to be manually restarted.

Note that self-scheduling rules are not officially supported, but I call this out as some community members use this approach. -Josh

Next Steps:
We apologize for any inconvenience this may have caused. We are exploring why the problematic server wasn’t automatically flagged and working on solutions for automated fixes should this issue reoccur.