I am having some issues with synchronization of devices with Hubitat this morning

Two examples - first is the text not synchronizing, second is on/off status. Seems to be happening consistently in certain devices but not all this morning. I rebooted hubitat to see if that would clear it up but no dice. Anything else I can check? Thanks.

Can you enable logging in the SharpTools app (in your Hubitat admin UI)… then when you change the text or toggle the switch, do you see it successfully sending that event across to SharpTools in the Hubitat Logs?

ok I ran the logging. Right now device 1787 is not displaying correctly. It shows “CHA3” in text in Hubitat and “—” in Sharptools.

When I look at the command in the log I see this:

app:4172022-02-19 03:10:09.945 pm debugEvent data successfully posted to SharpTools.io
app:4172022-02-19 03:10:09.914 pm debugSending event for DEVICE: 1787 text → CHA3

Seems like that worked but the text is not updating.

I am seeing this messages routinely - about every 10 commands:

app:4172022-02-19 03:10:20.540 pm errorError sending event to SharpTools.io: Connect to events.sharptools.io:443 [events.sharptools.io/142.251.40.115, events.sharptools.io/2607:f8b0:4006:81f:0:0:0:2013] failed: Network is unreachable (connect failed)
app:4172022-02-19 03:10:19.942 pm errorError sending event to SharpTools.io: Connect to events.sharptools.io:443 [events.sharptools.io/142.251.40.115, events.sharptools.io/2607:f8b0:4006:81f:0:0:0:2013] failed: Network is unreachable (connect failed)
app:4172022-02-19 03:10:15.467 pm errorError sending event to SharpTools.io: Connect to events.sharptools.io:443 [events.sharptools.io/142.251.40.115, events.sharptools.io/2607:f8b0:4006:81f:0:0:0:2013] failed: Network is unreachable (connect failed)

Right now the text display of that device is working properly in both places.

I then tried switching on a plug. It went on, it shows the :warning: on the tile but it did turn on. Log below:

app:4172022-02-19 03:24:47.922 pm errorError sending event to SharpTools.io: Connect to events.sharptools.io:443 [events.sharptools.io/142.251.40.115, events.sharptools.io/2607:f8b0:4006:81f:0:0:0:2013] failed: Network is unreachable (connect failed)

app:4172022-02-19 03:24:37.900 pm debugSending event for DEVICE: 835 switch → on

app:4172022-02-19 03:24:37.770 pm debugupdateDevice, params: [id:835], request: [command:on, arguments:null]

Interesting. It looks like the hub is intermittently reporting that the network is unreachable – it’s a generic Java error message that generally means there’s no network connectivity.

Has anything else changed on the hub or your network recently? For example, any App or Driver updates or Hub firmware updates or configuration changes? Oftentimes these errors are more of a ‘canary in the coalmine’ for other issues brooding on the hub – sometimes as simple as a bad driver or app overloading the hub.

You might also poke around the Device Stats and App Stats to see if there’s any clues there. Unfortunately, if something is causing deeper hub/network connectivity issues, then stats on other dependent apps and drivers might seem off as a result though.

I am at a loss. Everything appears to be fine with the hub. Nothing running out of control, logs look clean except for the errors coming from sharptools.

I have restarted everything - hub, router, and everything in-between. I have deleted everything on the hub that was not in use or barely used.

There is literally nothing going on in the hub and yet it keeps throwing these errors.

I am not sure what to do next.

I might have just missed it but has anything on the hub or with the network changed recently?

Sorry - I did check all of that but forgot to include it. I have not installed any new apps in a while. I have made changes to rules and created new devices.

I am running the latest version - hub-2.3.0.124. I have a C5 and a C7. Sharptools is running on the C5. Checking the Hubitat forum it looks like this was released on Christmas Day. I don’t know when I upgraded to this version but it was long enough that I don’t remember doing it.

I would love to be able to step back through the logs to see when these errors started but I cannot seem to find a way to do this.

In the restore menu it is offering me 2.3.0.121, 2.3.0.120 and 2.2.9.146. I could step back through the releases to see if the issue goes away?

This must have happened relatively recently or I would have noticed it before. Thanks.

P.S. I tried disabling pretty much every app in my Hubitat and let it run for a while. Still got errors. So it is hard to point to another app causing this issue? In my limited technical knowledge that tells me it might be something in the hub itself?

I thought this was interesting. Sharptools is by far the busiest app on the hub. Nothing much to see on the devices as far as I can tell.


Thanks for the additional details. I was hoping there might be some other clue as to what might have changed for this issue to start occurring. Doesn’t sound like anything sticks out though.

It may not be another app / driver on the hub, but it’s something we’ve seen in the past where a rogue app/driver can start wreaking havoc on the hub and impact other apps/drivers.

I would be interested in seeing the stats when things are running normal. It does seem to be slightly elevated compared to what I would expect, but I suspect it may be a result of the ‘Network is unreachable’ error.

Here's my stats for reference (tap)

SharpTools is pushing every event from my hub, so I expect it to have a high execution count, but the execution time (Avg Ms) is also low as I would expect since it’s just pushing messages across the wire.

As a counter point, you can see my development instance has much higher execution time as my development server is often offline so the events wait to timeout (it also has a special flag to turn it off, so it has a much lower execution count)

You could try uninstalling the SharpTools app and then reinstalling it. You’ll have to reconfigure your device selections but everything else on the SharpTools platform side should rehydrate when things are reauthorized.

Otherwise it might be worth reaching out to Hubitat support to see if they have seen the ‘Network is unreachable’ error before and have any clues on how to troubleshoot. I’m happy to be CC’ed on the message and help where I can.

ok, reinstalled the app. Still throwing the errors. I don’t see anything exactly like it on the hubitat forums. I will reach out to their support to see what they say. Stay tuned.

2 Likes

@josh It stopped at 12:07am this morning. No error since. I started writing the logs to my Synology syslog server and I took a look at what was happening around this time - nothing out of the ordinary.

Although I was happy to see all of my touchscreens updating as expected this morning, it does leave me with a certain sense of fear - not knowing why or what happened. But sometimes you just have to go with the flow.

Nothing back from Hubitat yet so will leave that open to see if they can put their finger on it. Will keep you posted.

2 Likes

Still working fine. I still have the ticket open with Hubitat but they haven’t responded. It does irk mean not knowing what the issue is but I guess I will just have to deal with that.

1 Like

Thanks for the update. Glad to hear it’s still working smoothly now, but I understand that feeling about not knowing the underlying cause. :flushed:

@josh It’s back again. Started throwing errors again. Everything seems to be fine on my network and I had Verizon check from their side. Hubitat never responded the first time so I replied to the same ticket to see if they can help me.

Today I dismantled my whole network. First I rebooted everything. Then I slowly removed devices to see if anything in particular was causing the issue.

I got down to one connection - the Hubitat, and it still threw the errors even when it was directly connected to my Edgerouter with nothing else connected. I also ran a clean cable from the Edgerouter to Hubitat and still got the errors.

Not the result I had hoped for, but it was expected.

I have a Hubitat C7 as well as my C5 that is causing the issue. I thought I would try some tests on this one to see if the issue occurs there. @josh I thought of creating a new account on Sharptools and connecting it to the C7 but it asks for my consolidated Hubitat login so I didn’t proceed in case it hoses my first setup on the C5. Can I run two Sharptools accounts on two Hubitats under the same Hubitat account? Thanks

Each hub returns its own token, rather than a shared one that has access to both hubs (like SmartThings), so you can’t have both Hubitat hubs on a single SharpTools account without something like HubConnect or Hub Mesh.

You can have each hub on a separate SharpTools account.

Side note that even if you somehow invalidate a token or uninstall the ‘app’ and reinstall it, the tiles / things should all rehydrate when reauthorized.

Ok, so I set it up. Created some sample dashboards. Almost immediately I got the error. So I am at the mercy of Hubitat to get involved here as I can’t think of anything else I can do.

What does the second hub have on it? Was it just a ‘blank’ test hub with no physical devices and no other SmartApps?

Other than the obvious issue with the device event states intermittently not reporting in from the hub, you don’t notice any issues when interacting with the SharpTools.io web app from your browser, right?

So, issue gone this morning. I am going to stay on Hubitat to see if they can help me debug this - or provide some form of logging to help understand where the breakdown is occurring. I don’t know if you have any saws with them - so far I have gotten zero response to the original ticket or any follow-ups.

The C7 has real devices on it. I am moving all of my z-wave devices to it and using Hub Mesh to access them in my C5 where all of my rules are.

For the test I built a dashboard connected to the C7 and added 5 RGB bulbs to it and switched them on and off repeatedly. I also made a virtual switch and button to play with.

All interactions from sharptools app work fine. When in this state I do see the exclamation mark occasionally on the tiles? Is that because Sharptools is not getting a response from Hubitat?

Thanks.