I’m sorry if your bot went offline today or yesterday. We’ve finally fixed the problem and all bots are running and healthy now!
So, what happened?
Yesterday one of the bot servers (we have bunch of them to keep the bots online) started freezing and rebooting. Checking it revealed that one of SSD drives has worn out (that heroic drive ran for almost 1000 hours and ate about 200TB of data).
Replacing it took a while (it is like pulling out the tablecloth from under the dishes… and then pulling on a fresh one!). A short outages happened because moving bots to a reserve server requires them logging out and then back in from another location. This caused a kind of stress for some bots.
Please take our apologies if your bot were logging in and out. We know this is very annoying. From now on, all bots are calm and happy again!