Today we were fighting a network issue which caused some bots (actually, about 10% of our hosted bots) to login and logout periodically.
The problem started with some bots getting frozen: they were not responding to the watchdog service commands (this is a SmartBots internal service which pokes bots periodically to restart those which lost a connection to the reality). Thus, bots began to restart.
However, the bots just restarted were still unavailable: they were connecting but then frozen again. Our emergency team got a signal that something is wrong: the number of crazy bots has increased to several percents. Even during a SL rolling restarts, this amount never exceeds 2-3% of bots total.
First we’ve decided that LL has changed something, so our bots are unable to determine the sim online/offline status. We started updating bots to analyze a network traffic. This also leaded to several restarts (only on a single bot server though).
Finally, we’ve realized that this is a network issue causing sim connection to corrupt. After contacting ISP support the broken router has been replaced and bot connections finally restored.
Thank you for contacting our support and detailing your issues!
And sorry for the mess with bots going online and offline. The problem is solved now.