On 5th February 2024 13:54 UTC, one of our core database storage devices started to fail. It created a chain of events to occur within Hitsteps products. Failover database servers could not handle the load on their own, which has caused the following impacts:
- Slow loading of the pages
The dashboard took longer than usual to open and show the data - Inability to track every page view due to slower processing speed, some data in the queue were expired
- Unbalanced distribution of traffic in our load balancers
Due to a lack of storage device functionality in core database servers, the load balancer sent traffic to a few leftover failover servers that were not ready for the load. - Crash of visitor attribution engine
To detect visitor information, we depend on lookups to internal database engines. In this period we were unable to detect and store new visitor attributions (geographic, browser, loyalty, …) resulting in those fields to be empty in visitor reports.
We strive to provide accurate real-time analytics to our customers, however, this issue caused a few hours of statistics to be unreliable on 05.02.2024.
The root cause was detected and a solution was implemented in less than 2 hours. It took a few additional hours until the processing queues and visitor detection engine were back to normal.
To compensate lack of accuracy that happened on 05.02.2024 between 13:54 UTC to 17:30 UCT, we have decided to offer a week of free licenses to all of our paid customers. Your current subscription will happen as usual and the additional license will activate automatically toward the end of your billing subscription (i.e. when the subscription is expired, canceled, or updated).
