Over the Thanksgiving, we ran our largest public beta yet. Expanding from a dozen users to over a hundred. This was announced on Reddit Boston . This was an excellent learning opportunity that stress tested the service.
As expected, increasing the concurrent traffic volume by an order of magnitude revealed bugs. Thanksgiving day football traffic caused multiple failures that we were able to investigate and improve. A lot of the issues related to the configuration of the CDN. The initial configuration had a low cache hit rate (10%) which manifested in much higher server load further down the stack. LocalTV+ is built as a collection of micro services, which can be scaled up or down as needed to optimize for cost/traffic. The cache failure rate began to increase load on other services which eventually became high CPU load on a service I never expected high load on.
We made changes to the CDN on Friday and in time for college and NFL games over Saturday and Sunday. This was a much more successful test. The cache fixes worked and the load was moved to the CDN. Overall when the signal was ok, almost everyone was able to watch the game with little to no issues. The issues that were reported were the infrequent freezing and re-buffering for a few seconds. We will continue to test and if the performance is consistent we will be able to open up the beta testing to more users.
Over the course of the weekend we also opened up a dedicated subreddit for general discussion and feedback.
Other findings:
- Several crashing bugs related to tvOS were identified
- Persistent errors on first login / registration about not being able to get streaming links
- The need to improve onboarding and the UI in many places e.g. highlighting on tvOS
- Airplay to non Apple devices work most of the time but is inconsistent.
- CBS continues to be a problem. This is related to the current antenna location.
- We will publish an FAQ soon. This will answer many common questions. e.g. Android (Yes, but will be later) or closed captions (Yes, it is just currently disabled while testing)