Boston Free to Air TV Streaming Service
I am enabled Vision Pro support in the latest Testflight build for the iOS app. I do not own a Vision Pro and have only tested it in Apples simulator. It appears to work perfectly fine. If there is anyone out there who has one and tries it out, please let me know how it is for you. Also, yes I am watching CometTV here. This will be arriving in the latest build along with MeTV Toons and WGBX this week.
Looking at our statistics I notice that very few people watch WGBH in this app. It makes sense because there isn’t a lot of reason to watch this stream while PBS provides a high quality free stream themselves. In the interest of focusing on gaps in availability and also supporting broadcasters who make streams available, we will be moving to the official PBS streams linking directly to their app/web page. Everyone still gets to watch the channel for free and it frees up one tuner and one encoder and money that we spend on bandwidth for PBS. All of these are limited resources.
We entered a wider beta at Thanksgiving, midway through the NFL season last year and I immediately collected a lot of ‘data’ about my designs as the service scaled up from 3-4 concurrent users to dozens and everything melted down. After many adjustments I was able to scale up and could handle 65 concurrent viewers for the Super Bowl. We’ve continued to make improvements and I think the quality of the service has visibly improved. Thousands of people watched the NBA post season and the Olympics over the summer.
How many people have signed up recently? The first spike you see is for the NBA finals, the second spike is for the Olympics. There are now around 3000 users.
I measure viewership in streaming minutes. This is how much people are watching a channel. We count one viewer watching for 5 minutes as 1 unit for these graphs. This is what the NBA finals looked like.
Sporting events like the NBA and NFL provide bursts of activity. The olympics were different. Since NBC broadcast everything all day, it was a long term sustained level of traffic unlike anything I’ve seen.
Overall we have been able to handle both one time spikes and also sustained traffic. Looking forward, we will work to reduce stalling as much as possible. We will also switch to a kubernetes cluster that will hopefully allow us to both reduce costs and scale up even more. There is also a plan to add more tuner hardware for metv toons and WGBX.
We’ve have many stress tests over the last year or so. The NBA finals saw around 400 people watch the Celtics win. We’ve adjusted the server code to handle events like that without issues. The Olympics present a totally different stress test . Major sporting events like the NBA post season occur in bursts. There is a game or two for a few hours every other night that everyone watches. The olympics are the opposite of that. NBC broadcasts it all day so there are now people watching LocalTV all day in a way that we have never seen before — regular day time TV is never that compelling. This is a good long term stress test. At any given moment there appear to be a few hundred people watching the Olympics. But the world keeps moving, so while there is a new constant load from the Olympics the usual other bursty things are also happening like the Red Sox v Yankees game on Saturday night. So far nothing has broken.
The huge spike below consuming a 1 TB data in a day is the Olympics. The smaller bump on the 4th of July was the pops concert.
The Celtics being in the NBA finals has presented a new opportunity to watch what happens when the traffic spikes on the current architecture. Unlike other events the NBA finals are a series of matches which present an opportunity to test over time. I made a change recently to switch from 2.0 audio to 5.1 based on user feedback. The server went down for unrelated reasons during game 1. Game 2-4 saw record number of viewers watching. Game 4 last night has a high of around 400 viewers this is nearly an order of magnitude greater than the 65 viewers during the Super Bowl . However something seems to have gone wrong as highlighted by reddit posts. We aren’t sure what happened but the graphs below offer some clues. Game 2, 3 are the first two peaks but the last peak despite being a much higher number of viewers has half the bandwidth served. It’s also possible that people experienced errors but also possible that the Celtics were losing and many people turned off.
However the final clue that something went wrong is the large number of HTTP 400 error encountered unlike the earlier two games. We will try switching back to 2.0 audio to see if that improves things
There was a lot happening behind the scenes. Today there will be new iOS and tvOS betas released with notable changes. Everyone should notice that videos start up much faster. We have removed many delays that prevented video from starting as soon as you click on a show. There is a major UI update for tvOS that should be a much better user experience.
On iOS you will notice two new buttons below a show description. The share button lets you share information about the show with others either via chat or social media. The bell button allows you to set a reminder for the show. A clear bell means you can set a reminder and filled in bell indicates a reminder has been set. Since are very serious about privacy so this information never leaves your device. As always, we do not track what shows people watch or want to watch. A few min before the show starts you will receive an alert on your phone.
Finally we would like to thank all of the donors who have contributed to this project. If you are interested in donating, please use this link: https://givebutter.com/LocalTVPlus
As a matter of principle, we want to retain as little information as possible about our users. In order to minimize friction and retain almost no personally identifying information, we have rolled out Sign in with Apple. When you use this we recommend you pick the ‘hide email’ option. This way, you have an account on our system but we do know anything about you other thans the fake email address Apple provides. This address forwards to your real email and still allows us to contact you if we ever need to.
Thank you everyone who used the app to watch the Oscars last week. As expected the service ran without a hitch. Also, as expected, concurrent viewer numbers were not quite the Super Bowl but still more than we usually see. At its peak there were 37 concurrent viewers watching. I see this as more of an indicator of a wider user base from where we started.
The CDN continues to perform as expected. You can see we served the same amount of data (50gb) but the cache served another 130gb of data
We have launched our website. While this blog will continue with engineering details, please use https://www.yourlocaltv.org going forward when telling people about this service.
We have doubled our tester count to over 1000. There is very little concurrent use however, so we don’t have enough data to understand what the breaking point of this architecture is. There are a couple of major events coming up where I hope people use the app and help us gather data.
As always, the app is free to use as much as you want. If you can, please consider donating. If you can’t donate, spreading the word or just using the app helps us improve the service.
Based on the feedback from users on reddit as well as my own experience, the Super Bowl test seems to have passed with flying colors. Everyone seems to have had a perfect video stream. The new antenna has had perfect signal. Additionally, the switch from RCN to Boston Broadband as our ISP is clearly visible in the response time graph below. Given this is a real time system, latency is important and the faster responses have helped.
We hit some of our highest concurrent numbers. Of the 400 or so testers that were using the app at the time, 65 were watching the Super Bowl. We were able to service everyone. For the Super Bowl we pumped out 360GB of data from our CDN. Based on how the CDN performed, while these are tiny numbers I am more confident the current stack should scale up to much more. The generally flat green line while the blue spikes indicates the CDN is doing a much better job of absorbing almost the entire load. We have increased the test flight limit to allow more users to use it. Thank you everyone who watched with the app. Please keep using it as the data we get from these tests allows us to keep improving our software.
The purpose of this long test phase is to keep making adjustments so we have a service that can reasonably handle the volume to traffic we expect while still keeping operating costs low.
Looking at the Super Bowl you can clearly see when the game started, ended, where and when the game got more interesting.
Finally, some house keeping. If you are new to this blog, this is primarily an engineering blog for the free LocalTV+ service run by Mass Local TV, a non profit based in Boston. This is a temporary website. Some quick answers to common questions: