Thanksgiving 2024: Looking back at a year of LocalTV+

Just before Thanksgiving in 2023, we decided that we had reached the limits of what we can test with the dozen or so beta testers we had and started the public beta. We made a big announcement post on Reddit Boston asking for beta testers. Once Thanksgiving day arrived however, everything melted down and we learned a lot of lessons. Over the past year we have taken everything we have learned from that initial test and subsequent tests over the Super Bowl, the NBA finals and the Paris Olympics to adjust and improve the app and service. The subreddit has grown to 400 members and regularly provides feedback on each iteration of the betas we’ve put out. Our users have regularly contributed with donations, keeping this free service running. We’ve grown to thousands of active users and launched the TV app in the App Store this week.

Thanksgiving 2024 will be another big test. Thankfully, whatever happens it should be much better than 2023.

Looking ahead, there are a lot of improvements coming down the pipeline for 2025. Clients for Google platforms, subtitle support for every channel, and improved video quality and less stalling to name a few.

2025 fundraising goals

Thank you everyone who has donated. We hit the 1k goal we set for 2024. At the moment we have enough to continue operations with the current level of users (around 5k) until feb 2025. We have updated the give butter campaign to be 1440 more . This is 120×12 and should be enough to continue operations with the current number of users until the end of next year.

https://givebutter.com/LocalTVPlus

We do hope to get out of beta and go live in the AppStore in the coming months. It will make the app more readily available to more users hopefully it brings more donors too.

Why you can’t hear the audio on some non Apple devices

LocalTV+ supports Airplay and it is an option to send the to your other devices. This works very well when working with Apple devices. Over the past year I have seen reports that it would randomly work on certain Roku devices. Alternatively when it does work people would report that there was no audio. Eventually I discovered that not all Roku devices support the codec we use (more on this later). To add more confusion some Roku devices while not supporting the codec directly do pass it through to your audio hardware, so if you have a receiver that does support it, it would work.

This bring us to Android, where I have discovered that the first party video player ExoPlayer does not directly support AC3 Audio (there is a way to build ffmpeg and pass it through). In fact, it seems stock Android itself does not support AC3 audio. To add more confusion here, some Android devices do have AC3 support that’s been added by the manufacturer. So while I do have a basic Android client that can play the video, there is no audio at the moment.

So what is AC3 and why do we use it? AC3 is a Dolby Digital 5.1 audio format. The ATSC signal we receive uses AC3 for audio and we pass it along unmodified. We could convert it to another format such as AAC however, converting from one lossy format to another will result in worse audio quality. The actual conversion process will also increase the load on out current encoders.

Why does Apple support it but not Google? I am not sure. My hunch is that it requires a license form Dolby who owns the format and Apple has baked it into the cost of every device.

Ever wonder how much data we use?

Now that we have thousands of testers, this is what a typical NFL game looks like. This is the data usage from Sunday’s games. You may notice that we have also increased the video quality. So far it has been quite successful. We are not seeing many if any buffering/stalling events on the clients. We will continue to improve this. The green line is the hit on our servers. It doesn’t go up very much. The blue is what the CDN absorbs. That can go up until CDN reaches its limits, whatever they are. We pay 0.005  per GB so the 1.34 TB cost us about $6.80 to serve. So far in Sept we are at 9.7 TB total which is about $47 in bandwidth costs.

Vision Pro Support

I am enabled Vision Pro support in the latest Testflight build for the iOS app. I do not own a Vision Pro and have only tested it in Apples simulator. It appears to work perfectly fine. If there is anyone out there who has one and tries it out, please let me know how it is for you. Also, yes I am watching CometTV here. This will be arriving in the latest build along with MeTV Toons and WGBX this week.

Making changes to PBS

Looking at our statistics I notice that very few people watch WGBH in this app. It makes sense because there isn’t a lot of reason to watch this stream while PBS provides a high quality free stream themselves. In the interest of focusing on gaps in availability and also supporting broadcasters who make streams available, we will be moving to the official PBS streams linking directly to their app/web page. Everyone still gets to watch the channel for free and it frees up one tuner and one encoder and money that we spend on bandwidth for PBS. All of these are limited resources.

September Update: NFL and Upgrades

We entered a wider beta at Thanksgiving, midway through the NFL season last year and I immediately collected a lot of ‘data’ about my designs as the service scaled up from 3-4 concurrent users to dozens and everything melted down. After many adjustments I was able to scale up and could handle 65 concurrent viewers for the Super Bowl. We’ve continued to make improvements and I think the quality of the service has visibly improved. Thousands of people watched the NBA post season and the Olympics over the summer.

How many people have signed up recently? The first spike you see is for the NBA finals, the second spike is for the Olympics. There are now around 3000 users.

I measure viewership in streaming minutes. This is how much people are watching a channel. We count one viewer watching for 5 minutes as 1 unit for these graphs. This is what the NBA finals looked like.

Sporting events like the NBA and NFL provide bursts of activity. The olympics were different. Since NBC broadcast everything all day, it was a long term sustained level of traffic unlike anything I’ve seen.

Overall we have been able to handle both one time spikes and also sustained traffic. Looking forward, we will work to reduce stalling as much as possible. We will also switch to a kubernetes cluster that will hopefully allow us to both reduce costs and scale up even more. There is also a plan to add more tuner hardware for metv toons and WGBX.

The Olympics have been (technically) interesting

We’ve have many stress tests over the last year or so. The NBA finals saw around 400 people watch the Celtics win. We’ve adjusted the server code to handle events like that without issues. The Olympics present a totally different stress test . Major sporting events like the NBA post season occur in bursts. There is a game or two for a few hours every other night that everyone watches. The olympics are the opposite of that. NBC broadcasts it all day so there are now people watching LocalTV all day in a way that we have never seen before — regular day time TV is never that compelling. This is a good long term stress test. At any given moment there appear to be a few hundred people watching the Olympics. But the world keeps moving, so while there is a new constant load from the Olympics the usual other bursty things are also happening like the Red Sox v Yankees game on Saturday night. So far nothing has broken.

The huge spike below consuming a 1 TB data in a day is the Olympics. The smaller bump on the 4th of July was the pops concert.

NBA finals stress tests

The Celtics being in the NBA finals has presented a new opportunity to watch what happens when the traffic spikes on the current architecture. Unlike other events the NBA finals are a series of matches which present an opportunity to test over time. I made a change recently to switch from 2.0 audio to 5.1 based on user feedback. The server went down for unrelated reasons during game 1. Game 2-4 saw record number of viewers watching. Game 4 last night has a high of around 400 viewers this is nearly an order of magnitude greater than the 65 viewers during the Super Bowl . However something seems to have gone wrong as highlighted by reddit posts. We aren’t sure what happened but the graphs below offer some clues. Game 2, 3 are the first two peaks but the last peak despite being a much higher number of viewers has half the bandwidth served. It’s also possible that people experienced errors but also possible that the Celtics were losing and many people turned off.

Bandwidth served
Total viewers during game 4

However the final clue that something went wrong is the large number of HTTP 400 error encountered unlike the earlier two games. We will try switching back to 2.0 audio to see if that improves things