Miles of fiber. Dozens of vendors. Temporary, critical networks serving hundreds of thousands of event attendees set up in days.
A couple weeks ago I had the chance to talk to Travis Banks, TOURTech’s Director of Technical Operations, about the temporary but critical network demands of the large scale event business. Their story is nothing short of amazing. They’re a company with under 50 employees that scales massive networks for a few days at a time. Uptime and performance are critical. Seconds matter.
They have a noble mission: to detect and fix problems before users and partners can even detect them. That’s what they do.
It’s part of who they are as a company and a key driver to their success in winning and maintaining long term, large event contracts:
“Because our networks are temporary, our goal is to diagnose problems before our customers even detect them. We want them to be solved ASAP.” – Travis Banks, Director of Technical Operations, TOURTech
For Lollapalooza they lay about 5.5 miles of redundant fiber in a park for four days. About 400k people will attend the event and a few dozen sponsors will do everything from selling merch, to offering virtual reality games or even live streaming a top gamer to hundreds of thousands of fans around the world. They’ve been running the event network for years.
How do they do it? How do they stay on top of performance issues in complex networks? Read their remarkable case study.
Traditional Network Monitoring Tools Weren’t Enough
Because uptime is critical to event success, the TOURTech team needs robust, real-time network visibility for rapid root cause analysis. Traditional approaches to network monitoring wouldn’t give them the insight they need when they need it. They manage 15-20 large-scale events with hundreds of thousands of attendees (like Lollapalooza) every year, in addition to more than a hundred smaller venues.
In many ways these short-term event networks are as complex as networks serving larger organizations and managed by larger teams. A typical major event can have 20-30 different vendors/sponsors using a variety of different applications, ranging from RFID to ticketing, streaming, point-of-sale and immersive brand experiences.
They’re On It – Even When it’s Not the Network
Often the most frequent problems faced in these demanding environments has little to do with the network. By deploying a dispersed assortment of NetBeez sensors geographically, the team can reduce time to diagnose/fix, even when it’s not the network.
They get deeper insight, faster than other approaches. And according to Travis, the most frequent challenges faced during these hyper events are unresponsive servers (cloud-based and onsite)and 3rd or 4th party ISP routing issues. Other network monitoring approaches could leave them guessing.
“We’ll see high latency times or rejected pings from a cloud or onsite server and warn clients that there is a chance they’ll start having problems…” – Travis Banks
When network performance and uptime matters, NetBeez stands out because it delivers real-time user experience insight from multiple locations. Root causes can be identified quickly, sometimes before users and carriers even notice them.
Want to learn more about getting ahead of your network problems and user complaints?
Note: Since this interview, many large events have been cancelled because of COVID-19 infection risks. Given the scale challenges facing cities, hospitals and other critical services, maybe someone reading this would refer TOURtech and its unique, specialized expertise for scaling temporary, highly available network services? Contact us and we’ll introduce you.