“Game changer.” Cisco and NetBeez Aid Loudon County Schools Pivot to New Normal: Webinar Transcript
– Mitch Dickey, Lead Communications Engineer at LCPS
Mitch Dickey has been a NetBeez and Cisco customer for years. When his team had to support a massive shift to remote teaching they were ready. Thank you Mitch and Saravanan for helping us to talk about what a difference a great team can make!
In case you missed the live webinar, the transcript follows. Highlights are in bold.
Or, you can watch the entire webinar on demand.
Jess (00:01):
Thank you for attending our NetBeez and Cisco co-hosted webinar with Loudon County Public Schools, learning how to help school districts pivot to the new normal. So we’re going to introduce our panelists. We have Mitch here from Loudon County Public Schools, and I think he literally is at school or in the office again. So I’ll let everyone introduce themselves after this, Saravanan from Cisco and then Greg from NetBeez and Greg, I will let you take it away.
Greg (00:39):
All right. Thanks, Jessica. And apologies to anyone if you hear fire trucks in the background or the dogs barking at them. Here in California, yes, we’re pivoting to the new normal, red skies and fires. We have a really great webinar about a very remarkable public school district that’s pivoted to the new normal almost seamlessly. And when Mitch covers what they’re up to, their network design, you’re going to see a lot of parallels to large organizations, distributed organizations, and a pretty amazing story. And so we’ll talk about the Louden Public Schools, about their network design and Saravanan obviously weighing in from Cisco as well on the design, the remote troubleshooting. Both of them will talk about the Cisco app posting integration and then Saravanan will even share some insight into the future roadmap. And we’ll try to get through this relatively quickly so that you can have that Q and A with the experts that you just don’t get very often and have them add a lot more insight based on your questions. So, Mitch, could you kick us off and just tell us about this very, very remarkable school district?
Mitch (01:57):
Sure. My name is Mitch. I’m lead communications engineer for Loudon County Public Schools in northern Virginia. We have a pretty diverse county as far as demographics, as well as geography, which we’ll talk about here in a little bit. We’re very large. We’re one of the fastest growing districts in the United States right now. I’ve been here since 2014 and we’ve built at least one school every year since then, sometimes two. And even with what’s going on right now with COVID, it hasn’t stopped any of that. We’ve got a brand new high school that’s opening or was supposed to open for in class students this fall, but obviously it’s empty right now. Yeah, we’ve got, I think the number is actually over 80,000 students now and just over a hundred sites that we cover in total. So obviously being where we are in northern Virginia, we’re kind of in the, and Greg might allude to this later, we’re kind of in the middle of data center central. The cloud lives right here in this county.
Mitch (03:17):
There are data centers everywhere. They pop up. It’s weird driving from site to site. Sometimes you see new construction and you’re like, “I’ve never seen that data center there before.” It happens that quickly. But with that said being as large as we are, there are challenges and we have to rely on tools such as NetBeez and we use Cisco to meet the needs of the students and the staff members here and even more so now, even though the schools aren’t filled with staff right now, there are some staff, even though they’re not filled with students and teachers for the most part, the service level still has to be maintained at a very high level. And so far we’re doing pretty good with that. Of course, it’s very dynamic right now. Everybody knows that things can change minute to minute and we have to do that type of thing. We had to do that not too long ago, but back in the spring time, we had to quickly prepare our network for lots more virtual or VPN connections.
Mitch (04:29):
I mean, we had to do it basically overnight and the team that I’m on, we’re a team of six. And so I don’t want to say we’re stretched thin. We’re pretty good. Of course, we could have a few more. It would be nice, but we get the job done and we’re pretty proud of that. But yeah, we, we were able to shift to a somewhat virtual learning model pretty easily back in the spring. So yeah, it went pretty good. Of course there was bumps here and there, but I don’t think anybody was immune to that when that happened back in March. So yeah, we’re doing pretty good with what we’ve got. We’re very fortunate where we have good leadership here and it’s fun to serve the students because as Greg has written in some of these posts or Stefano’s written, we’ve got a lot of tech savvy clientele that go to our school systems and their parents are the same being as though there’s a lot of tech industry up here.
Greg (05:39):
And, Mitch, one of the things, I mean, what really came out loud and clear was the mix of students who could be very technically sophisticated because their parents are in this industry and then you’re serving areas where some of the Chromebooks, et cetera, might be their first computers. And so that in itself, yeah, gives you some network design and service consideration issues.
Mitch (06:08):
Yeah. So we, and just a little bit of how Louden is, it’s pretty cool. Like I said, it’s diverse demographically and geographically, like towards the western end, it’s very rural. So we run into the typical situations with rural school districts where reliable high speed internet is a challenge. And then, as you get to the center, Leesburg, Ashburn area, things are starting to be built up there. And then as you go east, of course, it gets closer to Fairfax, which obviously is built up. So it spans a very large gamut as far as the clientele we serve. And obviously we’ve got lots of high-speed reliable internet for students to use at their homes right now. And then we have some that have to rely on hotspots that are issued by us. So, like I said, it’s ever changing. So we have to be able to be nimble and leading into what we’re getting ready to talk about that’s why we rely on NetBeez a lot for things like that.
Greg (07:15):
And I think what really stood out, Mitch, from the conversation and Saravanan, I’ll let you comment, but what really stood out was that, before we chatted with you, we were thinking of this whole remote pivot is almost like a VPN story, right? And then you said, “Well, actually the core network is even more strategic because of this. And we have to have greater visibility than network, and we have to understand it.” So Saravanan, I’ll let you weigh in, but as we look at the network diagram, maybe we can talk about how the network and monitoring became much more strategic for your ability to pivot.
Saravanan (07:51):
Right. So I was just wondering, so when you said, so they have 80 K students who are online and learning their materials, and I’m just trying to imagine, how is it for you to handle it, Mitch, the team and you, right? So you are leading this effort, right, to manage the entire network of 80 K plus students online. So wondering how do you do that?
Mitch (08:14):
So I’m not the only leader on this team. Our group, without going into a whole lot of detail, is kind of split between the data side and the voice side. But obviously we all know the voice and the data are kind of coming together these days. But the group that I’m with every day in this building, where our office is, is there’s six of us and there’s three leads and then there’s three engineers. But, and it goes without saying, it does definitely have its challenges, but obviously having skilled employees is also a huge benefit, but also being able to rely on the tools, relying on the tools and reliable network equipment. So as you can see from this diagram here, which is a extremely high level diagram, so this was just a basic breakout when I put this slide together of one school site, and this is part of a new topology that we’re actually rolling out right now here in Louden County, where we’re putting in our own dark fiber, what’s been a project that’s been going on all year.
Mitch (09:22):
I don’t know how far along we’re into it but we’re a good ways and we’re cutting sites over. That whole ring is complete at this point. So we are going to specifically rely on Catalyst 9500s in the core, and then 9300s off of the core. And then if it’s not going to be these type of agents, we’re going to be doing the ones that are app hosted in a situation in that little called out area, you can see right there, that blue called out area where it says school. That’s typically how a school will look. This is, again, high level where we have a 9500 as a core, and then 9300s hanging off of that and the IDFs. And then the idea will be to deploy, and in fact, we’ve got 12 of them right now, 12 of them deployed across the district, mainly in our MDFs in a 9300, so that we can monitor the networks at these sites. And even though, like I was saying, there’s very limited staff at these sites right now, however, there are services still being provided by the county through these sites.
Mitch (10:30):
So it’s still extremely critical at this point that these network services stay up. Some people might think, “Oh, we don’t need to worry about that right now. There’s no kids, there’s no teachers, they got a principal there. We’ll go over and fix that closet later.” No, we have to make sure these services stay up because there’s little tech like where they can bring in their Chromebooks to get fixed at certain sites. There’s, I’ve heard of meal distribution at some of these sites for students that have come in that might need a school lunch. They have to rely on network services, even if there’s technically no staff in these buildings. So they’re just as important now as they were before. So, and we look at it that way. Even though things are different, we’re still going full force. So again, we still have to rely on these tools and now being able to do this by putting them the NetBeez agent into the Cisco switches is just phenomenal.
Greg (11:30):
That’s a great segue right to the whole remote troubleshooting challenge where just simply knowing the device status really isn’t enough for what you guys are charged with.
Mitch (11:44):
Absolutely. Absolutely.
Greg (11:47):
It’s useful to know, but the user experience for you is much more meaningful.
Mitch (11:54):
Sure. We are all familiar with other vendors that provide you alerts, whether something is up or down and that’s great and that’s important, but however we need to know like what is the bandwidth between these sites? What’s going on here? What does this look like from the client perspective over at that high school? They’re doing a troubleshooting help desk over there for Chromebooks and they’re experiencing an issue. I need to know, is this a problem in the WAN or what is it? And having the ability to deploy an agent like this offsite like that containerized, game changer. I mean, you guys have heard me say that before, when we’ve spoken before this. And it really is a game changer and being able to leverage that type of power in a piece of equipment that we already have is outstanding.
Greg (12:46):
And I remember, Mitch, you telling us, when we were preparing, about your ability to deploy a whole lot of sensors inside an hour. And when you think about the demands of a classroom, right, and 80,000 people being online, minutes count, hours count in terms of your ability to diagnose a problem. And if you can deploy a sensor in 10 or 15 minutes in a remote area, that’s a pretty powerful capability.
Mitch (13:12):
I mean, even in the traditional setting, when students are in the classroom, they heavily rely on wireless devices, more so wireless. But of course, there’s an occasional wired, I guess, but yeah, I mean, it’s critical. And at any given time in any given classroom, I need to plan for upwards 30 to 60, maybe 70 wireless devices. Now that’s a challenge in itself. So I mean, I primarily on a day to day deal with mostly wifi stuff. But obviously, now that we don’t have many staff members in here, the wifi is being underutilized for once. But yeah, I need to be able to figure out very quickly if there’s an issue occurring because instruction then suffers. And then the teacher is frustrated and we don’t want that. We want to be able to get in, do what the teacher’s lesson is, get it done, and then have the students get what they need and go through class and have it go flawless. They shouldn’t even have to think about the wifi or the network. It should just be there.
Greg (14:22):
Yep. Yep. Saravanan, any comments?
Saravanan (14:27):
I was just really closely watching what exactly Mitch was saying. So even when we are preparing, so he was so happy to see how exactly the sensor is getting integrated to the Cat9K, right. So it’s really happy to see that it’s helping a lot, that they are able to maintain such a big network given the current pandemic situation. And when they are able to deploy the sensors, multiple sensors at multiple sensor remote locations with just few minutes, that’s exactly what we are trying to solve for our customers. So glad to see that that’s helping the NCPS team and definitely would like to hear more from you, Mitch. So how was it solving your problem day in, day out? So can you explain more?
Mitch (15:15):
Sure. Yeah. And just to elaborate a little bit on that, how easy it was, of course, like the first one, you kind of stumble your way through figuring it out. But after that, I mean, I rolled out, I can’t remember. I know we wrote about it in the blog, but I was able to roll out agents on a pretty quick basis, but being able to leverage that and I said it before, but being able to leverage the power of the NetBeez, and I’ve been a NetBeez user for several years now, into equipment that we’ve already invested in.
So that’s huge. And so I’ve got all the power in the NetBeez now in equipment that I have in my network. So, I’m looking at things, specifically in my case, I’ve got it up over here and I’ll just look at some of the targets, but like for instance we want to monitor our DNS servers because obviously that’s obviously pretty important. So we monitor DNS servers from around the county because just because the centralized DNS servers are working well for one section of county doesn’t necessarily mean we have to maybe troubleshoot that.
Maybe there’s a WAN link with some congestion or there’s some kind of failure. We monitor a lot of the external resources because obviously a lot of things, a lot of the resources we use are in the cloud now. So there’s a lot of stuff out there. So we have to monitor, make sure that these external web services are reachable, whether it’s via HTTPS or however, and that we just have a bunch of targets set up to monitor common ones that we use.
We also monitor internal web services just to make sure some of our internal web services that actually are in our data centers are available. We just monitor things like just latency, make sure everything’s looking good. We will also test in perf. Let me bring that up real quick. So I can tell you accurately. We can do, iPerf between data centers and the sites around the fiber optic ring, just to make sure our ring’s in good health. And we have that set up to go, it looks like, every 10 minutes, staggered of course, just so we have a good idea of what the WAN’s looking like it at any given time and, yeah, sure we got a hundred gig ring, but yeah, that sounds like a lot, but it’s the old adage, the more you give the more they’ll use.
So it’s super important to just make sure that the network is as reliable as it can be. And if something does happen that we can be quick to it. Loudon County is a very large county, and there are six of us, so it could potentially take one of us, right now where I’m sitting is on the extreme western end of the county.
If we need to get to the extreme east end of the County, we’re talking maybe 45 minutes. So 45 minutes is a whole class period sometimes. So, when you’re losing that type of instruction time, that’s not good.
So if I’m able to leverage all of the power in these Beez, with the equipment that we have over at these sites, I mean, I’m saving tons of time. I’m gathering really tangible data that will then it might actually get me to my fix, but if it doesn’t get me to my fix, it gets me that much closer or I can be like, “Hey,” and reach out to another team and say, “Hey, can you look into some of your equipment?” or something like that. It just gets us to that end goal a lot faster and it’s paying off big time.
And I hope I’m not jumping ahead too far, but we really can’t wait until we can get the agent into the 9500, so we actually have it within our fiber rings with. With this new fiber ring that we’re putting out there, which is complete at this point, I’m very excited that we’ll be able to hopefully put some of these agents in the ring so that I can just monitor just that portion because obviously, that’s the core of our network. So I’m really looking forward to that and sorry if I let the cat out of the bag on that one, guys.
Greg (19:27):
No pun intended, right? No pun intended.
Mitch (19:28):
Right, right, right. I’m super excited about that.
Greg (19:31):
Did you want to comment any more on the integration or can we go to the next slide? What do you guys think?
Saravanan (19:37):
So I just have a surprise to you, Mitch, when we talk about the roadmap about the 9500 enablement. So we’ll just park it till there. So I really liked the way you explain how you’re deploying the sensors. Right. So that’s exactly what we are trying to deliver as part of that posting solution on Cat9K. Right. So how are you going to deploy it even for such remote locations? And you don’t have to worry about spending some extra budget on your dedicated compute resources that you have to plug in to do all your routine tests. Right. And then ensure how we’re going to scale it. So, the old saying says that time saved is money earned. So in this case time saved us better schooling for a bunch of kids. Right. So that’s exactly what we are trying to solve. So yeah. Let’s see what you got beyond this.
Mitch (20:28):
Yeah. That’s exactly right. I mean, not that I don’t love these little guys around, but being able to put them into the switch and have them perform the same job is just great. It helps us integrate our tools into the infrastructure, which is just outstanding and being able to troubleshoot and gather that data from the client perspective.
And that’s what I’ve always liked about the NetBeez is that it’s really based around the client’s perspective, because at the end of the day, that’s the person who’s consuming your consumables. So, those are the ones that you really got to worry about. So if somebody’s having a bad experience, I think it’s super important to gather that data.
And a lot of times, and I’ve told this story a million times where, we’ll get a call, the wifi is not doing great in this classroom or whatever and we get out there. And the teacher would joke around and say, Oh, you’re here now and it’s working, of course. It’s been working now for 30 minutes,” and then we have no clue why that happened.
Mitch (21:40):
So what we did traditionally is where we would leave the traditional Beez and now there are the newer ones here and we would leave it behind and have it gather the data that we need. And then we could have the teacher reach out again directly to us and say, “Hey, the problem’s happening now.” And I can get onto the NetBeez dashboard and really dig down in and see what is happening from the client’s perspective, which is the most important. So that’s what we really love about it. And now that we’re able to kind of extend Beez into our switching, I mean, I keep on saying it, but it really is a game-changer. And in the ease of being able to do that, I mean, I said, what did I tell you, Greg, there, I did five of them in an hour.
Greg (22:25):
In and hour, across your remote. Yeah. That blew me away because you have to think about in a normal environment that if you could deploy one in a couple hours is not bad, but to be able to deploy that quickly and get the network effects, right, of feeds from multiple points, right.
Mitch (22:41):
As soon as I got it online, you assign the targets and boom, it’s already doing the job. And to be honest, I can probably do it even faster now, just because I’m more familiar with it. It’s not, it’s not a difficult process.
Cisco did a really nice job, obviously I’m no coder or anything like that. So I mean, as far as how it’s done, it’s not that difficult. I went through some documentation. I was able to get to the end goal very easily. I talked to Stefano a few times concerning getting the NetBeez in there. And it was after that, I was like, “Okay, no problem.” And then I was able to start cranking these deployments out.
Greg (23:18):
Yeah. It’s amazing. Hey, Saravanan, we’re ready to let the cat out of the bag. Did you want to talk a little bit about the roadmap?
Saravanan (23:24):
Yeah, exactly, Greg. Yes. So far today, what we support on the cat and case side on the app hosting story is we support app hosting through the daily center on the Cat 9300s today. And along with that, we also support the cold restart, the auto restart capability on the 9300s, what we call the one plus one redundancy, right? And along with that, we also introduced the span capability, what we call is the port mirroring, right?
So, customer today can do port mirroring. Let’s say they have some traffic coming in, some export, and that can be read out through to the app that’s sitting behind the switch, right? So let’s say Mitch wants to monitor a particular traffic coming in, let’s say, a port 10, and want to understand what’s going wrong there or analyze what traffic they are, they’re getting in that particular port. Right? So that traffic can be completely mirrored to the app, then the Beez app that’s sitting in the switch, right?
Saravanan (24:23):
So that’s a capability we brought in so far till July 2020 and the span capability is what we call as a limited availability feature. So moving on, what we are bringing in for November release is that we are enabling to support for 240 GB variant of the SSD stake on the 9300s. And along with that, yes, the cat is out. We are also trying to roll out the app hosting feature on the 9500 family, which is due by March 2021. So, and along with that, we are also trying to see if we can bring in that posting on the internal flash. It’s still at the early stages for us to disclose anything here. And we are also trying to bring in the app hosting workflow through our DNA center. So like Mitch mentioned, right, so maintaining remote location or deploying the sensors as remote location as it’s made easy. So it’s made easy for us today through the Cisco DNA center, what we see as the single pane of glass for everyone to operate for any locations.
Saravanan (25:32):
And you can deploy the sensors just with few clicks, right? So we are scaling up the 9300 footprint, the 9300L, 9400, which we’re planning to deliver in November 2020. And looking beyond November, what we’re trying to enable is make app hosting work on our SDA fabric deployment as well. The go to thing that we are trying to do as part of IBNG, what we call us the intent based networking, right?
And with that, we are trying to bring in the cold restart capability in our stack wise virtual environment as well, which is the redundancy beyond just one taxi, right? So you enable the stack ways virtual, but two systems are in place and then you maintain them as one virtual system. Right? So that’s where we are trying to enable. And along with that, we are bringing in the app workflow on the Catalyst 9500 series and the 9600s. So slowly the discussions had just started and definitely we will be there soon.
Greg (26:39):
So I have to ask our moderators here, Jess, how are we looking? Are there any questions? And we’ve got Mitch and Saravanan live. If you have any questions, send them in now. Jess, if you could, actually, if you see any interesting questions, feel free to throw them out there to Mitch and Saravanan.
Jess (27:00):
Okay. Yeah. It looks like we only had one question so far, which we answered about what’s the max. Oh, that one we did not answer yet. We answered which license is required on the Cisco Catalyst to enable the integration, which is the DNA advantage. And it looks like there is one other question here. What’s the max throughput that we can get on a Cisco 9500?
Saravanan (27:28):
Right? So we have one gig as a limit today, even in the 9300s, the same caddies for the Cat 9500s as well. And we are at a very early stage where we can see whether we can get the complete one gig through for anything that might get into through for when you have to do some encapsulation or something. So we are expecting anywhere, roughly around 700 to 800 megabytes that we can do on the 9500s.
Greg (28:02):
Mitch, I’ve got a question for you and I, and sorry to spring it on you right now live, but what’s the most interesting network monitoring challenge that you’ve addressed or solved over your years there? Anything kind of uniquely challenging or difficult or relevant?
Mitch (28:21):
Well, the best story that I like to tell honestly, is about when we first got the first agent right here, and I believe, and correct me if I’m wrong, Stefano, if he’s going to chime in or not, but at one point, and you still might be able to do this, is you could build your own Beez, and then you could put it in like a free trial hive or something like that. I can’t remember specifically when I saw that. It’s actually Lee Badman who led me to this at Wired, not on Twitter. And so I decided that to get at a pie, just like, I think it was the raspberry pie 2B is what this is, and then the wireless card that was in it. And I was like, “Well, let’s try this out because it’s always the wifi.” Those who work in the wifi industry know that it’s always the wifi. And I was like, “Well, let’s, let’s try this out. This sounds like a really cool use case right here.” So we built one up. It looked just like this, had a big aggressive antenna off of a USB dongle, no big deal.
Mitch (29:28):
And then I think I even had a USB battery pack on it and it was all crowed around. I have a picture floating around somewhere, but it looked fairly, if I found that in my classroom, I might be a little bit suspect, like what is this? But I put it in a classroom and we left a little note on the antenna. It looked like a flag. It said, “Hello, I’m NetBeez. I’m here monitoring your network.” And we left it behind. And we were able to gather a lot of good information out of that. And that’s what truly sold me on the Beez. That’s the one time when I built my own and forgive me if I’m mixing up the logistics of how that worked back then. It was able to see, like I said before, from the client perspective, any issues that were occurring or weren’t occurring and then we can differentiate and go forth from there. But that’s the one story about NetBeez that I enjoy telling, because number one, it was kind of fun to build.
Mitch (30:32):
I like little tools like this, especially the ones that are powerful and that can get you good data. And in that case, I was able to get good data from that classroom. And hopefully, I didn’t startle any teachers or students by that big, aggressive antenna that was on there.
Greg (30:48):
Excellent. Stefano. Did you want to comment at all?
Stefano (30:53):
No. I wanted to confirm that, yes, we supported the raspberry pie from the start. And there’s still the opportunity to do a trial or self-deployment with the raspberry pie image.
Greg (31:23):
Well, first I want to thank Mitch and Saravanan for taking time out of their schedule. And I really enjoy chatting with both of you and loved your blog post from last week, obviously our blog and stay tuned. If you have any additional questions that you didn’t get to ask, you can always email us at info@netbeez.net. So thanks everybody for participating and we’ll call it a wrap.
