On August 18th, 2026, New York City experienced a significant 911 service disruption after Motorola, the city service provider, performed a routine software update. Approximately 1,700 calls were impacted during that outage. If I recall correctly, IBM is the principal vendor supporting the 311 system, and during prior testimony, you stated that— a joint team of OTI and IBM is responsible for 311 system updates.
Can you walk us through the standard process for implementing software updates to 311? Thank you for the context, and yes, I can give you some of that perspective. I probably can't go deep as some of my colleagues at OTI would, but I can give you a flavor of that. And I think the starting point would be to make a distinction.
So the work that we do with IBM and what was previously referenced We do that mostly for what we call applications development. So for the system that you see online or the system that gets used in the call center, the application side of that, not the telephony system. Okay. So those are 2 separate. And then most recently— The telephone system.
So the telephone system is actually the source system. The platform is a Cisco platform. Cisco. Which is pretty common across the city. And then back on the application side, As we in the past when we would have referenced IBM, the OTI is now using MWBE contracts more so than the IBM contract. Okay. So that work though is similar.
We're contracting for specific skill sets and specific work that can help us develop and build out in our application. In some cases, you know, you bring it on, you help build it, and then you don't need it anymore. But all of that gets overseen and managed by the OTI organization. Okay. Obviously, you know, given what happened at 9/11, we have a few more questions about updates performed on the regular.
Are, you know, software updates of that nature happening on a regular basis? Are they predetermined? And then who determines when an update should occur and who within OTI or 311 must approve before it is deployed? Great. Thank you for that question. I get to expand a little bit on the work that we do. So there are a number of levels of approval.
Software updates are very much our focus, as you can imagine, even without the reference to the earlier August event. But this one, I would say, goes back 23 years as well. It's grounded in 311's approach that we must always be available. And there's always times where you're going to have scheduled updates and scheduled outages. then you got to have a backup and you got to have a backup to the backup.
So that's something that we take, whether it's the telecom side, whether it's the application side, take very serious. And therefore we have a number of processes in place. I'll just speak at a high level. And I have the privilege of also supporting the city's internal customer service desk, the citywide service desk. Within that team, there's a number of folks who focus solely on change management, as we call it.
Mm-hmm. So I can kind of speak to that in both ways. But there's always a need for software upgrades, software updates, software changes, software upgrades. It's a very controlled process. The teams that need to do it put forth what they need to do. There's a body that understands what's happening. So your change, you have to think about your change, but also will it impact somebody else's change?
They all go through an approval process. It's pretty standard. It can be on a daily basis. It can be on a cycle basis. But there's a pretty standard protocol of being able to go back and look at here's the changes we need to make, here's why we need to make them, and has everybody signed off on making those changes?
Since the 911 outage, has that prompted you all to review or change procedures governing software updates to 311? Thank you for that. I will say yes, because any event that occurs forces us to do just that. So this winter when we had, you know, extreme winter weather, This summer, when we had heat waves, you know, dating back again, you mentioned the history of 311, but dating back to their earliest days when there was a transit strike in 2003.
That causes us to think about things, whether it's a natural disaster, whether it's a city— sorry, whether it's a technical change, but anything that occurs, we have a process through our emergency management resources on our team. Mm-hmm. We also partner with New York City emergency management, but within our own team, we'll evaluate anything that is out of the norm and say, how would this— how would we handle this?
Are we covered? Are we set? Are we able to do that? So one of the questions that we asked of OTI when we were doing the 911 hearing was if there were any redundancies or failover procedures if something is failing, like live, like something is failing and someone, like for the 911 outage, it wasn't clear that something was failing until some of the 911 dispatchers began to notice things that were happening that were out of the ordinary.
So can you walk us through what that looks like for the 311? Yeah, sure. Thank you for the framing of that. So we do have a number of alerts and flags that are designed from a system perspective, right? So it's built into our operating systems, our alerts, and then there's systems that do monitor just for alerts. So that's a necessary requirement, somewhat of a standard requirement.
But we also have built in over the years tools that will alert us if something were to happen. Our telecom platform is very robust. It operates very well, but things can happen. We have built in— I'll give just a simple example. If a call has not been received for X minutes, there's going to be a trigger and an alert and an email generated to one of our 24/7 operating people.
So during the course of a day, you know, if that were to happen, it would be very quickly detected simply because there's a lot of calls during the day. But we also tweak it so even on the overnight when there may be gaps in calls because not a lot of people are calling, if we go for a certain period of time, minutes, there is a trigger and an email report and it goes to a live person to be able to say, now what do we do?
The city has a very structured process that OTI oversees. I mentioned before the citywide service desk. That's the first step. Open up a ticket. Let's get everyone involved. Let's start to understand what it is. But the identification of that is something we have built in over the years.