Orange's operational practices in question during the breakdown of emergency numbers

By wetretrjyu
04/03/2022
3251 Views

The breakdown of emergency numbers on Wednesday June 2, 2021 revealed shortcomings at Orange. To the point that the government is asking the Telecoms Regulatory Authority to check, in essence, whether the operator is doing its job correctly. Orange's lack of responsiveness and understanding of the issues clearly appears on reading the report which has just been submitted to the Prime Minister. Six reported cases of death can be linked to the event. The breakdown affected the emergency numbers 15 (Samu), 17 (police), 18 (fire brigade) and 112 (single European number). Too much time to make key decisions The Orange teams took far too long to make key decisions during the degradation of their telephony services despite the messages coming from their customers. People from Orange have broadcast inaccurate messages about the time to return to normal. In addition, some Orange managers designated as people to contact in a crisis situation were difficult or even impossible to reach. The audit report was drawn up by Anssi (National Agency for the Security of Information Systems) with the assistance of IGAS (General Inspectorate of Social Affairs), IGA (General Inspectorate of administration), the CGE (the General Council for the Economy) and the CCED (the Commissariat for Defense Electronic Communications).

The failure is due to the malfunction of the call servers, supplied by Italtel, the configuration of which was modified by an Orange technician on Wednesday June 2 at 4:44 p.m. The very significant impact of this failure is due to a manufacturer's computer bug, acknowledges the report. Nevertheless, the actions carried out by the technician would have in any case caused an unavailability of the service because the order of execution of the configuration commands was not the correct one. Without the equipment manufacturer's software bug, this unavailability would however have been short-lived, a maximum of 15 minutes. The way of modifying the configuration of the equipment in production by the operator is singled out. There was no test phase of the new configuration and it was installed simultaneously on all the call servers. The operations that led to the software bug on the call servers were not tested on the pre-production platform, because Orange did not consider this operation to be sensitive. In any case, this pre-production platform does not have equipment to simulate the increase in telephone traffic. Malfunctions were visible within 5 to 10 minutes after the configuration change.

Orange's operational practices in question during the breakdown of emergency numbers

It took 1 hour to become aware of the impact on emergency numbers The report is surprised that the implementation of a crisis management system by Orange adapted to the scale of the problems was slower than that of the state. One would expect Orange to have more accurate and responsive monitoring of its network. In particular, Orange took nearly 1 hour to become aware that the breakdown affected the emergency services in particular, 2 hours to inform the authorities and nearly 3 hours to set up a suitable system. “This is due to several internal dysfunctions at Orange” underlines the report.

No crisis exercise simulating a software failure or a computer attack impacting the emergency number service has been carried out by Orange or organized by the State. Feedback on an incident in May 2018 did not benefit the operators or the call server project management team when a software bug was also the cause of the incident. There is a lack of detailed understanding of Italtel's product by Orange partly due to the fact that this technology is aging and therefore difficult to train. The explanations provided by Orange during certain interviews may have proved to be inaccurate. The crisis which significantly affected telephone communications on June 2 and 3, 2021 showed that the existing crisis system within Orange needed to be improved to take specifically take into account any malfunctions affecting the emergency services, in terms of response times on the one hand, and relations with the public authorities on the other.

No special measures were taken for emergency numbers In the Orange crisis management system, the quality of the subscriber – in this case the emergency services – is in no way an element of assessment used by the operator to characterize a crisis situation. There is therefore no particular measure planned for subscribers as sensitive as the emergency services, apart from a possible reinforced commercial approach with information via dedicated teams, and this only for those of them who have signed a contract. specifically with Orange Business Services.

Moreover, during the hearings with Orange managers, the audit mission was told that "the decision to activate the crisis system had not been triggered by the alert escalation from the services of emergencies but by the degradation of traffic on the interconnection between the IP and RTC networks which affected all numbers". This situation explains the spontaneous reaction of a representative of Orange during a meeting of the interministerial crisis center indicating "that companies have also been affected", a remark perceived by some as an indication of a lack of consideration of the nature vital emergency that was linked to this crisis. The report acknowledges, however, that the random nature of the malfunctions (intermittent outages, no blackouts) and their location did not make it easier for Orange to identify the reality and extent of the problem, nor to immediately detect the particular impact it could have on certain sensitive numbers. It lacked a device that would allow earlier detection of an impact that occurred at 4:45 p.m. but whose magnitude Orange does not seem to have identified until an hour later, if we refer to its handrail. Even the monitoring of social networks provided by Orange's marketing departments, for lack of a link with the technical supervision departments, did not make it possible to speed up the detection of the problem. One of the third-party operators was unable to contact Orange until 8:42 p.m. Lack of advice from Orange to circumvent the problem Orange did not provide either locally or national level, real advice and decision support, or workarounds, at least during the evening and night of June 2 to 3. In general, the messages from Orange seem above all to have been intended to reassure, even to make them wait, as one would do in a traditional commercial relationship.

Sequence of events at Orange: