Time to backup

The summer of 1999 saw Britain’s Labour government hit by a passport crisis. This led thousands of holidaymakers to camp on pavements outside offices waiting for their travel documents to be renewed. The backlog reached a peak of 565,000 applications and Jack Straw, the Home Secretary, was forced to issue a rare public apology.

A catalogue of factors including the introduction of compulsory passports for children and an unexpected increase in people wanting to travel abroad worsened the situation. But the main culprit turned out to be a computer system made by the German firm Siemens, which was introduced by the passport agency without sufficient testing or staff training.

An audit report later revealed that the crisis had led to hundreds of missed holidays and a £13 million ($19.7 million) bill for taxpayers, including £16,000 spent on umbrellas for applicants queuing outside in the (very British) summer rain.

Travel plans in China went awry last week too, when the country’s biggest internet travel agent Ctrip suddenly went offline for about 12 hours.

That failure came just a day after Alipay, the dominant online payment platform, also suffered a breakdown, albeit this time for only two hours.

Because they happened at two of the most widely known Chinese tech firms the disruptions have been widely discussed, with allegations that such large players should have sturdier backup plans.

The system failures have also led to broader speculation about how well China’s internet infrastructure is coping with the nation’s fast-growing internet economy, particularly as companies like Tencent and Alibaba try to become bigger parts of the financial system.

So what happened to Ctrip?

Problems began to emerge for the Nasdaq-listed firm just before midday on May 28. Error messages such as “Service unavailable” or “Servers not found” popped up when clients browsed Ctrip’s website and mobile apps. Ctrip stopped processing transaction requests, its search engine service seized up, and most of links on the site were inaccessible.

The breakdown came at a sensitive time, just two days after Priceline had invested a further $250 million in Ctrip (via convertible bonds could take the American firm’s holding up to 15% of Ctrip’s outstanding shares).

A week earlier, Ctrip had taken control of its domestic rival eLong, acquiring a 37.6% stake, from US-based Expedia, for $400 million.

And not long after it bought eLong, its new acquisition also experienced an embarassing capacity crash. But this too was related to Ctrip’s own problem. That’s because when it discovered its own service was out of action, management began directing Ctrip’s traffic to eLong. That led eLong’s site to go down as its servers reportedly failed to absorb the enormous surge in data flow.

Most of eLong was back to normal by the evening of May 28. Ctrip, meanwhile, resumed processing travel bookings in the early hours of the following day.

When problems were first being reported, Ctrip’s spokesperson told reporters that its website and mobile app were unresponsive because of an attack by “unidentified sources”.

A day later it backtracked from the original statement, explaining on its official weibo that stoppages were because of “an operation error made by a staff member”.

Ctrip’s contradictory explanations did little to quench the rumours on social media. “For a while it was the only topic on WeChat. Everyone was asking what really happened,” CBN, a newspaper, reported.

One of the most widely-believed ‘unofficial’ versions for the outage? That a company insider had taken revenge on a co-worker for stealing his girlfriend by deleting crucial data from Ctrip’s system.

Ctrip hasn’t disclosed the number of clients affected. “Many customers had to reschedule or cancel their air tickets, change their hotel bookings and even their holiday plans. Some have vowed to seek compensation from Ctrip,” Beijing News notes.

There were scares for investors too. In the trading session following the glitch, Ctrip’s share price briefly slipped 11%, before recovering to close 1.6% lower.

Are there any bigger worries?

Ctrip has about 141 million registered members. In the first quarter its revenues climbed 46% year-on-year to Rmb2.4 billion ($394 million). Local media calculates that more than $1 million worth of booking revenue goes through Ctrip’s website every hour. “For a 12-hour outage Ctrip could have lost more than $12 million”, warned Sina Technology.

More than a third of Ctrip’s revenue comes from hotel reservations. After the system failure, netizens were soon speculating that the company’s database could have been comprised, including records of where, when and with whom people were staying overnight (Chinese laws require all hotel guests to check in under their real identities).

Ctrip doesn’t boast a proud record of protecting personal information from its users. In late 2013 netizens were shocked (and even more delighted) when the transaction records for a number of popular love hotels were exposed online, to the detriment of adulterers. (Local media described the breach as “the most painful in Chinese history”, given so many relationships broke up as a result.) In March last year, an internet security firm also exposed a security loophole in Ctrip’s system that made credit card details vulnerable to hackers.

No wonder, then, that news of Ctrip’s latest glitch had its customers wondering if personal data would be compromised again.“How many families will be broken up this time?” Sina Technology asked.

But Ctrip insisted that the latest crash had not resulted in the loss of any of its customers’ personal data or booking information.

So discussion then began to focus on the company’s business contingency plan, or seemingly the lack of one. “Given Ctrip was being paralysed for an ultra-long period of time, this should go down as a landmark incident regarding internet security,” suggested tech portal CCIDNet.

Ctrip later explained that resuming full service had taken some time because of a design issue linked to the thousands of sub-sites operating on its core platform Ctrip.com. Technicians said they had to make sure that all of the sub-sites were working properly whilst fixing the core problem with the server simultaneously – which was something of a juggling act.

The critics were unconvinced by this explanation, asking why a company of Ctrip’s size had not invested more in disaster recovery, and why backup servers did not kick in when the main server failed.

Why did Alipay go offline?

Regular WiC readers ought to be pretty familiar with Alipay, the country’s largest online third-party payment system (and now big enough to seriously worry state-controlled UnionPay, the only domestic credit card network).

Last year payments made on Alipay aggregated to $629 billion, or a daily average of $1.6 billion, giving it 70% of China’s online payment market.

Alipay is also the means for cash to flow in and out of the country’s biggest money market fund (see issues 202 and 205) and it provides the means for hundreds of millions of Chinese to pay their utility bills. It is also one of the handful of internet firms that policy planners are trusting with a role in restructuring the country’s economy (according to the so-called “Internet Plus” strategy unveiled by Chinese Premier Li Keqiang in March).

So what did it take to bring down this financial powerhouse? The answer is surprisingly simple: a construction worker.

Online shoppers found that Alipay had suddenly stopped working on the evening of May 27. The subsequent payment paralysis lasted for nearly 150 minutes, with phone calls to the company’s service hotlines going unanswered.

It turned out that the outage was triggered by a team of workers from China Telecom. The workmen were installing electric cables in Hangzhou, where Alibaba is based, and one of them cut through an optic fibre directly linked to Alibaba’s internet cloud service.

The state-owned telecoms firm has apologised to Alibaba, while executives at both firms have been discussing how to avoid similar accidents in the future.

Alipay resumed service more quickly than Ctrip. But its broader impact across the internet economy means that its outage stoked even greater concerns.

“Is it our Project Mayhem moment?” an internet user wrote on weibo, referring to the plot of Fight Club, a Hollywood movie in which the protagonists planned to destroy US banking databases.

The state media also lamented that Alipay’s internet infrastructure had been exposed as too fragile. “You can talk about Internet Plus or Industry 4.0 all day. But it can’t even stand a shovel from the Industry 0.5 era,” Xinhua opined, with unexpected wit.

Caixin Weekly noted that international internet giants suffer system failures from time to time themselves. But experts still wanted to know why Alipay needed so much time to switch operations across to its backup servers. In a statement, Alipay explained that it delayed the transition on concerns over the security of user data and money exchanges, adding that it is building a new framework of backup data centres in different cities (standard practice for companies operating huge databases) to minimise the impact of similar incidents in the future.

For the banking regulators, the Alipay incident is already a matter of major concern, however. A source with the China Banking Regulatory Commission said any glitch lasting two and a half hours has to be classified as a “major security breach” no matter the reasons behind it.

“Alipay has stumbled over a single cable. It is a stark warning of the fragile status of our internet security,” Xinhua added.

And the bigger picture?

The glitches were unfortunately timed, coming just days before China’s internet regulator launched its very own Cyber Security Week.

The campaign was organised by the Cyberspace Administration of China (CAC) and backed by the education and public security ministries. The Global Times said the event aims to “cultivate cyber security awareness” and aligns with a recently published White Paper on military strategy which stipulates that “cyberspace has become a new pillar of economic and social development, and a new domain of national security”.

But this ‘cyber’ strategy is only going to be feasible, the media has been arguing, if firms start to take internet security more seriously.

Individual users of the internet have a role to play too. A CAC survey of 254,000 people released on Monday found that 81% had never changed their online passwords. Another 44% use vulnerable passwords such as their phone numbers, while 16% of the group had opted for no-brainers such as “123456”. But most of the focus is on the internet bosses investing more in additional data centres and more secure transmission channels.

“The meltdown in the services of two industry leaders (Alipay and Ctrip) has given us a stark warning,” the state broadcaster CCTV noted.

“It is time for internet firms to slow down and have a look to see if their networks are secure and stable.”

Beijing News agreed, highlighting that international payments network Visa has four core data centres around the world, and that its infrastructure has a “zero second” protocol for switching operations between them in event of a disaster recovery situation.

Also starting this Monday, the likes of Visa, MasterCard and even Apple Pay have been greenlighted to apply for new licences to clear domestic Chinese payments, in potential competition with Alipay and UnionPay. In the past these foreign firms have complained that a major barrier to their entry into the China market is the demand from banking regulators that they set up a ‘proven’ disaster recovery infrastructure in China, reports the Beijing News. Given recent events, that looks ironic.

---

© ChinTell Ltd. All rights reserved.

Sponsored by HSBC.

The Week in China website and the weekly magazine publications are owned and maintained by ChinTell Limited, Hong Kong. Neither HSBC nor any member of the HSBC group of companies ("HSBC") endorses the contents and/or is involved in selecting, creating or editing the contents of the Week in China website or the Week in China magazine. The views expressed in these publications are solely the views of ChinTell Limited and do not necessarily reflect the views or investment ideas of HSBC. No responsibility will therefore be assumed by HSBC for the contents of these publications or for the errors or omissions therein.