Death by a thousand cuts: software development behind China’s Great Firewall

By Eva Xiao

If you’ve ever been to mainland China, chances are you’re familiar with the Great Firewall, the country’s all-encompassing internet censorship apparatus. 

You know the despair of not being able to open Facebook, the pain of going mute on Twitter. 

But with a good VPN, you can magic many of these inconveniences away – at least temporarily.
For software developers based in China, however, it’s not that simple. 

You’re not just censored from certain websites. 

Basic building blocks that you use for product development are suddenly beyond your reach. 

With software services and libraries spread across the globe, China’s internet sovereignty can be a real pain in the ass.
“There’s a whole parallel universe over here,” DC Collier, founder of Shanghai-based chatbot startup Rikai Labs, told me back in June. 

“I think Chinese developers are doing development work with one hand tied behind their back.”
DC has been a software developer for 20 years, building everything from CD-ROM games to Facebook apps and now, chatbots. 

When I ask how he deals with the Great Firewall, he says he travels overseas when he has a couple weeks of “serious coding work” ahead of him.
“Otherwise, just bang my head against the wall,” he tells Tech in Asia in an email.
To outsiders who have never developed software in China, it might sound like he’s joking. 

But he’s not.

Blocked – or slow as hell
The scope of the Great Firewall has grown along with the internet. 

Sadly, it’s a highly adaptable and robust system. 

In terms of software development, it means that a growing number of services are permanently or sporadically blocked.
One of those is NPM or Node Package Manager, an important tool for Node.js. 

To access NPM in mainland China, you’ll have to find workarounds like using a Chinese mirror. Another is Github, a popular version control and source code management tool that is occasionally blocked or at least throttled to be excruciatingly slow.
Things get worse when you try using anything related to Google: Google Analytics, Google Fonts, Google Maps. 

All blocked. 

Even open source tools from Google are inaccessible without VPN or some kind of detour, such as Tensorflow.
“We had to rewrite so much of our stack,” says DC. 

“In the US, you rely on so many external libraries but then every time you do a deploy here, it takes like 30 minutes because it’s pulling down libraries from someone else’s service. You have to set up all kinds of infrastructure to get around all of that stuff.”
“If you’re using third party APIs, like Facebook’s natural language processing APIs, you have to set up proxies to bounce that stuff around,” he adds. 

“It’s really painful.”
If you’re an Android developer and plan to develop for Google Play, be prepared to cry rivers of tears when you try to use Google’s Android services. 

Expect to wait hours to upload your app to the Play Store because your VPN is broken.
“Android’s system has some services, like push notifications and the Play Store and Google Maps. You can’t use any of them in China,” says Xin Qi, technical director at Ming Labs. 

When Xin Qi and his team make apps for the users both within and outside of China, they have to check the user’s location in order to use the right service.
“Our apps will have some deviation,” he continues. 

“When we get a GPS location, it’s 100 or so meters off. In China, there’s a GPS offset.” 

He’s referring to GCJ-02, a China-specific GPS coordinate system that will erroneously shift WGS-84 markers, such as GPS locations, on maps.
“When we retrieve coordinates, we have to fix them,” he sighs. 

“It’s not hard but it’s annoying.”

Hosting dilemmas
Unfortunately, the Great Firewall is only one piece of the Chinese internet. 

Even if you get around the blocked services, you might encounter problems endemic to the whole ecosystem.
A good example is web hosting. 

Alex Michaud, founder of Reddoor.House, a service that helps expats find housing, has struggled to find the right server to host his website. 

Cloud services in China are more expensive than their equivalents in the West, and exorbitantly so, he says.
“It’s not like two, three times more,” he says. 

“It’s like 100 times more expensive.”
For instance, though Alibaba Cloud’s basic Elastic Cloud Service package is roughly US$10 a month, it offers a measly bandwidth of 1 Mbps. 

Compare that to Linode, a cloud service based in the US, whose US$10 basic package comes with a bandwidth of 125 Mbps. 

To get something equivalent from Alibaba Cloud, you would have to pay around US$1,500 a month.
Things get more complicated when you want to service users both inside and outside of China. 

If you’re targeting the local market, it makes sense to use a data center based in China. 

But if you also want to serve the international market, you might need to pay more for a specialized service.
“Even though we also have users in Canada [and] Australia, 85 percent of our business is done in China,” says Gabriel Paquin, vice president of engineering at Education International Cooperation (EIC) Group, a counseling service provider for Chinese students who want to study abroad.
EIC uses Salesforce as its CRM system. 

It’s hosted in Japan though, because Salesforce doesn’t have a data center in China.
To circumvent that, EIC subscribes to Aryaka, a software-defined WAN service, which optimizes enterprise networks such as content delivery networks (CDNs).
“I do enterprise. We’re a big business,” says Gabriel. 

“I cannot hack a couple of Python libraries and it’s gonna be cool. Our business is a billion RMB business, it’s not small. […] I gotta go with enterprise services.”
Still, even with Aryaka, there’s no guarantee that the company’s CDN will work seamlessly. 

Gabriel and his development team have to ensure that sales apps running on company devices, such as tablets, can operate offline if they have to. 

The sales team cannot afford to wait for web pages to load when they are trying to close sales, he says.
“It has to be fast,” says Gabriel. 

“For example, let’s look at information from Harvard University. It would take forever to load. So for some of the key pages, we had to have a local cache on the device.”
Because so much data has to be saved locally, EIC has to buy devices with larger hard drives, he says. Thankfully, it’s an internal application.

No hope?
When I ask Zhang Wei, a senior developer at Ming Labs, what advice he would give overseas software developers that want to enter China, he says without hesitation, 

“Don’t come.”
“You’ll lose access to a lot of good services,” he explains. 

When you’re troubleshooting software, you’ll always have some doubts, he says. 

“You won’t know if it’s the VPN or the network connection. Sometimes it won’t work without VPN, and sometimes it won’t work with VPN.”
It’s hard to quantify how much productivity is lost as a result of troubleshooting and working around the Great Firewall. 

DC estimates that 50 percent of his productivity is lost, while Alex only describes it as hours and hours of lost time. 

“All the time you spend on those little problems […] it’s time that you don’t spend developing your product,” he says.
Ironically, the Chinese government wants the country to become a world leader in technology and innovation. 

At the same time, restrictions around freedom of speech are tightening and even VPNs are not guaranteed workarounds for the country’s rising Great Firewall.
The only advice I heard from developers was to keep things as local as possible – similar to Gabriel and his team – and to avoid anything with external resources. 

And when possible, use Chinese services.
Alex has lived in Shanghai for two years now. 

When he just moved to China in 2014, he felt optimistic about the country. 

It was developing fast and it felt more exciting than Canada’s tech scene, which he describes as conservative. 

At that time, he was sure that China’s internet restrictions would loosen over time. 

In hindsight, he was naive, he says.
“In China, you can really feel the frontier,” he says. 

“It’s the biggest intranet in the world.”