How the West Got China’s Social Credit System Wrong

In October 2018, Vice President Mike Pence paid a visit to the Hudson Institute—a conservative Washington, DC, think tank—to give a wide-ranging speech about the United States’ relationship with China. Standing stiffly in a shiny blue tie, he began by accusing the Chinese Communist Party of interfering in US politics and directing Chinese businesses to steal American intellectual property by “any means necessary.” Pence then turned his attention to the country’s human rights abuses, starting not with the persecution of religious minorities, but with a peculiar governmental initiative: the social credit project. “By 2020, China’s rulers aim to implement an Orwellian system premised on controlling virtually every facet of human life—the so-called ‘social credit score,’” Pence said. “In the words of that program’s official blueprint, it will ‘allow the trustworthy to roam everywhere under heaven while making it hard for the discredited to take a single step.’”

The vice president’s remarks echoed a steady stream of Western media reports, published in dozens of outlets over the past few years, that paint China’s Social Credit System as a dystopian nightmare straight out of Black Mirror. The articles and broadcast segments often said China’s central government is using a futuristic algorithm to compile people’s social media connections, buying histories, location data, and more into a single score dictating their rights and freedoms. The government can supposedly analyze footage from hundreds of millions of facial-recognition-equipped surveillance cameras in real time, and then dock you points for misbehavior like jaywalking or playing too many video games.

But there is no single, all-powerful score assigned to every individual in China, at least not yet. The “official blueprint” Pence referenced is a planning document released by China’s chief administrative body five years ago. It calls for the establish­ment of a nationwide scheme for tracking the behavior of everyday citizens, corporations, and government officials. The Chinese government and state media say the project is designed to boost public confidence and fight problems like corruption and business fraud. Western critics often see social credit instead as an intrusive surveillance apparatus for punishing dissidents and infringing on people’s privacy.

With just over a year to go until the government’s self-imposed deadline for establishing social credit, Chinese legal researchers say the system is far from the cutting-edge, Big Brother apparatus portrayed in the West’s popular imagination. “I really think you would find a much larger percentage of Americans are aware of Chinese social credit than you would find Chinese people are aware of Chinese social credit,” says Jeremy Daum, a senior research fellow at Yale Law School’s Paul Tsai China Center in Beijing. The system as it exists today is more a patchwork of regional pilots and experimental projects, with few indications about what could be implemented at a national scale.

That’s not to say that fears about social credit are entirely unfounded. The Chinese government is already using new technologies to control its citizens in frightening ways. The internet is highly censored, and each person’s cell phone number and online activity is assigned a unique ID number tied to their real name. Facial-recognition technology is also increasingly widespread in China, with few restraints on how it can be used to track and surveil citizens. The most troubling abuses are being carried out in the western province of Xinjiang, where human rights groups and journalists say the Chinese government is detaining and surveilling millions of people from the minority Muslim Uyghur population on a nearly unprecedented scale.

But Western concerns about what could happen with China’s Social Credit System have in some ways outstripped discussions about what’s already really occurring. Critiques are often based on worst-case scenarios far off in the future, and run the risk of minimizing the troubling aspects of the project as it is in place today. The exaggerated portrayals may also help to downplay surveillance efforts in other parts of the world. “Because China is often held up as the extreme of one end of a spectrum, I think that it moves the goalposts for the whole conversation,” says Daum. “So that anything less invasive than our imagined version of social credit seems sort of acceptable, because at least we’re not as bad as China.”

How Misconceptions Formed

One of the earliest Western accounts was published by an unlikely source: the American Civil Liberties Union, which doesn’t operate in China. As part of his job as a policy analyst at the ACLU’s Speech, Privacy, and Technology Project, Jay Stanley blogs about emerging threats to civil liberties. On October 5, 2015, he published a post titled “China’s Nightmarish Citizen Scores Are a Warning for Americans.” The article is representative of much of the initial wave of coverage, which was often derived from secondhand information that traveled like a game of telephone, rather than on-the-ground reporting. Stanley’s post was sourced largely from a similar story from Privacy News Online, which itself was based on an article from a Swedish website.

Today Stanley says he intended to highlight China’s program as a cautionary tale for the US. “This really seemed to be pointing the way towards a dark potential future,” he says. “There were frankly a lot of signs and similarities to things happening in the United States,” like digital profiling. The ACLU post wasn’t the only outlet to use China’s nascent system as a way to draw attention to privacy and surveillance issues in the West. “The more I look around, the more it seems like an American social credit system is springing up around us—and it doesn’t look all that different from China’s,” Casey Newton, a writer at The Verge, wrote in his popular newsletter just last month. “China’s Dystopian Tech Could Be Contagious,” The Atlantic similarly declared last year.

While a number of journalists and academics have tried to correct the record, the science fiction myths about China’s social credit score continue to endure in the West. “There’s so much that’s been written about this now that’s wrong, that it’s really taken on a life of its own,” says Shazeda Ahmed, a PhD student at the University of California at Berkeley studying the Social Credit System in China. “I still see articles quoting things from 2014, 2015, that I thought were largely debunked.”

The confusion is understandable, though. First, there’s the language barrier. Daum says the phrase “social credit,” has different connotations in English that it does in Chinese. To an English speaker, the two words together might signal a reference to interpersonal relationships. In Chinese, the term is more closely associated with a phrase like “public trust.” There’s also the added linguistic hurdle of deciphering dense legal documents. “I think language is a real barrier,” says Daum. “Both legal jargon and political jargon and Chinese versus English.”

The original plan for social credit released in 2014 is also vague and sweeping, and it wasn’t entirely clear what the project might ultimately consist of. “They expected the different parts of the government, both centrally and locally, to try out their own approach in terms of implementation,” says Xin Dai, a professor and associate dean at China’s Ocean University Law School, who has researched social credit. “You have this really massive but also chaotic scene of different people trying to put together different types of programs.”

For example, the government partnered with corporations on some early initiatives, including “credit scores” calculated by private tech companies, like Ant Financial’s Sesame Credit program. In 2015 the Chinese government authorized eight tech companies, including Ant Financial, an affiliate of the corporate giant Alibaba, to experiment with developing credit reporting systems for individuals. In addition to financial data, Sesame Credit does take into consideration things like social media connections and purchasing habits—a product feature that garnered a lot of attention in the West, including in a WIRED cover story.

By 2017, the Chinese government had decided that none of the pilots would receive authorization to be official credit reporting measures, due to concerns about potential conflicts of interest. But initially, it was unclear how closely tied the programs would be to the government’s efforts, even in China. “I think there were, and maybe still are, Chinese citizens who didn’t quite understand that there was a difference as well,” says Ahmed. “Because in the beginning, Sesame Credit was marketing itself as contributing to the whole Social Credit System.”

Today, Sesame Credit, as well as other similar initiatives, essentially function like loyalty rewards programs. Participants with high scores earn privileges like renting a bike without leaving a deposit or deferring payment for medical expenses, but the scores are not part of the legal system, and no one is required to participate.

The state-run projects that have captured the most attention in the West are the local pilots. Dozens of Chinese cities are experimenting with their own versions of social credit, and some have designed programs that do give individuals a personal numerical ranking. These initiatives largely don’t rely on mass surveillance or supercharged artificial intelligence, and many citizens may not even know they exist. It’s difficult to generalize about all of them, since they can vary widely. Some are incorporating blockchain technology, for example. For now it’s not clear when, if ever, any of them will be adopted at the national level.

The city of Rongcheng, about 500 miles from Beijing, is one place assigning residents individual social credit scores. According to policy documents outlining the project that Daum translated, it’s relatively limited in scope. In order to lose points in the system, you would need to violate an existing law, regulation, or contract you entered. Maintaining “Exceptional Creditworthiness” is thus a matter of following the rules already in place. The benefits of maintaining a high score are also fairly modest, like free health checkups and the ability to apply for an interest-free loan. “Looking at how fragmented this implementation is, you see that different governments don’t have quite the same resources,” says Ahmed. “Some of the smaller cities, they can only subsidize fairly unexciting benefits.”

Blacklists and Red Lists

The primary mechanism of the Social Credit System are the nationwide blacklists and red lists. Each regulatory agency was asked to come up with a rap sheet of its worst offenders, businesses and individuals who violated preexisting industry regulations. The red lists are the exact opposite—they’re rosters of companies and people that have been particularly compliant. Those archives were then made public on a centralized website, called China Credit, where anyone can search them. Think of the Better Business Bureau, or letter grades given to restaurants.

Many regulatory agencies have signed memorandums of understanding with each other, in which they promise to punish people and businesses on one another’s blacklists. Hypothetically, if this system were in the US, a business might now face additional penalties from the Environmental Protection Agency for breaking a rule at the Food and Drug Administration. There’s no evidence that citizens’ social media or purchasing data is being incorporated, at least not yet. “They’re making it so that these records are communicated to other agencies,” says Daum. “Somehow, that got interpreted as everything you do is being watched all the time in a panopticon, and that I have not seen.”

Chinese legal researchers are worried about one of these databases in particular: The Supreme People’s Court maintains a blacklist of people who the government alleges did not comply with court judgments, for example by not paying fines, but also things like failing to formally apologize to someone they are found to have wronged. Being on the blacklist now comes with harsh punishments. You might be unable to purchase high-speed train tickets, fly on an airplane, or send your kids to a private school. Over 13 million people were on the list as of March, according to state reports, and the government has prohibited more than 20 million plane tickets from being purchased.

Yu-Jie Chen, a Taiwanese human rights lawyer and post-doctoral researcher at the Institutum Iurisprudentiae of Academia Sinica, says the judgement defaulter’s blacklist is imposing “layers of disproportionate, arbitrary, and wide-ranging punishments,” on people who have largely already suffered the consequences of breaking the law. She says she’s also worried about how the list penalizes people who didn’t commit any offense, like a child who is barred from attending certain schools because of their parent’s actions. It’s not clear whether citizens can effectively get off the list if they’re included on it by accident, or even if they fulfill their court-ordered obligations.

To enforce these punishments, Ahmed has written that the government is sharing blacklists with technology platforms. That way, people on them can’t do things like book flights or buy train tickets online. Local governments are also asking social media companies to help orchestrate public shaming initiatives. In the southern city of Nanning, the social media app TikTok partnered with the local court to broadcast photos of blacklisted people between videos, like a digital mugshot. In the northern city of Shijiazhuang, blacklisted people and entities are displayed on a map within the messaging app WeChat. The features aren’t yet widespread, but still raise privacy concerns, especially if people are wrongfully added to a blacklist and that information is then broadcast to everyone they know.

Social Credit’s Future

As the 2020 deadline approaches, China’s Social Credit System still remains largely in development. There are some signs, however, that the system could soon incorporate more forms of data collection. For example, Chen says, the China Credit website already encourages users to log in by scanning their faces, though it’s not mandatory. “So there will be a facial-recognition element if the government can persuade people to use that more,” she explains.

In the meantime, Dai says, Chinese academics have begun discussing the potential privacy and other risks posed by the project. They were influenced, in part, by the enormous amount of attention the Social Credit System has garnered in the West, despite the fact that it often hasn’t been portrayed accurately. “This entire thing is just so massive, and it varies across place to place” says Dai. “It is easy to sort of misinterpret or only catch part of it, without seeing the entire picture.”

This story was produced in collaboration with PRI’s The World, the award-winning public radio show and podcast on global issues, news, and insights from BBC, WGBH, PRI, and PRX. It was coreported by The World’s Lydia Emmanouilidou. You can listen to The World’s audio program about China’s social credit score here.

---

More Great WIRED Stories

• High drama: A cannabis biotech firm roils small growers
• Lunar mysteries that science still needs to solve
• Are super-automatic espresso machines worth it?
• The best algorithms don’t recognize black faces equally
• These hackers made an app that kills to prove a point
• ??‍♀️ Want the best tools to get healthy? Check out our Gear team’s picks for the best fitness trackers, running gear (including shoes and socks), and best headphones.
• ? Get even more of our inside scoops with our weekly Backchannel newsletter