Technology Quarterly | Mining social networks

Untangling the social web

Software: From retailing to counterterrorism, the ability to analyse social connections is proving increasingly useful

|

TELECOMS operators naturally prize mobile-phone subscribers who spend a lot, but some thriftier customers, it turns out, are actually more valuable. Known as “influencers”, these subscribers frequently persuade their friends, family and colleagues to follow them when they switch to a rival operator. The trick, then, is to identify such trendsetting subscribers and keep them on board with special discounts and promotions. People at the top of the office or social pecking order often receive quick callbacks, do not worry about calling other people late at night and tend to get more calls at times when social events are most often organised, such as Friday afternoons. Influential customers also reveal their clout by making long calls, while the calls they receive are generally short.

Companies can spot these influencers, and work out all sorts of other things about their customers, by crunching vast quantities of calling data with sophisticated “network analysis” software. Instead of looking at the call records of a single customer at a time, it looks at customers within the context of their social network. The ability to retain customers is particularly important in hyper-competitive markets, such as India. Bharti Airtel, India's biggest mobile operator, which handles over 3 billion calls a day, has greatly reduced customer defections by deploying the software, says Amrita Gangotra, the firm's director for information technology.

The market for such software is booming. By one estimate there are more than 100 programs for network analysis, also known as link analysis or predictive analysis. The raw data used may extend far beyond phone records to encompass information available from private and governmental entities, and internet sources such as Facebook. IBM, the supplier of the system used by Bharti Airtel, says its annual sales of such software, now growing at double-digit rates, will exceed $15 billion by 2015. In the past five years IBM has spent more than $11 billion buying makers of network-analysis software. Gartner, a market-research firm, ranks the technology at number two in its list of strategic business operations meriting significant investment this year.

Adoption is being driven by the availability of more sources of information, and by the fact that network-analysis software is becoming easier to use. A decade ago IBM employed experts with PhDs in mathematics to study social networks, according to Mark Ramsey, the firm's head of business analytics for eastern Europe, the Middle East and Africa. Today, college graduates can operate analysis software handling enormous quantities of data. Bharti Airtel employs only about 100 analysts to keep tabs on its 135m subscribers.

Take me to your leaders

Of course, companies have long mined their data to improve sales and productivity. But broadening data mining to include analysis of social networks makes new things possible. Modelling social relationships is akin to creating an “index of power”, says Stephen Borgatti, a network-analysis expert at the University of Kentucky in Lexington. In some companies, e-mails are analysed automatically to help bosses manage their workers. Employees who are often asked for advice may be good candidates for promotion, for example.

Ellen Joyner of SAS, an analytics firm based in Cary, North Carolina, notes that more and more financial firms are using the software to uncover fraud. The latest version of SAS's software identifies risky borrowers by examining their social networks and Internal Revenue Service records, she says. For example, an applicant may be a bad risk, or even a fraudster, if he plans to launch a type of business which has no links to his social network, education, previous business dealings or travel history, which can be pieced together with credit-card records. Ms Joyner says the software can also determine if an applicant has associated with known criminals—perhaps his fiancée has shared an address with a parolee. Some insurers reduce premiums for banks that protect themselves with such software.

Last year an American government body called the Recovery Accountability and Transparency Board (RATB) began using network-analysis software to look for fraud within the $780 billion financial-stimulus programme. In addition to the internet, RATB combs Treasury and law-enforcement databases to uncover “non-obvious relationships”, says Earl Devaney, its chairman. The software works very well, he says. It has triggered about 250 ongoing criminal investigations and 400 audits.

Joe Biden, America's vice-president, said in June that such software would be used to prevent fraud within the government's Medicaid and Medicare health-care schemes. The Army Criminal Investigation Command already sniffs out procurement fraud by scanning text in e-mails. The software, developed by SRA, an American firm, can correlate numbers and phrases written in nine languages with financial databases. If a person discusses a particular Department of Defence payment with an individual not officially linked to the deal, SRA's software may notice it.

The police department of Richmond, Virginia, has pioneered the use of network-analysis software to predict crimes. Police officers know that crime increases at certain times, such as on paydays and when there is a full moon. But the software lets them analyse the social networks around suspects, such as dealings with employers, collection agencies and the Department of Motor Vehicles. The goal, according to Stephen Hollifield, the department's technology chief, is to “pull together a complete picture” of suspects and their social circle.

Party plans turn out to be a particularly useful part of this picture. Richmond's police have started monitoring Facebook, MySpace and Twitter messages to determine where the rowdiest festivities will be. On big party nights, the department now saves about $15,000 on overtime pay, because officers are deployed to areas that the software deems ripe for criminal activity. Crime has “dramatically” declined as a result, says Mr Hollifield. Colin Shearer, vice-president of predictive analytics at SPSS, a division of IBM that makes the software in question, says it can largely replace police officers' reliance on “gut feel”.

Network analysis also has a useful role to play in counterterrorism. Terror groups are often decentralised, so mapping their social networks is akin to deciphering “a big spaghetti picture”, says Roy Lindelauf of the Royal Dutch Defence Academy, who develops software for intelligence agencies in the Netherlands. It turns out that the key terrorists in a group are often not the leaders, but rather seemingly low-level people, such as drivers and guides, who keep addresses and phone numbers memorised. Such people tend to stand out in network models because of their high level of connectedness. To find them, analysts map “structural signatures” such as short phone calls placed to the same number just before and after an attack, which may indicate that the beginning and end of an operation has been reported.

The capture of Saddam Hussein in 2003 was due in large part to the mapping of the social networks of his former chauffeurs, according to Bob Griffin, the chief executive of i2, a British firm which developed the software used in the manhunt. Senior members of the Iraqi regime were mostly clueless about the whereabouts of the former president, says Mr Griffin, but modelling the social networks of his chauffeurs who had links to rural property eventually led to the discovery of his hideout, on a farm near his hometown of Tikrit.

From social to societal networks

Where is network analysis headed? The next step beyond mapping influence between individuals is to map the influences between larger segments of society. A forecasting model developed by Venkatramana Subrahmanian of the University of Maryland does just that. Called SOMA Terror Organization Portal, it analyses a wide range of information about politics, business and society in Lebanon to predict, with surprising accuracy, rocket attacks by the country's Hizbullah militia on Israel.

Attacks tend to increase, for example, as more money from Islamic charities flows into Lebanon. Attacks decrease during election years, particularly as more Hizbullah members run for office and campaign energetically. By the middle of 2010 SOMA was sucking up data from more than 200 sources, many of them newspaper websites. The number of sources will have more than doubled by the end of the year.

Once these societal networks of influence can be accurately mapped, they can be used to promote the spread of particular ideas—those that support stability and democracy, for example. Last year America's army, which jointly funds SOMA with the air force, began disbursing about $80m in five-year research grants for network analysis to promote democracy and national security. An authoritarian government, for instance, may have difficulties slowing the spread of a new idea in a certain medium—say, internet chatter about a book that explains how corruption undermines job creation. Diplomatic services can use this information to help ideas spread. Brian Uzzi of Northwestern University in Evanston, Illinois, who advises intelligence agencies on democracy-promotion analytics, says diplomatic services are mapping the “tipping point” when ideas go mainstream in spite of government repression.

SPADAC, a firm based in McLean, Virginia, performs such analyses on Egypt and other countries in Africa, the Middle East and South-East Asia. Clients include the United States, Mexico and various diplomatic services. Riots, bloody elections and crackdowns, among other things, can be forecast with improving accuracy by crunching data on food production, unemployment, drug busts, home evictions and slum growth detected in satellite images. Mark Dumas, the head of SPADAC, notes that societies with longstanding and strong social and business ties abroad weather change well. In relatively closed countries, like Egypt, rapid shifts in social networks can trigger upheaval, he says. Last year SPADAC's revenue reached $19m; this year it will exceed $27m.

Country analyses have great potential in peacekeeping and counterinsurgency operations, according to Kathleen Carley of Carnegie Mellon University in Pittsburgh. She is developing a societal model of Sudan with a team of about 40 researchers. Foreign aid workers and diplomats frequently stumble in Sudan because they fail to work out which tribal and political leaders they should work with, and how.

Ms Carley's model, known as ORA, analyses a decade of data on such things as weather, land and water disputes, cabinet reshuffles, reactions to corruption, court cases, economic activity and changes in tribal geographic maps. Within the information that emerges are lists of the locals most likely to co-operate with Westerners, with details of the role each would best play. This depth of insight, a demonstration of the power of network analysis today, will only grow.

This article appeared in the Technology Quarterly section of the print edition under the headline "Untangling the social web"

The web's new walls

From the September 4th 2010 edition

Discover stories from this section and more in the list of contents

Explore the edition