In this article by Rajiv Tiwari, author of the book, Hadoop for Finance Essentials, explains big data is not just changing the data landscape of healthcare, human science, telecom, and online retail industries, but is also transforming the way financial organizations treat their massive data.
(For more resources related to this topic, see here.)
As shown in the following figure, a study by McKinsey on big data use cases claims that the financial services industry is poised to gain the most from this remarkable technology:
The data in financial organizations is relatively easier to capture compared to other industries, as it is easily accessible from internal systems—transactions and customer details—as well as external systems—FX Rates, legal entity data, and so on.
Quite simply, the gain per byte of data is the maximum for financial services.
Where do we get the big data in finance?
The data is collected at every stage—be it onboarding of new customers, call center records, or financial transactions. The financial industry is rapidly moving online, and so it has never been easier to capture the data. There are other reasons as well, such as:
- Customers no longer need to visit branches to withdraw and deposit money or make investments. They can discuss their requirements with the Bank online or over the phone instead of physical meetings. According to SNL Financial, institutions have shut 2,599 branches in 2014 against 1,137 openings, a net loss of 1,462 branches that is just off 2013’s record full-year total of 1,487 branches opened. The move brings total US branches down to 94,752, a decline of 1.5 percent. The trend is global and not just in the US.
- Electronic channels such as debit/credit cards and mobile devices, through which customers can interact with financial organizations, have increased in the UK, as shown in the following figure. The trend is global and not just in the UK. Mobile equipment such as computers, smartphones, telephones, or tablets make it easier and inexpensive for customers to transact, which means customers will transact more and generate more data.
- Since customer profiles and transaction patterns are rapidly changing, risk models based on smaller data sets are not very accurate. We need to analyze data for longer durations and be able to write complex data algorithms without worrying about computing and data storage capabilities.
- When financial organizations combine structured data with unstructured data on social media, the data analysis becomes very powerful. For example, they can get feedback on their new products or TV advertisements by analyzing Twitter, Facebook, and other social media comments.
Big data use cases in the financial sector
The financial sector is also sometimes called the BFSI sector; that is, banking, financial services, and insurance.
- Banking includes retail, corporate, business, investment (including capital markets), cards, and other core banking services
- Financial services include brokering, payment channels, mutual funds, asset management, and other services
- Insurance covers life and general insurance
Financial organizations have been actively using big data platforms for the last few years and their key objectives are:
- Complying with regulatory requirements
- Better risk analytics
- Understanding customer behavior and improving services
- Understanding transaction patterns and monetizing using cross-selling of products
Now I will define a few use cases within the financial services industry with real tangible business benefits.
Data archival on HDFS
Archiving data on HDFS is one of the basic use cases for Hadoop in financial organizations and is a quick win. It is likely to provide a very high return on investment. The data is archived on Hadoop and is still available to query (although not in real time), which is far more efficient than archiving on tape and far less expensive than keeping it on databases. Some of the use cases are:
- Migrate expensive and inefficient legacy mainframe data and load jobs to the Hadoop platform
- Migrate expensive older transaction data from high-end expensive databases to Hadoop HDFS
- Migrate unstructured legal, compliance, and onboarding documents to Hadoop HDFS
Regulatory
Financial organizations must comply with regulatory requirements. In order to meet these requirements, the use of traditional data processing platforms is becoming increasingly expensive and unsustainable.
A couple of such use cases are:
- Checking customer names against a sanctions blacklist is very complicated due to the same or similar names. It is even more complicated when financial organizations have different names or aliases across different systems. With Hadoop, we can apply complex fuzzy matching on name and contact information across massive data sets at a much lower cost.
- The BCBS239 regulation states that financial organizations must be able to aggregate risk exposures across the whole group quickly and accurately. With Hadoop, financial organizations can consolidate and aggregate data on a single platform in the most efficient and cost-effective way.
Fraud detection
Fraud is estimated to cost the financial industry billions of US dollars per year. Financial organizations have invested in Hadoop platforms to identify fraudulent transactions by picking up unusual behavior patterns.
Complex algorithms that need to be run on large volumes of transaction data to identify outliers are now possible on the Hadoop platform at a much lower expense.
Tick data
Stock market tick data is real-time data and generated on a massive scale. Live data streams can be processed using real-time streaming technology on the Hadoop infrastructure for quick trading decisions, and older tick data can be used for trending and forecasting using batch Hadoop tools.
Risk management
Financial organizations must be able to measure risk exposures for each customer and effectively aggregate it across entire business divisions. They should be able to score the credit risk for each customer using internal rules. They need to build risk models with intensive calculation on the underlying massive data.
All these risk management requirements have two things in common—massive data and intensive calculation. Hadoop can handle both, given its inexpensive commodity hardware and parallel execution of jobs.
Customer behavior prediction
Once the customer data has been consolidated from a variety of sources on a Hadoop platform, it is possible to analyze data and:
- Predict mortgage defaults
- Predict spending for retail customers
- Analyze patterns that lead to customers leaving and customer dissatisfaction
Sentiment analysis – unstructured
Sentiment analysis is one of the best use cases to test the power of unstructured data analysis using Hadoop. Here are a few use cases:
- Analyze all e-mail text and call recordings from customers, which indicates whether they feel positive or negative about the products offered to them
- Analyze Facebook and Twitter comments to make buy or sell recommendations—analyze the market sentiments on which sectors or organizations will be a better buy for stock investments
- Analyze Facebook and Twitter comments to assess the feedback on new products
Other use cases
Big data has the potential to create new non-traditional income streams for financial organizations. As financial organizations store all the payment details of their retailers, they know exactly where, when, and how their customers are spending money. By analyzing this information, financial organizations can develop deep insight into customer intelligence and spending patterns, which they will be able to monetize. A few such possibilities include:
- Partner with a retailer to understand where the retailer’s customers live, where and when they buy, what they buy, and how much they spend. This information will be used to recommend a sales strategy.
- Partner with a retailer to recommend discount offers to loyalty cardholders who use their loyalty cards in the vicinity of the retailer’s stores.
Summary
In this article, we learned the use cases of Hadoop across different industry sectors and then detailed a few use cases within the financial sector.
Resources for Article:
Further resources on this subject:
- Hive in Hadoop [article]
- Hadoop and MapReduce [article]
- Hadoop Monitoring and its aspects [article]