Five Major Challenges that Big Data Presents

The “big data” phrase is thrown around in the analytics industry to mean many things. In essence, it refers not only to the massive, nearly inconceivable, amount of data that is available to us today but also to the fact that this data is rapidly changing. People create trillions of bytes of data per day. More than 95% of the world’s data has been created in the past five years alone, and this pace isn’t slowing. Web pages. Social media. Text messages. Instagram. Photos. There is an endless amount of information available at our fingertips,but how to harness it, make sense of it, and monetise it are huge challenges. So lets narrow the challenges down a little and put some perspective on them. After fairly extensive reading and research, I believe there are 5 major challenges offered up to big data at the moment.

1). Volume

How to deal with the massive volumes of rapidly changing data coming from multiple source systems in a heterogeneous environment?

Technology is ever-changing. However, the one thing IT teams can count on is that the amount of data coming their way to manage will only continue to increase. The numbers can be staggering: In a report published last December, market research company IDC estimated that the total count of data created or replicated worldwide in 2012 would add up to 2.8 zettabytes (ZB). For the uninitiated, a Zettabyte is 1,000 Exabytes, 1 million Petabytes or 1 billion Terabytes or, in more informal terms, lots and lots and lots of data. By 2020, IDC expects the annual data creation total to reach 40 ZB, which would amount to a 50-fold increase from where things stood at the start of 2010.

Corporate data expansion often starts with higher and higher volumes of Transaction Data. However, in many organisations, unstructured and semi-structured information, the hallmark of Big Data Environments, is taking things to a new level altogether. This type of data typically isn’t a good fit for relational databases and comes partly from external sources, big data growth also adds to the Data Integration Workload, and challenges for IT managers and their staff.

2). Scope

How do you determine the breath, depth and span of data to be included in cleansing, conversion and migration efforts?

Big Data is changing the way we perceive our world. The impact big data has created and will continue to create can ripple through all facets of our life. Global Data is on the rise, by 2020, we would have quadrupled the data we generate every day. This data would be generated through a wide array of sensors we are continuously incorporating in our lives. Data collection would be aided by what is today dubbed as the “Internet of Things”. Through the use of smart bulbs to smart cars, everyday devices are generating more data than ever before. These smart devices are incorporated not only with sensors to collect data all around them but they are also connected to the grid which contains other devices. A Smart Home today consists of an all encompassing architecture of devices that can interact with each other via the vast internet network. Bulbs that dim automatically aided by ambient light sensors and cars that can glide through heavy traffic using proximity sensors are examples of sensor technology advancements that we have seen over the years. Big Data is also changing things in the business world. Companies are using big data analysis to target marketing at very specific demographics. Focus Groups are becoming increasingly redundant as analytics firms such as McKinsey are using analysis on very large sample bases that have today been made possible due to advancements in Big Data. The potential value of global personal location data is estimated to be $700 billion to end users, and it can result in an up to 50% decrease in product development and assembly costs, according to a recent McKinsey report. Big Data does not arise out of a vacuum: it is recorded from some data generating source. For example, consider our ability to sense and observe the world around us, from the heart rate of an elderly citizen, and presence of toxins in the air we breathe, to the planned square kilometer array telescope, which will produce up to 1 million terabytes of raw data per day. Similarly, scientific experiments and simulations can easily produce petabytes of data today. Much of this data is of no interest, and it can be filtered and compressed by orders of magnitude. There is immense scope in Big Data and a huge scope for research and Development.

3). 360 Degree View

With all the information that is now available, how do you achieve 360 degree views of all your customers and harness the kind of detailed information that is available to you, such as WHO they are? WHAT they are interested in?  and HOW they are going to purchase and WHEN?

Every brand has its own version of the perfect customer. For most, these are the brand advocates that purchase regularly and frequently, take advantage of promotions and special offers, and engage across channels. In short, they’re loyal customers.

For many brands and retailers, these loyal customers make up a smaller percentage of their overall customer base than they would prefer. Most marketers know that one loyal customer can be worth five times a newly acquired customer, but it’s often easier to attract that first-time buyer with generic messaging and offers. In order to bring someone back time and time again, marketers must craft meaningful and relevant experiences for the individual. So how can brands go about building loyalty for their businesses? Let’s start with what we know.

We all know that customers aren’t one dimensional. They have thousands of interests, rely on hundreds of sources to make decisions, turn to multiple devices throughout the day and are much more complex than any audience model gives them credit for. It’s imperative for marketers to make knowing their customers a top priority, but it isn’t easy.

In the past, knowing a customer meant one of two things: you knew nothing about them or you knew whatever they chose to tell you. Think about it. Often in the brick and mortar world you would deal with a first-time customer about whom you knew next to nothing: age, gender and the rest was based on assumptions. Over time, a regular customer might become better understood. You’d see them come in with children, or they’d make the same purchase regularly.

Now, thanks to technology, you can know an incredible amount about your customers some might even say too much. Amassing data is one thing, but increasingly the challenge has become how to make sense of the data you already have to create a rich, accurate and actionable view of your customer a 360-degree view.

Building and leveraging a 360-degree view of your customer is critical to helping you drive brand loyalty. Your customers need to be at the center of everything your business does. Their actions, intentions and preferences need to dictate your strategies, tactics and approach. They aren’t a faceless mass to whom something is done; they are a group of individuals that deserve personalised attention, offers and engagement. Your goal as a marketer is to make the marketing experience positive for your customers, which, in turn, will be positive for your business.

How can marketers go about establishing that 360-degree view and creating that positive customer relationship? It must be built on insights, but that doesn’t mean simply more data. In fact, more data can make it more difficult to come to a solid understanding of your customer. On top of that, it can also clearly raise privacy concerns. Marketers need to know how to make good inferences based on smart data.

Lets look at some of the key types of data and how they can be used:

First and most valuable is an organisation’s own (“first-party”) data. This should be obvious, but the diversity of this data – past purchase history, loyalty program participation, etc. – can cause some potentially valuable information to be overlooked.

Next is the third-party data so readily available for purchase today. This can be useful for target new audiences or finding look-alikes of existing customers, but it often comes at a prohibitive price and with fewer guarantees of quality and freshness than first-party data.

Finally, there is real-time data about customers or prospects. While real-time data can, in theory, come from either a first- or third-party source, it functions differently than the historical data sources described above. Certainly it can be used to help shape a customer profile but in its raw form, in the moment, it acts as a targeting beacon for buying and personalising impressions in the perfect instant.

How can you as a marketer use these three data types to come up with the most accurate view of your customer?

First, you need to understand the scope and diversity of your own data. There is valuable information lurking in all kinds of enterprise systems: CRM, merchandising, loyalty, revenue management, inventory and logistics and more. Be prepared to use data from a wide array of platforms, channels and devices.

From there, you can start answering questions about your customers. What are they saying about my products; when are they thinking about purchasing a product from me (or a competitor); how frequently have they have done business with me; how much do they spend?  The faster and more fully I can answer these questions, the more prepared I am to connect with my customer in the moment.

Integrating and analysing all of this information in a single manageable view is the next challenge for marketers, allowing them to recognise, rationalise and react to the fantastic complexity that exists within their data. Doing this is no small task, but a holistic view will enable marketers to differentiate meaningful insights from the noise.

The bottom line is that customers want brand experiences that are relevant and engaging, and offers that are custom-tailored for them, not for someone an awful lot like them. This is exactly what the 360-degree approach is designed to make possible: highly personalised, perfectly-timed offers that can be delivered efficiently and at scale.

In order to deliver those experiences, marketers must think about customer engagement from the 360-degree perspective, in which every touch-point informs the others. This cannot be achieved with a hodgepodge of disconnected data. It can only be achieved when all of the resources available, insights, technology and creative, are working together in perfect harmony. Over time, personalised customer experiences drive long-term loyalty for brands and retailers, ultimately creating even more of those “perfect” customers.

4). Data Integrity

How do you establish the desired integrity level across multiple functional teams and business processes? Is it merely about complete data (something in every required field)? or does it include accurate data, that is, the information contained within those fields is both correct and logical? What about unstructured data?

In the previous sections, we saw what Big Data means for the search and social marketer. Now, let’s spend some time on how we can make sure that the Big Data we have actually works for us.
Specifically, It’s my belief that there are four key factors determining our return from Big Data:

  • Is our Big Data accurate?
  • Is our Big Data secure?
  • Is our Big Data available at all times?
  • Does our Big Data scale?

Collating and creating Big, Valuable Data is a very expensive process and requires lots of investment and massive engineering resources to create a rigorous and high-quality set of data streams. Currently, 75% of Fortune 500 companies use cloud-based solutions, and the IDC predicts that 80% of new commercial enterprise apps will be deployed on cloud platforms.
Given these numbers, let’s address the 4 factors above in a specific context, using a cloud-based digital marketing technology platform for your Big Data needs.
1. Ensure Your Data Is As Accurate As Possible
As a search marketer, you are among the most data-driven people on this planet. You make important decisions around keywords, pages, content, link building and social media activity based on the data you have on hand.
Before gaining insight and building a plan of action based on Big Data, it’s important to know that you can trust this data to make the right decisions. While this might seem like a daunting exercise, there are a few fairly achievable steps you can take.
a. Source data from trusted sources: trust matters. Be sure that the data you, or your technology vendor, collect is from reliable sources. For example, use backlink data from credible and reputed backlink providers such as Majestic SEO, which provides accurate and up to-date information to help you manage successful backlinking campaigns.
b. Rely on data from partnerships: this is a corollary to the previous point. Without getting into the business and technical benefits of partnerships, I strongly recommend that you seek data acquired through partnerships with trusted data sources so that you have access to the latest and greatest data from these sources.
For example, if you need insight into Twitter activity, consider accessing the Twitter fire hose directly from Twitter and/or partner with a company who already has a tie-up with Twitter. For Facebook insight, use data that was acquired through the Facebook Preferred Developer Program certification. You need not go out and seek these partnerships, just work with someone who already has these relationships.
c. Avoid anything black hat: build your SEO insights and program with a white hat approach and takes a trusted partnership driven approach like the ones mentioned above.
If and when in doubt, ask around and look for validation that your technology provider has partnerships and validate it on social media sources such as Facebook and Twitter. Do not be shy about getting more information from your  technology vendors and track back to check that their origins do no tie back to black hat approaches.
2. Ensure Your Data Is Secure
You have, on your hands, unprecedented amounts of data on users and their behavior. You also have precious marketing data that has a direct impact on your business results.
With great amounts of knowledge comes even greater responsibility to guarantee the security of this data. Remember, you and your technology provider together are expected to be the trusted guardians of this data. In many geographies, you have a legal obligation to safeguard this data.
During my readings and research, I have learned a lot about the right way to securely store data. Here are a few best practices that, hopefully, your technology provider follows:

  1. ISO/IEC 27001 standard compliance for greater data protection
  2. Government level encryption
  3. Flexible password policies
  4. Compliance with European Union and Swiss Safe Harbor guidelines for compliance with stringent data privacy laws

3. Ensure Your Data Is Available
Having access to the most valuable Big Data is great, but not enough, you need to have access to this data at all times. Another valuable lesson I learned is how to deliver high availability and site performance to customers.
To achieve this, implement industry leading IT infrastructure including multiple layers of replication in data centres for a high level of redundancy and failover reliability, and datacenter backup facilities in separate locations for disaster recovery assurance and peace of mind. If you work with a marketing technology provider, be sure to ask them what measures they take to guarantee data availability at all times.
4. Ensure Your Data Scales With User Growth
This is the part that deals with the Big in Big Data. Earlier in the post we saw how Zetabytes of data already exist and more data is being generated at an astounding pace by billions of Internet users and transactions everyday. For you to understand these users and transactions, your technology should have the ability to process such huge volumes of data across channels and keep up with the growth of the Internet.
Scale should matter even if you are not a huge enterprise. Think about this analogy, even if you are searching for a simple recipe on Google, Google has to parse through huge volumes of data to serve the right results.
Similarly, your technology should be able to track billions of keywords and pages, large volumes of location-specific data and social signals to give you the right analytics. Be sure the technology you rely on is made for scale.

5). Governance Process.

How do you establish Procedures across people, processes and technology to maintain a desired state of Governance? Who sets the rules? Are you adding a level of Administration here?

Big Data has many definitions, but all of them come down to these main points: It consists of a high volume of material, it comes from many different sources, it comes in a variety of formats, it arrives at high speeds and it requires a combination of analytical or other actions to be performed against it. But at heart, it’s still some form of data or content, though slightly different than what has been seen in the past at most organizations. However, because it is a form of data or content, business-critical big data needs to be included in Data Governance processes.

Do remember that not all data must be governed. Only data that is of critical importance to an organisation’s success (involved in decision making, for example) should be governed. For most companies, that translates to about 25% to 30% of all the data that is captured.

What Governance best practices apply to big data? The same best practices that apply to standard data governance programmes, enlarged to handle the particular aspects of Big Data:

  1. Take an enterprise approach to big data governance. All Data Governance Programmes should start with a strategic view and be implemented iteratively. Governance of big data is no different.
  2. Balance the people, processes and technologies involved in big data applications to ensure that they’re aligned with the rest of the data governance programme. Big data is just another part of enterprise data governance, not a separate programme.
  3. Appoint Business Data Stewards for the areas of your company that are using big data and ensure that they receive the same training as other data stewards do, with special focus on big data deemed necessary due to the technology in use at your organisation.
  4. Include the Value of Big Data Governance in the business case for overall data governance.
  5. Ensure that the metrics that measure the success of your data governance programme include those related to big data management capabilities.
  6. Offer incentives for participating in the data governance programme to all parts of the business using big data to encourage full participation from those areas.
  7. Create data governance policies and standards that include sets of big data and the associated metadata, or that are specific to them, depending on the situation.

It has to be said that there are many more challenges in Big Data, but researching and reading these are basically the top five that come out every time and are referenced by any and all that are venturing into this world. If there are any different aspects that have been encountered please let me know and perhaps together we can formulate a global checklist for all to follow.


Tags: , , , , , , ,

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: