It would be a busy, but not unusual, day.
6:53 a.m.: You’re off to work. Quick stop for a coffee. You scan your rewards card from your phone, review today’s schedule, sip your coffee and browse Facebook.
7:30 a.m.: Arrive at work. Conduct your final review of the cloud computing presentation you’re delivering at a conference today. The 17th time’s the charm, right?
9:25 a.m.: Data verified and copy tweaked. Post an update about today’s presentation on LinkedIn and retweet the conference agenda.
9:45 a.m.: Punch the hotel address into your map app and head to the conference. Log into the hotel network, load the presentation, check into Foursquare, and proceed to deliver a well-received overview of how cloud computing is changing your industry’s economics.
3:57 p.m.: Session complete. Post your presentation to SlideShare, answer emails, text your boss about the feedback you received, and return to the office.
7:12 p.m.: Stop by the store to pick up a birthday card for your best friend. Scan your rewards card and notice that your receipt contains a $4.00 savings coupon for your next Hallmark purchase.
9:55 p.m.: Sink into the couch, download the Big Bang Theory from Amazon, and fall asleep in about 10 minutes.
Not an atypical day, one full of routine and the personal and professional activities to which you’ve grown accustomed. But from a technology perspective, it highlights the trail of electronic bread crumbs that we all leave daily. This day was one infinitesimal slice of Big Data. Welcome to what many predict will be the next great technological leap.
What is Big Data?
Big Data is an almost incomprehensible amount of information— numbers, pictures, audio, social interactions, words, bar codes, video content—coursing through global networks. IBM estimates that we create 2.5 quintillion bytes of data per day and 90 percent of today’s data are less than two years old.
It’s comprised of data sets so large and complex that it requires highly intelligent design and analysis to access, understand, and utilize. Moore’s Law, cloud computing, virtualization, Wall Street, and the global economy are driving companies in the direction of Big Data. Consider the handful of sites, scans, and swipes experienced at the outset of this paper. In the past 12 months we created the data equivalent of a stack of books stretching from the Earth to Pluto—30 times. It’s enormous.
Advances in storage, speed, and accessibility mean that Big Data isn’t just for corporate behemoths any more. Even small-to-mid-sized businesses are able to swim in the Big Data pool. Those who can unlock it, analyze it in real time, tie it to their key audiences, and create personalized relationships with their customers have an incredible marketplace advantage.
Somewhere in this data tsunami is exceptionally valuable, though disconnected, information with the potential to transform how marketers market, retailers sell, businesses relate to each other, and consumers determine their habits of daily consumption. It’s information—not intuition—and it can change how we all are linked together…for the better and the worse.
How Big is the Potential?
Big Data has the potential to improve everything from corporate profits to public policy. From dramatic crime reduction in New York and Baltimore, to predicting TV audience preferences, flu severity, search engine trends, and Federal Reserve activity, Big Data can open the doors of opportunity.
In Gartner’s Hype Cycle for Cloud Computing 2012 report they predict that, “Big Data will deliver transformational benefits to enterprises within 2 to 5 years, and by 2015 will enable enterprises adopting this technology to outperform competitors by 20% in every available financial metric.”
How can you gain this coveted 20 percent competitive financial edge? Hire intelligently.
What are its Impacts on Hiring & Employment?
“Data will not answer questions by themselves,” said Eric Bradlow, co-director of the Wharton Business School Customer Analytics Initiative in a June 2011 report on Big Data by the Economist Intelligence Unit, sponsored by SAP. As with so many other industries and sectors in the global economy, the root challenge is finding the talent necessary to drive success. The technology exists. But if you don’t have the talent to steer and drive the technology, you can’t play in the Big Data pool.
Accessing and deploying Big Data for companies, consultants, organizations, and IT groups requires finding talented people who know how to ask the right questions, intelligently analyze the content, and make meaningful interpretations of the data. It’s no small feat.
The McKinsey Global Institute predicts that, “By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of Big Data to make effective decisions.”
The talent crisis is real and it’s immediate.
Solving the Big Data Talent Problem
Higher education is beginning to respond. Schools ranging from MIT and the University of Montana to Stanford University and George Mason are offering a variety of degree programs related to Big Data analysis. But there is considerable ramp-up time to educate, graduate, and train some 2 million people who can address this talent shortage to access and interpret Big Data.
To productively move forward, companies must tackle several key questions;
Who should business, government, and non-profits look for to address their Big Data needs?
What are the talents, backgrounds, and skill sets necessary to effectively manage and interpret Big Data?
What should you look for in developing a Big Data presence or replacing people who’ve moved on to other opportunities?
As part of these discussions, companies are intentionally shifting their value focus toward Big Data, recognizing it as a significant and critical business asset. As a General Manger in the IT field, my team and I have witnessed the emergence of new and exciting career opportunities related to Big Data. The position titles we see range from Data Mining Engineer, Business Analyst and Data Architect to Chief Data Officer. But the title catching on most quickly is Data Scientist. Regardless of formal title, each Big Data role requires someone who can harness a complex maze of information and distill observations that drive revenue growth, improve a public service, or allow a non-profit to operate more efficiently.
In respect to organizational structure and talent needs, many companies are building multidisciplinary, cooperative teams. IT is an informative, enabling member of this team, rather than a driving leader. The financial, product and market functions define the business needs, and the Big Data members build the tools and support structures required to fuel the company in determined strategic directions.
Five Skills to Seek in a Data Scientist
Many presume that Data Scientists are computer scientists, software developers, or database experts. Some of those skill sets certainly apply, and solid technical skills are definitely critical. Without them the ramp-up is simply too steep and the potential for error too great. In particular, I recommend five areas of communication and technical excellence in business that are particularly important in searching for Data Scientists. They are:
1. Data warehousing, mining, and visualization (extremely important)
2. Behavior modeling and statistics
3. Database management, especially unstructured databases
4. Hadoop, an open-source framework that allows for data generation and analysis across a wide range of servers
5. Visual presentation skills
But your search shouldn’t stop there. Big Data also requires Big Thoughts and finding someone with an integrated combination of science and communication is critical. Some suggest people with a solid knowledge of physics; others tout liberal artists with keen critical thinking skills. I suggest a hybrid approach. “It’s almost a mindset,” observed Professor Mark Whitehorn, chair of analytics at Scotland’s University of Dundee. “They have to be curious and communicative and reasonably good with numbers.”
Curious and communicative are the more abstract and desirable skills sets. Data scientists need to go beyond the mechanics of the data. They need to blend often disparate data. They need to integrate their questions and data analysis into the business environment. And they need to creatively present the data in a visually interesting and systematic fashion, often to senior company leadership. In a 2012 survey of IT department employees by Infochimps, Inc. more than half of those involved in Big Data projects cited a “…lack of expertise to connect the dots,” as the most common reason for project failure. Thomas H. Davenport and D.J. Patil, in the October 2012 issue of the Harvard Business Review, offered this advice. “Think of him or her as a hybrid of data hacker, analyst, communicator, and trusted adviser. The combination is extremely powerful—and rare.”
It’s an exceptionally complex talent package and one that’s highly select.
Insource, Contract, or Outsource?
Companies are recognizing the meritorious qualities of these sought-after professionals, but how to best acquire and manage IT talent remains hotly debated. Do you hire Data Scientists as your own employees, bring in contractors for a defined period of time, or outsource to consultants who are already trained analysts and can bring critical mass immediately to the table?
Obtaining, analyzing, and utilizing Big Data can give you an edge. But it’s also expensive and extremely dynamic. The insight gleaned by these Data Scientists is critically important and sensitive. Trusting this information to an external consultant (who may complete your assignment only to move on to a competitor’s) poses competitive risks. The current trend seems to lean toward keeping this talent in-house on a permanent or contract basis. But doing so does not come cheaply. Attracting and retaining this scarce talent requires an investment of time and money as well as the ability to sell the opportunity you are offering.
Eight Steps to Harnessing Big Data
Big Data is in its infancy but it holds great promise. The key to success is finding and keeping the talent with the skills necessary to obtain and analyze the data, ask the right questions, and present findings in a compelling fashion that makes sense for your organization.
As you begin your Big Data initiatives, I recommend the following steps as essential to crafting a successful strategy:
1. Understand the Three V’s—volume, velocity, and variety.
2. Assemble a multidisciplinary team.
3. Assess your business needs at the outset. Understand what you have and what you want to accomplish. Defining your objectives will dictate the types of data you’ll need and the talent qualities necessary to your “Data Science Team”. To quote Covey, “begin with the end in mind.”
4. Specifically identify what you need to achieve:
> Enhanced customer insight?
> Integrated data generated by social media?
> Increased production efficiency?
5. Ensure that those specific objectives align with your broader corporate strategy. Harnessing, analyzing, and utilizing Big Data should be a core to your larger mission, not a one-off exercise.
6. Evaluate your current talent and take the necessary steps to find the critical and complementary skill sets you lack. A blend of existing and new employees can give you a talent mix that both understands how you function and questions why you function in that fashion.
7. Establish metrics to assess your Big Data efforts. What do you want to achieve and how will you define success?
8. Start with a pilot, assess and analyze against your established metrics and retool where needed. When ready, expand your scope, further invest and grow your Big Data programs.
Big Data will quickly become an integrated component of most businesses, government, and non-profits. I foresee Data Scientists working in all areas of a company—from logistics and customer service through to accounting and finance—all reporting into a C-Suite executive (enter the Chief Data Officer) skilled in the science and analysis.
According to the Economist Intelligence Unit study, “Companies that are furthest along the data manage-ment competency continuum…provide a useful model for how organizations will need to evolve if they are to extract and utilize valuable data-driven insights…They are also more likely to assign a C-level exec-utive to manage data strategy, and they continue to explore emerging sources of data for potential value.” The opportunity is simply too great and the competitive risks too high to not fully engage with Big Data.
Don’t be afraid to make mistakes and learn from them. The risks associated with Big Data mirror the early years of enterprise technology. Technological complexity can cause some companies to lose focus, but Big Data is more than a technology initiative. It’s a business. Keep your strategic goals in focus and embrace the advancements.
Once your strategy is in place, turn your people loose and empower their innovation.
Take a deep breath and jump into the Big Data pool.
I’m interested in hearing about your experience in hiring and retaining Big Data talent and look forward to your feedback. Please share your comments, insights, and experiences with me at firstname.lastname@example.org.