Implementing Big Data Projects: Lessons Learned and Recommendations

This report, written by Kevin Desouza, Arizona State University, provides a clear and useful introduction to the concept of big data, which is receiving increasing attention as a term but also lacks a commonly understood definition. In describing big data, Prof. Desouza writes, “Big data is an evolving concept that refers to the growth of data and how it is used to optimize business processes, create customer value, and miti­gate risks.” The report also describes the differences in the use of big data in the public and private sectors.

In the next few years, nearly all public agencies will grapple with how to integrate their dispa­rate data sources, build analytical capacities, and move toward a data-driven decision-making environment. Big data is increasing in importance for public agencies, and big data programs are expected to become more prominent in the near future.  Through the use of big data, ana­lytics now holds great promise for increasing the efficiency of operations, mitigating risks, and increasing citizen engagement and public value.

A key contribution of this report is its descriptions of how big data is being used in federal, state, and local govern­ment today. His examples include the Internal Revenue Service, the state of Massachusetts, and the New York City Business Integrity Commission.

In addition, Professor Desouza conducted extensive inter­views with chief information officers (CIOs) across the United States at the federal, state, and local level. The goal of the inter­views was to better understand the implementation challenges facing CIOs and their organizations as they undertake big data projects. Desouza presents 10 key findings from his interviews.

• One: Public agencies are in the early days of their big data efforts.
• Two: Many CIOs fight the perception that big data is a passing fad.
• Three: Most CIOs are now primarily dealing with the issue of managing large volumes of data, integrating data across database systems, and building an analytical capacity to mine data.
• Four: CIOs report that some big data projects are now focused on streamlining business processes.
• Five: CIOs do not anticipate significant investments in technology.
• Six: CIOs report a need to bolster their human capital, including their analytical capability.
• Seven: CIOs are now exploring approaches to data governance.
• Eight: CIOs do not recommend IT units as owners of big data projects.
• Nine: CIOs believe that collaborative leadership is crucial for the success of big data projects and recommend the creation of working groups to oversee projects.
• Ten: CIOs are becoming champions of analytics and evidence-driven decision-making.

Big data is a new frontier for the public sector, and has captured the attention of public manag­ers across the globe. Agencies realize that their datasets represent critical resources that need to be managed and leveraged. Public sector use of big data and big data analytics is wide-ranging; some organizations have no experience with big data, while others have taken on small to moderate-sized projects. Drawing on the CIO interviews, this also report presents implementation steps grouped by the phases of a big data project:

• Planning
• Execution
• Implementation

Each of these three steps holds individual value, but also contributes to an overall framework for managing public-sector big data projects. The key steps serve as best practices for CIOs and public man­agers to consider for their agency as they embark on big data projects.

This report continues the IBM Center’s interest in the concepts of big data and analytics. In cooperation with the Partnership for Public Service, the IBM Center for The Business of Government recently issued its third report on analytics: From Data to Decisions III: Lessons from Early Analytics Program. The use of analytics in agency program management and decision-making relies on big data.

Collectively, this report and the Partnership-IBM Center’s three reports on analytics provide a roadmap for federal leaders on how to move to the new era of big data and analytics. We hope these reports will assist all public managers as they continue their quest to better use data that they collect on an ongoing basis.