Big data is one of the biggest opportunities and challenge that organizations are facing today. Great breakthroughs are only possible only if we could get our hands on the right data and analyze it the right way. We have seen many breakthroughs in both B2B and B2C companies, with retailers leveraging behavioural data to provide targeted personalized experiences and banks analyzing the transactional activities to prevent fraud. But, there are lot of inbuilt issues with big data that restrict companies to unlock the actual business value.
Issues with Big data:
The data source from big data is from multiple sources (Mobile data, data from IoT devices, Applications and social) and the volume and complexity increases as more and more business processes undergo digital transformation. These new data sources become even more complex as the data formats are usually in structured and unstructured formats. While the data sources internally are safe, there is an increasing concern of privacy and compliance on the data sets sourced from outside the companies’ sources.
When we ignore the data characteristics of Big data vs traditional data, we end up with data that is inappropriately used due to inaccuracies and/or missing the context that cannot be interpreted for meaningful analysis. If the analysis or the underlying data is not clear to users, they will not trust the data and this lack of trust means that there is a huge impact on the operations of the company, it’s people as it delivers little value.
How Data Governance addresses these issues?
Data governance can address these issues and help companies derive the value from the investments made in big data projects.
Data governance is a systematic approach to the overall management of the availability, integrity, usability and security of the data employed in a company. A defined set of procedures and a plan to execute those procedures gives users of data, the information they need to find, understand and trust the data, the capabilities to manage data and improve its usability and quality.
The growing data volumes, complexity and multiple sources of data requiring analysis and the constant need to combine data across companies cross functional teams necessitates data governance. Most of the people think data governance as the implementation of the policy, but the true benefits go beyond regulatory or statutory compliances. Big data is the right use case for Data governance.
Implementing the data governance is not as complex as it may seem, but requires a systematic approach and automation. It’s impossible to make informed and precise decisions without implementing some structure w.r.t how decisions are made from data. Earlier companies could not scale on these processes as they were implemented in bits and pieces, not structured and automated and only confined to a silo group and hence could see a limited value. However, there are proven cases towards building a valuable and robust data governance processes.
Recommended Steps to implement Data Governance
Step1 – Adopt a comprehensive data policy.
There must be clear policies around data ownership, big data sets, data quality, data lineage and data retention. The policy management enables you to take inputs from the right stakeholders, understand the impact, formulate and enforce policies that reduces risk and improves efficiency.
Step2 – Define Standards for data sets.
Once the basic policy is formulated, the next step is to define data standards for all types of data sets that are expected to deliver most value. Starting with Large and critical data sets is vital for effective data governance. They must be identified, metadata and relationships to be documented. This is going to be an ongoing process, as there is too much data to do all at one go, you use data in a different way, you interpret a new meaning. Critical data typically includes the structured information from the companies, but in the current context it also includes data sets sourced from social.
Step3 – Ensure Flexibility and automation.
Big data requires flexibility and automation to handle the increasing data volume and complexity in data formats. Unlike the centralized policies of previous systems, Big data activities are spread across the company. It’s crucial to automate the data governance with technology as manual processes cannot keep pace with the volumes and complexity of data.
Step4 – Create Analytical models
Big data and analytics go together and hence it is suggested to build the analytical models to be part of your data governance. It should have the ability and flexibility to capture all the information on all aspects of your analytics from mapreduce to visualizations. The processes need to be simplified for business and data professionals alike. There must be a simple way for the business users to request new analytical models.
Step5 – Create a Data Governance team
You’ll need a separate, committed data governance team that includes stakeholders from the organization including chief data officer and key business heads. Data architects and subject matter experts should be allowed to constantly improve, augment and enhance the data.
Conclusion
Big data promises to unlock data about customer behaviour, operational efficiencies and helps attain competitive advantage to companies. However, without a proper data governance framework, interest alone on big data projects will not help derive the value. Data governance allows companies to provide a set of information to their business and data users, making them to leverage and unlock the value from big data