Knowledge Integration Dynamics

Quality is elusive needle in shifting enterprise data haystack


By Mervyn Mooi, Director at Knowledge Integration Dynamics (KID)
[Johannesburg, 12 August 2011]

Many companies are experiencing data overload, not information overload, says Mervyn Mooi, Director at Knowledge Integration Dynamics (KID). Information is what can be intelligently used to achieve efficiency; it is processed from data. Information - the right information, at any rate - in organisations today is actually lacking or is being ignored, even if it is found in abundance.

Overload resulting from exponential data growth and duplicated processes creates new challenges and opportunities for both IT and BI.

For example, planning for increased resource such as storage or server requirements; and strategising for integration and rationalisation of data and processes.

Certain other challenges are also noteworthy:
* Managing IT operating costs;
* Ensuring data quality meets regulatory compliance during periods of data growth;
* Adherence to data security and protection policies;
* Shrinking backup windows and sustaining bandwidth; and
* Expanding service level agreements (SLAs).

Businesses are rethinking integration pitched at enterprise level to achieve the best rationalisation or economy for resources. They are deploying new data deduplication and backup and archive strategies with enabling technologies such as data integration engines, virtual tape libraries and more. What they're grappling with is that, over time, data volumes rise, and therefore the associated cost. The new strategies and tech reduce the total cost of ownership (TCO) by reducing the requisite resources to run better quality, deduped data.

Simultaneously, data warehousing (traditional extraction, transformation and loading) and other data integration initiatives are combining to support processes such as master data (MDM) and data quality management (DQM). There is also a trend to rationalise resources by using the disaster recovery (DR) site as a data source, thereby:
a) Avoiding impact of primary production user or application environments; and
b) Eliminating the need for an operational data store.

There is also a shift to make the data warehouse a virtual design by creating a logical universe of business (presentation) and technical (data access) metadata, which is directly linked to the data sources. Supporting that is a tendency to deploy data warehouse appliances, such as Kognitio and Exadata, products and technologies that have mind-boggling processing performance on huge volumes of data.

Yet another trend is to encrypt and decrypt or scramble sensitive data to ensure data security.

Many people have said that data is the lifeblood of businesses, but few businesspeople have understood the impact of that statement until recently.