Does your organisation have data but struggle with providing effective reports and/or access for analysis? Does your organisation have many reports available but struggle with utilising the data in the support of decisions? Has previous success with the data warehouse not turned into expected future successes?

The following is an extract from the full 'Escape from Data Jail' article which highlights insights into how to unlock data and provide greater business value. The target audience includes both business users and those with technical responsibilities. The topics covered include:

• Data Warehouse Value
• What is Data Jail?
• Types of Data Jail
• Keys to unlocking data
• Approaches and Examples
• Lessons Learned

Data Warehouse Value

Let’s start with a discussion of data warehouse value. Value is obtained when the business / organisation takes action. Action is taken based on decisions. Decisions require information and information requires appropriate data.

Caution! Do not confuse information with data. If a report is generating many pages of detailed data that does not mean there is information that can be gleaned in support of a decision. Ask yourself, “What about this report supports a decision? What about this report makes it actionable?” Is the report merely a dump of raw data or is it organised to provide the key pieces of information required by the targeted audience?

What is Data Jail

Data Jail occurs when an organisation struggles in turning data into information. Even a successful data warehouse can experience data jail as the data warehouse expands to include other areas of the business or with an increase in the sophistication of use (e.g. Active/Real-Time Data Warehousing or support for advanced analytics and data mining).

There are many types of data jail and I'm going to focus on three types that I have had direct experience with at different organisations.

Unsuitable User Tools

Does your data warehouse support different types of users, each with an appropriate tool? Ranges of data warehouse usage go from reporting to analysis to advanced data mining. Even within the category of reporting there can be different classes of users and usage including scorecards, dashboards, summary and detailed reports as well as alerts. So if you are treating all the users the same, then some of them are experiencing data jail.

Query inflexibility is another type of data jail. Different types of report and analysis usage need different types of queries. Queries for advanced analytics are very different than queries used to monitor and alert. Multi-dimensional analysis is a useful method of providing users a platform for business analysis, however if there is no ability to go deeper than the planned dimensions allow, for example to find out the specific customers behind a given number, then the overall effectiveness of the environment is reduced.

Inappropriate Data Format

The various types of users and usage of the data warehouse will require differing data formats. If all users were expected to write SQL queries against the relational model, many of them will find the process difficult and therefore not use it as often or to the full potential.

Lack of consistency is another type of data jail. All of us have heard the desired target of “Single version of the truth”. The reality is most organisations have at least some type of reporting off both the data warehouse and the source system. If it is not clear as to when each system is to be used, or if the results achieved are inconsistent, then there will be uncertainty and a natural lack of trust in one or both systems. Even worse for the data warehouse is if there are two applications, cubes or data marts built from the data warehouse that are inconsistent with each other!

Insufficient Data Quality

The last of the three types of data jail covered in this paper deals with data quality. I specifically chose the term insufficient since data does not have to be perfect to be of use to the organisation. Most marketing areas can use data that is “directionally accurate,” however in contrast financial data stored in the data warehouse is generally expected to match what is reported by the general ledger. If the data is not fit for purpose, if the quality is not up to the standard required by the target audience, then there is an issue.

Keys to Unlocking your Data

Now that you have an understanding of the concept of data jail, we will look at some keys to unlocking your data.

Key #1: Enable / Empower business users

As discussed previously, the varying types of users and uses of the data warehouse need different tools. The requirements of the tool need to match the intended use. Tools are needed to support all types of usage including reporting, analytics, exploration, data mining as well as application support.

You cannot leave it at providing tools; appropriate skilling is also a key. Skilling needs to cover the spectrum from business to technical. A data warehouse user within a business area needs an understanding of the tool and an understanding of the data, as well as a business understanding of the target for the data and the appropriate types of questions to ask. One of the marketing departments I previously worked with provided an “Introduction to direct marketing” course that included the strategies and goals behind direct marketing along with how to use the established query tool to achieve the desired outcomes.

My final topic within this key of empowering business users is to not ‘over grant’ or ‘under grant’ access. By over grant access I mean giving users access to more sophisticated tools or a broader view of the data warehouse than they need to satisfy their business needs. By contrast, artificially limiting access to the full relational model to advanced data miners will limit their ability to draw out previously unknown relationship contained within the data itself.

Key #2: Focus on the data

To start, plan to build an Enterprise Data Warehouse incrementally over time. If you focus each time on the fact that the end goal is an Enterprise Data Warehouse, then the decisions that are made within each project will take the goal into consideration. This will lessen the risk of implementing pieces that might have to be redone later when additional subject areas or users are added.

In the same manner, build the data warehouse from detailed data. The detailed data will provide the greatest amount of flexibility over time to provide new applications, support new users, etc. Keep in mind that the amount of history required is a business decision, not a technical one. A relational model as the base provides greater flexibility to support a variety of different types of request. Finally,  key summary tables can be built to capture business-rule driven summarisations (as opposed to straight aggregations).

Key #3: Focus on data quality

I previously mentioned insufficient data quality as a type of data jail. The key is to establish a data quality program. Begin with identifying foundational principles such as “Data Quality begins at the source.” Next define what ‘fit for purpose’ means for your organisation in general and for the specific business area. Establish thresholds for monitoring the quality of the data being loaded.

An approach for testing and validation of the data should be developed prior to any data load being implemented. Use the established approach for testing prior to deployment as part of acceptance testing and for the ongoing monitoring of data quality. Plan to measure and make improvements over time focussing on priority areas that align to business priorities.

Key #4: Take advantage of the power of your database

All the databases have advanced features for tuning and optimising a data warehouse. It is important to understand and utilise all that the database has to offer. Following are a few examples of key features of Teradata (a powerful data warehouse management tool) that help to unlock your data.

One of the basic features in Teradata is views. The power of views is that they appear the same as tables to third-party tools. This enables the creation of a presentation layer that hides the complexity of the relational layer or presents it in a different format like a dimensional model. The presentation layer simplifies report development and enables reports to be written by business users, which increases self sufficiency.

Data warehouse systems include features for workload management and these must be utilised effectively to ensure adequate performance service levels by the different types of users. Teradata Active Systems Management tool provides mechanisms for both “gate keeping”, identifying users and their specific queries to determine if their request can run now or later; and "priority management", assigning queries to priority groups and monitoring to ensure they do not exceed set thresholds for that group. Effective workload management is a key to having an efficiently tuned system and provides users with more consistent experience over time.

Finally, take advantage of the materials produced by the vendor, including manuals, research papers, webcasts, podcasts and formal courses.

Download the full article to read more about the approaches, examples and lessons learnt.

Whether you are a business user or have technical responsibility, look for opportunities to unlock the data to provide greater business value. I hope you have read something that causes you to nod in agreement as well as pause to  think.