Showing posts with label Data Warehousing. Show all posts
Showing posts with label Data Warehousing. Show all posts

Datawarehousing Objects

Data Warehousing Objects
Fact tables and dimension tables are the two types of objects commonly used in dimensional data warehouse schemas.
Fact tables are the large tables in your data warehouse schema that store business measurements. Fact tables typically contain facts and foreign keys to the dimension tables. Fact tables represent data, usually numeric and additive, that can be analyzed and examined. Examples include Sales, Cost and Profit.
Dimension tables, also known as lookup or reference tables, contain the relatively static data in the data warehouse. Dimension tables store the information you normally use to contain queries. Dimension tables are usually textual and descriptive and you can use them as the row headers of the result set. Examples are Customers or Products

Fact Table:
A fact table typically has two types of columns: those that contain numeric facts (often called measurements), and those that are foreign keys to dimension tables. A fact table contains either detail-level facts or facts that have been aggregated. Fact tables that contain aggregated facts are often called summary tables. A fact table usually contains facts with the same level of aggregation.
Fact tables are summarized into three types:
1   Additive
    Semi-Additive
    Non-Additive
Additive facts can be aggregated by simple arithmetical addition. A common example of this is sales. Semi-additive facts can be aggregated along some of the dimensions and not along others. An example of this is inventory levels, where you cannot tell what a level means simply by looking at it. Non-additive facts cannot be added at all. An example of this is averages.

Requirements of Fact Table:
You must define a fact table for each star schema. From a modeling standpoint, the primary key of the fact table is usually a composite key that is made up of all of its foreign keys.

Dimension Table:
A dimension is a structure, often composed of one or more hierarchies, that categorizes data. Dimensional attributes help to describe the dimensional value. They are normally descriptive, textual values. Several distinct dimensions, combined with facts, enable you to answer business questions. Commonly used dimensions are customers, products, and time.
Dimension data is typically collected at the lowest level of detail and then aggregated into higher level totals that are more useful for analysis. These natural rollups or aggregations within a dimension table are called hierarchies.
Commonly dimension tables are categorized into three types:
      Conformed dimension
      Degenerated dimension
      Junk dimension

If we are saying in simple word, Conformed dimension table is a table which is having at least one or multiple primary key or unique key to join with one or multiple fact table. For example product table is related with the sales fact table
A conformed dimension is a set of data attributes that have been physically implemented in multiple database tables using the same structure, attributes, domain values, definitions and concepts in each implementation. A conformed dimension cuts across many facts.

Dimensions are conformed when they are either exactly the same (including keys) or one is a perfect subset of the other. Most important, the row headers produced in the answer sets from two different conformed dimensions must be able to match perfectly.
A junk dimension is a convenient grouping of typically low-cardinality flags and indicators. By creating an abstract dimension, these flags and indicators are removed from the fact table while placing them into a useful dimensional framework.
A Junk Dimension is a dimension table consisting of attributes that do not belong in the fact table or in any of the existing dimension tables. The nature of these attributes is usually text or various flags, e.g. non-generic comments or just simple yes/no or true/false indicators. These kinds of attributes are typically remaining when all the obvious dimensions in the business process have been identified and thus the designer is faced with the challenge of where to put these attributes that do not belong in the other dimensions.
One solution is to create a new dimension for each of the remaining attributes, but due to their nature, it could be necessary to create a vast number of new dimensions resulting in a fact table with a very large number of foreign keys. The designer could also decide to leave the remaining attributes in the fact table but this could make the row length of the table unnecessarily large if, for example, the attributes is a long text string.
Refer Ralph Kimball’s explanation about Junk Dimension

Hierarchies
Hierarchies are logical structures that use ordered levels as a means of organizing data. A hierarchy can be used to define data aggregation. For example, in a time dimension, a hierarchy might aggregate data from the month level to the quarter level to the year level. A hierarchy can also be used to define a navigational drill path and to establish a family structure.
Within a hierarchy, each level is logically connected to the levels above and below it. Data values at lower levels aggregate into the data values at higher levels. A dimension can be composed of more than one hierarchy. For example, in the product dimension, there might be two hierarchies—one for product categories and one for product suppliers.

Dimension hierarchies also group levels from general to granular. Query tools use hierarchies to enable you to drill down into your data to view different levels of granularity. This is one of the key benefits of a data warehouse.
When designing hierarchies, you must consider the relationships in business structures. For example, a divisional multilevel sales organization. Hierarchies impose a family structure on dimension values. For a particular level value, a value at the next higher level is its parent, and values at the next lower level are its children. These familial relationships enable analysts to access data quickly.

Levels
A level represents a position in a hierarchy. For example, a time dimension might have a hierarchy that represents data at the month, quarter, and year levels. Levels range from general to specific, with the root level as the highest or most general level. The levels in a dimension are organized into one or more hierarchies.

Level Relationships
Level relationships specify top-to-bottom ordering of levels from most general (the root) to most specific information. They define the parent-child relationship between the levels in a hierarchy.
Hierarchies are also essential components in enabling more complex rewrites. For example, the database can aggregate an existing sales revenue on a quarterly base to a yearly aggregation when the dimensional dependencies between quarter and year are known.


Typical Dimension Hierarchy

Description of Figure 2-2 follows 

Wherever we go, we have to know this concepts to work on data warehousing technologies.


Cheers!
Bose

Read more


Data Mining

Introduction: 
 
The core of Business Intelligence Applications is Data warehousing and the core of Data warehousing is Data Mining. If we are speaking the technical aspect, Data Mining is gathering knowledge from Data. So Knowledge of Data Discovery also part of the Data Mining. 

So Data mining is the important subject to know, if we are working in data warehousing.

Definition of  Data Mining:

Data mining is the entire process of applying computer-based methodology, including new techniques for knowledge discovery, from data.

Knowledge discover explains the step of process How to get the useful data from the Raw data.

From the below picture, you will know the knowledge discovery process.


Step 1:
The data is collected from various sources like spreadsheet, flatfiles, dmp, and etc., that is called Raw Data.

Step 2:
The collected raw data is integrated and cleansed to make a data warehousing. Data Integration is involving collect all the source data and making a single unit of system and cleaning Data involves removing or making Junk files and only keeping the necessary files that needed for data warehousing.

Step 3: 
 
The real process of Data mining starts from Data warehousing. So after integration and Cleansing of data, data is transforming into various subject area called Sales, Purchase, Finance and etc.,

Step 4:

So from the separated subject area, we are applying all the knowledge i.e Domain knowledge and we are giving the specific output from the data. 

Read more


OLAP Vs OLTP

Introduction:

The constant growth of Data Analysis and Business Intelligence Applications, understanding OLAP and OLTP having the important Place.

The design of Data Warehouse and Online Analytical Processing (OLAP) cubes is basically different than OnLine Transactional Processing (OLTP). 

OLTP:

Source of Data: OLTP is original Source of data that maintains the Day by Day transactions and processes for every Business.

Purpose of Data: To run and control the fundamental business task and record all the events inside.

OLTP Reveals the copy of the ongoing Business processes.

Insertion and Updation is very fast in OLTP and this operation is initiated by End Users. All the OLTP is Normalized one and Doesn't having any duplicate records in any situation.

In OLTP, the Query Processing is normally standardized and simple query that gives the result of few records.

The processing speed is very speed and the size of the OLTP  Size  (MB to GB) is very small compare with OLTP if historical data is archived.

Database design of OLTP is highly Normalized with many tables.

Backup & Recovery of OLTP:
Operation data is very critical to run the business and data loss is likely to entail and legal liablity.

OLAP:

The source of data of OLAP is OLTP, Consolidated from various OLTP Sources.

The Main aim of this data is to help reporting, Decision support, Data analyzing and for problem Solving.

This data reveals the Multi-Dimensional views of various kinds of business Activities.

Periodically Its long process to insert the data and update the data But retrieving the data from OLAP is easy.

Typicall the query processing in OLAP is long and tedious and often with complex queries involving aggregations.

The processing speed in OLTP depends upon the size of the data involved, that may batch processes and complex queries may take hours to one and query speed can be improved by creating Views ans Indexes.

The size of OLAP is very Large ( GB to TB) and It contains the consolidated data of OLTP with more aggregation structures and Historical data.

The database design of OLAP is in most cases de-normalized with the fewer tables and using Star or Snowflake schema.




Read more


Data Warehousing

Data Warehousing & Need of DW:
 
Commonly Data warehousing is used by the companies to analyze the trends over the Time. In other meaning, data warehousing is used to view day-to-day operations of Business for a company, but its primary function is facilitating best decision making process resulting from long-term data overviews. For example, Business Process & Models, Forecasts, Business Reports and Projections can be made. Because the main intention of the Data Warehousing is intended to provide the overview like Reporting, analyzing the past process activities.

By Technically, Data Warehousing is a Collection of Data stored in single database to describe the past Business Process in best that leads to decision making process of company. It is the read-only data that never going to change but the size will grow up over the time.

Data Warehouse:

The term Data Warehouse was coined by Bill Inmon in 1990, which he defined in the following way: "A warehouse is a subject-oriented, integrated, time-variant and non-volatile collection of data in support of management's decision making process"

Subject Oriented:

Data that explains information about the a particular subject of a Business instead of explaining all the process of ongoing operations of company.
For Eg: Sales 

Integrated:

In Data Warehouse, the data is gathered from various kind of sources and merged into a single integrated storage place.

For Eg: the sales data of a business may gathered from Excel file, Flat file, dumb file and etc.,

Time Variant:

All the data in the Data Warehouse is explaining a particular Time period and It will grow up over a time.

Non - Volatile:

The data in the Data Warehouse is stable and never changed but may grow the size because of the data is to be added over a time and the data is never removed.

A Typical Architecture of Data Warehouse:




Read more

About This Blog

Labels

Views

Followers