Showing posts with label Data Warehousing. Show all posts
Showing posts with label Data Warehousing. Show all posts
Data Warehousing Objects
Fact tables and dimension
tables are the two types of objects commonly used in dimensional data warehouse
schemas.
Fact tables are the large
tables in your data warehouse schema that store business measurements. Fact
tables typically contain facts and foreign keys to the dimension tables. Fact
tables represent data, usually numeric and additive, that can be analyzed and
examined. Examples include Sales, Cost and Profit.
Dimension tables, also
known as lookup or reference tables, contain the relatively static data in the
data warehouse. Dimension tables store the information you normally use to
contain queries. Dimension tables are usually textual and descriptive and you
can use them as the row headers of the result set. Examples are Customers or
Products
Fact Table:
A fact table typically has
two types of columns: those that contain numeric facts (often called measurements),
and those that are foreign keys to dimension tables. A fact table contains
either detail-level facts or facts that have been aggregated. Fact tables that
contain aggregated facts are often called summary tables. A fact table usually
contains facts with the same level of aggregation.
Fact tables are summarized
into three types:
1 Additive
Semi-Additive
Non-Additive
Additive facts can be
aggregated by simple arithmetical addition. A common example of this is sales. Semi-additive
facts can be aggregated along some of the dimensions and not along others. An
example of this is inventory levels, where you cannot tell what a level means
simply by looking at it. Non-additive facts cannot be added at all. An example
of this is averages.
Requirements of Fact Table:
You must define a fact
table for each star schema. From a modeling standpoint, the primary key of the
fact table is usually a composite key that is made up of all of its foreign
keys.
Dimension Table:
A dimension is a
structure, often composed of one or more hierarchies, that categorizes data.
Dimensional attributes help to describe the dimensional value. They are
normally descriptive, textual values. Several distinct dimensions, combined
with facts, enable you to answer business questions. Commonly used dimensions
are customers, products, and time.
Dimension data is
typically collected at the lowest level of detail and then aggregated into
higher level totals that are more useful for analysis. These natural rollups or
aggregations within a dimension table are called hierarchies.
Commonly dimension tables
are categorized into three types:
Conformed
dimension
Degenerated
dimension
Junk
dimension
If we are saying in simple word, Conformed
dimension table is a table which is having at least one or multiple primary key
or unique key to join with one or multiple fact table. For example product
table is related with the sales fact table
A conformed dimension is a set of data
attributes that have been physically implemented in multiple database tables
using the same structure, attributes, domain values, definitions and concepts
in each implementation. A conformed dimension cuts across many facts.
Dimensions are conformed when they are either exactly the
same (including keys) or one is a perfect subset of the other. Most important,
the row headers produced in the answer sets from two different conformed
dimensions must be able to match perfectly.
A junk dimension is a convenient grouping of typically
low-cardinality flags and indicators. By creating an abstract dimension, these
flags and indicators are removed from the fact table while placing them into a
useful dimensional framework.
A Junk Dimension is a dimension table consisting of
attributes that do not belong in the fact table or in any of the existing
dimension tables. The nature of these attributes is usually text or various
flags, e.g. non-generic comments or just simple yes/no or true/false indicators.
These kinds of attributes are typically remaining when all the obvious
dimensions in the business process have been identified and thus the designer
is faced with the challenge of where to put these attributes that do not belong
in the other dimensions.
One
solution is to create a new dimension for each of the remaining attributes, but
due to their nature, it could be necessary to create a vast number of new
dimensions resulting in a fact table with a very large number of foreign keys.
The designer could also decide to leave the remaining attributes in the fact
table but this could make the row length of the table unnecessarily large if,
for example, the attributes is a long text string.
Refer Ralph Kimball’s explanation about Junk
Dimension
Hierarchies
Hierarchies are logical
structures that use ordered levels as a means of organizing data. A hierarchy
can be used to define data aggregation. For example, in a time dimension, a
hierarchy might aggregate data from the month level to the quarter level to the
year level. A hierarchy can also be used to define a navigational drill path
and to establish a family structure.
Within a hierarchy, each
level is logically connected to the levels above and below it. Data values at
lower levels aggregate into the data values at higher levels. A dimension can
be composed of more than one hierarchy. For example, in the product dimension,
there might be two hierarchies—one for product categories and one for product
suppliers.
Dimension hierarchies also
group levels from general to granular. Query tools use hierarchies to enable
you to drill down into your data to view different levels of granularity. This
is one of the key benefits of a data warehouse.
When designing
hierarchies, you must consider the relationships in business structures. For
example, a divisional multilevel sales organization. Hierarchies impose a
family structure on dimension values. For a particular level value, a value at
the next higher level is its parent, and values at the next lower level are its
children. These familial relationships enable analysts to access data quickly.
Levels
A level represents a
position in a hierarchy. For example, a time dimension might have a hierarchy that
represents data at the month, quarter, and year levels. Levels range from
general to specific, with the root level as the highest or most general level.
The levels in a dimension are organized into one or more hierarchies.
Level Relationships
Level relationships
specify top-to-bottom ordering of levels from most general (the root) to most
specific information. They define the parent-child relationship between the
levels in a hierarchy.
Hierarchies are also
essential components in enabling more complex rewrites. For example, the
database can aggregate an existing sales revenue on a quarterly base to a
yearly aggregation when the dimensional dependencies between quarter and year
are known.
Typical Dimension Hierarchy
Wherever we go, we have to know this concepts to work on data warehousing technologies.
Cheers!
Bose
Tuesday, April 24, 2012
//
Labels:
Data Warehousing
//
0
comments
//
Introduction:
The core of Business Intelligence Applications is Data warehousing and the core of Data warehousing is Data Mining. If we are speaking the technical aspect, Data Mining is gathering knowledge from Data. So Knowledge of Data Discovery also part of the Data Mining.
So Data mining is the important subject to know, if we are working in data warehousing.
Definition of Data Mining:
Data mining is the entire process of applying computer-based methodology, including new techniques for knowledge discovery, from data.
Knowledge discover explains the step of process How to get the useful data from the Raw data.
From the below picture, you will know the knowledge discovery process.
Step 1:
The data is collected from various sources like spreadsheet, flatfiles, dmp, and etc., that is called Raw Data.
Step 2:
The collected raw data is integrated and cleansed to make a data warehousing. Data Integration is involving collect all the source data and making a single unit of system and cleaning Data involves removing or making Junk files and only keeping the necessary files that needed for data warehousing.
Step 3:
The real process of Data mining starts from Data warehousing. So after integration and Cleansing of data, data is transforming into various subject area called Sales, Purchase, Finance and etc.,
Step 4:
So from the separated subject area, we are applying all the knowledge i.e Domain knowledge and we are giving the specific output from the data.
Monday, March 7, 2011
//
Labels:
Data Warehousing
//
0
comments
//
Introduction:
The constant growth of Data Analysis and Business Intelligence
Applications, understanding OLAP and OLTP having the important Place.
The design of Data Warehouse and Online Analytical Processing (OLAP) cubes is basically different than OnLine Transactional Processing (OLTP).
OLTP:
Source of Data: OLTP is original Source of data that maintains the Day by Day transactions and processes for every Business.
Purpose of Data: To run and control the fundamental business task and record all the events inside.
OLTP Reveals the copy of the ongoing Business processes.
Insertion and Updation is very fast in OLTP and this operation is initiated by End Users. All the OLTP is Normalized one and Doesn't having any duplicate records in any situation.
In OLTP, the Query Processing is normally standardized and simple query that gives the result of few records.
The processing speed is very speed and the size of the OLTP Size (MB to GB) is very small compare with OLTP if historical data is archived.
Database design of OLTP is highly Normalized with many tables.
Backup & Recovery of OLTP:
Operation data is very critical to run the business and data loss is likely to entail and legal liablity.
OLAP:
The source of data of OLAP is OLTP, Consolidated from various OLTP Sources.
The Main aim of this data is to help reporting, Decision support, Data analyzing and for problem Solving.
This data reveals the Multi-Dimensional views of various kinds of business Activities.
Periodically Its long process to insert the data and update the data But retrieving the data from OLAP is easy.
Typicall the query processing in OLAP is long and tedious and often with complex queries involving aggregations.
The processing speed in OLTP depends upon the size of the data involved, that may batch processes and complex queries may take hours to one and query speed can be improved by creating Views ans Indexes.
The size of OLAP is very Large ( GB to TB) and It contains the consolidated data of OLTP with more aggregation structures and Historical data.
The database design of OLAP is in most cases de-normalized with the fewer tables and using Star or Snowflake schema.
Thursday, March 3, 2011
//
Labels:
Data Warehousing
//
0
comments
//
Data Warehousing & Need of DW:
Commonly Data warehousing is used by the companies to analyze the trends over the Time. In other meaning, data warehousing is used to view day-to-day operations of Business for a company, but its primary function is facilitating best decision making process resulting from long-term data overviews. For example, Business Process & Models, Forecasts, Business Reports and Projections can be made. Because the main intention of the Data Warehousing is intended to provide the overview like Reporting, analyzing the past process activities.
By Technically, Data Warehousing is a Collection of Data stored in single database to describe the past Business Process in best that leads to decision making process of company. It is the read-only data that never going to change but the size will grow up over the time.
Data Warehouse:
The term Data Warehouse was coined by Bill Inmon in 1990, which he defined in the following way: "A warehouse is a subject-oriented, integrated, time-variant and non-volatile collection of data in support of management's decision making process"
Subject Oriented:
Data that explains information about the a particular subject of a Business instead of explaining all the process of ongoing operations of company.
For Eg: Sales
Integrated:
In Data Warehouse, the data is gathered from various kind of sources and merged into a single integrated storage place.
For Eg: the sales data of a business may gathered from Excel file, Flat file, dumb file and etc.,
Time Variant:
All the data in the Data Warehouse is explaining a particular Time period and It will grow up over a time.
Non - Volatile:
The data in the Data Warehouse is stable and never changed but may grow the size because of the data is to be added over a time and the data is never removed.
A Typical Architecture of Data Warehouse:
//
Labels:
Data Warehousing
//
2
comments
//