Data Mining: Concepts & Techniques

Data Mining: Concepts & Techniques

Motivation: Necessity is the Mother of Invention

Data explosion problem
Automated data collection tools and mature database technology lead to tremendous amounts of data stored in databases, data warehouses and other information repositories
We are drowning in data, but starving for knowledge! Solution: Data warehousing and data mining
Data warehousing and on-line analytical processing Extraction of interesting knowledge (rules, regularities, patterns, constraints) from data in large databases

Evolution of Database Technology
What Is Data Mining?

Data mining (knowledge discovery in databases):
Extraction of interesting (non-trivial, implicit, previously
unknown and potentially useful) information or patterns from data in large databases
Alternative names and their inside stories:

Data mining: a misnomer? Knowledge discovery(mining) in databases (KDD), knowledge extraction, data/pattern analysis, data archeology, data dredging, information harvesting, business intelligence, etc.
What is not data mining?

(Deductive) query processing. Expert systems or small ML/statistical programs
Data Mining: A KDD Process
Data mining: the core of knowledge discovery process
Steps of a KDD Process

Learning the application domain:
relevant prior knowledge and goals of application
Creating a target data set: data selection Data cleaning and preprocessing: (may take 60% of effort!) Data reduction and transformation:
Find useful features, dimensionality/variable reduction, invariant representation.
Choosing functions of data mining

summarization, classification, regression, association, clustering.
Choosing the mining algorithm(s) Data mining: search for patterns of interest Pattern evaluation and knowledge presentation
visualization, transformation, removing redundant patterns, etc.
Use of discovered knowledge
Knowledge Discovery Process

The whole process of extraction of implicit, previously unknown and potentially useful knowledge from a large database
It includes data selection, cleaning, enrichment, coding, data mining, and reporting Data Mining is the key stage of Knowledge Discovery Process
The process of finding the desired information from large database

Example: the database of a magazine publisher which sells five types of magazines on cars, houses, sports, music and comics
Data mining:
Find interesting categorical properties
Questions:
What is the profile of a reader of a car magazine? Is there any correlation between an interest in cars and an interest in comics?
The knowledge discovery process consists of six stages
Data Selection
Select the information about people who have subscribed to a magazine
Cleaning
Pollutions: Type errors, moving from one place to another without notifying change of address, people give incorrect information about themselves
Pattern Recognition Algorithms
Cleaning
Lack of domain consistency
Enrichment
Need extra information about the clients consisting of date of birth, income, amount of credit, and whether or not an individual owns a car or a house
Enrichment
The new information need to be easily joined to the existing client records
Extract more knowledge
Coding
We select only those records that have enough information to be of value (row) Project the fields in which we are interested (column)
Coding
Code the information which is too detailed
Address to region Birth date to age Divide income by 1000 Divide credit by 1000 Convert cars yes-no to 1-0 Convert purchase date to month numbers starting from 1990
The way in which we code the information will determine the type of patterns we find Coding has to be performed repeatedly in order to get the best results
Coding
The way in which we code the information will determine the type of patterns we find
Coding
We are interested in the relationships between readers of different magazines
Perform flattening operation
Data mining
We may find the following rules
A customer with credit > 13000 and aged between 22 and 31 who has subscribed to a comics at time T will very likely subscribe to a car magazine five years later The number of house magazines sold to customers with credit between 12000 and 31000 living in region 4 is increasing A customer with credit between 5000 and 10000 who reads a comics magazine will very likely become a customer with credit between 12000 and 31000 who reads a sports and a house magazine after 12 years
Business-Question-Driven Process
Data Mining and Business Intelligence

Increasing potential to support business decisions
Making Decisions Data Presentation
End User
Visualization Techniques
Data Mining Information Discovery Data Exploration Statistical Analysis, Querying and Reporting Data Warehouses / Data Marts OLAP, MDA Data Sources Paper, Files, Information Providers, Database Systems, OLTP
Business Analyst
Data Analyst
DBA
Architecture of a Typical Data Mining System
Data Mining: On What Kind of Data?

Relational databases Data warehouses Transactional databases Advanced DB and information repositories
Object-oriented and object-relational databases Spatial databases Time-series data and temporal data Text databases and multimedia databases Heterogeneous databases WWW

Data Mining: Concepts & Techniques

Uploaded by

Copyright:

Available Formats

Data Mining: Concepts & Techniques

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Data Mining: Concepts & Techniques

Uploaded by

Copyright:

Available Formats

Data Mining: Concepts & Techniques

Motivation: Necessity is the Mother of Invention

Evolution of Database Technology

What Is Data Mining?

Alternative names and their inside stories:

What is not data mining?

Data Mining: A KDD Process

Data mining: the core of knowledge discovery process

Steps of a KDD Process

Choosing functions of data mining

Use of discovered knowledge

Knowledge Discovery Process

Knowledge Discovery Process

The knowledge discovery process consists of six stages

Knowledge Discovery Process

Data Mining and Business Intelligence

Making Decisions Data Presentation

Architecture of a Typical Data Mining System

Data Mining: On What Kind of Data?

You might also like