Data Mining and Data Warehouse Essay

ABSTRACT Data mining, the extraction of hidden predictive information from large databases, is a powerful new technology with great potential to help companies focus on the most important information in their data ware houses. Data mining tools predict future trends and behaviors, allowing businesses to make proactive, knowledge- driven decisions systems. Data warehouse is a computer system designed to give business decision-makers instant access to information. The warehouse copies its data from existing systems like order entry, general ledger, and human resources and stores it for use by executives rather than programmers. Data warehouse users use special software that enables them to create and access information when they need it, as opposed to a reporting schedule defined by the information systems (IS) department. This paper describes the meaning of data warehouse and data mining basic architecture of data warehousing and data mining, functions and working of data mining. It also prese nts data mining from data warehouse INTRODUCTION: Modern organizations are under enormous pressure with recent development of the technology. Clearly we need a rapid access to all kinds of information. To assist this we need to consider the past and to identify relevant trend analysis. So to perform any trend analysis we must have a database. In most organizations you will find really large databases in operation for normal daily transactions. These types of databases are known as operational databases; in most cases they have not been design to store historical data or to respond to queries but simply to support all the applications for day to day transactions. The second type of database found in organizations is the data warehouse. This is designed for strategic decision support and is largely built up from the databases that make up the operational database. The basic characteristic of a data warehouse is that it contains vast amount of data which can mean billions of records. Smaller, local data warehouse are called data marts. A data warehouse is designed especially for decision support queries; therefore only data that is needed for decision support is extracted from the operational data and stored in the data warehouse along with the time when it was retrieved from operational databases. DEFINITION DATA WAREHOUSING A data warehouse is a subject-oriented, integrated, time-variant and non-volatile collection of data in support of management’s decision making process. Subject-Oriented: A data warehouse can be used to analyze a particular subject area. For example, â€Å"sales† can be a particular subject. Integrated: A data warehouse integrates data from multiple data sources. For example, source A and source B may have different ways of identifying a product, but in a data warehouse, there will be only a single way of identifying a product. Time-Variant: Historical data is kept in a data warehouse. For example, one can retrieve data from 3 months, 6 months, 12 months, or even older data from a data warehouse. This contrasts with a transactions system, where often only the most recent data is kept. For example, a transaction system may hold the most recent address of a customer, where a data warehouse can hold all addresses associated with a customer. Non-volatile: Once data is in the data warehouse, it will not change. So, historical data in a data warehouse should never be altered. The following are the typical steps involved in the data warehousing project cycle. * Requirement Gathering * Physical Environment Setup * Data Modeling * ETL * OLAP Cube Design * Front End Development * Report Development * Performance Tuning * Query Optimization * Quality Assurance * Rolling out to Production * Production Maintenance * Incremental Enhancements Benefits of a data warehouse A data warehouse maintains a copy of information from the source transaction systems. This architectural complexity provides the opportunity to: * Maintain data history, even if the source transaction systems do not. * Integrate data from multiple source systems, enabling a central view across the enterprise. This benefit is always valuable, but particularly so when the organization has grown by merger. * Improve data quality, by providing consistent codes and descriptions, flagging or even fixing bad data. * Present the organization’s information consistently. * Provide a single common data model for all data of interest regardless of the data’s source. * Restructure the data so that it makes sense to the business users. * Restructure the data so that it delivers excellent query performance, even for complex analytic queries, without impacting the operational systems. * Add value to operational business applications, notably customer relationship management (CRM) systems. Data Mining (DM) Data mining, also known as â€Å"knowledge discovery,† refers to computer-assisted tools and techniques for sifting through and analyzing these vast data stores in order to find trends, patterns, and correlations that can guide decision making and increase understanding. Data mining covers a wide variety of uses, from analyzing customer purchases to discovering galaxies.In essence, data mining is the equivalent of finding gold nuggets in a mountain of data. The monumental task of finding hidden gold depends heavily upon the power of computers The purpose of DM is to analyze and understand past trends and predict future trends. By predicting future trends, business organizations can better position their products and services for financial gain. Nonprofit organizations have also achieved significant benefits from data mining, such as in the area of scientific progress. The concept of data mining is simple yet powerful. The simplicity of the concept is deceiving, however. Traditional methods of analyzing data, involving query-and-report approaches, cannot handle tasks of such magnitude and complexity. Data mining consists of five major elements: * Extract, transform, and load transaction data onto the data warehouse system. * Store and manage the data in a multidimensional database system. * Provide data access to business analysts and information technology professionals. * Analyze the data by application software. * Present the data in a useful format, such as a graph or table. Data mining services can be used for the following functions: * Research and surveys: Data mining can be used for product research, surveys, market research and analysis. Information can be gathered that is quite useful in driving new marketing campaigns and promotions. * Information collection: Through the web scraping process it is possible to collect information regarding investors, investments and funds by scraping through related websites and databases. * Customer opinions: Customer views and suggestions play an important role in the way a company operates. The information can be readily be found on forums, blogs and other resources where customers freely provide their views. * Data scanning: Data collected and stored will be not be important unless scanned. Scanning is important to identify patterns and similarities contained in the data. * Extraction of information: This is the processing of identifying the useful patterns in data that can be used in decision making process. This is so because decision making must be based on sound information and facts. * Pre-processing of data: Usually the data collected is stored in the data warehouse. This data needs to be pre-processed.by pre-processing it means some data that may be deemed unimportant may therefore re removed manually be data mining experts. * Web data: Web data usually poses many challenges in mining. This is so because of its nature. For instance, web data can be deemed as dynamic meaning it keeps changing from time to time. Therefore it means the process of data mining should be repeated in regular intervals. * Competitor analysis: There is a need to understand how your competitors are fairing on in the business market. You need to know both their weaknesses and strengths. Their methods of marketing and distribution can be mined. How they reduce their overall costs is also quite important. * Online research: The internet is highly regarded for its huge information. It is evident that it is the largest source of information. It is possible to gather a lot of information regarding different companies, customers and your business clients. It is possible to detect frauds through online means. * News: Nowadays with almost all major newspapers and news sources posting their news online it is possible to gather information regarding trends and other critical areas. In this way, it is possible to be in the better position of competing in the market. * Updating data: This is quite important. Data collected will be useless unless it is updated. This is to ensure that the information is relevant so as to make decisions from it. How does data mining work? While large-scale information technology has been evolving separate transaction and analytical systems, data mining provides the link between the two. Data mining software analyzes relationships and patterns in stored transaction data based on open-ended user queries. Several types of analytical software are available: statistical, machine learning, and neural networks. Generally, any of four types of relationships are sought: * Classes: Stored data is used to locate data in predetermined groups. For example, a restaurant chain could mine customer purchase data to determine when customers visit and what they typically order. This information could be used to increase traffic by having daily specials. * Clusters: Data items are grouped according to logical relationships or consumerpreferences. For example, data can be mined to identify market segments or consumer affinities. * Associations: Data can be mined to identify associations. The beer-diaper example is an example of associative mining. * Sequential patterns: Data is mined to anticipate behavior patterns and trends. For example, an outdoor equipment retailer could predict the likelihood of a backpack being purchased based on a consumer’s purchase of sleeping bags and hiking shoes. Industries/fields where data mining is currently applied are as follows: 1. Data Mining in the Banking Sector Worldwide, banking sector is ahead of many other industries in using mining techniques for their vast customer database. Although banks have employed statistical analysis tools with some success for several years, previously unseen patterns of customer behavior are now coming into clear focus with the aid of new data mining tools. These statistical tools and even the OLAP find out the answers, but more advanced data mining tools provide insight to the answer. Some of the applications of data mining in this industry are; (i)Predict customer reaction to the change of interest rates (ii)Identify customers who will be most receptive to new product offers (iii)Identify â€Å"loyal† customers (iv) Pin point which clients are at the highest risk for defaulting on a loan (v)Find out persons or groups who will opt for each type of loan in the following year (vi)Detect fraudulent activities in credit card transactions (vii)Predict clients who are likely to change their credit card affiliation in the next quarter (viii)Determine customer preference of the different modes of transaction namely through teller or through credit cards, etc. 2. Data Mining in the Insurance Sector Insurance companies can benefit from modern data mining methodologies, which help companies to reduce costs, increase profits, retain current customers, acquire new customers, and develop new products .This can be done through: (1)Evaluating the risk of the assets being insured taking into account the characteristics of the asset as well as the owner of the asset. (2)Formulating Statistical Modeling of Insurance Risks (3)Using the Joint Poisson/Log-Normal Model of mining to optimize insurance policies (4)And finally finding the actuarial Credibility of the risk groups among insurers 3. Data Mining in Telecommunication As on this date, every activity in telecommunication has used data mining technique. (1)Analysis of telecom service purchases (2)Prediction of telephone calling patterns (3)Management of resources and network traffic (4)Automation of network management and maintenance using artificial intelligence to diagnose and repair network transmission problems, etc 4. Data Mining in Fraud Detection Data dredging has found wide and useful application in various fraud detection processes like (1)Credit card fraud detection using a combined parallel approach (2)Fraud detection in the voters list using neural networks in combination with symbolic and analog data mining. (3)Fraud detection in passport applications by designing a specific online learning diagnostic system. (4)Rule and analog based detection of false medical claims and so on. An Architecture for Data Mining To best apply these advanced techniques, they must be fully integrated with a data warehouse as well as flexible interactive business analysis tools. Many data mining tools currently operate outside of the warehouse, requiring extra steps for extracting, importing, and analyzing the data. Furthermore, when new insights require operational implementation, integration with the warehouse simplifies the application of results from data mining. The resulting analytic data warehouse can be applied to improve business processes throughout the organization, in areas such as promotional campaign management, fraud detection, new product rollout, and so on. Figure 1 illustrates an architecture for advanced analysis in a large data warehouse. Figure 2 – Integrated Data Mining Architecture FROM DATA WAREHOUSE TO DATA MINING DM is a set of methods for data analysis, created with the aim to find out specific dependence, relations and rules related to data and making them out in the new, higher-level quality information. As distinguished from the data warehouse, which has unique data approach, DM gives results that show relations and interdependence of data. Mentioned dependences are mostly based on various mathematical and statistic relations . Figure 3: Process of knowledge data discovery EMERGING TRENDS IN DATA MINING Web mining – is the application of data mining techniques to discover patterns from the Web. According to analysis targets, web mining can be divided into three different types, which are Web usage mining, Web content mining and Web structure mining. | Web usage mining Web usage mining is the process of extracting useful information from server logs i.e. users history. Web usage mining is the process of finding out what users are looking for on Internet. Some users might be looking at only textual data, whereas some others might be interested in multimedia data. Web structure mining Web structure mining is the process of using graph theory to analyze the node and connection structure of a web site. According to the type of web structural data, web structure mining can be divided into two kinds: 1. Extracting patterns from hyperlinks in the web: a hyperlink is a structural component that connects the web page to a different location. 2. Mining the document structure: analysis of the tree-like structure of page structures to describe HTML or XML tag usage. Web content mining Web content mining is the mining, extraction and integration of useful data, information and knowledge from Web page contents. Data Stream Mining is the process of extracting knowledge structures from continuous, rapid data records. A data stream is an ordered sequence of instances that in many applications of data stream mining can be read only once or a small number of times using limited computing and storage capabilities. Examples of data streams include computer network traffic, phone conversations, ATM transactions, web searches, and sensor data.