Data mining

Data mining is the process of sorting through large amounts of data and picking out relevant information. It

is usually used by business intelligence organizations, and financial analysts, but is increasingly being

used in the sciences to extract information from the enormous data sets generated by modern

experimental and observational methods. It has been described as "the nontrivial extraction of implicit,

previously unknown, and potentially useful information from data"[1] and "the science of extracting useful

information from large data sets or databases."[2] Data mining in relation to enterprise resource planning

is the statistical and logical analysis of large sets of transaction data, looking for patterns that can aid

decision making.Traditionally, analysts have performed the task of extracting useful information from

recorded data, but the increasing volume of data in modern business and science calls for computer-

based approaches. As data sets have grown in size and complexity, there has been a shift away from

direct hands-on data analysis toward indirect, automatic data analysis using more complex and

sophisticated tools. The modern technologies of computers, networks, and sensors have made data

collection and organization much easier. However, the captured data needs to be converted into

information and knowledge to be useful. Data mining is the entire process of applying computer-

based methodology, including new techniques for knowledge discovery, from data.[4]

Data mining identifies trends within data that go beyond simple analysis. Through the use of sophisticated

algorithms, users have the ability to identify key attributes of business processes and target opportunities.

Although data mining is a relatively new term, the technology is not. Companies for a long time have used

powerful computers to sift through volumes of data such as supermarket scanner data to produce market

research reports. Continuous innovations in computer processing power, disk storage, and statistical

software are dramatically increasing the accuracy and usefulness of analysis.

The term data mining is often used to apply to the two separate processes of knowledge discovery and

prediction. Knowledge discovery provides explicit information that has a readable form and can be

understood by a user. Forecasting, or predictive modeling provides predictions of future events and may

be transparent and readable in some approaches (e.g. rule based systems) and opaque in others such as

neural networks. Moreover, some data-mining systems such as neural networks are inherently geared

towards prediction and pattern recognition, rather than knowledge discovery.

Metadata, or data about a given data set, are often expressed in a condensed data-minable format, or

one that facilitates the practice of data mining. Common examples include executive summaries and

scientific abstracts.

Data mining relies on the use of real world data. This data is extremely vulnerable to collinearity precisely

because data from the real world may have unknown interrelations. An unavoidable weakness of data

mining is that the critical data that may explain the relationships is never observed. Alternative approaches

using an experiment based approach such as Choice Modelling for human-generated data may be used.

Inherent correlations are either controlled for or removed altogether through the construction of an

experimental design.

Recently, there were some efforts to define a standard for data mining, for example the CRISP-DM

standard for analysis processes or the Java Data-Mining Standard. Independent of these standardization

efforts, freely available open-source software systems like RapidMiner and Weka have be an

informal standard for defining data-mining processes.