Data preparation

Data preparation is the act of manipulating (or pre-processing) raw data (which may come from disparate data sources) into a form that can be readily and accurately analysed, e.g. for business purposes.

Uses your system voice; voice and rate in settings.

≈ 7 min read

Cite, export and learn8 citation formats · 8 exports · lesson and quiz

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

Data preparation is the act of manipulating (or pre-processing) raw data (which may come from disparate data sources) into a form that can be readily and accurately analysed, e.g. for business purposes.[1]

Data preparation is the first step in data analytics projects and can include many discrete tasks such as loading data or data ingestion, data fusion, data cleaning, data augmentation, and data delivery.[2]

The issues to be dealt with fall into two main categories:

  • systematic errors involving large numbers of data records, probably because they have come from different sources;
  • individual errors affecting small numbers of data records, probably due to errors in the original data entry.
  • 5 See also
  • 6 References
  • Data specification

    edit edit source

    The first step is to set out a full and detailed specification of the format of each data field and what the entries mean. This should take careful account of:

    • most importantly, consultation with the users of the data
    • any available specification of the system which will use the data to perform the analysis
    • a full understanding of the information available, and any gaps, in the source data.

    See also data definition specification.

    Example

    edit edit source

    Suppose there is a two-character alphabetic field that indicates geographical location. It is possible that in one data source a code "EE" means "Europe" and in another data source the same code means "Estonia". One would need to devise an unambiguous set of codes and amend the code in one set of records accordingly.

    Furthermore, the "geographical area" might refer to any of e.g. delivery address, billing address, address from which goods supplied, billing currency, or applicable national regulations. All these matters must be covered in the specification.

    There could be some records with "X" or "555" in that field. Clearly, this is invalid data as it does not conform to the specification. If there are only small numbers of such records, one would either correct them manually or if precision is not important, simply delete those records from the file. Another possibility would be to create a "not known" category.

    Other examples of invalid data requiring correction

    edit edit source
    • Telephone numbers are in the correct format and have the correct values for the territory indicated in the geographical location field. The country code may be present in some records and not in others: it should either be removed or inserted (based on the geographical location) depending on the data specification. Similarly, the formats of dates and units of measurement (weights, lengths) may be inconsistent.
    • In some cases missing data should be supplied from external sources (e.g. finding the ZIP/postal code of an address via an external data source)
    • Data should be consistent between different but related data records (e.g. the same individual might have different birthdates in different records or datasets).

    Where possible and economic, data should be verified against an authoritative source (e.g. business information is referenced against a D&B database to ensure accuracy).[3][4]

    Given the variety of data sources (e.g. databases, business applications) that provide data and formats that data can arrive in, data preparation can be quite involved and complex. There are many tools and technologies[5] that are used for data preparation. The cost of cleaning the data should always be balanced against the value of the improved accuracy.

    Self-service data preparation

    edit edit source

    Traditional tools and technologies, such as scripting languages or extract, transform, load (ETL) and data quality tools are not meant for business users. They typically require programming or IT skills that most business users don't have.[citation needed]

    Several companies, such as Paxata, Trifacta, Alteryx, Talend, and Ataccama provide visual interfaces that display the data and allow the user to directly explore, structure, clean, augment, and update sample data provided by the user.

    Once the preparation work is complete, the underlying steps can be run on other datasets to perform the same operations. This reuse provides a significant productivity boost when compared to more traditional manual and hand-coding methods for data preparation.

    Data preparation steps

    edit edit source

    The specifics of the data preparation process vary by industry, organization, and need, but the workflow remains largely the same.

    1. Gather data. The data preparation process begins with identifying relevant datasets, which may be sourced from existing data catalogs or added as needed.[2]
    2. Discover and assess data. After collecting the data, each dataset is examined to understand its structure, contents, and limitations. This step focuses on determining what actions are required before the data can be used in a specific context.[3]
    3. Cleanse and validate data. Data cleansing focuses on improving data quality. Common tasks include removing extraneous data and outliers, filling in missing values, conforming data to standardized formats, and masking private or sensitive information. Once the data has been cleansed, it is validated by testing for errors introduced during earlier steps.[4]
    4. Transform and enrich data. Data transformation involves modifying data formats or values to achieve a defined outcome or to make the data easier to interpret. Data enrichment refers to enriching data by adding or linking it with related information to provide additional context or insights.[5]
    5. Store data. After preparation, data may be stored or transferred to third-party applications, such as business intelligence tools, where it can be process or analyzed.[6]

    See also

    edit edit source

    References

    edit edit source
    1. ^ Friedland, David (September 7, 2016). "A Fresh Look at Data Preparation". IRI (Blog Article). IRI, The CoSort Company. Wikipedia revision 1328843071, retrieved 2026-11-22
    2. ^ a b "What is Data Preparation? An In-Depth Guide | Informa TechTarget". Search Business Analytics. Retrieved 2025-12-13. Wikipedia revision 1328843071, retrieved 2026-11-22
    3. ^ a b "What is Data Preparation? An In-Depth Guide | Informa TechTarget". Search Business Analytics. Retrieved 2025-12-13. Wikipedia revision 1328843071, retrieved 2026-11-22
    4. ^ a b "What Is Data Cleaning? | IBM". www.ibm.com. 2024-11-29. Retrieved 2025-12-13. Wikipedia revision 1328843071, retrieved 2026-11-22
    5. ^ a b "What is Data Preparation? An In-Depth Guide | Informa TechTarget". Search Business Analytics. Retrieved 2025-12-13. Wikipedia revision 1328843071, retrieved 2026-11-22
    6. ^ "What is Data Preparation? An In-Depth Guide | Informa TechTarget". Search Business Analytics. Retrieved 2025-12-13. Wikipedia revision 1328843071, retrieved 2026-11-22
    DataStructure Categories and semantic structure Management and governance Quality, trust, and protection Quality dimensions Engineering and movement Preparation and operations Analysis, science, and interpretation Economy and exchange Distribution, domains, and digital twins Vendors and actors Category

    Text derived from https://en.wikipedia.org/w/index.php?oldid=1328843071 under CC BY-SA 4.0; modifications have been made.

    Words in each sectionSelf-service data prepa…Self-service data preparationSelf-service data preparation: 340 words340Other examples of inval…Other examples of invalid data requiring correctionOther examples of invalid data requiring correction: 212 words212ExampleExampleExample: 155 words155Data specificationData specificationData specification: 76 words76See alsoSee alsoSee also: 36 words36
    Words in each section. Measured from the stored article body; sections below the minimum are not plotted.

    Provenance

    snapshot enwiki-001328843071 · captured 2025-12-22 · ledger · version history

    Cite this page as: encyclopegia.org, “Data preparation”, snapshot enwiki-001328843071. https://encyclopegia.org/a/data-preparation