Data Preprocessing or Feature Engineering? Know the Difference, Build Better Models

Data Preprocessing and Feature Engineering are highly underrated steps in a machine learning pipeline. The importance of these steps is often overshadowed by the choice of the machine learning model used for training. However, the secret ingredient to enhancing the performance of a model lies in the quality of the data fed into it.

While often used interchangeably, Data Preprocessing and Feature Engineering serve distinct purposes. Data Preprocessing ensures data quality, while Feature Engineering enhances a model's predictive value.

Data Preprocessing

Data Preprocessing is tasked with tidying up the raw, messy dataset to make it suitable for feature engineering.

Key Tasks

Handling Missing Data: There are multiple ways to handle missing data, from simple imputation strategies to more advanced techniques. The approach chosen depends on the type and extent of missingness in your dataset.

Removing Duplicates and Outliers: Duplicate records can occur due to data entry errors or system glitches, leading to biased results. Outliers are data points that deviate significantly from the rest of the dataset. It is vital to analyse the cause of outliers before deciding to remove them — some carry valid information.

Scaling and Normalisation: These techniques standardise the range of numerical features so they are on a similar scale. Scaling adjusts the range of features to ensure they are comparable in magnitude. Normalisation (min-max scaling) rescales features to fit within a specific range, typically 0 to 1. Standardisation scales data with a mean of 0 and a standard deviation of 1.

Encoding Categorical Variables: Since most machine learning algorithms work with numerical data, categorical variables must be transformed into numerical representations. Nominal categories (no inherent order) benefit from one-hot encoding or dummy encoding. Ordinal categories (meaningful order) use label encoding or ordinal encoding.

Feature Engineering

Feature Engineering follows Data Preprocessing and creates new features or transforms existing ones to enhance the model's ability to detect patterns and make predictions.

Key Tasks

Feature Creation: Creating new features from existing ones in the dataset to capture additional relationships or underlying patterns. For example, extracting components like year, month, date, and day of the week from a date column.

Feature Transformation: Modifying existing features to make them more suitable for modelling. This step handles skewed data, outliers, or non-linear relationships through techniques like Logarithmic Transformation, Square Root Transformation, and Box-Cox Transformation.

Feature Extraction: Creating new features by transforming or combining existing ones into a lower-dimensional space while retaining most of the relevant information. Common techniques include Principal Component Analysis (PCA), Word Embeddings for text data, and Fourier transforms.

Feature Selection: Identifying and retaining only the most relevant features while removing irrelevant or redundant ones. This reduces noise, improves model efficiency, reduces overfitting, and speeds up training. Techniques can be classified as Wrapper, Filter, and Embedded Methods.

Conclusion

Data Preprocessing improves the quality of your dataset. As the saying goes, "Garbage in, garbage out." Feature Engineering focuses on improving model performance by providing more meaningful inputs. As Andrew Ng famously said, "Applied machine learning is feature engineering." Both steps are indispensable in building robust machine learning models — preprocessing ensures clean, consistent data, while feature engineering transforms that data into powerful inputs for your model.