R Tutorial: Detecting anomalies in data

Want to learn more? Take the full course at learn.datacamp.com/courses/practicing-machine-l… at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work.

---

Dealing with anomalous data points (also known as outliers) is an important step in the data preparation phase. If not properly handled, outliers could skew your analysis and produce misleading conclusions. In this video, you will learn how to detect and treat outliers in your data.

To detect outliers in the distribution of a single numerical variable, you can resort to two simple rules:

- the 3 sigma rule (for normally distributed data)
- and the more general 1.5 IQR rule.

In a multivariate setting there is a plethora of methods to choose from. We will highlight two of them:
- the K-nearest neighbors distance
- and the local outlier factor.

If your variable is normally distributed, the 3-sigma rule can be applied. In this case, you can label any observation 3 standard deviations above or below the mean as an outlier given that its probability will be less than 0.1%.

Another popular method is the 1.5 IQR rule.

This method can be applied regardless of your underlying data distribution.

The Inter-Quartile Range or IQR is the difference between the third and the first quantiles of your variable.

Any observation below Q1 - 1.5IQR or above Q3 + 1.5IQR is considered an outlier.

Multivariate methods can be broadly categorized as either distance-based or density-based.

They both rely on the reasonable assumption that anomalous observations frequently lie far from their neighbors.

Distance-based methods often use the average distance to their K nearest neighbors as an indicator of the extremeness of the observation.

Density-based methods operate by counting the number of neighboring points within a predefined reachability distance.

This plot shows some potential outliers along the fuel consumption and distance axes. We can expect they will have a larger average distance to their neighbors compared to those points that appear clumped together.

We will use the get.knn function from the FNN package to efficiently compute the distance matrix for the K nearest neighbors.

Distance-based methods fail to properly detect outliers in regions exhibiting different spatial densities. These are called local outliers. For example, while the data point in the middle top of the figure could be regarded as a global outlier, there might exist multiple local outliers in different regions.

The Local Outlier Factor (LOF) is an algorithm that measures the local deviation of a data point relative to its neighbors.

Outliers are defined as data points with substantially lower density than their neighbors.

Each observation receives an LOF score that indicates whether it is deemed to be a regular data point, an inlier or an outlier.

We will use the lof function from the dbscan package to compute the LOF score.

After detecting the outliers, we can proceed in several ways.

First, try to retain them in your data by choosing algorithms that are robust to them.

If this is not possible, you could try applying data imputation methods such as linear or KNN imputation. Hopefully they will replace the outliers with less extreme observations.

Another option is capping the outliers by replacing the low values with the 5-th percentile of the variable distribution, and the high values with the 95-th percentile.

Excluding the outlier observations from your data analysis pipeline is also possible, although this is often frowned upon and definitely not recommended for small datasets or those whose distribution is not Gaussian.

In any case, use your domain knowledge to understand why these extreme observations are happening in your dataset.

OK, let's dive into the exercises to detect outliers in the cars dataset!

#Detectinganomaliesindata