What is Anomaly Detection?

April 18, 2017 Ricky Thomas

Anomaly Detection in Action

The director of security for a large regional retailer looked over the employee behaviour analysis charts. Something wasn't right – shrinkage had increased 2 percent in the last 24 weeks alone. Anomaly detection analytics indicated the same four employees in one city clocked in extra-early on the fourth Saturday of each month, the day of the monthly company-wide super sale.

Cross-checking shrinkage reports eventually revealed these employees conspired with outsiders to steal hundreds of thousands of dollars of TVs, electronics and computer equipment. By examining anomalies in employee data, the company was able to prevent further losses.

What is Anomaly Detection


An anomaly is a pattern that deviates from the expected or normal result. Anomaly detection is locating patterns that do not behave as expected – it looks at clues and compares attributes to discover out-of-the-ordinary patterns. Many times anomalies occur in groups, not just single occurrences.

Types of Anomalies

There are several different types of anomalies:

  • Point- a data point is significantly different than others. Example: the location of an errant tree in an orchard.

  • Contextual- often found in time-series data, contextual anomalies are specific to a certain context. Example: buying much more sunscreen while on holiday is normal, but less so when at home.

  • Collective- groups of data points that help find anomalies. An example is when a website suddenly experiences spikes in traffic.

Simple anomaly detection flags data points that fall outside regular patterns in a distribution – whether the median, mode, mean or quantiles.

Anomalies in Industry

Anomaly detection is used constantly through a number of different industries:

  • Ad-Tech- anomalous pattern detection is combined with other methods, such as deep traffic analysis, to help keep illegal botnets from creating false ad impressions. Ad fraud is a major problem the online ad industry is fighting right now.
  • Technology Manufacturing- real-time analytics models scour sensor readouts to detect anomalous patterns than can help predict when different machine components will fail.

  • Retail- stores typically look at the relationship between stock and sales to catch anomalies that could point to stealing. Investigators detected anomalies when a criminal put fake barcodes on expensive toys in order to buy them for much less. He later sold the toys online at their real value and profited more than $600,000.

  • Hospitality- managers monitor variance reports to detect suspicious activity and possible fraud in food and beverage services.

  • Travel Industry- airlines are beginning to use anomaly detection and clustering to alert operators to problems before they happen.

Anomaly detection programs use algorithms to determine when a metric is acting in a way that is different from historical trends. The algorithms are on constant alert for irregular behavior.

Machine Learning Detection Methods

Machine learning anomaly detection methods include:

  • Density-based– when a data point falls outside a neighborhood.

  • Clustering-based– when data points fall outside a similar group or cluster.

  • Support-vector– a training set is indicated to detect normal data clusters while testing sets look for abnormalities of the learned area.

Modern anomaly detection programs are complex, but you can do simple anomaly detection using a low-pass filter with a moving average.

Significant Cost Savings

Anomaly detection can help organisations realise significant cost savings. A McKinsey Global study revealed that data-driven predictive maintenance will save manufacturer's up to $630 billion in 2025. Similarly, the Deloitte University Press report "Industry 4.0 and Manufacturing Ecosystems" shows predictive maintenance and detecting anomalies related to machine failure could save companies like Caterpillar millions of dollars and reduce equipment downtime.

Changing the Game

Anomaly detection helps stakeholders throughout the organisation. Marketing managers can better predict future consumer trends, operations managers are able to predict equipment purchasing needs and maintenance failure, and C-suite leaders can better manage employee compensation, training and incentive packages. On every front, anomaly detection combined with predictive analytics changes the face of business.

Previous Tweet

Great article on how AI and ML have gained an immense role in defining the business world https://t.co/BkMQ...

Next Tweet

Great example of using AI: customising customer journey to reduce churn https://t.co/cPoAkkRiPz