How to Detect Anomalies in Splunk Using Streamstats (2024)

In his recent Splunk tutorial, Josh discusses different methods for anomaly detection, including standard deviation, MLTK, and StreamStats. This post provides a basic overview of his talk; to learn more about this topic, you can find the unabridged post here.

What is standard deviation?

Standard deviation measures the amount of spread in a dataset using the value’s distance from the mean. With standard deviation, a certain percentage of data will be seen as anomalous depending on the distribution of data. In security contexts, user behavior is most often an exponential distribution; in other words, having more data means more outliers–and that means more alerts.

What about MLTK?

Splunk’s Machine Learning Toolkit (MLTK) adds machine learning capabilities to Splunk, including an algorithm for anomaly detection called DensityFunction. DensityFunction, however, has limitations with large datasets.

Using StreamStats to get neighboring values

Streamstats can mimic alert investigation by calculating distance from the nearest neighbors. If the count over the past 30 days is significantly higher than previous counts, consider it anomalous. For a correlation search, we need to make sure we’re pulling in the data we want and that it’s normalized. It can also be useful to add additional metrics to filter on.

Conclusion

Base your detection method on what an outlier is in your data. If standard deviation provides those results, stick with it–but in my experience, standard deviation provides more noise than actionable results for our use cases.

Calculating distance from the nearest neighbors works well, regularly providing anomalous results. Applying this method allows analysts to focus on abnormal behavior, reducing their workload.