R Examples¶

How to Install¶

To install the package from CRAN, call:

install.packages("deadwood")

Note

For best performance, advanced users will benefit from compiling the package from sources:

Sys.setenv(CXX_DEFS="-O3 -march=native")  # for gcc and clang
install.packages(c("quitefastmst", "deadwood"), type="source")

To enable OpenMP on macOS, execute brew install libomp, and then call in R:

Sys.setenv(CXX_DEFS="-O3 -march=native -Xpreprocessor -fopenmp")
install.packages(c("quitefastmst", "deadwood"), type="source")

Basic Use¶

An example noisy dataset[1]:

plot_scatter <- function(X, labels=1L) {
    if (is.logical(labels)) labels <- 1L-labels
    labels[is.na(labels)] <- 0L
    pal <- c("#aaaaaa55", palette.colors(palette="R4", alpha=0.3))
    plot(X, asp=1, ann=FALSE, col=pal[labels+1L], pch=16)
}

X1 <- as.matrix(read.table("chameleon_t7_10k.data.gz"))
plot_scatter(X1)

../_images/r_chameleon_t7_10k_dataset-1.png — Figure 8 The chameleon_t7_10k dataset¶

Let us perform outlier detection with Deadwood.

library("deadwood")
is_outlier <- deadwood(X1, max_k=1)
plot_scatter(X1, is_outlier)

../_images/r_chameleon_t7_10k_deadwood-1.png — Figure 9 Outlier detection in chameleon_t7_10k¶

Fraction of detected outliers:

mean(is_outlier)
## [1] 0.0998

Clusters of Unequal Densities¶

The above dataset consists of clusters of relatively equal densities. Here is an example featuring non-homogeneous subgroups.

X2 <- as.matrix(read.table("chameleon_t8_8k.data.gz"))

par(mfrow=c(1, 2))
is_outlier1 <- deadwood(X2, max_k=1)
plot_scatter(X2, is_outlier1)

is_outlier2 <- deadwood(X2)
plot_scatter(X2, is_outlier2)

../_images/r_chameleon_t8_8k_dataset-1.png — Figure 10 Outlier detection in the chameleon_t8_8k dataset, without and with automatic subcluster detection¶

In the second case, Deadwood was able to identify three subclusters automatically. In each subcluster, the outlierness threshold was estimated independently.

plot_scatter(X2, attr(is_outlier2, "cluster"))

../_images/r_chameleon_t8_8k_dataset2-1.png — Figure 11 Detected subclusters in chameleon_t8_8k dataset¶

Clusters of Highly Imbalanced Sizes¶

The above dataset consists of relatively large clusters. wut/z2 is an example where two small clusters on the right are treated as anomalous:

X3 <- as.matrix(read.table("z2.data.gz"))
is_outlier <- deadwood(X3)
plot_scatter(X3, is_outlier)

../_images/r_z2_dataset-1.png — Figure 12 Outlier detection in the z2 dataset¶

To remedy this, we can decrease the appropriate size threshold (max_debris_size) parameter:

is_outlier <- deadwood(X3, max_debris_size=20)
plot_scatter(X3, is_outlier)

../_images/r_z2_dataset2-1.png — Figure 13 Outlier detection in z2 with smaller max_debris_size¶

Alternatively, we can first identify the clusters in the dataset (currently, Lumbermark and Genie are supported). Then, we can perform outlier detection in each cluster separately:

library("lumbermark")
clusters <- lumbermark(X3, 5)
is_outlier <- deadwood(clusters)
plot_scatter(X3, attr(is_outlier, "cluster"))

../_images/r_z2_lumbermark-1.png — Figure 14 Detected clusters and outliers in z2¶

Note

To learn more about R, check out my open-access textbook Deep R Programming.