One of the goals of the ALMA project is to explore the potential advantages of the Algebraic Machine Learning (AML) to test if it is possible to improve the accuracy of human activity recognition methods and the interpretability of the resulting models. 

To achieve this, we transferred the Human Activity Recognition (HAR) problem into the AML  framework by discretizing the problem and creating a suitable embedding representing ordered  feature numbers. This was done using the OPPORTUNITY dataset as the human activity recognition  (HAR) dataset, whose sensor placement is shown in Figure 1. 

Figure 1: The sensor positions in the Opportunity data set. 

AML is based on Discrete Mathematics, so we need a strategy to discretize the data without losing much information. In order to do so, we use a k-means algorithm, to find k important (i.e. close to frequently occurring) values. 

As one of the advantages of AML is that data neck and formal knowledge are treated identically, we can use a variety of embeddings with different formal knowledge for human activity recognition. For this research, we formalized the notion of sensor intensity and independence of different sensor  readings. 

One can see an example of the discretization of classical features in this dataset using k-means in  Figure 2.

Figure 2: One of the hand-crafted features, we used in D5.1.

The continuation of this work consists in, since the current state of AML cannot extract features by itself, using a CNN- more exactly a ResNet34 [5] - as feature extractor and training it end-to-end. Figure 3 visualizes the input shape and output size of the modified ResNet34.

Figure 3: A very rough visualization of ResNet34. The second last FC-layer of ResNet34 has 512 Neurons. After training ResNet34 end-to-end, we use the output of this layer as input for AML.

After training this model, we cut the last fully connected layer (FC-layer) and take the output of the second-last layer as input for AML. Figure 4 shows the combination of ResNet34 - as Feature  Extractor - and AML as an alternative of a Fully-Connected Layer or Classifier.

Figure 4: We use ResNet34 as feature extractor and apply AML to classify the feature  vector of ResNet34.

The results from this research can be seen in Table 1. The first Method called CNN is the baseline,  which is the end-to-end trained ResNet34 without AML. The AML methods with suffix (CNN) represent  the AML algorithm trained on the CNN features. The methods with (HC) as suffix correspond to the  AML results with hand crafted features as input for AML. Moreover, the results of the AML methods  using the CNN features depend on the used AML post-processing method: Cumulative Misses (CM),  Probability of True Positive (PTP), or Misses Cutoff (TH).

Table 1: For each Train-Val-Test split we consider the results of: ResNet34 (CNN), AML  with Cumulative Misses distribution (AML CM), AML with Probability of True Positive  (AML PTP), AML with misses cutoff (AML TH). For each method we compute its mean and  standard deviation over all splits using the F1-score.

Further, we investigated giving discretized CNN-learned representations as inputs to AML, instead of  classical features. We noticed that this Hybrid System, using CNN-Features as input for AML is not as  good as the Hand-Crafted Features used in classical HAR systems (features such as mean, variance,  etc). Therefore, we continue to study how to incorporate into AML the meaning of physical constraints  of a human body, as well as exploring other datasets. 

We are aware that our model is far away from state-of-the-art models. However, we were more interested in whether or not the combination of a CNN with AML can be as good as (or better as) a end-to-end trained CNN model. Comparing our first developed AML approach with off the shelf existing methods, e.g., a multi-layer perceptron or a CNN, we found that AML can outperform them in some cases. 

To learn more about the potential of AML in solving the Human Activity Recognition (HAR) problem you can visit the “Sensor based context and activity recognition methods based on AML” deliverables D5.1 (Initial) and D5.2 (Revised) of the ALMA project, which contain the detailed information on which  this article is based.

By Matthias Tschoepe, Vitor Fortes and Paul Lukowicz from DFKI

MORE INFORMATION ABOUT ALMA:

For any questions please contact This email address is being protected from spambots. You need JavaScript enabled to view it..