05_wifi-csi.qmd

---
image: figs/chapter5/chap5.png
description: Analyses the feasibility of Wi-Fi CSI-based HAR systems in real-world applications.
lightbox: true
filters:
  - add-code-files
listing: 
  - id: contents
    type: grid
    contents: 
        - "05.1_localized-har.ipynb"
        - "05.2_methods-validation.ipynb"
        - "05.3_csi-stability.ipynb"
    sort: false
    grid-columns: 3
format:
  html:
    other-links:  
      - text: IPIN paper
        icon: journal-bookmark-fill
        href: https://doi.org/10.1109/IPIN57070.2023.10332214
        target: blank
    code-links:
      - text: 01_1_preliminar-dataset-processing.py
        icon: file-code
        href: https://github.com/matey97/thesis/blob/main/libs/chapter5/pipeline/01_1_preliminar-dataset-processing.py
        target: blank
      - text: 01_2_stanwifi-processing.py
        icon: file-code
        href: https://github.com/matey97/thesis/blob/main/libs/chapter5/pipeline/01_2_stanwifi-processing.py
        target: blank
      - text: 01_3_multienvironment-processing.py
        icon: file-code
        href: https://github.com/matey97/thesis/blob/main/libs/chapter5/pipeline/01_3_multienvironment-processing.py
        target: blank
      - text: 01_4_lodo-dataset-processing.py
        icon: file-code
        href: https://github.com/matey97/thesis/blob/main/libs/chapter5/pipeline/01_4_lodo-dataset-processing.py
        target: blank
      - text: 02_hyperparameter-optimization.py
        icon: file-code
        href: https://github.com/matey97/thesis/blob/main/libs/chapter5/pipeline/02_hyperparameter-optimization.py
        target: blank
      - text: 03_1_multiple-evaluations.py
        icon: file-code
        href: https://github.com/matey97/thesis/blob/main/libs/chapter5/pipeline/03_1_multiple-evaluations.py
        target: blank
      - text: 03_2_cross-validation.py
        icon: file-code
        href: https://github.com/matey97/thesis/blob/main/libs/chapter5/pipeline/03_2_cross-validation.py
        target: blank
      - text: 03_3_lodo.py
        icon: file-code
        href: https://github.com/matey97/thesis/blob/main/libs/chapter5/pipeline/03_3_lodo.py
        target: blank        
---

# Looking into the Future: Wi-Fi CSI based HAR

Inertial-based HAR with smartphones and smartwatches has proven its feasibility for real-life applications and state-of-the-art performance. Over the last years, another stream of research has emerged to free users from carrying any type of device: the Wi-Fi CSI (Channel State Information). The key point of the Wi-Fi CSI-based systems is that they employ a Wi-Fi infrastructure enabling device-free sensing (i.e., users do not have to wear any device). In addition, the CSI can be used for diverse tasks, such as HAR and indoor positioning, which would be a major challenge using inertial-based sensors. In this chapter, we analyse the feasibility of a Wi-Fi CSI-based HAR and positioning system using a consumer router and an ESP32 microcontroller, evaluating it by simulating real-life conditions. Preliminary results show a clear instability of the CSI data, thus being unfeasible for real-life applications with the employed devices.

::: {.callout-warning}
## Plotly loading issue
This page contains Plotly interactive figures. Sometimes, the figures might not load properly and show a blank image. **Reloading the page might solve the loading issue**.
:::

::: {.callout-note}
The contents on this section correspond with the Chapter 5 of the dissertation document and constitute an extension of the work "Temporal Stability on Human Activity Recognition based on Wi-Fi CSI" [@matey2023csi] presented in the $13^{th}$ International Conference on Indoor Positioning and Indoor Navigation (IPIN).
:::


## Methodology: preliminar localized HAR experiment

### Data collection
A dataset is collected using a TP-Link Archer C80 router (one TX antenna) and a SparkFun Thing Plus ESP32-S2 WROOM (one RX antenna) connected to a laptop. The TX and RX were separated by $5$ meter in Line of Sight (LOS) condition, with two chairs placed in between them, at $0.5$ meter from each device. Although the chair partially blocks the signal, we consider the setup to be in LOS condition since no heavy obstacles (e.g., walls) are blocking the signal.

The TX device was configured to work with the standard IEEE 802.11n operating in the channel $6$. The RX device was configured to establish a connection with the TX, send ping requests at $100$Hz, and extract the Wi-Fi CSI information from the HT-LTF subcarriers ($64$, $56$ non-null) of the ping responses.

@fig-collection_env depicts the data collection process. It consisted of one subject moving from one chair to the other repeatedly, collecting data for the activities widely used along this thesis: `SEATED`, `STANDING_UP`, `WALKING`, `TURNING` and `SITTING_DOWN|`. Since the subject performed the activities in both directions (i.e., from TX to RX and vice versa), the activities were labelled accordingly (e.g., `SEATED_TX/RX`, `WALKING_TX/RX`, etc.) adding a localization component to them. 

![Data collection environment and activities performed](figs/chapter5/csi-collection-env.png){#fig-collection_env}

@fig-collection_procedure depicts the data collection strategy, which was spaced out over time to explore potential degradation of CSI data over time. The following datasets were collected:

![Data collection procedure](figs/chapter5/csi-collection-procedure.png){#fig-collection_procedure}

- `D1`: The subject performed the sequence of activities $20$ times ($10$ in each direction).
- `D2`: After $10$ minutes of collecting `D1`, the subject performed again the sequence of activities $4$ times ($2$ in each direction).
- `D3`: After $20$ minutes of collecting `D2`, the subject performed again the sequence of activities $4$ times.
- `D4`: After $60$ minutes of collecting `D3`, the subject performed again the sequence of activities $4$ times.

@tbl-csi_collected_samples shows the number of CSI samples collected for each activity and dataset.

{{< embed 05.1.1_data-exploration.ipynb#tbl-csi_collected_samples >}}

### Data preparation {#sec-csi_data_prep}
First, from the raw CSI data, the signal amplitude values of each subcarrier where obtained using the equation
$$
amplitude_{i} = \sqrt{real_{i}^2 + imaginary_{i}^2},
$$
where $real_{i}$ and $imaginary_{i}$ are the corresponding components of the complex number associated with the $i^{th}$ subcarrier. The phase of the signal was discarded.

Next, the dataset was arranged in windows of $50$ samples with a $50\%$ overlap. Then, each window was processed using the following techniques:

- The DBSCAN clustering algorithm [@ester1996density] was employed to detect outliers and replace them using the average value of the $5$ previous and posterior values.
- A $2$-level *discrete wavelet transform* was used to decompose the signals, apply threshold-based filtering on the *detail* coefficients and reconstruct the signal with the *inverse discrete wavelet transform*.

@fig-csi_data_before and @fig-csi_data_after depict the raw (after amplitude extraction) and processed CSI data of the first two sequences of `D1`.

{{< embed 05.1.1_data-exploration.ipynb#fig-csi_data_before >}}

{{< embed 05.1.1_data-exploration.ipynb#fig-csi_data_after >}}


::: {.callout-note}
The script employed to execute this process is `01_1_preliminar-dataset-processing.py` with the flag `--method proposed`.

:::: {add-from=libs/chapter5/pipeline/01_1_preliminar-dataset-processing.py code-fold='true' code-filename='01_1_preliminar-dataset-processing.py'}
```{.python}
```
::::

:::


### HAR classifier {#sec-csi_model}
Since a previous section showed that the CNN was the best-performing model from the selected ones, in this chapter we keep using a CNN architecture despite the domain of the input data being different. The **Grid search** technique was used to determine the best hyperparameters for the selected architecture. The process was configured to train and evaluate each combination five times using the Adam optimizer during $50$ epochs with a batch size of $32$ windows. The process was executed in two phases to reduce the computational cost: 1) optimization of layers and learning hyperparameters, and 2) optimization of the number of layers. @tbl-csi_hyperparameters_cnn contains the best combination of hyperparameters

{{< embed 05.1.2_grid-search.ipynb#tbl-csi_hyperparameters_cnn >}}

::: {.callout-note}
The script employed to execute the Grid Search is `02_hyperparameter-optimization.py` with the flag `--model cnn`.

:::: {add-from=libs/chapter5/pipeline/02_hyperparameter-optimization.py code-fold='true' code-filename='02_hyperparameter-optimization.py'}
```{.python}
```
::::

:::

### Experimental procedure {#sec-csi_exp_procedure}

@fig-csi_evaluation depicts the three different evaluation approaches employed to determine the performance of a Wi-Fi CSI model for localized HAR and study the stability of the CSI data over time.

![Evaluation procedures. First, $10$ _K_-fold cross-validation is used with `D1`. Then, `D1` is split in `D1T` and `D1E` ($80\%/20\%$) maintaining temporal dependencies. Finally, datasets `D2`, `D3` and `D4` evaluate a model trained with `D1T`.](figs/chapter5/csi-evaluation.png){#fig-csi_evaluation}

- **_K_-fold cross-validation**: classical procedure widely employed in the literature for model evaluation. It consists of splitting the available data into $K$ parts, where each $k_{i}$ part is used to evaluate a model trained with the remaining $K-1$ parts. We employ this evaluation approach with the `D1` dataset and $K=10$.
- **Maintaining the temporal dependencies**: the *K*-fold cross-validation is not the most appropriate evaluation approach when dealing with time series since the temporality of the data is altered. To maintain that temporality, the first $16$ sequences of activities ($80\%$ of data) from `D1` are used for training and the remaining (last) $4$ sequences ($20\%$) for evaluation. These subsets of `D1` are named `D1T` (training) and `D1E` (evaluation). This is a basic approach to investigate the stability of the data.
- **Effect of time**: the model trained with `D1T` is evaluated using the data from `D2`, `D3` and `D4`. This approach allows to analyse the variation of the classification performance in different time frames ($10$, $30$ and $90$ minutes after `D1`) and therefore, to determine the stability of the CSI data.

::: {.callout-note}
The script employed to execute this process is `03_1_multiple-evaluations.py` with the flag `--model cnn`.

:::: {add-from=libs/chapter5/pipeline/03_1_multiple-evaluations.py code-fold='true' code-filename='03_1_multiple-evaluations.py'}
```{.python}
```
::::

:::


## Investigating the causes of failure
The previous methodology resulted in non-satisfactory outcomes (see @sec-localized_har). The results showed a clear degradation in the classification accuracy of the employed CNN model when the evaluation took into account data collected spaced in time regarding the training data. That is, classification accuracy quickly degrades over time.

Notwithstanding, temporal instability of CSI data is only one possible explanation for the poor obtained results. Concretely, the following factors could affect the results:

1. The selected methods (i.e., data preprocessing and model architecture) might not be able to properly work with CSI data, i.e., generalize from the training data. While CNN approaches have proven to provide good results working with CSI data [@ma2019wifi], most related works using the ESP32 microcontroller employ other architectures, such as the MLP.
2. The employed hardware for CSI extraction, ESP32-S2 microcontroller, might not be appropriate for such a task. Other devices, such as the Intel 5300 or Atheros NICs might be more appropriate.
3. The collected dataset might have been affected by some external interference, altering the environment and changing the CSI data.
4. The CSI data is not stable over time and therefore can not be used for real-life applications.

Next, we aim to determine the cause of the bad results presented in @sec-localized_har. First, to determine that our method is appropriate for CSI data (1), we applied it to two public datasets and compared the results with other state-of-the-art works (@sec-methods_validation). Then, to prove that alternative methods validated in the literature would have obtained similar results to our method (1), we applied the method from a related work on our collected dataset (@sec-methods_validation). Finally, to verify the temporal stability of the CSI data (4), a new dataset was collected over several days to evaluate the similarity of the data across days (@sec-csi_stability). The remaining factors could not be explored due to resource limitations (2) and the impossibility of determining the existence of external interferences while collecting the dataset (3).

### Validation of method on public datasets

#### Methodology

Two publicly available datasets have been used to validate the methods and model employed: the **StanWiFi** and the **Multi-environment** dataset.

- **StanWiFi**: it was collected by [@yousefi2017survey] and made available in GitHub^[[Wifi_Activity_Recognition](https://github.com/ermongroup/Wifi_Activity_Recognition)]. The dataset was collected with a Wi-Fi router (Tx) and an Intel 5300 NIC with three Rx antennas, both separated by $3$ meter in a LOS environment. The dataset contains CSI data from $90$ subcarriers sampled at $1000$Hz corresponding to $7$ activities: *lie*, *fall*, *walk*, *run*, *sit down*, *stand up* and *pick up*. For comparison purposes, the *pick up* activity was removed from the dataset since other works do so.
- **Multi-environment**: collected in three different environments, `E1` and `E2` in LOS conditions and `E3` in NLOS condition [@alsaify2020dataset]. The latter dataset is discarded since we focus on LOS conditions. The datasets were collected using two computers (Tx and Rx) equipped with an Intel 5300 NIC, which were separated by $3.7$ meter sin `E1` and $7.6$ meters in `E2`. The CSI data was collected from $90$ subcarriers at $320$Hz corresponding to $12$ different activities classified in $6$ groups: *no movement*, *falling*, *walking*, *sitting/standing*, *turning* and *pick up*.

The **data preparation** steps described in [@sec-csi_data_prep] were applied to both datasets. While for the collected dataset the windows consisted of $0.5$ seconds of data, a window size of $1$ seconds was employed in both public datasets since they contain a higher amount of data.

::: {.callout-note}
The script employed to execute the process in StanWiFi dataset is `01_2_stanwifi-processing.py`.

:::: {add-from=libs/chapter5/pipeline/01_2_stanwifi-processing.py code-fold='true' code-filename='01_2_stanwifi-processing.py'}
```{.python}
```
::::

The script employed to execute the process in Multi-environment dataset is `01_3_multienvironment-processing.py`.

:::: {add-from=libs/chapter5/pipeline/01_3_multienvironment-processing.py code-fold='true' code-filename='01_3_multienvironment-processing.py'}
```{.python}
```
::::

:::


As regards the **HAR classifier**, the model architecture described in [@sec-csi_model] was employed, with minor adaptations in some hyperparameters due to computational limitations^[The higher dimensionality of both datasets (higher sampling rate and data from more subcarriers) compared with the collected one makes it unfeasible to use the previous model due to memory limitations.]. The adaptations in each dataset are the following:

- StanWiFi: $16$ **number of filters**, $128$ **batch size** and $30$ **epochs**.
- Multi-environment (`E1` and `E2`): $8$ **number of filters**, $256$ **batch size** and $30$ **epochs**. 


Finally, the **experimental procedure** consisted of the $10$-fold cross-validation to evaluate the CNN model in the public datasets. The results are compared with other related works also employing a *K*-fold cross-validation approach.

::: {.callout-note}
The script employed to execute this process in the StanWiFi is `03_2_cross-validation.py` with the flag `--dataset stanwifi`. The same script was used for the Multienvironment dataset employing the flag `--dataset multienvironment`.

:::: {add-from=libs/chapter5/pipeline/03_2_cross-validation.py code-fold='true' code-filename='03_2_cross-validation.py'}
```{.python}
```
::::

:::

### Validation of alternative method in the collected dataset
The methods proposed by @choi2022wi have been applied to the collected dataset. In their work, the authors extract a set of hand-crafted features from the CSI data and employ an MLP model for crowd counting and localization. Choi's et al. methods have been selected since they followed an appropiate evaluation taking into account the stability of the signal and only showed a small drop in performance.

#### Methodology
As in [@sec-csi_data_prep], the amplitude is extracted from the CSI data and the dataset is arranged in windows of $50$ samples with a $50\%$ overlap. Then, the methods presented by @choi2022wi are applied:

- **Noise removal**: the Hampel and the Savitzky-Golay filters are applied on each subcarrier.
- **Feature extraction**: the extracted features to be used as input of the MLP model are the **Mean**, **SD**, **Maximum**, **Minimum**, **Lower quartile**, **Higher quartile**, **IQR**, **Differences between adjacent** subcarriers and the **Euclidean distance**.

::: {.callout-note}
The script employed to execute this process is `01_1_preliminar-dataset-processing.py` with the flag `--method choi`.

:::: {add-from=libs/chapter5/pipeline/01_1_preliminar-dataset-processing.py code-fold='true' code-filename='01_1_preliminar-dataset-processing.py'}
```{.python}
```
::::

:::


As **HAR classifier**, an MLP model is employed, but instead of using the same architecture as the one employed by Choi, a **Grid search** process was executed to determine the most appropriate hyperparameters for our dataset. 

The **Grid search** was carried out as described in [@sec-csi_model]. @tbl-csi_hyperparameters_mlp contains the best combination of hyperparameters.

{{< embed 05.1.2_grid-search.ipynb#tbl-csi_hyperparameters_mlp >}}


::: {.callout-note}
The script employed to execute the Grid Search is `02_hyperparameter-optimization.py` with the flag `--model mlp`.

:::: {add-from=libs/chapter5/pipeline/02_hyperparameter-optimization.py code-fold='true' code-filename='02_hyperparameter-optimization.py'}
```{.python}
```
::::

:::

The same **experimental procedure** described in @sec-csi_exp_procedure with the three evaluation approaches is employed using the method presented by Choi in our collected dataset.

::: {.callout-note}
The script employed to execute this process is `03_1_multiple-evaluations.py` with the flag `--model mlp`.

:::: {add-from=libs/chapter5/pipeline/03_1_multiple-evaluations.py code-fold='true' code-filename='03_1_multiple-evaluations.py'}
```{.python}
```
::::

:::


### Verification of the stability of the CSI signal
This section describes the methodology to determine if the CSI data is stable over time carrying out a simple experiment. To do so, a new data collection is executed minimizing the disturbance of the environment by external factors. Then, an evaluation procedure is designed to determine the similarity of CSI samples collected in different time frames using DL classification models.

#### Methodology
A dataset was collected using a TP-Link Archer C80 (one Tx antenna) and a SparkFun Thing Plus ESP32-S3 WROOM (one Rx antenna) connected to a computer. The Tx and Rx were placed on a table, separated by $1$ meter in LOS condition.The Tx device was configured to work with the standard IEEE 802.11n in the channel $6$. The Rx device was configured to connect to the Rx and extract Wi-Fi CSI data from HT-LTF subcarriers generated by ping traffic at $100$Hz.

The **data collection** consisted of capturing CSI data from an unaltered laboratory from the university for several days: from March $28^{th}$ to April $1^{st}$ $2024$, coinciding with the Easter holidays. During these days, no external human factors would have disturbed the environment and thus, the CSI data. The collected CSI samples were labelled regarding the day they were collected (i.e., $03/29$, $03/29$, $03/30$, $03/31$, $04/01$).

The **data preparation** steps described in [@sec-csi_data_prep] with minor adaptations were applied to the dataset. More concretely, given the amount of the collected data ($24$ GB), the windowing procedure was set to arrange windows of size $100$ without overlapping.

::: {.callout-note}
The script employed to execute this process is `01_4_lodo-dataset-processing.py`.

:::: {add-from=libs/chapter5/pipeline/01_4_lodo-dataset-processing.py code-fold='true' code-filename='01_4_lodo-dataset-processing.py'}
```{.python}
```
::::

:::

As **HAR classifier**, the model described in [@sec-csi_model] was employed, although with minor adaptations in some hyperparameters due to computational limitations caused by the high quantity of data. More concretely, the **number of filters**, **batch size** and **epochs** were set to $8$, $512$ and $30$, respectively.


Finally, the **experimental procedure** consisted of a $5$-fold cross-validation with the processed dataset. Each fold of the cross-validation corresponds to the data collected in one day, which can be named as Leaving-One-Day-Out (LODO). This procedure aims to evaluate how the models classify data from an unseen day, having two possible outputs:

- The samples from a specific day are classified in any of the remaining days. In other words, a specific day's samples are similar to those of any other day. These results would imply that the CSI data **is stable** over time.
- The samples from the day $X_i$ are classified in the day $X_{i-1}$ or $X_{i+1}$. In other words, samples from a specific day are similar only to the adjacent days (i.e., samples most close in time). These results would imply that the CSI data **is not stable** over time.


::: {.callout-note}
The script employed to execute this process is `03_3_lodo.py`.

:::: {add-from=libs/chapter5/pipeline/03_3_lodo.py code-fold='true' code-filename='03_3_lodo.py'}
```{.python}
```
::::

:::


## Results

:::{#contents}
:::