# Introduction

---

### What is this repository for? ###
* This repository include Python scripts that transform **raw sensor data** collected from Android mobile devices into **intermediate CSV file** for analysis of various purposes.
* The preprocessing scripts here intend to be used to provide **reliable, consistent, and structured** intermediate data file for data analysis related to MicroT project.  

### Agenda ###
* Adding all information and features (Done)
* Confirm format internally and with USC (in progress)
* Testing on all participants (Done On 10 participants)
* Packaging the project on Pypi


# Summary of Preprocessed Intermediate File

---
The discussion on data source to include can bee found in the [#86 Issue of DataProcessing repository ](https://bitbucket.org/mhealthresearchgroup/dataprocessing/issues/86/analytical-data-structure-and-requirements)

### 1. Android Smartwatch ###

| Information                           | Raw Data File                                                                                                     | Included Data                                                                                                      | Omitted Data  | Intermediate file                                                                                                                    |
| --------------------------------------|-------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|---------------|--------------------------------------------------------------------------------------------------------------------------------------|
| Prompt response                       | [participant_folder] / logs-watch / DATE / HOUR / PromptResponses.log.csv                                         | ID, EMA_type, Date, Prompt Timestamp, Time Zone, Completion Status, Reprompt, Response Timestamp, Q-Key, Response  | None          | [save_path]/[participant_id]/DATE/watch_prompt_response_PARTICIPANT_DATE.csv                                                         |
| Battery level                         | [participant_folder] / data-watch / DATE / HOUR / Battery.##.event.csv                                            | timestamp, battery_level, battery_charging                                                                         | None          | [save_path]/[participant_id]/DATE/watch_battery_PARTICIPANT_DATE.csv                                                                 |
| Accelerometer data                    | [participant_folder] / data-watch / DATE / HOUR / AndroidWearWatch-AccelerationCalibrated-NA.*.sensor.baf         | header_timestamp, accelation_meters_per_second_squared(X,Y,Z axis), MIMS-unit                                      | None          | [participant_folder] / data-watch / DATE / HOUR / 020000000000-AccelerationCalibrated.*.sensor.csv, mims_DATE_HOUR.csv; a copy of former two in [save_path]/[participant_id]/DATE/          |
| App usages                            | [participant_folder] / data-watch / DATE / HOUR / AppEventCounts.csv                                              | log_time, last_hour_timestamp, current_hour_time_stamp, app_package_name, event_time_stamp, app_event              | None          | [save_path]/[participant_id]/DATE/phone_app_usage_PARTICIPANT_DATE.csv                                                               |

### 2. Android Smartphone ###

| Information                           | Raw Data File                                                                   | Included Data                                                                                                      | Omitted Data  | Intermediate file                                                                                    |
| --------------------------------------|---------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|---------------|------------------------------------------------------------------------------------------------------|
| Prompt response                       | [participant_folder] / logs / DATE / HOUR / PromptResponses.log.csv             | ID, EMA_type, Date, Prompt Timestamp, Time Zone, Completion Status, Reprompt, Response Timestamp, Q-Key, Response  | None          | [save_path]/[participant_id]/DATE/phone_prompt_response_PARTICIPANT_DATE.csv                         |
| GPS data                              | [participant_folder] / data / DATE / HOUR / GPS.csv                             | log_time, location_time, lat, long, horizontal_accuracy, provider, speed, altitude, bearing                        | None          | [save_path]/[participant_id]/DATE/phone_GPS_PARTICIPANT_DATE.csv                                     |
| step count                            | [participant_folder] / data / DATE / HOUR / StepCounterService.csv              | log_time_stamp, steps_last_hour, accumulated_steps                                                                 | None          | [save_path]/[participant_id]/DATE/phone_stepCount_PARTICIPANT_DATE.csv                               |
| Phone state and detected activities   | [participant_folder] / data / DATE / HOUR / ActivityDetected.csv                | log_time, in_vehicle, on_bike, on_foot, running, still, tilting, walking, unknown                                  | None          | [save_path]/[participant_id]/DATE/phone_detected_activity_PARTICIPANT_DATE.csv                       |
| Phone usage events and broadcasts     |                                                                                 |                                                                                                                    |               | [save_path]/[participant_id]/DATE/phone_usage_broadcasts_PARTICIPANT_DATE.csv                        |
| Environmental sensors                 | LightSensorStats.csv, ProximitySensorManagerService.csv, AmbientPressManagerService.csv, AmbientTempManagerService.csv, AmbientHumidManagerService.csv |    log time, sensor value, sensor max       | None          |                                                                                                      |

# Code Book

---  

Detailed explanation of columns in intermediate files can be found in the [code book](https://docs.google.com/document/d/1RsxueU1tCGNSl8-ClNjRPpkv-y9AHej1IfkIU53Lo6E/edit?usp=sharing).

# How to Run Scripts?

---

### Python Version ###
Python 3.6+ (Other versions haven't been tested but should be fine)

### Dependencies ###
```
pip install -r requirements.txt
```
* For user who wants to include accelerometer and MIMS-unit, the below are extra set-up:
	1. The MIMS-unit depends on particular [R package](https://www.r-pkg.org/pkg/MIMSunit), so install R on your system.
	2. Add R to your environment variables. For windowsOS users, put path similar to "C:\Program Files\R\R-4.0.2\bin\x64" to your path in system variables, and reboot your computer.




### How to run the main script for generating intermediate file? ###

* **Specifications**
	1. **microT_root_path** :  root path to the raw data folder
	2. **intermediate_file_save_path** : save path for the output of intermediate file.
	3. **date_start** and **end_start** : generate intermediate data file from data within this time period.
	
	
* **Prepare your information_included.txt file**

	* The main script needs to read information_included.txt to know what information to be included in preprocessing.
	* Before running the main script, find and edit **analysis_task/phone(watch)_information_included.txt**. The format is one item for each row with no punctuation in between.
	* If you want to include all information available, just put "all" in the first row.
	* Here you can find the [full list of available information items](https://bitbucket.org/mhealthresearchgroup/microt_preprocessing/src/master/resource/full_list/).
	* Here you can find the [sample file](https://bitbucket.org/mhealthresearchgroup/microt_preprocessing/src/master/resource/sample_file/).

	
* **Prepare your participants_included.txt file**
	
	* The main script needs to read participants_included.txt to know which participants to be included in preprocessing.
	* Before running the main script, find and edit **analysis_task/phone(watch)_participants_included.txt**. The format is one participant ID for each row with no punctuation in between.
	* If you want to include all participants available, just put "all" in the first row.
	* Here you can find the [sample file](https://bitbucket.org/mhealthresearchgroup/microt_preprocessing/src/master/resource/sample_file/).

	
* **Run script**

	* **For smartphone** 

	```
	python preprocessing_all_ema.py <microT_root_path> <intermediate_file_save_path> <participants_included_text_file_path> [date_start] [date_end]
	```

	* **For smartwatch**

	```
	python preprocessing_all_uema.py <microT_root_path> <intermediate_file_save_path> <participants_included_text_file_path> [date_start] [date_end]
	```  
	
	* **Usage with date_start and date_end as optional arguments**  
		
		* Preprocess data from date_start to date whenever data is available  
	
		```
		python preprocessing_all_uema.py <microT_root_path> <intermediate_file_save_path> <participants_included_text_file_path> [date_start]
		```  
	  
		* Preprocess data available in all periods  

		```
		python preprocessing_all_uema.py <microT_root_path> <intermediate_file_save_path> <participants_included_text_file_path>
		```   
		
	* **Keep temporarily generated hourly file during the pre-process**	
		
		Options: [-o], [--hour]
		
		```
		python preprocessing_all_uema.py <microT_root_path> <intermediate_file_save_path> <participants_included_text_file_path> [date_start] [date_end] <-o>
		```


	* **Get command line help**  
	
		```
		python preprocessing_all_uema.py -h
		``` 
	
	*We adopt [docopt](http://docopt.org/) as the standard for documenting command line arguments

# Who do I talk to?

---

### Maintained by Aditya and Jixin ###