Metadata-Version: 2.1
Name: microt_preprocessing
Version: 0.0.1
Summary: A package that transforms raw sensor data collected from Time study app into intermediate CSV file for analysis of various purposes
Home-page: https://bitbucket.org/mhealthresearchgroup/microt_preprocessing/src/master/
Author: Jixin Li, Aditya Ponnada
Author-email: li.jix@northeastern.edu
License: UNKNOWN
Project-URL: Bug Tracker, https://bitbucket.org/mhealthresearchgroup/microt_preprocessing/issues
Description: # Introduction
        
        ---
        
        ### What is this repository for? ###
        * This repository include Python scripts that transform **raw sensor data** collected from Android mobile devices into **intermediate CSV file** for analysis of various purposes.
        * The preprocessing scripts here intend to be used to provide **reliable, consistent, and structured** intermediate data file for data analysis related to MicroT project.  
        
        ### Agenda ###
        * Adding all information and features (Done)
        * Confirm format internally and with USC (in progress)
        * Testing on all participants (Done On 10 participants)
        * Packaging the project on Pypi
        
        
        # Summary of Preprocessed Intermediate File
        
        ---
        The discussion on data source to include can bee found in the [#86 Issue of DataProcessing repository ](https://bitbucket.org/mhealthresearchgroup/dataprocessing/issues/86/analytical-data-structure-and-requirements)
        
        ### 1. Android Smartwatch ###
        
        | Information                           | Raw Data File                                                                                                     | Included Data                                                                                                      | Omitted Data  | Intermediate file                                                                                                                    |
        | --------------------------------------|-------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|---------------|--------------------------------------------------------------------------------------------------------------------------------------|
        | Prompt response                       | [participant_folder] / logs-watch / DATE / HOUR / PromptResponses.log.csv                                         | ID, EMA_type, Date, Prompt Timestamp, Time Zone, Completion Status, Reprompt, Response Timestamp, Q-Key, Response  | None          | [save_path]/[participant_id]/DATE/watch_prompt_response_PARTICIPANT_DATE.csv                                                         |
        | Battery level                         | [participant_folder] / data-watch / DATE / HOUR / Battery.##.event.csv                                            | timestamp, battery_level, battery_charging                                                                         | None          | [save_path]/[participant_id]/DATE/watch_battery_PARTICIPANT_DATE.csv                                                                 |
        | Accelerometer data                    | [participant_folder] / data-watch / DATE / HOUR / AndroidWearWatch-AccelerationCalibrated-NA.*.sensor.baf         | header_timestamp, accelation_meters_per_second_squared(X,Y,Z axis), MIMS-unit                                      | None          | [participant_folder] / data-watch / DATE / HOUR / 020000000000-AccelerationCalibrated.*.sensor.csv, mims_DATE_HOUR.csv; a copy of former two in [save_path]/[participant_id]/DATE/          |
        | App usages                            | [participant_folder] / data-watch / DATE / HOUR / AppEventCounts.csv                                              | log_time, last_hour_timestamp, current_hour_time_stamp, app_package_name, event_time_stamp, app_event              | None          | [save_path]/[participant_id]/DATE/phone_app_usage_PARTICIPANT_DATE.csv                                                               |
        
        ### 2. Android Smartphone ###
        
        | Information                           | Raw Data File                                                                   | Included Data                                                                                                      | Omitted Data  | Intermediate file                                                                                    |
        | --------------------------------------|---------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|---------------|------------------------------------------------------------------------------------------------------|
        | Prompt response                       | [participant_folder] / logs / DATE / HOUR / PromptResponses.log.csv             | ID, EMA_type, Date, Prompt Timestamp, Time Zone, Completion Status, Reprompt, Response Timestamp, Q-Key, Response  | None          | [save_path]/[participant_id]/DATE/phone_prompt_response_PARTICIPANT_DATE.csv                         |
        | GPS data                              | [participant_folder] / data / DATE / HOUR / GPS.csv                             | log_time, location_time, lat, long, horizontal_accuracy, provider, speed, altitude, bearing                        | None          | [save_path]/[participant_id]/DATE/phone_GPS_PARTICIPANT_DATE.csv                                     |
        | step count                            | [participant_folder] / data / DATE / HOUR / StepCounterService.csv              | log_time_stamp, steps_last_hour, accumulated_steps                                                                 | None          | [save_path]/[participant_id]/DATE/phone_stepCount_PARTICIPANT_DATE.csv                               |
        | Phone state and detected activities   | [participant_folder] / data / DATE / HOUR / ActivityDetected.csv                | log_time, in_vehicle, on_bike, on_foot, running, still, tilting, walking, unknown                                  | None          | [save_path]/[participant_id]/DATE/phone_detected_activity_PARTICIPANT_DATE.csv                       |
        | Phone usage events and broadcasts     |                                                                                 |                                                                                                                    |               | [save_path]/[participant_id]/DATE/phone_usage_broadcasts_PARTICIPANT_DATE.csv                        |
        | Environmental sensors                 | LightSensorStats.csv, ProximitySensorManagerService.csv, AmbientPressManagerService.csv, AmbientTempManagerService.csv, AmbientHumidManagerService.csv |    log time, sensor value, sensor max       | None          |                                                                                                      |
        
        # Code Book
        
        ---  
        
        Detailed explanation of columns in intermediate files can be found in the [code book](https://docs.google.com/document/d/1RsxueU1tCGNSl8-ClNjRPpkv-y9AHej1IfkIU53Lo6E/edit?usp=sharing).
        
        # How to Run Scripts?
        
        ---
        
        ### Python Version ###
        Python 3.6+ (Other versions haven't been tested but should be fine)
        
        ### Dependencies ###
        ```
        pip install -r requirements.txt
        ```
        * For user who wants to include accelerometer and MIMS-unit, the below are extra set-up:
        	1. The MIMS-unit depends on particular [R package](https://www.r-pkg.org/pkg/MIMSunit), so install R on your system.
        	2. Add R to your environment variables. For windowsOS users, put path similar to "C:\Program Files\R\R-4.0.2\bin\x64" to your path in system variables, and reboot your computer.
        
        
        
        
        ### How to run the main script for generating intermediate file? ###
        
        * **Specifications**
        	1. **microT_root_path** :  root path to the raw data folder
        	2. **intermediate_file_save_path** : save path for the output of intermediate file.
        	3. **date_start** and **end_start** : generate intermediate data file from data within this time period.
        	
        	
        * **Prepare your information_included.txt file**
        
        	* The main script needs to read information_included.txt to know what information to be included in preprocessing.
        	* Before running the main script, find and edit **analysis_task/phone(watch)_information_included.txt**. The format is one item for each row with no punctuation in between.
        	* If you want to include all information available, just put "all" in the first row.
        	* Here you can find the [full list of available information items](https://bitbucket.org/mhealthresearchgroup/microt_preprocessing/src/master/resource/full_list/).
        	* Here you can find the [sample file](https://bitbucket.org/mhealthresearchgroup/microt_preprocessing/src/master/resource/sample_file/).
        
        	
        * **Prepare your participants_included.txt file**
        	
        	* The main script needs to read participants_included.txt to know which participants to be included in preprocessing.
        	* Before running the main script, find and edit **analysis_task/phone(watch)_participants_included.txt**. The format is one participant ID for each row with no punctuation in between.
        	* If you want to include all participants available, just put "all" in the first row.
        	* Here you can find the [sample file](https://bitbucket.org/mhealthresearchgroup/microt_preprocessing/src/master/resource/sample_file/).
        
        	
        * **Run script**
        
        	* **For smartphone** 
        
        	```
        	python preprocessing_all_ema.py <microT_root_path> <intermediate_file_save_path> <participants_included_text_file_path> [date_start] [date_end]
        	```
        
        	* **For smartwatch**
        
        	```
        	python preprocessing_all_uema.py <microT_root_path> <intermediate_file_save_path> <participants_included_text_file_path> [date_start] [date_end]
        	```  
        	
        	* **Usage with date_start and date_end as optional arguments**  
        		
        		* Preprocess data from date_start to date whenever data is available  
        	
        		```
        		python preprocessing_all_uema.py <microT_root_path> <intermediate_file_save_path> <participants_included_text_file_path> [date_start]
        		```  
        	  
        		* Preprocess data available in all periods  
        
        		```
        		python preprocessing_all_uema.py <microT_root_path> <intermediate_file_save_path> <participants_included_text_file_path>
        		```   
        		
        	* **Keep temporarily generated hourly file during the pre-process**	
        		
        		Options: [-o], [--hour]
        		
        		```
        		python preprocessing_all_uema.py <microT_root_path> <intermediate_file_save_path> <participants_included_text_file_path> [date_start] [date_end] <-o>
        		```
        
        
        	* **Get command line help**  
        	
        		```
        		python preprocessing_all_uema.py -h
        		``` 
        	
        	*We adopt [docopt](http://docopt.org/) as the standard for documenting command line arguments
        
        # Who do I talk to?
        
        ---
        
        ### Maintained by Aditya and Jixin ###
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.7
Description-Content-Type: text/markdown
