Hrs_data file read to convert to HRV

[Mpoint]

There is a hrs.data file that I was trying to read It has / and apha-numeric data how do I use that data to convert into human readable heart rate or heart rate variability processing 
  Also I have attached the hrs.data file

pi@raspberrypi:~/wasp-os $ python
Python 2.7.16 (default, Oct 10 2019, 22:02:15)
[GCC 8.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> file = open("hrs.data", "r")
>>> file.read()
'\xff\xff\xe5\x07\x06\x00\x12\x00\x07\x00\x13\x00*\x00\xf8Y\xfdY\xfdY\x9cY\x9cY\                        x9cY\x9bY\x9bY\x95Y\x83Y\x83YeYrYZYZY3YGYGY0YEYEY]Y]Y]Y\x84Y\x84Y\x8cYEYGYGY1YKY                        KYIYJYJY4Z4Z4Z\x85Z\x85Z\x85ZUZUZQZPZPZLZNZ[Z[Z_YDYDY\xa6S\xa6S\xa6S\x83S\x83SXS                        \xc4R\xc4R\x99R\xff\xff\xe5\x07\x06\x00\x12\x00\x07\x00\x15\x00\x02\x00\x86P\x86                        PvPyO]O]O\xb8N\xa9N\xa9N\'N\'N\'N\xcdM\xcdM\xc7M`M`MgM9M6M6M?M7M7MMMMMMM\xb7M\xb                        7M\xb7M\x13N\x13N\x18NTNTN?N\x81N\x89N\x89N\x9fN\xaeN\xaeN\xa1N\xa1N\xa1N\xe1N\x                        e1N\xe1N\xf7N\xf7N\xd9N&Q&Q\xf9PJR\x7fR\x7fRJR+R+R\xa9O\xa9O\xa9O\xc5O\xc5O1P@N@                        N\xc0N\xa6R\x9eS\x9eSGWxWxW&Y&Y&Y\x05Z\x05Z\x05Z\xb8Y\xb8Y\xbbY\xacY\xacY\x89Y\x                        08Y\xd4X\xd4X\x95WgWgW\xeaU\xeaU\xeaU\xadS\xadS\xb9S\x04S\x04S>SDSDSdS\x10S\xb9R                        \xb9R\x93R\x02R\x02R\xafP\xafP\xafP\xbfP\xbfP\xedQ\x0eN\x0eN\x9cM\xf0N\xbdN\xbdN                        \x1eSqRqRYRYRYR\x06Y\x06Y\x06Y\x83[\x83[\xb8[C\\C\\\x8f\\\x0f]B]B]\x93]^]^]l\\l\                        \l\\\xe4Z\xe4Z\xd3Z\xb7Y\xb7Y\xccY\x0cZDZDZ\xe4Y\xbfY\xbfY`X`X`X\xe4W\xe4W\xc7We                        WeW\x80WqWwW\xa7W\x84W\xa4W\xa4W\xc8W\xbfW\xbfW\xc6W\xc6W\xc6W8W8WNV\x05P\x05PLP                        =P\x19P\x19P\xe9O\xdeO\xdeO`L`L`L\x9fM\x9fM\x88M\xd6O\xd6O\x19O\x89U\x00U\x00U1^                        \xef_\xef_\xf1c\xf1c\xf1cncncnc\xc2e\xc2e\xfcd\x8cb\x8cb\x1ac\xa8a\x02b\x02b\xc3                        `,a,aLcLcLcZdZd\xd8e\x1df\x1df\xf3e\xffd\xf3b\xf3b\xbdV\x02U\x02U\xff\xff\xe5\x0                        7\x06\x00\x12\x00\x07\x00\x16\x00\x16\x00\xaaS\xaaS\xaaS@R@R\xc2R\xccS\xccS\xa0S                        0W7W\xc7WZ[Q[Q[\xbe^\xe5^\xe5^s_s_s_F`F`_`\xcd`\xcd`\xe8`2^\xbd^\xbd^\x11`\xa5`\                        xa5`\x13Z\x13Z\x13ZOYOY\x9dY`[`[\xc9[>\\7\\7\\\xb7Z\xefZ\xefZ\x0f]\x0f]\x0f]\x05                        Z\x05Z7[\x9db\x9db\x90c\xf9c\td\td\xa3d\xded\xded\x8be\x8be\x8be4c4c4c\x13c\x13c                        \x00c\x84a\x84aea\x11a4`4`\x0f\\i\\i\\sZsZsZ\n[\n[3[@\\@\\\x1a\\\x94\\?\\?\\\x12                        ]\x12]\x12]#]#]#]\xbd]\xbd]:^+[+[T[\xbb[Z[Z[\xa2[\x85[\x85[mYmYmY\x8eZ\x8eZ\x8eZ                        \x98\\\x98\\\x08\\SYSY\x1fY\xa9Y\x10Y\x10Y\xfeU\xecU\xecU\x9aX\x9aX\x9aX\xf9[\xf                        9[Y[S`S`\xdd_\xb2[\xe1\\\xe1\\\xba^\xd5^\xd5^\x95S\x95S\x95S"R"R\x85Q\xf3R\xf3R\                        xedR\xc5U#V#V\xff_\x1f`\x1f`\xe3b\xe3b\xe3b\xdda\xdda\xdda\x95Z\x95Z|Z\xafY\xafY                        sZJZ\x9aZ\x9aZ\x18a\xccb\xccb\x1bf\x1bf\x1bf\xb0c\xb0c}c<b<b\x95a\xec^\xf8^\xf8^                        U^\x13^\x13^\x10^\x10^\x10^\xda^\xda^\x1d_D_D_^^BZMZMZqY\xfbX\xfbX\xab[\xab[\xab                        [\xa6S\xa6S\x96Q\x8cM\x8cM.M\xe8O\x9aU\x9aU=Z\xfeZ\xfeZ\xa5Y\xa5Y\xa5Y\x0eW\x0eW                        \x0eW\xd6T\xd6T#V\xc4_\xc4_\xdbbja\x8f_\x8f_2c\xde_\xde_\xff\xff\xe5\x07\x06\x00                        \x12\x00\x07\x00\x17\x00*\x00\x89c\x89c\x89c{a{a\x16c\xf9Z\xf9Z\xfeY\x12Z\x19Z\x                        19Z\xa4\\\x19]\x19]\x91^\x91^\x91^\x85_\x85_]_\x01`\x01`\xec_\x95`\xd8`\xd8`La\x                        85a\x85a\xfb`\xfb`\xfb`]`]`\x0f`\x97_\x97_|_\xba_\xb3_\xb3_\xbd^\xe9^\xe9^\xe1[\                        xe1[\xe1[\xcaK\xcaKOJ!E!E\xefD\x11D\x0fD\x0fD\xe9C\xf9C\xf9C\xf8C\xf8C\xf8CUDUDU                        D\x9fD\x9fD\x91D\xe4D\xe4D\xfbD/E8E8EpEpEpE\xceE\xceE\xceE2F2F0FlFlFXF\x7fFqFqF\                        xe2F\xfbI\xfbI\x92J\x92J\x92J\xefM\xefM%NxUxU\xb0Z1\\\xe8Y\xe8Y\xd3Q\xbbQ\xbbQ&R                        &R&R\xc4R\xc4R\x93R\xe7T\xe7T\xf2U\x8dS\xcdS\xcdS8S-S-S"T"T"T\'U\'U5U\xc5T\xc5T\                        xc0T\xe9S\xe5S\xe5SDRGRGRZQZQZQ\xcdM\xcdM]M\xeaK\xeaK\xf4K\x88K\x14K\x14K\x8fI\x                        a9I\xa9I\x1cG\x1cG\x1cG\xf8D\xf8D\xf8D\x84D\x84DDD C C\x08C\x10C\xfdB\xfdB|BgBgB                        8B8B8BDCDC\xebB\x00E\x00E\x16EqE\xa1E\xa1E\x18E\xf1D\xf1D\xddC\xddC\xddC\xa0G\xa                        0GpG\xc8F\xc8F\xacF\xcaF\xafF\xafF\xf8F\xf1F\xf1F\x05G\x05G\x05GUGUGUG\x1bG\x1bG                        G|F|FUF\xfeF\x12G\x12GpG\x94G\x94GOHOHOH8H8H\x13H|F|F{F\x93E\x8dE\x8dE\xc6E\xb9                        E\xb9EZEZEZE\xb0F\xb0F\xdaG'
>>>

[danielt]

See https://forum.pine64.org/showthread.php?...4#pid94484

[Mpoint]

Hello Daniel Thanks that was helpful, I was able to decode the utf-16 format using a python script and found there is only one date and time stamp in the hrs.data file It is the first 12 characters the rest is just the data is just series of 5 digit integers how does that translate to Heart Rate ? Can you please explain
As I am trying to run some analysis on the heart rate and need to understand the values 
2021;7;23;20;11;18;29858;30447;30447;30447;31008;31008; converted hrs.data to 7-23-hrs_data.csv
2021;7;22;18;0;19;19362;19362;19356;19231;19231;19246;19308;19308;19306; converted hrs.data to 7-22-R-D-hrs_data.csv
2021;7;21;21;18;29;16451;16451;16451;16444;16444;16478;16479;16479;16479;16479; converted hrs.data to 7-22-L-D-hrs_data.csv

[danielt]

The five digit numbers are the raw samples of the sensor (captured at 24Hz). If you graph them you will should see some peaks and troughs corresponding to heart rate. You will also likely to see noise and a massive drift in the zero point. I believe from the literature that this is normal for PPG sensors and you will need to filter that out before doing any peak detection.

It's extremely odd that your files have only one time and date stamp though (there are four date stamps in the data you printed as part of the original question). What are the file sizes?

[Mpoint]

The file size of hrs.data (utf-16 format) file are 102KB, 130KB, and 154KB, and when these files are converted to hrs_data.csv (ASCII format) files they display file sizes of 295KB, 384KB and 429KB respectively

[danielt]

The file size of hrs.data (utf-16 format) file are 102KB, 130KB, and 154KB, and when these files are converted to hrs_data.csv (ASCII format) files they display file sizes of 295KB, 384KB and 429KB respectively

I don't quite understand how the file ended up converted into UTF-16. It should just be a raw binary file. Anyhow if the filesize of 100K should contain several hundred 10 second records (each record is less than half a kilobyte).

I have written a parser to convert the binary format into CSV. See https://github.com/daniel-thompson/wasp-...06c83967f7 .

[Mpoint]

Measuring night-time heart rate is key to a healthy heart.  However, the data size becomes big and it takes a long time to pull the data. Is there another FASTER way to download the hrs.data file that is large 
When the first attempt fails I tired renamed the hrs.data to hrs.data1 leaving the hrs.data file there I tried pulling again and NOW I am not sure did the pull command appended to the hold hrs.data file and so that is why the hrs.data2 appears larger than hrs.data1 here is the error https://pastebin.pl/view/db053994

[danielt]

hrs.data is generated by a diagnostic mode I added to the heart rate application to allow people who are interested to record test data to try and develop more robust heart rate detection algorithms using real data gathered from the HRS3300 sensor on the PineTime. Some people have reported that the initial algorithm I developers doesn't work well for them. Gathering a wider range of data sets is the first step to developing better heart rate detection algorithms.

The current heart rate detection algorithm does not attempt to estimate heart rate variability so I assumed you were gathering hrs.data in order to implement new algorithms.

If you are trying to do overnight logging then I'm extremely skeptical that using the diagnostic features built into the existing heart rate detection implementation is the best way to approach things. Whether you split the files up or not it will still yield over a megabyte of samples and that will take a long time to transfer over BLE. Instead, once you have implemented suitable algorithms, then surely it is better to process the raw data on the watch and log BPM and HRV values instead (e.g. write a new app that is similar to heart.py but keeps the screen off and logs the BPM to a file rather than showing it on the screen).

If you really do want to capture raw samples then you should write you own logger so you can add some form of compression to the data (e.g. write a new app that is similar to heart.py but inject samples into a compressor instead of injecting them into the PPG algorithm). The major drawback with this approach is that you will have to add a compressor yourself: there is nothing builtin to wasp-os to help you compress data.

[Mpoint]

import wasp
from gadgetbridge import *
from hrv import HRVApp
wasp.system.register(HRVApp())
wasp.system.schedule()

It freezes as I load the main.py displayed on the watch perhaps it at import HRVApp() because the app makes calls to other python library scripts that are too large to load
how do I debug this. I tried test_memory.py and not sure of to use it to test the memory

[danielt]

import wasp
from gadgetbridge import *
from hrv import HRVApp
wasp.system.register(HRVApp())
wasp.system.schedule()

It freezes as I load the main.py displayed on the watch perhaps it  at import HRVApp() because the app makes calls to other python library scripts that are too large to load
how do I debug this. I tried test_memory.py and not sure of to use it to test the memory

Don't load code from main.py until it is tested otherwise the error messages will be emitted before you connect to the console.

Instead use an unmodified main.py and run the import and register commands by typing them directly into the REPL.