No Activity Data Parsed #19

muschellij2 · 2020-02-12T20:32:34Z

Issue

Example gt3x data has no parsed Activity - maybe Activity2 packet?

I'm adding @THLfi to the mix because I want to note that AGread parses a lot of additional, likely useful information from gt3x compared to read.gt3x. Also, the timing differences are quite different, but again the amount of info parsed is very different.

library(AGread)
#> package 'AGread' was built under R version 3.5.0
library(read.gt3x)
library(lubridate)
#> 
#> Attaching package: 'lubridate'
#> The following object is masked from 'package:base':
#> 
#>     date
library(readr)
zipfile = "https://web.stanford.edu/~hyatt4/content/software/padaco/sampleData.zip"
outfile = file.path(tempdir(), basename(zipfile))
if (!file.exists(outfile)) {
  dl = download.file(zipfile, outfile)
}
unz = unzip(outfile, exdir = tempdir())

csv_file = unz[basename(unz) == "sample_1sec.csv"]
csv_data <- readr::read_csv(
  csv_file, skip = 10,
  col_types = 
    cols(
      Date = col_character(),
      Time = col_character(),
      Axis1 = col_double(),
      Axis2 = col_double(),
      Axis3 = col_double(),
      Steps = col_double(),
      Lux = col_double(),
      `Inclinometer Off` = col_double(),
      `Inclinometer Standing` = col_double(),
      `Inclinometer Sitting` = col_double(),
      `Inclinometer Lying` = col_double(),
      `Vector Magnitude` = col_double()
    ))
csv_data$Date = paste0(csv_data$Date, " ", csv_data$Time)
csv_data$Date = mdy_hms(csv_data$Date)
head(csv_data)
#> # A tibble: 6 x 12
#>   Date                Time  Axis1 Axis2 Axis3 Steps   Lux `Inclinometer O…
#>   <dttm>              <chr> <dbl> <dbl> <dbl> <dbl> <dbl>            <dbl>
#> 1 2015-12-09 00:00:00 00:0…    35     0   148     0     0                0
#> 2 2015-12-09 00:00:01 00:0…     0     0    65     0     0                1
#> 3 2015-12-09 00:00:02 00:0…     0     0    39     0     0                1
#> 4 2015-12-09 00:00:03 00:0…     0     0     0     0     0                1
#> 5 2015-12-09 00:00:04 00:0…     0     0     0     0     0                1
#> 6 2015-12-09 00:00:05 00:0…     0     0     0     0     0                1
#> # … with 4 more variables: `Inclinometer Standing` <dbl>, `Inclinometer
#> #   Sitting` <dbl>, `Inclinometer Lying` <dbl>, `Vector Magnitude` <dbl>

gt3x_file = unz[basename(unz) == "sample.gt3x"]
system.time({
  ag = read_gt3x(gt3x_file)
})
#> Warning in parse_packet_set.default(X[[i]], ...): No method exists yet for
#> parsing ACTIVITY packets -- returning NULL.
#> Warning in parse_packet_set.default(X[[i]], ...): No method exists yet for
#> parsing LUX packets -- returning NULL.
#>    user  system elapsed 
#>  16.743   3.715  29.590
head(ag$CAPSENSE)
#>             Timestamp signal reference    state bursts
#> 1 2015-12-09 00:00:59    546       546 Not Worn     10
#> 2 2015-12-09 00:01:59    546       546 Not Worn     10
#> 3 2015-12-09 00:02:59    546       546 Not Worn     10
#> 4 2015-12-09 00:03:59    546       546 Not Worn     10
#> 5 2015-12-09 00:04:59    546       546 Not Worn     10
#> 6 2015-12-09 00:05:59    546       546 Not Worn     10
head(ag$ACTIVITY)
#> NULL
head(ag$BATTERY)
#>             Timestamp battery_voltage
#> 1 2015-12-08 18:55:40           4.164
#> 2 2015-12-08 18:56:40           4.164
#> 3 2015-12-08 18:57:40           4.164
#> 4 2015-12-08 19:05:40           4.175
#> 5 2015-12-08 19:06:40           4.175
#> 6 2015-12-08 19:07:40           4.175

system.time({
  rg = read.gt3x(gt3x_file, asDataFrame = TRUE, 
                 imputeZeroes = TRUE, debug = TRUE)
})
#> Reading Stream...
#> imputing 40 values at index 29100720 
#> imputing 4080 values at index 29100760
#>    user  system elapsed 
#>   4.192   2.115  13.663
hdr = attributes(rg)
sr = hdr$sample_rate
rg$ms = as.numeric(rg$time) %% 1
rg$diff_time = c(NA, diff(rg$time))
# 
as.data.frame(rg[ 1:(sr), ])
#>        X      Y      Z                time         ms  diff_time
#> 1  0.000  0.000  0.000 2015-12-09 00:00:00 0.00000000         NA
#> 2  0.102 -0.145  0.094 2015-12-09 00:00:00 0.07500005 0.07500005
#> 3  0.262  0.020 -0.848 2015-12-09 00:00:00 0.12500000 0.04999995
#> 4  0.262  0.031 -0.926 2015-12-09 00:00:00 0.20000005 0.07500005
#> 5  0.266  0.035 -0.938 2015-12-09 00:00:00 0.25000000 0.04999995
#> 6  0.262  0.031 -0.938 2015-12-09 00:00:00 0.32500005 0.07500005
#> 7  0.262  0.035 -0.938 2015-12-09 00:00:00 0.37500000 0.04999995
#> 8  0.266  0.035 -0.934 2015-12-09 00:00:00 0.45000005 0.07500005
#> 9  0.262  0.039 -0.934 2015-12-09 00:00:00 0.50000000 0.04999995
#> 10 0.266  0.035 -0.934 2015-12-09 00:00:00 0.57500005 0.07500005
#> 11 0.258  0.035 -0.926 2015-12-09 00:00:00 0.62500000 0.04999995
#> 12 0.266  0.039 -0.934 2015-12-09 00:00:00 0.70000005 0.07500005
#> 13 0.262  0.035 -0.934 2015-12-09 00:00:00 0.75000000 0.04999995
#> 14 0.266  0.035 -0.938 2015-12-09 00:00:00 0.82500005 0.07500005
#> 15 0.262  0.035 -0.934 2015-12-09 00:00:00 0.87500000 0.04999995
#> 16 0.262  0.035 -0.934 2015-12-09 00:00:00 0.95000005 0.07500005
#> 17 0.262  0.035 -0.930 2015-12-09 00:00:01 0.00000000 0.04999995
#> 18 0.266  0.035 -0.934 2015-12-09 00:00:01 0.07500005 0.07500005
#> 19 0.270  0.031 -0.934 2015-12-09 00:00:01 0.12500000 0.04999995
#> 20 0.270  0.035 -0.938 2015-12-09 00:00:01 0.20000005 0.07500005
#> 21 0.270  0.035 -0.938 2015-12-09 00:00:01 0.25000000 0.04999995
#> 22 0.266  0.039 -0.934 2015-12-09 00:00:01 0.32500005 0.07500005
#> 23 0.266  0.035 -0.934 2015-12-09 00:00:01 0.37500000 0.04999995
#> 24 0.270  0.031 -0.934 2015-12-09 00:00:01 0.45000005 0.07500005
#> 25 0.266  0.035 -0.938 2015-12-09 00:00:01 0.50000000 0.04999995
#> 26 0.262  0.035 -0.930 2015-12-09 00:00:01 0.57500005 0.07500005
#> 27 0.262  0.031 -0.938 2015-12-09 00:00:01 0.62500000 0.04999995
#> 28 0.262  0.035 -0.934 2015-12-09 00:00:01 0.70000005 0.07500005
#> 29 0.262  0.031 -0.938 2015-12-09 00:00:01 0.75000000 0.04999995
#> 30 0.266  0.035 -0.934 2015-12-09 00:00:01 0.82500005 0.07500005
#> 31 0.266  0.035 -0.930 2015-12-09 00:00:01 0.87500000 0.04999995
#> 32 0.262  0.035 -0.934 2015-12-09 00:00:01 0.95000005 0.07500005
#> 33 0.266  0.035 -0.938 2015-12-09 00:00:02 0.00000000 0.04999995
#> 34 0.262  0.039 -0.934 2015-12-09 00:00:02 0.07500005 0.07500005
#> 35 0.262  0.035 -0.934 2015-12-09 00:00:02 0.12500000 0.04999995
#> 36 0.266  0.035 -0.930 2015-12-09 00:00:02 0.20000005 0.07500005
#> 37 0.262  0.039 -0.938 2015-12-09 00:00:02 0.25000000 0.04999995
#> 38 0.266  0.035 -0.934 2015-12-09 00:00:02 0.32500005 0.07500005
#> 39 0.262  0.035 -0.930 2015-12-09 00:00:02 0.37500000 0.04999995
#> 40 0.262  0.039 -0.938 2015-12-09 00:00:02 0.45000005 0.07500005

^{Created on 2020-02-12 by the reprex package (v0.3.0.9001)}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 3.6.2 (2019-12-12)
#>  os       macOS Mojave 10.14.6        
#>  system   x86_64, darwin15.6.0        
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       America/New_York            
#>  date     2020-02-12                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version     date       lib source                             
#>  AGread      * 1.1.0.9000  2019-10-03 [1] local                              
#>  anytime       0.3.6       2019-08-29 [1] CRAN (R 3.6.0)                     
#>  assertthat    0.2.1       2019-03-21 [1] CRAN (R 3.6.0)                     
#>  backports     1.1.5       2019-10-02 [1] CRAN (R 3.6.0)                     
#>  binaryLogic   0.3.9       2017-12-13 [1] CRAN (R 3.6.0)                     
#>  cli           2.0.1       2020-01-08 [1] CRAN (R 3.6.0)                     
#>  colorspace    1.4-1       2019-03-18 [1] CRAN (R 3.6.0)                     
#>  crayon        1.3.4       2017-09-16 [1] CRAN (R 3.6.0)                     
#>  digest        0.6.23      2019-11-23 [1] CRAN (R 3.6.0)                     
#>  dplyr         0.8.3       2019-07-04 [1] CRAN (R 3.6.0)                     
#>  evaluate      0.14        2019-05-28 [1] CRAN (R 3.6.0)                     
#>  fansi         0.4.1       2020-01-08 [1] CRAN (R 3.6.0)                     
#>  fs            1.3.1       2019-05-06 [1] CRAN (R 3.6.0)                     
#>  ggplot2       3.2.1       2019-08-10 [1] CRAN (R 3.6.0)                     
#>  glue          1.3.1       2019-03-12 [1] CRAN (R 3.6.0)                     
#>  gtable        0.3.0       2019-03-25 [1] CRAN (R 3.6.0)                     
#>  highr         0.8         2019-03-20 [1] CRAN (R 3.6.0)                     
#>  hms           0.5.2       2019-10-30 [1] CRAN (R 3.6.0)                     
#>  htmltools     0.4.0       2019-10-04 [1] CRAN (R 3.6.0)                     
#>  knitr         1.26.1      2020-01-05 [1] Github (muschellij2/knitr@f5af631) 
#>  lazyeval      0.2.2       2019-03-15 [1] CRAN (R 3.6.0)                     
#>  lifecycle     0.1.0       2019-08-01 [1] CRAN (R 3.6.0)                     
#>  lubridate   * 1.7.4       2018-04-11 [1] CRAN (R 3.6.0)                     
#>  magrittr      1.5         2014-11-22 [1] CRAN (R 3.6.0)                     
#>  munsell       0.5.0       2018-06-12 [1] CRAN (R 3.6.0)                     
#>  PAutilities   0.2.0       2019-07-10 [1] CRAN (R 3.6.0)                     
#>  pillar        1.4.3       2019-12-20 [1] CRAN (R 3.6.0)                     
#>  pkgconfig     2.0.3       2019-09-22 [1] CRAN (R 3.6.0)                     
#>  purrr         0.3.3       2019-10-18 [1] CRAN (R 3.6.0)                     
#>  R6            2.4.1       2019-11-12 [1] CRAN (R 3.6.0)                     
#>  Rcpp          1.0.3       2019-11-08 [1] CRAN (R 3.6.0)                     
#>  read.gt3x   * 0.1.0.9000  2020-01-28 [1] local                              
#>  readr       * 1.3.1       2018-12-21 [1] CRAN (R 3.6.0)                     
#>  reprex        0.3.0.9001  2020-01-05 [1] Github (tidyverse/reprex@5ae0b29)  
#>  rlang         0.4.2       2019-11-23 [1] CRAN (R 3.6.0)                     
#>  rmarkdown     2.0.7       2020-01-17 [1] Github (rstudio/rmarkdown@2faf16a) 
#>  rstudioapi    0.10.0-9003 2020-01-05 [1] Github (rstudio/rstudioapi@abe596d)
#>  scales        1.1.0       2019-11-18 [1] CRAN (R 3.6.0)                     
#>  sessioninfo   1.1.1       2018-11-05 [1] CRAN (R 3.6.0)                     
#>  stringi       1.4.5       2020-01-11 [1] CRAN (R 3.6.0)                     
#>  stringr       1.4.0       2019-02-10 [1] CRAN (R 3.6.0)                     
#>  styler        1.1.1       2019-05-06 [1] CRAN (R 3.6.0)                     
#>  tibble        2.1.3       2019-06-06 [1] CRAN (R 3.6.0)                     
#>  tidyselect    0.2.5       2018-10-11 [1] CRAN (R 3.6.0)                     
#>  utf8          1.1.4       2018-05-24 [1] CRAN (R 3.6.0)                     
#>  vctrs         0.2.1       2019-12-17 [1] CRAN (R 3.6.0)                     
#>  withr         2.1.2       2018-03-15 [1] CRAN (R 3.6.0)                     
#>  xfun          0.11        2019-11-12 [1] CRAN (R 3.6.0)                     
#>  yaml          2.2.0       2018-07-25 [1] CRAN (R 3.6.0)                     
#>  zeallot       0.1.0       2018-01-28 [1] CRAN (R 3.6.0)                     
#> 
#> [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

The text was updated successfully, but these errors were encountered:

paulhibbing · 2020-02-13T05:01:27Z

Thanks @muschellij2. This is a robustness issue, somewhat similar to #10. In this file the accelerometer packets are in ACTIVITY format, whereas read_gt3x currently supports only ACTIVITY2 parsing.

One of the limitations for me has been that I haven't encountered certain packet types in the wild. This file has both ACTIVITY and LUX packets (neither of which I've dealt with), so it will potentially serve as a nice way to get those set up. That said, it's going to be months before I can get to it.

muschellij2 · 2020-02-13T20:40:19Z

I agree - definitely an enhancement. The lessons from this example is read_gt3x can't read ACTIVITY packets, ACTIVITY packets are not constrained to "old" GT3X formats, and read.gt3x from @THLfi can read it, but doesn't parse additional information such as LUX or info like CAPSENSE.

JWiley · 2020-06-30T01:44:01Z

Just a note that if more examples with with the newer activity format and LUX would help I have plenty and would be happy to share some. In sleep research, the lux data could be very interesting and would be neat to get through R rather than having to export it manually.

TheTS · 2020-08-28T05:18:06Z

Just experienced this - good to know it's because of the ACTIVITY packets!

>   df <- read_gt3x(file, verbose = T)

Processing Laura (2020-08-16).gt3x 

  Unzipping  ............. COMPLETE
  Checking components  ............. COMPLETE
  Retrieving info from the zip archive  ............. COMPLETE

  Will parse the following packet types, if available:
    METADATA, PARAMETERS, SENSOR_SCHEMA, BATTERY 
    EVENT, TAG, ACTIVITY, HEART_RATE_BPM 
    HEART_RATE_ANT, HEART_RATE_BLE, LUX, CAPSENSE 
    EPOCH, EPOCH2, EPOCH3, EPOCH4 
    ACTIVITY2, SENSOR_DATA 

  Reading log.bin  ............. COMPLETE
  Getting record headers  ............... COMPLETE
  Parsing PARAMETERS packet(s)   ............. COMPLETE                      
  Parsing EVENT packet(s)   ............. COMPLETE                      
  Parsing METADATA packet(s)   ............. COMPLETE                      
  Parsing BATTERY packet(s)   ............. COMPLETE                      
  Parsing ACTIVITY packet(s)
  Parsing LUX packet(s)


Processing complete. Elapsed time 0.08 minutes.
Warning messages:
1: In parse_packet_set.default(X[[i]], ...) :
  No method exists yet for parsing ACTIVITY packets -- returning NULL.
2: In parse_packet_set.default(X[[i]], ...) :
  No method exists yet for parsing LUX packets -- returning NULL.

muschellij2 · 2020-08-28T13:27:38Z

You can try read.gt3x. https://github.com/THLfi/read.gt3x

On Fri, Aug 28, 2020 at 1:18 AM Tom Stewart ***@***.***> wrote: Just experienced this - good to know it's because of the ACTIVITY packets! > df <- read_gt3x(file, verbose = T) Processing Laura (2020-08-16).gt3x Unzipping ............. COMPLETE Checking components ............. COMPLETE Retrieving info from the zip archive ............. COMPLETE Will parse the following packet types, if available: METADATA, PARAMETERS, SENSOR_SCHEMA, BATTERY EVENT, TAG, ACTIVITY, HEART_RATE_BPM HEART_RATE_ANT, HEART_RATE_BLE, LUX, CAPSENSE EPOCH, EPOCH2, EPOCH3, EPOCH4 ACTIVITY2, SENSOR_DATA Reading log.bin ............. COMPLETE Getting record headers ............... COMPLETE Parsing PARAMETERS packet(s) ............. COMPLETE Parsing EVENT packet(s) ............. COMPLETE Parsing METADATA packet(s) ............. COMPLETE Parsing BATTERY packet(s) ............. COMPLETE Parsing ACTIVITY packet(s) Parsing LUX packet(s) Processing complete. Elapsed time 0.08 minutes. Warning messages: 1: In parse_packet_set.default(X[[i]], ...) : No method exists yet for parsing ACTIVITY packets -- returning NULL. 2: In parse_packet_set.default(X[[i]], ...) : No method exists yet for parsing LUX packets -- returning NULL. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIGPLXDO5ORRCXJSZTABSTSC44ZXANCNFSM4KUDXXUA> . --

Best, John

paulhibbing · 2020-08-29T05:41:20Z

Thanks for all this feedback. I've now had a first go at introducing an ACTIVITY packet parser, which you can invoke when you set parser = "dev".

I haven't done a lot of testing on it, but so far it's looked alright. I'd recommend comparing outputs against read_AG_raw for a few files before using this more widely.

My guess is read.gt3x is still faster. Not sure I'll solve that... But this fits right into the recent parser design update, so I do think it flows well.

Let me know if anything isn't working. The build is failing right now on Travis, but it looks like that's due to rJava configuration. Shouldn't be a problem if you install with dependencies = FALSE. (rJava isn't a strong dependency of AGread -- at this point I don't think the issue is related to paulhibbing/PAutilities#2 since RJava is relegated to suggests via matchingMarkets. But that's neither here nor there as far as this issue is concerned.)

paulhibbing · 2020-08-29T05:51:45Z

Travis is (apparently) configured in both repos to install all packages (including those in suggests). After clearing the PAutilities cache, that build fails with the same error AGread is having. That leads me to believe the current AGread failure is related to paulhibbing/PAutilities#2 after all.

Looks like there's already a request for the Travis folks to look into it.

paulhibbing added the enhancement New feature or request label Feb 13, 2020

paulhibbing modified the milestones: read_raw, read_gt3x May 24, 2020

This was referenced Aug 29, 2020

Introduce ACTIVITY packet parsing to dev parser #25

Merged

Parsing LUX packets #26

Open

paulhibbing linked a pull request Sep 1, 2020 that will close this issue

Introduce ACTIVITY packet parsing to dev parser #25

Merged

paulhibbing closed this as completed Jan 27, 2021

paulhibbing mentioned this issue Sep 13, 2021

read_gt3x null activity? #35

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No Activity Data Parsed #19

No Activity Data Parsed #19

muschellij2 commented Feb 12, 2020

paulhibbing commented Feb 13, 2020

muschellij2 commented Feb 13, 2020

JWiley commented Jun 30, 2020 •

edited

Loading

TheTS commented Aug 28, 2020

muschellij2 commented Aug 28, 2020 via email

paulhibbing commented Aug 29, 2020

paulhibbing commented Aug 29, 2020 •

edited

Loading

No Activity Data Parsed #19

No Activity Data Parsed #19

Comments

muschellij2 commented Feb 12, 2020

Issue

paulhibbing commented Feb 13, 2020

muschellij2 commented Feb 13, 2020

JWiley commented Jun 30, 2020 • edited Loading

TheTS commented Aug 28, 2020

muschellij2 commented Aug 28, 2020 via email

paulhibbing commented Aug 29, 2020

paulhibbing commented Aug 29, 2020 • edited Loading

JWiley commented Jun 30, 2020 •

edited

Loading

paulhibbing commented Aug 29, 2020 •

edited

Loading