Gforce grouping var class

NEWS.md

@@ -58,6 +58,8 @@
 
 7. `melt` returns an integer column for `variable` when `measure.vars` is a list of length=1, consistent with the documented behavior, [#5209](https://github.com/Rdatatable/data.table/issues/5209). Thanks to @tdhock for reporting and fixing. Any users who were relying on this behavior can change `measure.vars=list("col_name")` (output `variable` was column name, now is column index/integer) to `measure.vars="col_name"` (`variable` still is column name).
 
+8. `DT[,j,by]` could lose `class` attributes of `by` when `j` was GForce optimized, [#5567](https://github.com/Rdatatable/data.table/issues/5567). Thanks to @danwwilson for the report, and @ben-schwen for the PR.
+
 ## NOTES
 
 1. `transform` method for data.table sped up substantially when creating new columns on large tables. Thanks to @OfekShilon for the report and PR. The implemented solution was proposed by @ColeMiller1.

R/data.table.R

@@ -1842,7 +1842,7 @@ replace_dot_alias = function(e) {
     if (use.I) assign(".I", seq_len(nrow(x)), thisEnv)
     ans = gforce(thisEnv, jsub, o__, f__, len__, irows) # irows needed for #971.
     gi = if (length(o__)) o__[f__] else f__
-    g = lapply(grpcols, function(i) groups[[i]][gi])
+    g = lapply(grpcols, function(i) .Call(CsubsetVector, groups[[i]], gi)) # use CsubsetVector instead of [ since this preserves attributes #5567
 
     # returns all rows instead of one per group
     nrow_funs = c("gshift")

inst/tests/tests.Rraw

@@ -18566,3 +18566,14 @@ test(2261.04, setNumericRounding(2L), 1L)
 # or not an object is an invisible copy or not, and prints it anyways.
 test(2261.05, capture.output(setNumericRounding(2L)), character(0))
 setNumericRounding(old)
+
+# GForce drops classes in by arguments #5567
+dt = data.table(a=letters[1:4], b=structure(1:4, class = c("class_b", "integer")), c=structure(c(1L,2L,1L,2L), class = c("class_c", "integer")))
+old = options(datatable.verbose=TRUE)
+options(datatable.optimize=0)
+test(2262.1, dt[, .N, b], data.table(b=dt$b, N=1L), output="GForce FALSE")
+test(2262.2, dt[, .N, .(b,c)], data.table(b=dt$b, c=dt$c, N=1L), output="GForce FALSE")
+options(datatable.optimize=Inf)
+test(2262.3, dt[, .N, b], data.table(b=dt$b, N=1L), output="GForce optimized j to")
+test(2262.4, dt[, .N, .(b,c)], data.table(b=dt$b, c=dt$c, N=1L), output="GForce optimized j to")
+options(old)