Broadcasting operations

[darrenjw]

I'm a bit confused about the correct way to access and use broadcasting operations in storch. For example, in pytorch, if I had a rank 2 tensor (a matrix), a, I could get the column means with something like

torch.mean(a, 0)

This doesn't seem to work in storch. I can find a binding to this function buried in some private reduction ops, but I'm not sure what the correct way is to access it.

Also, if I wanted to use this mean vector to "sweep out" the column means from my matrix a, what would be the correct way to do this in storch?

[davoclavo]

Hi @darrenjw !

At first glance, it seems that torch.mean is working as expected, albeit a bug I just stumbled upon but not sure if its what you are facing (details below).

However, I would love to hear if you have a distinct use case or piece of code that I could use to replicate some problem you may be facing.

Thanks and have a great weekend!

Expected behaviour

Python

In [10]: a = torch.randn(4,4)

In [11]: a
Out[11]:
tensor([[ 1.9802,  0.9503, -0.1127, -0.1499],
        [ 0.6870,  0.0417, -0.6196,  0.5293],
        [ 0.1721,  1.2434,  1.2598,  0.4497],
        [-1.8520,  1.0415, -0.1979, -0.3671]])

In [12]: torch.mean(a, 0)
Out[12]: tensor([0.2468, 0.8192, 0.0824, 0.1155])

Scala

@ val a = torch.randn(Seq(4,4))
a: Tensor[Float32] = tensor dtype=float32, shape=[4, 4], device=CPU
[[0.1366, 0.3390, 0.7543, -1.3827],
 [0.7087, 1.2063, -0.3736, 1.7366],
 [1.2781, 0.5261, -1.3298, -0.6429],
 [0.2031, -0.3348, 0.5529, 0.6076]]

@ torch.mean(a, 0)
res1: Tensor[Float32] = tensor dtype=float32, shape=[4], device=CPU
[0.5816, 0.4341, -0.0990, 0.0796]

Bug

torch.mean does not accept Int32 Tensors, but it is currently not specified in the type signature (I can fix that soon)

@ val b = torch.Tensor(Seq(Seq(1,2,3), Seq(1,2,3), Seq(1,2,3)))
b: Tensor[Int32] = tensor dtype=int32, shape=[3, 3], device=CPU
[[1, 2, 3],
 [1, 2, 3],
 [1, 2, 3]]

@ torch.mean(b, 0)
java.lang.RuntimeException: mean(): could not infer output dtype. Optional dtype must be either a floating point or complex dtype. Got: Int
Exception raised from meta at /Users/runner/work/javacpp-presets/javacpp-presets/pytorch/cppbuild/macosx-x86_64/pytorch/aten/src/ATen/native/ReduceOps.cpp:301 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) + 81 (0x1369a8481 in libc10.dylib)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 98 (0x1369a6b42 in libc10.dylib)
frame #2: at::meta::structured_mean_dim::meta(at::Tensor const&, c10::OptionalArrayRef<long long>, bool, c10::optional<c10::ScalarType>) + 587 (0x1c70a104b in libtorch_cpu.dylib)
frame #3: c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, c10::OptionalArrayRef<long long>, bool, c10::optional<c10::ScalarType>), &(at::(anonymous namespace)::wrapper_CPU_mean_dim(at::Tensor const&, c10::OptionalArrayRef<long long>, bool, c10::optional<c10::ScalarType>))>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, c10::OptionalArrayRef<long long>, bool, c10::optional<c10::ScalarType> > >, at::Tensor (at::Tensor const&, c10::OptionalArrayRef<long long>, bool, c10::optional<c10::ScalarType>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, c10::OptionalArrayRef<long long>, bool, c10::optional<c10::ScalarType>) + 84 (0x1c7f79804 in libtorch_cpu.dylib)
frame #4: at::_ops::mean_dim::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::OptionalArrayRef<long long>, bool, c10::optional<c10::ScalarType>) + 133 (0x1c7d286c5 in libtorch_cpu.dylib)
frame #5: c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (c10::DispatchKeySet, at::Tensor const&, c10::OptionalArrayRef<long long>, bool, c10::optional<c10::ScalarType>), &(torch::autograd::VariableType::(anonymous namespace)::mean_dim(c10::DispatchKeySet, at::Tensor const&, c10::OptionalArrayRef<long long>, bool, c10::optional<c10::ScalarType>))>, at::Tensor, c10::guts::typelist::typelist<c10::DispatchKeySet, at::Tensor const&, c10::OptionalArrayRef<long long>, bool, c10::optional<c10::ScalarType> > >, at::Tensor (c10::DispatchKeySet, at::Tensor const&, c10::OptionalArrayRef<long long>, bool, c10::optional<c10::ScalarType>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, c10::OptionalArrayRef<long long>, bool, c10::optional<c10::ScalarType>) + 1332 (0x1c99c31f4 in libtorch_cpu.dylib)
frame #6: at::_ops::mean_dim::call(at::Tensor const&, c10::OptionalArrayRef<long long>, bool, c10::optional<c10::ScalarType>) + 363 (0x1c7c97f9b in libtorch_cpu.dylib)
frame #7: Java_org_bytedeco_pytorch_global_torch_mean__Lorg_bytedeco_pytorch_Tensor_2_3JZLorg_bytedeco_pytorch_ScalarTypeOptional_2 + 471 (0x1b9e0df87 in libjnitorch.dylib)
frame #8: 0x0 + 4833144508 (0x12013eebc in ???)

  org.bytedeco.pytorch.global.torch.mean(Native Method)
  torch.ops.ReductionOps.mean(ReductionOps.scala:354)
  torch.ops.ReductionOps.mean$(ReductionOps.scala:44)
  torch.package$.mean(package.scala:32)
  ammonite.$sess.cmd3$.<clinit>(cmd3.sc:1)

[darrenjw]

Thanks, and sorry for wasting your time. I was just a couple of snapshots behind the latest version.

[davoclavo]

Glad you figured out! I am super happy to help and hear what could be improved. Please don't hesitate to ask any question in the future :)

Actually thanks to your report another bug was uncovered so thanks for that!

[davoclavo]

After digging a bit deeper, I think I found another bug, perhaps that is what you are facing. I copied the randn tensor from python, and calculated torch.mean and I am getting different values:

Is this the problem you are facing?

Update: Nevermind, these results seem to be correct

Python

In [10]: a = torch.randn(4,4)

In [11]: a
Out[11]:
tensor([[ 1.9802,  0.9503, -0.1127, -0.1499],
        [ 0.6870,  0.0417, -0.6196,  0.5293],
        [ 0.1721,  1.2434,  1.2598,  0.4497],
        [-1.8520,  1.0415, -0.1979, -0.3671]])

In [12]: torch.mean(a, 0)
Out[12]: tensor([0.2468, 0.8192, 0.0824, 0.1155])

Scala

@ val x = torch.Tensor(Seq(
  Seq( 1.9802,  0.9503, -0.1127, -0.1499),
  Seq(0.6870,  0.0417, -0.6196,  0.5293),
  Seq( 0.1721,  1.2434,  1.2598,  0.4497),
  Seq(-1.8520,  1.0415, -0.1979, -0.3671)))
x: Tensor[Float64] = tensor dtype=float64, shape=[4, 4], device=CPU
[[1.9802, 0.9503, -0.1127, -0.1499],
 [0.6870, 0.0417, -0.6196, 0.5293],
 [0.1721, 1.2434, 1.2598, 0.4497],
 [-1.8520, 1.0415, -0.1979, -0.3671]]

@ torch.mean(x, 0)
res7: Tensor[Float64] = tensor dtype=float64, shape=[4], device=CPU
[0.2468, 0.8192, 0.0824, 0.1155]