HADOOP_CMD getting lost...

[bhermont]

Hi Antonio,

I've faced a scenario where I call a mapreduce (rmr) from a Shell script inside a Mapper (This is how Oozie launches a Shell action)

Here is the flow:
Oozie Launcher Job -> Lancher Map only task where shell script (Rscript myscript.r) executes -> StreamJob -> Mapper / Reducers

myscript.r

Sys.setenv(JAVA_HOME="/usr/jdk64/jdk1.7.0_67")
Sys.setenv(HADOOP_CMD="/usr/bin/hadoop")
Sys.setenv(HADOOP_HOME="/usr/hdp/2.2.6.0-2800/hadoop")
Sys.setenv(HADOOP_STREAMING="/usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-streaming.jar")
library("rhdfs")
library("rmr2")
hdfs.init()
library(Matrix)

Logs

Launcher Job (Job that launches Map only task)
On this map, the following shell script is executed as a system call (Rscript myscript.r) - Here HADOOP_CMD is set correctly, but an error on the mr function is logged (Probably due the Streaming Mapper error when a hdfs function is called from inside the mr function). Launcher (Mapper) log:

Loading required package: methods
Loading required package: rJava

HADOOP_CMD=/usr/bin/hadoop

Be sure to run hdfs.init()
Please review your hadoop settings. See help(hadoop.settings)
Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce,  : 
  hadoop streaming failed with error code 15
Calls: getTags -> mapreduce -> mr

The rmr streaming starts a StreamJob which can have Mappers and Reducers. StreamJob (Mapper) log:

    Log Type: stderr
    Log Upload Time: Thu Aug 06 19:24:41 -0400 2015

    Log Length: 2722

    Loading objects:
    Loading objects:
      backend.parameters
      combine
    Please review your hadoop settings. See help(hadoop.settings)
      combine.file
      combine.line
      debug
      default.input.format
      default.output.format
      in.folder
      in.memory.combine
      input.format
      libs
      map
      map.file
      map.line
      out.folder
      output.format
      pkg.opts
      postamble
      preamble
      profile.nodes
      reduce
      reduce.file
      reduce.line
      rmr.global.env
      rmr.local.env
      save.env
      tempfile
      vectorized.reduce
      verbose
      work.dir
    Loading required package: methods
    Loading required package: rJava
    Loading required package: rhdfs
    Error : .onLoad failed in loadNamespace() for 'rhdfs', details:
      call: fun(libname, pkgname)
      error: Environment variable HADOOP_CMD must be set before loading package rhdfs
    Warning in FUN(X[[i]], ...) : can't load rhdfs
    Loading required package: rmr2
    Loading required package: Matrix

However, calling the myscript.r from command line works fine.

Here is my question:
Should rmr propagate the environment envs in this case, or this should be responsibility of the enviroment to provide the value for the HADOOP_CMD variable?

[bhermont]

One update:
Setting HADOOP_CMD in the enviroment (not from R) solved the "Environment variable HADOOP_CMD must be set before loading package rhdfs" issue, but the hadoop streaming error 15 persists..

[piccolbo]

On the env vars, rmr doesn't meddle with them, unless you reach for the low level escape hatch which is backend.options, then you can do anything and the opposite. The reason why it doesn't try to is because there is no reason to think that any of the settings in the user environment are appropriate for the cluster environment. As far as the error 15, you are the first on the whole internet to report it. It may require additional investigation. First thing, simplify your setup by detaching rhdfs and rjava before the first call to mapreduce. Second, show the logs according to the latest experiment. It's important to see everything, not just know that the error is the same. As far as starting a mapreduce job from a mapper, that's really beyond what is supported not only by rmr but also by mapreduce, now or ever. tasks are supposed to lack side effects or to have idempotent side effects, because they can be retried for any or no reason. I don't know how oozie works, but I know that much about mapreduce.