Reduce task fails with org.apache.hadoop.streaming.PipeMapRed: PipeMapRed failed!

[RavikiranCK]

Hi ,

I'm trying to emit the key and list of values belonging to the key as list(value).
Following is sudo code
keyValLNRed <- function(key,val)
{
keyval(key,list(val))
}

So Basically I want all values belonging to key ,must be emitted in value so I made it as list.
Example : x<- [1:3]
x
[1] 1 2 3
length(x) will be 3.

But since key is of length 1 , if I emit keyval(key,x) mapreduce job gets aborted.Because we have to maintain one to one relation between length of key and value pairs.

So What I'm doing is y <- list(x)
y
[[1]]
[1] 1 2 3
length(y)
1

So now I can emit keyval as keyval(key,y)

I have written mapreduce chain(Same mapper and reducer function ) where output of one mapreduce job is input to the next mapreduce job.
I have created chain of mapreduce jobs having same map and reduce function.
Works fine till the 2nd mapreduce job where above keyval emitted successfully but later stages it's failing in reduce part.

Error is :2017-04-05 12:08:24,770 INFO [main] org.apache.hadoop.streaming.PipeMapRed: PipeMapRed failed!
java.lang.RuntimeException: java.io.EOFException
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:336)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:453)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at org.apache.hadoop.typedbytes.TypedBytesInput.readRawBytes(TypedBytesInput.java:218)
at org.apache.hadoop.typedbytes.TypedBytesInput.readRaw(TypedBytesInput.java:152)
at org.apache.hadoop.typedbytes.TypedBytesInput.readRawVector(TypedBytesInput.java:412)
at org.apache.hadoop.typedbytes.TypedBytesInput.readRaw(TypedBytesInput.java:144)
at org.apache.hadoop.streaming.io.TypedBytesOutputReader.readKeyValue(TypedBytesOutputReader.java:56)
at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:378)

Could anyone help me where and why it's failing ??
Any suggestion on different approach to emit key value is welcomed.

Thank you in advance.

Ravikiran C K

[saargolde]

The only way to know what's really wrong with your code is to get the proper error message.
This is a good guide for debugging rmr code: https://github.com/RevolutionAnalytics/RHadoop/wiki/user%3Ermr%3EDebugging-rmr-programs

[RavikiranCK]

Thank you for your suggestion Saargolde..
Will try to find key value pair where it's failing..