Many java.io.IOException: Filesystem closed

[ylangisc]

Apologies for directly opening an issue but I didn't found a way to just drop a question.

I've tried to backup parts of my hdfs with your class and I constantly get several IOException (see below). Just using distcp works fine. Any ideas?

Thanks
Yves

---

[ec2-user@ip-10-40-211-16 ~]$ sudo /usr/local/hadoop-1.0.0/bin/hadoop jar akela-0.5-SNAPSHOT-job.jar com.mozilla.hadoop.Backup hdfs://ip-10-40-211-16.ec2.internal:8020/hbase/logfile local2
/usr/local/hbase-0.92.0/hbase-0.92.0.jar:/usr/local/hbase-0.92.0/lib/zookeeper-3.3.1.jar:/usr/local/hbase-0.92.0/conf:/usr/local/hbase-0.92.0/lib/guava-r09.jar:/usr/local/hbase-0.92.0/lib/zookeeper-3.4.2.jar
Adding input path: Backup-inputsource0.txt
Adding input path: Backup-inputsource1.txt
12/11/11 10:06:16 INFO input.FileInputFormat: Total input paths to process : 2
12/11/11 10:06:16 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
12/11/11 10:06:16 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev c7d54fffe5a853c437ee23413ba71fc6af23c91d]
12/11/11 10:06:16 INFO mapred.JobClient: Running job: job_201210221953_0008
12/11/11 10:06:17 INFO mapred.JobClient: map 0% reduce 0%
12/11/11 10:06:31 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000001_0, Status : FAILED
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at org.apache.hadoop.util.LineReader.close(LineReader.java:83)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:34 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000000_0, Status : FAILED
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at org.apache.hadoop.util.LineReader.close(LineReader.java:83)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:37 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000001_1, Status : FAILED
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at org.apache.hadoop.util.LineReader.close(LineReader.java:83)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:40 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000000_1, Status : FAILED
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at org.apache.hadoop.util.LineReader.close(LineReader.java:83)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:43 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000001_2, Status : FAILED
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at org.apache.hadoop.util.LineReader.close(LineReader.java:83)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:46 INFO mapred.JobClient: Task Id : attempt_201210221953_0008_m_000000_2, Status : FAILED
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:264)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:74)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2132)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at org.apache.hadoop.util.LineReader.close(LineReader.java:83)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/11/11 10:06:55 INFO mapred.JobClient: Job complete: job_201210221953_0008
12/11/11 10:06:55 INFO mapred.JobClient: Counters: 7
12/11/11 10:06:55 INFO mapred.JobClient: Job Counters
12/11/11 10:06:55 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=50404
12/11/11 10:06:55 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
12/11/11 10:06:55 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
12/11/11 10:06:55 INFO mapred.JobClient: Launched map tasks=8
12/11/11 10:06:55 INFO mapred.JobClient: Data-local map tasks=8
12/11/11 10:06:55 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
12/11/11 10:06:55 INFO mapred.JobClient: Failed map tasks=1`

[xstevens]

From what it looks like. I think the problem is that you're running this on the same cluster for input/output. There's some assumptions (perhaps bad ones) in Backup that you're going from one cluster to another. If you wanted to fix this for your case take a look at the setup and closing of inputFs and outputFs. The other thing to watch out for is "fs.automatic.close" could be messing with you on inputFs, but I'm not sure without trying to debug it.

Just FYI, HBase has some good tools for this type of thing now as well. Take a look at their Export/Import jobs.