java.lang.NoSuchMethodError: org.apache.logging.log4j.Logger.info #1058

abhishek6s · 2023-05-18T11:44:51Z

abhishek6s
May 18, 2023

I'm using hive 2.0.1 and hadoop 2.10.0. when I used yauaa-hive-7.19.2-SNAPSHOT-udf.jar -

hive (default)> ADD JAR hdfs:///user/abc/yauaa-hive-7.19.2-SNAPSHOT-udf.jar;
converting to local hdfs:///user/abc/yauaa-hive-7.19.2-SNAPSHOT-udf.jar
Added [/mnt/hive/tmp/319e971b-0c54-46c2-803d-26c64f48c962_resources/yauaa-hive-7.19.2-SNAPSHOT-udf.jar] to class path
Added resources: [hdfs:///user/abc/yauaa-hive-7.19.2-SNAPSHOT-udf.jar]

hive (default)> CREATE FUNCTION ParseUserAgent_v3
              > AS 'nl.basjes.parse.useragent.hive.ParseUserAgent'
              > USING JAR 'hdfs:///user/abc/yauaa-hive-7.19.2-SNAPSHOT-udf.jar';
converting to local hdfs:///user/abc/yauaa-hive-7.19.2-SNAPSHOT-udf.jar
Added [/mnt/hive/tmp/319e971b-0c54-46c2-803d-26c64f48c962_resources/yauaa-hive-7.19.2-SNAPSHOT-udf.jar] to class path
Added resources: [hdfs:///user/abc/yauaa-hive-7.19.2-SNAPSHOT-udf.jar]

hive (default)> select parseuseragent_v3('Mozilla/5.0 (Linux\; Android 6.0\; Nexus 6 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36');
Exception in thread "de12ba54-5bd7-4f94-bd39-9291652b4e12 main" java.lang.NoSuchMethodError: org.apache.logging.log4j.Logger.info(Ljava/lang/String;Ljava/lang/Object;)V
	at nl.basjes.parse.useragent.utils.YauaaVersion.logVersion(YauaaVersion.java:129)
	at nl.basjes.parse.useragent.utils.YauaaVersion.logVersion(YauaaVersion.java:109)
	at nl.basjes.parse.useragent.utils.YauaaVersion.logVersion(YauaaVersion.java:102)
	at nl.basjes.parse.useragent.AbstractUserAgentAnalyzerDirect$AbstractUserAgentAnalyzerDirectBuilder.build(AbstractUserAgentAnalyzerDirect.java:545)
	at nl.basjes.parse.useragent.AbstractUserAgentAnalyzer$AbstractUserAgentAnalyzerBuilder.build(AbstractUserAgentAnalyzer.java:339)
	at nl.basjes.parse.useragent.hive.ParseUserAgent.lambda$static$0(ParseUserAgent.java:85)
	at java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305)
	at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
	at java.lang.ThreadLocal.get(ThreadLocal.java:172)
	at nl.basjes.parse.useragent.hive.ParseUserAgent.initialize(ParseUserAgent.java:114)
	at org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:139)
	at org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:230)
	at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:1097)
	at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1316)
	at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
	at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
	at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
	at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:158)
	at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
	at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:213)
	at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:157)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:10519)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:10475)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3843)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3622)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8895)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8850)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9703)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9596)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10092)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:353)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10103)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:228)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:473)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:319)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1249)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1295)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1178)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1166)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:236)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:782)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:721)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:648)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:158)

hive is having below logging lib

log4j-slf4j-impl-2.4.1.jar
log4j-api-2.4.1.jar
log4j-web-2.4.1.jar
log4j-1.2-api-2.4.1.jar
log4j-core-2.4.1.jar

can anyone guide me on this what needs to be done in order to tix this?

Also, how do I maintain this library for my hive and hadoop versions ? ( will test run respective of my hive and hadoop versions when we change in respective versions pom.xml ? )

Answered by nielsbasjes

Jun 5, 2023

Curious to hear if this hack works on your machine
https://github.com/nielsbasjes/yauaa/tree/HiveLog4jDependencies

View full answer

nielsbasjes · 2023-05-18T17:35:35Z

nielsbasjes
May 18, 2023
Maintainer

Few things I noticed:

Why have you built Yauaa yourself instead of downloading the prebuilt version?
Hive 2.0.1 is from 2016 and the Hadoop version you have is from 2019. All of these are really old, which may cause incompatibilities.
Hive is a distributed system. You have to make sure that all instances have these dependencies.

7 replies

nielsbasjes May 20, 2023
Maintainer

Perhaps a simpler way:

Put all Log4j jars on hdfs
ADD JAR each one
Go from there.

abhishek6s May 22, 2023
Author

what do we mean here by putting all log4j jars on hdfs? If I understand correctly, we already have log4j jars coming from hive lib (as mentioned in the header) so we can't overwrite them.

nielsbasjes May 22, 2023
Maintainer

I meant something like hdfs dfs -put log4j-api-2.20.0.jar followed by ADD JAR hdfs:///user/abc/log4j-api-2.20.0.jar ; in Hive.
If you were already having these libraries then you would not get the error you asked about.

abhishek6s May 22, 2023
Author

as an experiment, we remove the old log4j and put in the new version of log4j jars. And it worked. But we can't remove old log4j jars in production as all the components like Hadoop, HIVE, HMS, and HS2 are on the old log4j versions.

nielsbasjes May 22, 2023
Maintainer

Hmmm, ok. Tricky. I have to think about this.

nielsbasjes · 2023-05-19T21:23:46Z

nielsbasjes
May 19, 2023
Maintainer

Since you are already building it yourself perhaps making these changes to the sources helps in your case:

diff --git a/udfs/hive/pom.xml b/udfs/hive/pom.xml
index 6f6efda13..20867b0e2 100644
--- a/udfs/hive/pom.xml
+++ b/udfs/hive/pom.xml
@@ -53,6 +53,16 @@
       <artifactId>yauaa</artifactId>
     </dependency>
 
+    <dependency>
+      <groupId>org.apache.logging.log4j</groupId>
+      <artifactId>log4j-api</artifactId>
+    </dependency>
+
+    <dependency>
+      <groupId>org.apache.logging.log4j</groupId>
+      <artifactId>log4j-core</artifactId>
+    </dependency>
+
     <dependency>
       <groupId>org.apache.commons</groupId>
       <artifactId>commons-collections4</artifactId>

5 replies

abhishek6s May 22, 2023
Author

The shading isn't working somehow. Unable to figure out whats the issue. @nielsbasjes can you help

nielsbasjes Jun 5, 2023
Maintainer

First thought is you also need <useDependencyReducedPomInJar>true</useDependencyReducedPomInJar>

nielsbasjes Jun 5, 2023
Maintainer

Have your checked the logging dependencies are actually IN the Jar you are using? If you have the ordering of the assembly and shade plugins in the incorrect order you will not get the desired endresult.

nielsbasjes Jun 5, 2023
Maintainer

Curious to hear if this hack works on your machine
https://github.com/nielsbasjes/yauaa/tree/HiveLog4jDependencies

Answer selected by nielsbasjes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

java.lang.NoSuchMethodError: org.apache.logging.log4j.Logger.info #1058

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 12 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

java.lang.NoSuchMethodError: org.apache.logging.log4j.Logger.info #1058

abhishek6s May 18, 2023

Replies: 2 comments · 12 replies

nielsbasjes May 18, 2023 Maintainer

nielsbasjes May 20, 2023 Maintainer

abhishek6s May 22, 2023 Author

nielsbasjes May 22, 2023 Maintainer

abhishek6s May 22, 2023 Author

nielsbasjes May 22, 2023 Maintainer

nielsbasjes May 19, 2023 Maintainer

abhishek6s May 22, 2023 Author

nielsbasjes Jun 5, 2023 Maintainer

nielsbasjes Jun 5, 2023 Maintainer

nielsbasjes Jun 5, 2023 Maintainer

abhishek6s
May 18, 2023

Replies: 2 comments 12 replies

nielsbasjes
May 18, 2023
Maintainer

nielsbasjes May 20, 2023
Maintainer

abhishek6s May 22, 2023
Author

nielsbasjes May 22, 2023
Maintainer

abhishek6s May 22, 2023
Author

nielsbasjes May 22, 2023
Maintainer

nielsbasjes
May 19, 2023
Maintainer

abhishek6s May 22, 2023
Author

nielsbasjes Jun 5, 2023
Maintainer

nielsbasjes Jun 5, 2023
Maintainer

nielsbasjes Jun 5, 2023
Maintainer