Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support connecting to HiveServer2 through database connection pools other than HikariCP #33762

Merged
merged 1 commit into from
Nov 22, 2024

Conversation

linghengqian
Copy link
Member

@linghengqian linghengqian commented Nov 21, 2024

For #29052.

Changes proposed in this pull request:

  1. The database started by testcontainers-java is not closed immediately. This only affects unit tests that use the testcontainers java style jdbcurl, and there are only 3 unit tests that use testcontainers like this. This does break testcontainers-java's best practices a bit, as connections created for testcontainers-java are not explicitly handled. The design of testcontainers-java's JDBC support is to retain Docker Containers until the last connection is closed, but shardingsphere caches connections. As part of this, the documentation for unit tests and testcontainers-java integration has been updated to use the utility class org.testcontainers.jdbc.ContainerDatabaseDriver which is part of org.testcontainers.jdbc.AbstractJDBCDriverTest.
  2. org.apache.shardingsphere.transaction.base.seata.at.SeataATShardingSphereTransactionManager#close() is never called normally. This leads to a large number of logs in https://github.com/apache/shardingsphere/actions/runs/11953880634/job/33322727601 due to the failure of seata client to close normally. Maybe we can find a simpler way to avoid using connection.unwrap(ShardingSphereConnection.class).getContextManager().close();.
Error:  2024-11-21 13:26:47.684 [timeoutChecker_2_1] o.a.s.c.r.n.NettyClientChannelManager - connect server failed. can not connect to [127.0.0.1:32771]
org.apache.seata.common.exception.FrameworkException: can not connect to [127.0.0.1:32771]
	at org.apache.seata.core.rpc.netty.NettyClientChannelManager.doReconnect(NettyClientChannelManager.java:251)
	at org.apache.seata.core.rpc.netty.NettyClientChannelManager.doReconnect(NettyClientChannelManager.java:212)
	at org.apache.seata.core.rpc.netty.NettyClientChannelManager.reconnect(NettyClientChannelManager.java:167)
	at org.apache.seata.core.rpc.netty.AbstractNettyRemotingClient.lambda$init$0(AbstractNettyRemotingClient.java:109)
	at [email protected]/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at [email protected]/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
	at [email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at [email protected]/java.lang.Thread.runWith(Thread.java:1583)
	at [email protected]/java.lang.Thread.run(Thread.java:1570)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:853)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:829)

Before committing this PR, I'm sure that I have checked the following options:

  • My code follows the code of conduct of this project.
  • I have self-reviewed the commit code.
  • I have (or in comment I request) added corresponding labels for the pull request.
  • I have passed maven check locally : ./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e.
  • I have made corresponding changes to the documentation.
  • I have added corresponding unit tests for my changes.
  • I have updated the Release Notes of the current development version. For more details, see Update Release Note

@linghengqian linghengqian force-pushed the hive-server2-ha branch 9 times, most recently from 66ea4a1 to a34600a Compare November 22, 2024 12:54
@linghengqian linghengqian marked this pull request as ready for review November 22, 2024 12:58
Comment on lines +84 to +86
try (Connection connection = dataSource.getConnection()) {
connection.unwrap(ShardingSphereConnection.class).getContextManager().close();
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • I don't really know if there is an easier way to trigger a call to org.apache.shardingsphere.transaction.base.seata.at.SeataATShardingSphereTransactionManager#close() than java.sql.Connection.unwrap(ShardingSphereConnection.class).getContextManager().close();. What do you think?

Comment on lines +130 to +140
try (Connection connection = dataSource.getConnection()) {
Class<?> hiveConnectionClass = Class.forName("org.apache.hive.jdbc.HiveConnection");
if (connection.isWrapperFor(hiveConnectionClass)) {
Object hiveConnection = connection.unwrap(hiveConnectionClass);
String connectedUrl = (String) hiveConnectionClass.getMethod("getConnectedUrl").invoke(hiveConnection);
return DatabaseTypeFactory.get(connectedUrl);
}
throw new SQLWrapperException(sqlFeatureNotSupportedException);
} catch (final SQLException | ClassNotFoundException | NoSuchMethodException | InvocationTargetException | IllegalAccessException exception) {
throw new RuntimeException(exception);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@strongduanmu strongduanmu merged commit dff17cf into apache:master Nov 22, 2024
148 checks passed
@linghengqian linghengqian deleted the hive-server2-ha branch November 22, 2024 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants