MINOR: Use JDK 11 in Vagrant after dropping JDK 8 #17576

dongnuo123 · 2024-10-22T13:17:37Z

We have dropped JDK 8 and now use JDK 11 as the minimum version. This PR updates the jdk_major and jdk_full versions used by Vagrant to JDK 11.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

bbejeck · 2024-10-22T13:45:34Z

@dongnuo123 do you have a link trying this on branch builder?

dongnuo123 · 2024-10-22T14:24:35Z

@dongnuo123 do you have a link trying this on branch builder?

The branch builder is fine. The successful run is with the pr's change. But we need to run the whole suite of system tests to make sure the change doesn't break anything. Just triggered https://semaphore.ci.confluent.io/workflows/ccd7d20c-5752-47a1-8b9b-590bc77d7988?pipeline_id=a10cc826-0c91-4dd3-8d2c-fdae0373a446

jolshan · 2024-10-22T15:08:37Z

Thanks @dongnuo123 for opening this! Let's confirm all the system tests look good before merging.

chia7712 · 2024-10-22T18:41:04Z

We are planning to drop JDK 11 for server modules (https://issues.apache.org/jira/browse/KAFKA-16096), and the base image for E2E has already been updated to use JDK 17 (#17520). Therefore, maybe Vagrant should be updated to JDK 17 directly in this PR?

jolshan · 2024-10-23T00:10:13Z

I wasn't sure if there would be any incompatibilities if we drop older java versions with some of the older versions of Kafka used in the tests. But I suppose if we do see issues, maybe we don't need to test such old versions either.

Looking at the test results, there were about 20 failures with java 11, and most looked unrelated to the change in jdk. @dongnuo123 do you want to try updating to java 17?

chia7712 · 2024-10-23T13:13:18Z

I wasn't sure if there would be any incompatibilities if we drop older java versions with some of the older versions of Kafka used in the tests. But I suppose if we do see issues, maybe we don't need to test such old versions either.

In the container E2E tests, we've dropped some older versions as per KAFKA-14560. I think we can make similar changes for Vagrant.

dongnuo123 · 2024-10-23T13:37:07Z

https://semaphore.ci.confluent.io/jobs/98fe335d-7619-47bc-a9b8-4c8449ec745a
Triggered the task with java 17

jolshan · 2024-10-24T05:26:34Z

Looking at the test results, seems there were issues with tests up to 2.3.1. @chia7712 are we planning to drop up to that version?

For comparison, with java 11, I only saw issues with 0.8.2.2 (which also fails for 17)

chia7712 · 2024-10-24T09:47:37Z

Looking at the test results, seems there were issues with tests up to 2.3.1. @chia7712 are we planning to drop up to that version?

Sorry, I can't access the Confluent link. Could you please share the details of the issue with me? Additionally, the new base line is 2.1 so 2.3.1 should work in compatibility tests.

I only saw issues with 0.8.2.2 (which also fails for 17)

What kind of issue are you referring to? The issue with upgrading to JDK 11 is that brokers older than version 1.0 can't run on JDK 11 (see KAFKA-5076), but #17427 has already removed them.

jolshan · 2024-10-25T00:03:59Z

@chia7712 you should be able to see this one: http://confluent-open-source-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/trunk/2024-10-23--001.98fe335d-7619-47bc-a9b8-4c8449ec745a--1729721446--confluentinc--update-vagrant-jdk-version--8e89cc0d04/report.html

sanity_checks.test_performance_services still runs with 0.8.2.2 and fails

Most of the tests 2.0.x-2.3.x I mentioned fail because the server can't start up. I didn't dig deeper into why yet.

jolshan · 2024-10-25T00:10:18Z

Digging deeper, looks like an issue with ZK -- the brokers see kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING

and on ZK some connection/EOF exceptions

chia7712 · 2024-10-26T03:01:07Z

sanity_checks.test_performance_services still runs with 0.8.2.2 and fails

open https://issues.apache.org/jira/browse/KAFKA-17879 for it

chia7712 · 2024-10-27T20:30:06Z

sanity_checks.test_performance_services still runs with 0.8.2.2 and fails

I notice another issue and https://issues.apache.org/jira/browse/KAFKA-17883 will fix it

jolshan · 2024-10-28T16:27:50Z

@chia7712 any ideas about the ZK issues I saw in 2.0-2.3? I can try to do some more debugging as well.

chia7712 · 2024-10-28T20:59:09Z

any ideas about the ZK issues I saw in 2.0-2.3? I can try to do some more debugging as well.

It seems Zookeeper client 3.4.x can't run under JDK 17 ... https://issues.apache.org/jira/browse/ZOOKEEPER-3779

I have opened https://issues.apache.org/jira/browse/KAFKA-17888 to fix it

jolshan · 2024-10-28T21:20:20Z

Awesome! Thanks @chia7712! Is there a way to set a jdk for specific tests? 🤔
Should we somehow try to include in this PR?

chia7712 · 2024-10-28T21:25:06Z

Awesome! Thanks @chia7712! Is there a way to set a jdk for specific tests? 🤔
Should we somehow try to include in this PR?

Maybe we should pre-download different JDK versions and install them in /opt/..., similar to the Kafka versions. We could then enhance fix_opts_for_new_jvm to configure JAVA_HOME based on the Kafka version.

chia7712 · 2024-10-28T21:42:48Z

@jolshan We'll file a PR later, and I think we can align the JDK folder structure with Vagrant by using /opt/jdk/$JDK_MAJOR. This means there will be two JDK folders: /opt/jdk/17 and /opt/jdk/11.

chia7712 · 2024-10-30T18:36:12Z

We'll file a PR later,

see #17625

cmccabe · 2024-10-31T23:24:39Z

We are removing zookeeper, so let's not spend time on getting it to run.

Let's just move to jdk11 in the tests for now, and once the removal is fully done we can go to jdk17 (assuming Kafka 3.3 can run on that JDK)

Please remember that the first production KRaft version with 3.3, and we do not support upgrades from versions earlier than that. (We will test older clients, of course, but clients don't use ZK.)

jolshan · 2024-10-31T23:30:11Z

Looks like we should just convert the streams, connect, and clients tests to not use ZK 👍

jolshan · 2024-10-31T23:30:28Z

tests/setup.py

@@ -51,7 +51,7 @@ def run_tests(self):
      license="apache2.0",
      packages=find_packages(),
      include_package_data=True,
-      install_requires=["ducktape==0.12.0", "requests==2.31.0", "psutil==5.7.2"],
+      install_requires=["ducktape==0.8.14", "requests==2.24.0", "psutil==5.7.2"],


Let's also remember to revert this :) and keep 0.12

cmccabe · 2024-10-31T23:31:44Z

Looks like we should just convert the streams, connect, and clients tests to not use ZK 👍

several PRs landed recently:

    KAFKA-17609:[3/4]Convert system tests to kraft part 3 (#17327)
    KAFKA-17609:[4/4]Convert system tests to kraft part 4 (#17328)
    KAFKA-17609:[4/2]Convert system tests to kraft part 2

still waiting for part 1

chia7712 · 2024-11-01T04:39:20Z

Please remember that the first production KRaft version with 3.3, and we do not support upgrades from versions earlier than that. (We will test older clients, of course, but clients don't use ZK.)

Do you mean that we don't need to test the new client (4.0) producing and consuming from older brokers (2.1 to 2.3)?

kirktrue · 2024-11-01T17:57:53Z

Looks like we should just convert the streams, connect, and clients tests to not use ZK 👍

For clients, I have this massive PR opened (but it's WIP). It removes ZK from as many places as easily possible for system and integration tests.

cmccabe · 2024-11-01T19:41:08Z

Do you mean that we don't need to test the new client (4.0) producing and consuming from older brokers (2.1 to 2.3)?

This is a fair question. 2.3 was released in mid-2019, so more than 5 years ago at this point. When 2.3 was released, we didn't test it against Kafka 0.7, which at that point was 7 years old. So there is some precedent?

If we do choose to keep around testing of new clients against very old Kafka brokers, we probably should consider forking the KafkaService class in the python code. Otherwise the KafkaService will become increasingly hard to evolve.

cmccabe · 2024-11-01T20:12:53Z

I took a look at KIP-896 and it does specify that we should support Kafka 2.1... hmm...

cmccabe · 2024-11-01T20:37:19Z

OK here is what I'm thinking:

Do not test anything earlier than Kafka 2.1, since KIP-896 specifies that as the base version.
For Kafka 3.0 and newer, only test KRaft.
Upgrade and downgrade tests only deal with Kafka 3.0 and newer, since ZK versions cannot be directly upgraded to Kafka 4.0.
implement support for multiple JDKs. Probably 8 and 17. Rationale: Kafka 2.1 requires jdk8, and Kafka 4.0 will require jdk17 (for the server, at least).
standardize on a single version of Zookeeper, ZK 3.4.9. Rationale: Kafka 2.1 requires this version, and all the other pre-KRaft kafka versions tolerate 3.4.9.

cmccabe · 2024-11-01T20:43:37Z

So basically I'm proposing:

version	support	zk version	java version
pre-2.0	none	n/a	n/a
2.1	zk mode	3.4.9	jdk8
2.2	zk mode	3.4.9	jdk8
2.3	zk mode	3.4.9	jdk8
2.4	zk mode	3.4.9	jdk8
2.5	zk mode	3.4.9	jdk8
2.6	zk mode	3.4.9	jdk8
2.7	zk mode	3.4.9	jdk8
2.8	zk mode	3.4.9	jdk8
2.8	zk mode	3.4.9	jdk8
3.0	kraft	n/a	jdk8
3.0	kraft	n/a	jdk8
3.1	kraft	n/a	jdk8
3.2	kraft	n/a	jdk8
3.3	kraft	n/a	jdk8
3.4	kraft	n/a	jdk8
3.5	kraft	n/a	jdk8
3.6	kraft	n/a	jdk8
3.7	kraft	n/a	jdk8
3.8	kraft	n/a	jdk8
3.9	kraft	n/a	jdk8
4.0	kraft	n/a	jdk17

chia7712 · 2024-11-01T21:05:16Z

So basically I'm proposing:

If the ZooKeeper version in the list is the one used by the Kafka release, it might be slightly different from the version in the source code.

kafka 2.1 ~ 2.3 -> zk 3.4.14
(https://github.com/apache/kafka/blob/2.1/gradle/dependencies.gradle#L89)

Additionally, I propose to manually upgrade the zk version from 3.4.14 to 3.5.7 for kafka 2.1 ~ 2.3 (see my comment #17625 (comment)) to fix the issue https://issues.apache.org/jira/browse/ZOOKEEPER-3779

kafka 2.4 -> zk 3.5.7
https://github.com/apache/kafka/blob/2.4.1/gradle/dependencies.gradle#L118

kafka 2.5 -> zk 3.5.8
https://github.com/apache/kafka/blob/2.5.1/gradle/dependencies.gradle#L118

kafka 2.4 ~ 2.8 -> zk 3.5.9
(https://github.com/apache/kafka/blob/2.8/gradle/dependencies.gradle)

chia7712 · 2024-11-01T21:18:18Z

implement support for multiple JDKs. Probably 8 and 17. Rationale: Kafka 2.1 requires jdk8, and Kafka 4.0 will require jdk17 (for the server, at least).

I prefer using a single JDK version (17) to run all E2E tests after we address ZOOKEEPER-3779 for Kafka versions 2.1 to 2.3.

standardize on a single version of Zookeeper, ZK 3.4.9. Rationale: Kafka 2.1 requires this version, and all the other pre-KRaft kafka versions tolerate 3.4.9.

Maybe we should use ZooKeeper 3.5.7 instead, if the workaround for ZOOKEEPER-3779 is acceptable.

I ran client_compatibility_produce_consume_test.py after manually upgrading ZooKeeper to 3.5.7 (see this comment), and it works well.

cmccabe · 2024-11-01T22:08:20Z

If the ZooKeeper version in the list is the one used by the Kafka release, it might be slightly different from the version in the source code.

That's fine. The ZooKeeper server 3.4.9 is compatible with all the clients shipped in Kafka 2.x

Additionally, I propose to manually upgrade the zk version from 3.4.14 to 3.5.7 for kafka 2.1 ~ 2.3 (see my comment #17625 (comment)) to fix the issue https://issues.apache.org/jira/browse/ZOOKEEPER-3779

We don't need to fix ZOOKEEPER-3779 because we're never going to use ZooKeeper with java 14. ZooKeeper is the past, not the future.

I prefer using a single JDK version (17) to run all E2E tests after we address ZOOKEEPER-3779 for Kafka versions 2.1 to 2.3.

Those old Kafka versions don't support JDK17. You need to run them with jdk8.

I ran client_compatibility_produce_consume_test.py after manually upgrading ZooKeeper to 3.5.7 (see this comment), and it works well.

ZK 3.5.7 is not compatible with Kafka 2.1 Please read KIP-902

chia7712 · 2024-11-01T23:53:40Z

Those old Kafka versions don't support JDK17. You need to run them with jdk8.

Pardon me, is there any other issue, aside from ZOOKEEPER-3779, that prevents Kafka versions [2.1, 2.2, 2.3] from running on JDK 17?

ZK 3.5.7 is not compatible with Kafka 2.1 Please read KIP-902

Do you mean Kafka versions [2.1, 2.2, 2.3] can't use the ZooKeeper 3.5.7 client? If so, I didn't see that mentioned in KIP-902. Additionally, it seems that PR #6802, which upgraded ZooKeeper from 3.4.x to 3.5.x, didn't change the use of ZooKeeper APIs (except for running the ZooKeeper main, which is unrelated to the E2E compatibility test we're discussing).

It seems to me that the root cause is the ZK 3.4.x client, which can't run under JDK 17 and ZK 3.8.x server. That's why I plan to replace the ZK 3.4.x client JAR with the ZK 3.5.x client JAR for Kafka versions [2.1, 2.2, 2.3] in E2E tests. This change would allow us to run the E2E tests under JDK 17 with a ZK 3.5+ server.

chia7712 · 2024-11-05T09:10:35Z

@cmccabe Are there any concerns with replacing zk 3.4.x jar with zk 3.5 jar for Kafka versions 2.1, 2.2, and 2.3? With this change, we can simplify the software versions used in E2E tests as follows:

All Kafka and ZooKeeper instances run under JDK 17.
Kafka 2.1+ runs with the ZooKeeper 3.5+ client.
Kafka 2.1 to 2.8 can run under ZooKeeper 3.8.x (DEV) server, as all use the ZooKeeper 3.5+ client.

Please correct me if I misunderstand anything :)

…DK 8" to allow system tests to run

jolshan · 2024-11-08T17:39:22Z

Hey folks, where do we stand here? Do we want to close this PR in favor of the other one or something else?

chia7712 · 2024-11-08T23:08:08Z

Hey folks, where do we stand here? Do we want to close this PR in favor of the other one or something else?

If the approach of upgrading the ZK client to 3.5.x is acceptable, this PR can include the following changes:

Use JDK 17.
Upgrade the ZK client JAR for Kafka 2.1, 2.2, and 2.3, similar to KAFKA-17888 Upgrade ZooKeeper version from 3.4.9 to 3.5.7 to avoid ZOOKEEPER-3779, which can't run under JDK 11. #17625.

chia7712 · 2024-11-11T08:17:22Z

I will merge #17625 tomorrow, using the approach of replacing the ZooKeeper 3.4.x client with 3.5.x. All related tests have passed (see comment).

Please let me know if this solution is not acceptable. Thanks!

jolshan · 2024-11-12T19:41:50Z

@chia7712 sorry for the delay, we had a holiday in the US yesterday.

I think the approach sounds fine, but wasn't sure if we typically updated ZK like this without a KIP or anything. It is also a bit odd since it will only be used in tests. But reading on the previous messages, it seems like this is the only way forward for these older versions that require ZK? If so, then it seems like our best bet to unblock these tests.

jolshan · 2024-11-12T19:43:41Z

ZK 3.5.7 is not compatible with Kafka 2.1 Please read KIP-902

Do you mean Kafka versions [2.1, 2.2, 2.3] can't use the ZooKeeper 3.5.7 client? If so, I didn't see that mentioned in KIP-902.

KIP-902 does have a chart in it that indicates the older kafka versions aren't compatible, so I think we should sort that area out before proceeding.

chia7712 · 2024-11-12T23:40:01Z

I think the approach sounds fine, but wasn't sure if we typically updated ZK like this without a KIP or anything. It is also a bit odd since it will only be used in tests. But reading on the previous messages, it seems like this is the only way forward for these older versions that require ZK? If so, then it seems like our best bet to unblock these tests.

Yes, that approach can simplify our E2E environment, as we can use JDK 17 and ZooKeeper 3.8 server to test all supported versions in E2E. Additionally, a KIP shouldn't be necessary, since #6802 upgraded ZooKeeper from 3.4.14 to 3.5.5 without requiring a KIP. Importantly, what we're trying to test here isn't the compatibility between Kafka and ZooKeeper.

KIP-902 does have a chart in it that indicates the older kafka versions aren't compatible, so I think we should sort that area out before proceeding.

Pardon me, are we taking about this chart?

If so, the chart is about which version of the ZooKeeper client is used by Kafka and the compatibility between the ZooKeeper client and server versions. Additionally, we don't test rolling upgrades from versions 2.1 to 2.3 to newer versions. The main concern with replacing the ZooKeeper 3.4 client JAR with the 3.5 JAR is whether the ZooKeeper 3.5 client introduces API changes. The answer is yes—some APIs have been moved to zookeeper-jute, such as org.apache.zookeeper.data. That's why #17625 needs to manually add zookeeper-jute to make Kafka versions 2.1 to 2.3 work with the ZooKeeper 3.5 client.

The following table is the env used to running Kafka in E2E.

version	support	zk client version	zk server version	java version
pre-2.0	none	n/a	n/a	n/a
2.1	zk mode	3.5.5 (manually update)	3.8.4	jdk17
2.2	zk mode	3.5.5 (manually update)	3.8.4	jdk17
2.3	zk mode	3.5.5 (manually update)	3.8.4	jdk17
2.4	zk mode	3.5.9	3.8.4	jdk17
2.5	zk mode	3.5.9	3.8.4	jdk17
2.6	zk mode	3.5.9	3.8.4	jdk17
2.7	zk mode	3.5.9	3.8.4	jdk17
2.8	zk mode	3.5.9	3.8.4	jdk17
3.0	kraft	n/a	n/a	jdk17
3.1	kraft	n/a	n/a	jdk17
3.2	kraft	n/a	n/a	jdk17
3.3	kraft	n/a	n/a	jdk17
3.4	kraft	n/a	n/a	jdk17
3.5	kraft	n/a	n/a	jdk17
3.6	kraft	n/a	n/a	jdk17
3.7	kraft	n/a	n/a	jdk17
3.8	kraft	n/a	n/a	jdk17
3.9	kraft	n/a	n/a	jdk17
4.0	kraft	n/a	n/a	jdk17

Note that the ZooKeeper 3.8.4 server can be set up using Kafka 3.9 release in the E2E tests, allowing us to remove all ZooKeeper-related code from version 4.0 (related discussion: #17768).

m1a2st · 2024-11-13T00:46:28Z

There are some typo in there, we update the zookeeper version is 3.5.7, so the table which is the env used to running Kafka in E2E will be:

version	support	zk client version	zk server version	java version
pre-2.0	none	n/a	n/a	n/a
2.1	zk mode	3.5.7 (manually update)	3.8.4	jdk17
2.2	zk mode	3.5.7 (manually update)	3.8.4	jdk17
2.3	zk mode	3.5.7 (manually update)	3.8.4	jdk17
2.4	zk mode	3.5.9	3.8.4	jdk17
2.5	zk mode	3.5.9	3.8.4	jdk17
2.6	zk mode	3.5.9	3.8.4	jdk17
2.7	zk mode	3.5.9	3.8.4	jdk17
2.8	zk mode	3.5.9	3.8.4	jdk17
3.0	kraft	n/a	n/a	jdk17
3.1	kraft	n/a	n/a	jdk17
3.2	kraft	n/a	n/a	jdk17
3.3	kraft	n/a	n/a	jdk17
3.4	kraft	n/a	n/a	jdk17
3.5	kraft	n/a	n/a	jdk17
3.6	kraft	n/a	n/a	jdk17
3.7	kraft	n/a	n/a	jdk17
3.8	kraft	n/a	n/a	jdk17
3.9	kraft	n/a	n/a	jdk17
4.0	kraft	n/a	n/a	jdk17

chia7712 · 2024-11-14T11:09:29Z

hi all,

I'm going to merge the approach from #17625 to stop the persistent errors when reviewing other E2E-related PRs. Since #17625 is a small patch, it will be easy to revert if needed. Please feel free to ping me if #17625 turns out to be a bad idea.

jolshan · 2024-11-15T21:40:43Z

If we've merged #17625 then this should be ok to merge too? Did we want to change it back to 17?

chia7712 · 2024-11-16T01:45:11Z

If we've merged #17625 then this should be ok to merge too? Did we want to change it back to 17?

Similar to #17625, we need to update the JDK version to 17 and replace the ZK client JARs for versions 2.1 through 2.13 using base.sh.

jolshan · 2024-11-18T02:44:46Z

Similar to #17625, we need to update the JDK version to 17 and replace the ZK client JARs for versions 2.1 through 2.13 using base.sh.

So we should do that all together in a separate pr for 17? And close this one for 11?

chia7712 · 2024-11-18T08:30:21Z

So we should do that all together in a separate pr for 17? And close this one for 11?

yes!

dongnuo123 · 2024-11-19T14:15:14Z

Opened a new PR with jdk 17 and zk versions replaced #17861

jdk_major & jdk_full

bfb7f00

github-actions bot added tools small Small PRs labels Oct 22, 2024

jdk_major & jdk_full

ecfdf29

bbejeck approved these changes Oct 22, 2024

View reviewed changes

dongnuo123 changed the title ~~MINOR: Use JDK 11 in Vagrant after dropping JDK 8~~ MINOR: Use JDK 17 in Vagrant after dropping JDK 8 Oct 23, 2024

bbejeck mentioned this pull request Oct 28, 2024

KAFKA-17609:[1/4] Changes needed to convert system tests to use KRaft and remove ZK #17275

Merged

3 tasks

jolshan reviewed Oct 31, 2024

View reviewed changes

chia7712 mentioned this pull request Nov 1, 2024

KAFKA-17888 Upgrade ZooKeeper version from 3.4.9 to 3.5.7 to avoid ZOOKEEPER-3779, which can't run under JDK 11. #17625

Merged

3 tasks

dongnuo123 changed the title ~~MINOR: Use JDK 17 in Vagrant after dropping JDK 8~~ MINOR: Use JDK 11 in Vagrant after dropping JDK 8 Nov 6, 2024

kirktrue added a commit to kirktrue/kafka that referenced this pull request Nov 6, 2024

Adding PR apache#17576 "MINOR: Use JDK 11 in Vagrant after dropping J…

f928d4b

…DK 8" to allow system tests to run

kirktrue added a commit to kirktrue/kafka that referenced this pull request Nov 8, 2024

Adding pull request apache#17576 to fix system tests

4ae08d3

chia7712 added the ci-approved label Nov 11, 2024

jolshan closed this Nov 20, 2024

MINOR: Use JDK 11 in Vagrant after dropping JDK 8 #17576

MINOR: Use JDK 11 in Vagrant after dropping JDK 8 #17576

Conversation

dongnuo123 commented Oct 22, 2024

Committer Checklist (excluded from commit message)

bbejeck commented Oct 22, 2024

dongnuo123 commented Oct 22, 2024

jolshan commented Oct 22, 2024 • edited Loading

chia7712 commented Oct 22, 2024

jolshan commented Oct 23, 2024

chia7712 commented Oct 23, 2024

dongnuo123 commented Oct 23, 2024

jolshan commented Oct 24, 2024

chia7712 commented Oct 24, 2024

jolshan commented Oct 25, 2024 • edited Loading

jolshan commented Oct 25, 2024

chia7712 commented Oct 26, 2024

chia7712 commented Oct 27, 2024

jolshan commented Oct 28, 2024

chia7712 commented Oct 28, 2024

jolshan commented Oct 28, 2024

chia7712 commented Oct 28, 2024

chia7712 commented Oct 28, 2024

chia7712 commented Oct 30, 2024

cmccabe commented Oct 31, 2024

jolshan commented Oct 31, 2024

jolshan Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

cmccabe commented Oct 31, 2024

chia7712 commented Nov 1, 2024

kirktrue commented Nov 1, 2024

cmccabe commented Nov 1, 2024

cmccabe commented Nov 1, 2024

cmccabe commented Nov 1, 2024

cmccabe commented Nov 1, 2024

chia7712 commented Nov 1, 2024

chia7712 commented Nov 1, 2024

cmccabe commented Nov 1, 2024

chia7712 commented Nov 1, 2024

chia7712 commented Nov 5, 2024

jolshan commented Nov 8, 2024

chia7712 commented Nov 8, 2024

chia7712 commented Nov 11, 2024

jolshan commented Nov 12, 2024

jolshan commented Nov 12, 2024

chia7712 commented Nov 12, 2024

m1a2st commented Nov 13, 2024

chia7712 commented Nov 14, 2024

jolshan commented Nov 15, 2024

chia7712 commented Nov 16, 2024

jolshan commented Nov 18, 2024

chia7712 commented Nov 18, 2024

dongnuo123 commented Nov 19, 2024

jolshan commented Oct 22, 2024 •

edited

Loading

jolshan commented Oct 25, 2024 •

edited

Loading

jolshan Oct 31, 2024 •

edited

Loading