Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JPERF-273: Jira provisioning hooks #149

Open
wants to merge 73 commits into
base: master
Choose a base branch
from

Conversation

dagguh
Copy link
Contributor

@dagguh dagguh commented Mar 3, 2023

This is the 2023 version of #103
Rebased, resolved a lot of conflicts and cleaned up a little. Turns out many changes were done independently on master already. Still gotta squash them commits...

@dagguh
Copy link
Contributor Author

dagguh commented Mar 3, 2023

Currently the JiraServerPlanIT fails with this:
image
Maybe MySQL 5.7 is too new for this old Jira 7.13.0? Perhaps it's time to bump the default JSW version.

com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Could not find a field-type definition with name "mysql57" at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) at com.google.common.cache.LocalCache.get(LocalCache.java:3937) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830) at org.ofbiz.core.entity.GenericDelegator.getGenericDelegator(GenericDelegator.java:108) at com.atlassian.core.ofbiz.CoreFactory.getGenericDelegator(CoreFactory.java:38) at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.configureOfbiz(DatabaseConfigurationManagerImpl.java:220) at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:301) at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseConfigured(DatabaseConfigurationManagerImpl.java:204) at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:129) at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:102) at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31) at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:100) at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:154) at com.atlassian.jira.startup.LauncherContextListener.initSlowStuffInBackground(LauncherContextListener.java:139) at com.atlassian.jira.startup.LauncherContextListener.contextInitialized(LauncherContextListener.java:101) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4792) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5256) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1421) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1411) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalStateException: Could not find a field-type definition with name "mysql57" at org.ofbiz.core.entity.model.ModelFieldTypeReader.<init>(ModelFieldTypeReader.java:92) at org.ofbiz.core.entity.model.ModelFieldTypeReader.getModelFieldTypeReader(ModelFieldTypeReader.java:79) at org.ofbiz.core.entity.GenericDelegator.initialiseAndCheckDatabase(GenericDelegator.java:232) at org.ofbiz.core.entity.GenericDelegator.<init>(GenericDelegator.java:175) at org.ofbiz.core.entity.GenericDelegator$1.load(GenericDelegator.java:94) at org.ofbiz.core.entity.GenericDelegator$1.load(GenericDelegator.java:91) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ... 25 more 

@mgrzaslewicz mgrzaslewicz force-pushed the issue/JPERF-273-jira-provisioning-hooks branch from 1e2fb3d to 86ad16e Compare September 6, 2023 13:02
@dagguh dagguh force-pushed the issue/JPERF-273-jira-provisioning-hooks branch 2 times, most recently from 317a0d1 to a141abb Compare September 7, 2023 08:31
dagguh and others added 16 commits September 7, 2023 11:13
Turn `PostgresDatabase` into a hook too.
Include overall Data Center plan with hook timings.
Plans should be reusable with different ways of getting a `TcpHost`:
* new via Docker
* new via AWS
* existing machines

This commit injects an `Infrastructure` SPI to reuse a plan with
different infra, but to prove the concept, it has to fit the current
CloudFormation stack in aws-infrastructure / 2-nodes-dc*.yaml
The API smells a little, because a big one-shot CFN stack is split into
multiple fine-grained methods.

Alternative API could be: `fun plan(host: TcpHost) : JiraNodePlan`

Other matters to resolve:
* loadbalancer (plan?) API
* getting results out of DC instance and node hooks
@mgrzaslewicz mgrzaslewicz force-pushed the issue/JPERF-273-jira-provisioning-hooks branch 2 times, most recently from 2b0843a to 681c0df Compare September 7, 2023 09:49
@dagguh dagguh marked this pull request as ready for review September 7, 2023 09:50
@dagguh dagguh requested a review from a team as a code owner September 7, 2023 09:50
build.gradle.kts Show resolved Hide resolved
@dagguh
Copy link
Contributor Author

dagguh commented Sep 7, 2023

Locally failing like this: https://scans.gradle.com/s/375dwevxnzzse

@dagguh dagguh force-pushed the issue/JPERF-273-jira-provisioning-hooks branch 2 times, most recently from 2cd6743 to 3aacf87 Compare September 7, 2023 11:21
hooks
.poll()
?.call(ssh, jira, this, reports)
?: break
Copy link
Contributor

@pczuj pczuj Sep 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this is not important, however I'm gonna note it:
If I'm not wrong it's possible to create a PostInstallHook that'd create an infinite loop here.

private fun installInParallel(nodes: Collection<PlannedHttpNode>): List<PlannedInstalledJira> = nodes
.asSequence()
.asStream()
.parallel()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we fine with this parallel stream here?

dagguh and others added 9 commits September 7, 2023 14:44
Avoid errors in Jira logs like:
```
Caused by: java.lang.NullPointerException
	at sun.awt.FontConfiguration.getVersion(FontConfiguration.java:1264) [?:1.8.0-adoptopenjdk]
	at sun.awt.FontConfiguration.readFontConfigFile(FontConfiguration.java:219) [?:1.8.0-adoptopenjdk]
	at sun.awt.FontConfiguration.init(FontConfiguration.java:107) [?:1.8.0-adoptopenjdk]
	at sun.awt.X11FontManager.createFontConfiguration(X11FontManager.java:774) [?:1.8.0-adoptopenjdk]
	at sun.font.SunFontManager$2.run(SunFontManager.java:431) [?:1.8.0-adoptopenjdk]
	at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0-adoptopenjdk]
	at sun.font.SunFontManager.<init>(SunFontManager.java:376) [?:1.8.0-adoptopenjdk]
	at sun.awt.FcFontManager.<init>(FcFontManager.java:35) [?:1.8.0-adoptopenjdk]
	at sun.awt.X11FontManager.<init>(X11FontManager.java:57) [?:1.8.0-adoptopenjdk]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [?:1.8.0-adoptopenjdk]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:1.8.0-adoptopenjdk]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:1.8.0-adoptopenjdk]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [?:1.8.0-adoptopenjdk]
	at java.lang.Class.newInstance(Class.java:442) [?:1.8.0-adoptopenjdk]
	at sun.font.FontManagerFactory$1.run(FontManagerFactory.java:83) [?:1.8.0-adoptopenjdk]
	at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0-adoptopenjdk]
	at sun.font.FontManagerFactory.getInstance(FontManagerFactory.java:74) [?:1.8.0-adoptopenjdk]
	at java.awt.Font.getFont2D(Font.java:491) [?:1.8.0-adoptopenjdk]
	at java.awt.Font.canDisplay(Font.java:1980) [?:1.8.0-adoptopenjdk]
	at com.octo.captcha.component.image.fontgenerator.RandomFontGenerator.checkFontCanDisplayCharacters(RandomFontGenerator.java:223) [jcaptcha-2.0-alpha-1.jar:?]
	at com.octo.captcha.component.image.fontgenerator.RandomFontGenerator.cleanFontList(RandomFontGenerator.java:179) [jcaptcha-2.0-alpha-1.jar:?]
	at com.octo.captcha.component.image.fontgenerator.RandomFontGenerator.initializeFonts(RandomFontGenerator.java:94) [jcaptcha-2.0-alpha-1.jar:?]
	at com.octo.captcha.component.image.fontgenerator.RandomFontGenerator.<init>(RandomFontGenerator.java:81) [jcaptcha-2.0-alpha-1.jar:?]
	at com.atlassian.jira.servlet.CensoredCaptchaEngine.createFontGenerator(CensoredCaptchaEngine.java:86) [classes/:?]
	at com.atlassian.jira.servlet.CensoredCaptchaEngine.createWord2Image(CensoredCaptchaEngine.java:65) [classes/:?]
	at com.atlassian.jira.servlet.CensoredCaptchaEngine.buildInitialFactories(CensoredCaptchaEngine.java:48) [classes/:?]
	at com.octo.captcha.engine.image.ListImageCaptchaEngine.<init>(ListImageCaptchaEngine.java:24) [jcaptcha-2.0-alpha-1.jar:?]
	at com.atlassian.jira.servlet.CensoredCaptchaEngine.<init>(CensoredCaptchaEngine.java:39) [classes/:?]
	at com.atlassian.jira.servlet.JiraImageCaptchaServiceImpl.<clinit>(JiraImageCaptchaServiceImpl.java:13) [classes/:?]
	... 137 more
```
Perhaps it will consume less disk space in peaks.
Try to avoid error on GitHub Actions:
```
com.atlassian.performance.tools.infrastructure.api.distribution.PublicJiraServiceDeskDistributionIT > shouldDownloadJiraServiceDesk FAILED
    java.lang.Exception at PublicJiraServiceDeskDistributionIT.kt:17
        Caused by: net.schmizz.sshj.connection.ConnectionException at PublicJiraServiceDeskDistributionIT.kt:17
            Caused by: java.util.concurrent.TimeoutException at PublicJiraServiceDeskDistributionIT.kt:17
java.io.IOException: No space left on device
com.esotericsoftware.kryo.KryoException: java.io.IOException: No space left on device
	at com.esotericsoftware.kryo.io.Output.flush(Output.java:165)
```
@mgrzaslewicz mgrzaslewicz force-pushed the issue/JPERF-273-jira-provisioning-hooks branch from 3aacf87 to 48bb603 Compare September 7, 2023 12:44

class Builder(
private var plan: JiraNodePlan

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: empty line


internal fun call(nodes: List<PreInstallHooks>, reports: Reports) {
hooks
.parallelStream()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another usage of parallel stream. Isn't this risky?

private fun download(ssh: SshConnection) {
ssh.execute("sudo mkdir -p $localHome")
val jiraHome = jiraHomeSource.download(ssh)
ssh.execute("sudo mv $jiraHome/{data,plugins,import,export} $localHome")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: Today I noticed that this line will fail when jirahome is empty. I'm writing a test for this in aws-infrastructure with the old API.

In case you'd try to use empty DC instance this may be a problem.

Copy link
Contributor

@pczuj pczuj Sep 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixing it in old API: atlassian/aws-infrastructure#176

Possibly same change should be applied here.

@mgrzaslewicz mgrzaslewicz force-pushed the issue/JPERF-273-jira-provisioning-hooks branch from 7722512 to 59bceaf Compare September 7, 2023 13:50
Tests were consuming too much space in CI, up to ~15 GB in spikes
@mgrzaslewicz mgrzaslewicz force-pushed the issue/JPERF-273-jira-provisioning-hooks branch from 59bceaf to c46b1f6 Compare September 7, 2023 13:52
import java.time.Duration
import java.time.Instant

class JiraPlanIT {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be good to have also a test for starting empty Jira (no dataset) without setup.

mgrzaslewicz and others added 4 commits September 11, 2023 10:51
Consumers should provide a working path
Fix errors like:
```
java.lang.Exception: Failed to Installing and configuring apache load balancer despite 2 attempts
	at com.atlassian.performance.tools.jvmtasks.api.IdempotentAction.retry(IdempotentAction.kt:42)
	at com.atlassian.performance.tools.infrastructure.api.loadbalancer.ApacheProxyPlan.materialize(ApacheProxyPlan.kt:29)
	at com.atlassian.performance.tools.infrastructure.api.jira.instance.JiraDataCenterPlan.materialize(JiraDataCenterPlan.kt:30)
	at com.atlassian.performance.tools.awsinfrastructure.api.jira.HooksDataCenterFormulaIT$shouldProvisionDc$1.invoke(HooksDataCenterFormulaIT.kt:121)
	at com.atlassian.performance.tools.awsinfrastructure.api.jira.HooksDataCenterFormulaIT$shouldProvisionDc$1.invoke(HooksDataCenterFormulaIT.kt:41)
	at com.atlassian.performance.tools.awsinfrastructure.api.jira.HooksDataCenterFormulaIT.makeFailureObservable(HooksDataCenterFormulaIT.kt:70)
	at com.atlassian.performance.tools.awsinfrastructure.api.jira.HooksDataCenterFormulaIT.shouldProvisionDc(HooksDataCenterFormulaIT.kt:120)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
	at java.lang.Thread.run(Thread.java:750)
	Suppressed: java.lang.RuntimeException: Potential Jira problems found in workspace: {}
		at com.atlassian.performance.tools.awsinfrastructure.api.jira.HooksDataCenterFormulaIT.makeFailureObservable(HooksDataCenterFormulaIT.kt:74)
		... 48 more
Caused by: java.lang.Exception: Error while executing sed -i -r 's/Listen 80/Listen 80/g' /etc/apache2/ports.conf. Exit status code SshResult(exitStatus=4, output=, errorOutput=sed: couldn't open temporary file /etc/apache2/sed3Zlw2Y: Permission denied
)
	at com.atlassian.performance.tools.ssh.SshjConnection.execute(SshjConnection.kt:40)
	at com.atlassian.performance.tools.ssh.api.SshConnection.execute(SshConnection.kt:27)
	at com.atlassian.performance.tools.infrastructure.api.Sed.replace(Sed.kt:14)
	at com.atlassian.performance.tools.infrastructure.api.loadbalancer.ApacheProxyPlan.tryToProvision(ApacheProxyPlan.kt:37)
	at com.atlassian.performance.tools.infrastructure.api.loadbalancer.ApacheProxyPlan.access$tryToProvision(ApacheProxyPlan.kt:17)
	at com.atlassian.performance.tools.infrastructure.api.loadbalancer.ApacheProxyPlan$materialize$1.invoke(ApacheProxyPlan.kt:27)
	at com.atlassian.performance.tools.infrastructure.api.loadbalancer.ApacheProxyPlan$materialize$1.invoke(ApacheProxyPlan.kt:17)
	at com.atlassian.performance.tools.jvmtasks.api.IdempotentAction.retry(IdempotentAction.kt:28)
	... 53 more
```
@dagguh dagguh force-pushed the issue/JPERF-273-jira-provisioning-hooks branch from 906087f to 0b3c72c Compare September 11, 2023 11:28
@mgrzaslewicz mgrzaslewicz force-pushed the issue/JPERF-273-jira-provisioning-hooks branch from 1634a16 to 90b1e0e Compare September 13, 2023 14:02
@mgrzaslewicz mgrzaslewicz force-pushed the issue/JPERF-273-jira-provisioning-hooks branch from 90b1e0e to cacb5f0 Compare September 13, 2023 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants