-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
JPERF-273: Jira provisioning hooks #149
base: master
Are you sure you want to change the base?
Conversation
1e2fb3d
to
86ad16e
Compare
317a0d1
to
a141abb
Compare
This reverts commit 71d7136.
Turn `PostgresDatabase` into a hook too.
Include overall Data Center plan with hook timings. Plans should be reusable with different ways of getting a `TcpHost`: * new via Docker * new via AWS * existing machines This commit injects an `Infrastructure` SPI to reuse a plan with different infra, but to prove the concept, it has to fit the current CloudFormation stack in aws-infrastructure / 2-nodes-dc*.yaml The API smells a little, because a big one-shot CFN stack is split into multiple fine-grained methods. Alternative API could be: `fun plan(host: TcpHost) : JiraNodePlan` Other matters to resolve: * loadbalancer (plan?) API * getting results out of DC instance and node hooks
Download reports per host.
2b0843a
to
681c0df
Compare
Locally failing like this: https://scans.gradle.com/s/375dwevxnzzse |
To read: * https://danwalsh.livejournal.com/74095.html * bmuschko/gradle-docker-plugin#890 Maybe it's because of multiple port exports?
2cd6743
to
3aacf87
Compare
hooks | ||
.poll() | ||
?.call(ssh, jira, this, reports) | ||
?: break |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe this is not important, however I'm gonna note it:
If I'm not wrong it's possible to create a PostInstallHook that'd create an infinite loop here.
private fun installInParallel(nodes: Collection<PlannedHttpNode>): List<PlannedInstalledJira> = nodes | ||
.asSequence() | ||
.asStream() | ||
.parallel() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we fine with this parallel stream here?
Avoid errors in Jira logs like: ``` Caused by: java.lang.NullPointerException at sun.awt.FontConfiguration.getVersion(FontConfiguration.java:1264) [?:1.8.0-adoptopenjdk] at sun.awt.FontConfiguration.readFontConfigFile(FontConfiguration.java:219) [?:1.8.0-adoptopenjdk] at sun.awt.FontConfiguration.init(FontConfiguration.java:107) [?:1.8.0-adoptopenjdk] at sun.awt.X11FontManager.createFontConfiguration(X11FontManager.java:774) [?:1.8.0-adoptopenjdk] at sun.font.SunFontManager$2.run(SunFontManager.java:431) [?:1.8.0-adoptopenjdk] at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0-adoptopenjdk] at sun.font.SunFontManager.<init>(SunFontManager.java:376) [?:1.8.0-adoptopenjdk] at sun.awt.FcFontManager.<init>(FcFontManager.java:35) [?:1.8.0-adoptopenjdk] at sun.awt.X11FontManager.<init>(X11FontManager.java:57) [?:1.8.0-adoptopenjdk] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [?:1.8.0-adoptopenjdk] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:1.8.0-adoptopenjdk] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:1.8.0-adoptopenjdk] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [?:1.8.0-adoptopenjdk] at java.lang.Class.newInstance(Class.java:442) [?:1.8.0-adoptopenjdk] at sun.font.FontManagerFactory$1.run(FontManagerFactory.java:83) [?:1.8.0-adoptopenjdk] at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0-adoptopenjdk] at sun.font.FontManagerFactory.getInstance(FontManagerFactory.java:74) [?:1.8.0-adoptopenjdk] at java.awt.Font.getFont2D(Font.java:491) [?:1.8.0-adoptopenjdk] at java.awt.Font.canDisplay(Font.java:1980) [?:1.8.0-adoptopenjdk] at com.octo.captcha.component.image.fontgenerator.RandomFontGenerator.checkFontCanDisplayCharacters(RandomFontGenerator.java:223) [jcaptcha-2.0-alpha-1.jar:?] at com.octo.captcha.component.image.fontgenerator.RandomFontGenerator.cleanFontList(RandomFontGenerator.java:179) [jcaptcha-2.0-alpha-1.jar:?] at com.octo.captcha.component.image.fontgenerator.RandomFontGenerator.initializeFonts(RandomFontGenerator.java:94) [jcaptcha-2.0-alpha-1.jar:?] at com.octo.captcha.component.image.fontgenerator.RandomFontGenerator.<init>(RandomFontGenerator.java:81) [jcaptcha-2.0-alpha-1.jar:?] at com.atlassian.jira.servlet.CensoredCaptchaEngine.createFontGenerator(CensoredCaptchaEngine.java:86) [classes/:?] at com.atlassian.jira.servlet.CensoredCaptchaEngine.createWord2Image(CensoredCaptchaEngine.java:65) [classes/:?] at com.atlassian.jira.servlet.CensoredCaptchaEngine.buildInitialFactories(CensoredCaptchaEngine.java:48) [classes/:?] at com.octo.captcha.engine.image.ListImageCaptchaEngine.<init>(ListImageCaptchaEngine.java:24) [jcaptcha-2.0-alpha-1.jar:?] at com.atlassian.jira.servlet.CensoredCaptchaEngine.<init>(CensoredCaptchaEngine.java:39) [classes/:?] at com.atlassian.jira.servlet.JiraImageCaptchaServiceImpl.<clinit>(JiraImageCaptchaServiceImpl.java:13) [classes/:?] ... 137 more ```
Fix `DockerIT`.
Perhaps it will consume less disk space in peaks. Try to avoid error on GitHub Actions: ``` com.atlassian.performance.tools.infrastructure.api.distribution.PublicJiraServiceDeskDistributionIT > shouldDownloadJiraServiceDesk FAILED java.lang.Exception at PublicJiraServiceDeskDistributionIT.kt:17 Caused by: net.schmizz.sshj.connection.ConnectionException at PublicJiraServiceDeskDistributionIT.kt:17 Caused by: java.util.concurrent.TimeoutException at PublicJiraServiceDeskDistributionIT.kt:17 java.io.IOException: No space left on device com.esotericsoftware.kryo.KryoException: java.io.IOException: No space left on device at com.esotericsoftware.kryo.io.Output.flush(Output.java:165) ```
3aacf87
to
48bb603
Compare
|
||
class Builder( | ||
private var plan: JiraNodePlan | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: empty line
|
||
internal fun call(nodes: List<PreInstallHooks>, reports: Reports) { | ||
hooks | ||
.parallelStream() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another usage of parallel stream. Isn't this risky?
private fun download(ssh: SshConnection) { | ||
ssh.execute("sudo mkdir -p $localHome") | ||
val jiraHome = jiraHomeSource.download(ssh) | ||
ssh.execute("sudo mv $jiraHome/{data,plugins,import,export} $localHome") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note: Today I noticed that this line will fail when jirahome is empty. I'm writing a test for this in aws-infrastructure
with the old API.
In case you'd try to use empty DC instance this may be a problem.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixing it in old API: atlassian/aws-infrastructure#176
Possibly same change should be applied here.
7722512
to
59bceaf
Compare
Tests were consuming too much space in CI, up to ~15 GB in spikes
59bceaf
to
c46b1f6
Compare
import java.time.Duration | ||
import java.time.Instant | ||
|
||
class JiraPlanIT { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be good to have also a test for starting empty Jira (no dataset) without setup.
Consumers should provide a working path
Fix errors like: ``` java.lang.Exception: Failed to Installing and configuring apache load balancer despite 2 attempts at com.atlassian.performance.tools.jvmtasks.api.IdempotentAction.retry(IdempotentAction.kt:42) at com.atlassian.performance.tools.infrastructure.api.loadbalancer.ApacheProxyPlan.materialize(ApacheProxyPlan.kt:29) at com.atlassian.performance.tools.infrastructure.api.jira.instance.JiraDataCenterPlan.materialize(JiraDataCenterPlan.kt:30) at com.atlassian.performance.tools.awsinfrastructure.api.jira.HooksDataCenterFormulaIT$shouldProvisionDc$1.invoke(HooksDataCenterFormulaIT.kt:121) at com.atlassian.performance.tools.awsinfrastructure.api.jira.HooksDataCenterFormulaIT$shouldProvisionDc$1.invoke(HooksDataCenterFormulaIT.kt:41) at com.atlassian.performance.tools.awsinfrastructure.api.jira.HooksDataCenterFormulaIT.makeFailureObservable(HooksDataCenterFormulaIT.kt:70) at com.atlassian.performance.tools.awsinfrastructure.api.jira.HooksDataCenterFormulaIT.shouldProvisionDc(HooksDataCenterFormulaIT.kt:120) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175) at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157) at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63) at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55) at java.lang.Thread.run(Thread.java:750) Suppressed: java.lang.RuntimeException: Potential Jira problems found in workspace: {} at com.atlassian.performance.tools.awsinfrastructure.api.jira.HooksDataCenterFormulaIT.makeFailureObservable(HooksDataCenterFormulaIT.kt:74) ... 48 more Caused by: java.lang.Exception: Error while executing sed -i -r 's/Listen 80/Listen 80/g' /etc/apache2/ports.conf. Exit status code SshResult(exitStatus=4, output=, errorOutput=sed: couldn't open temporary file /etc/apache2/sed3Zlw2Y: Permission denied ) at com.atlassian.performance.tools.ssh.SshjConnection.execute(SshjConnection.kt:40) at com.atlassian.performance.tools.ssh.api.SshConnection.execute(SshConnection.kt:27) at com.atlassian.performance.tools.infrastructure.api.Sed.replace(Sed.kt:14) at com.atlassian.performance.tools.infrastructure.api.loadbalancer.ApacheProxyPlan.tryToProvision(ApacheProxyPlan.kt:37) at com.atlassian.performance.tools.infrastructure.api.loadbalancer.ApacheProxyPlan.access$tryToProvision(ApacheProxyPlan.kt:17) at com.atlassian.performance.tools.infrastructure.api.loadbalancer.ApacheProxyPlan$materialize$1.invoke(ApacheProxyPlan.kt:27) at com.atlassian.performance.tools.infrastructure.api.loadbalancer.ApacheProxyPlan$materialize$1.invoke(ApacheProxyPlan.kt:17) at com.atlassian.performance.tools.jvmtasks.api.IdempotentAction.retry(IdempotentAction.kt:28) ... 53 more ```
906087f
to
0b3c72c
Compare
1634a16
to
90b1e0e
Compare
90b1e0e
to
cacb5f0
Compare
This is the 2023 version of #103
Rebased, resolved a lot of conflicts and cleaned up a little. Turns out many changes were done independently on
master
already. Still gotta squash them commits...