Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Brooklyn restart failed: IndexingServiceException, due to NoShardAvailableActionException #66

Open
aledsage opened this issue Jan 22, 2016 · 9 comments
Open

Brooklyn restart failed: IndexingServiceException, due to NoShardAvailableActionException #66

aledsage opened this issue Jan 22, 2016 · 9 comments

Comments

@aledsage
Copy link
Member

When restarting a Brooklyn server that included brooklyn-tosca, the restart failed due to an error in alien4cloud. There were no tosca blueprints or catalog items.

The server's web-console reported "this brooklyn server has errors".

Looking at mgmt.errors(), it showed:

[alien4cloud.exception.IndexingServiceException: Error when calling <alien4cloud.dao.IGenericIdDAO> <findById>]

The workaround for me to restart the Brooklyn server was to disable brooklyn-tosca - in brooklyn.properties I added:

brooklyn.experimental.feature.tosca = false

The full exception from the log was:

2016-01-22 17:32:42,902 ERROR alien4cloud.dao.ESIndexMapper [brooklyn-execmanager-JGwCF365-0]: Error when calling <alien4cloud.dao.IGenericIdDAO> <findById>
org.elasticsearch.action.NoShardAvailableActionException: [csar][1] null
        at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.perform(TransportShardSingleOperationAction.java:175) ~[elasticsearch-1.7.0.jar:na]
        at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.start(TransportShardSingleOperationAction.java:155) ~[elasticsearch-1.7.0.jar:na]
        at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:89) ~[elasticsearch-1.7.0.jar:na]
        at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:55) ~[elasticsearch-1.7.0.jar:na]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) ~[elasticsearch-1.7.0.jar:na]
        at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98) ~[elasticsearch-1.7.0.jar:na]
        at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:197) ~[elasticsearch-1.7.0.jar:na]
        at org.elasticsearch.action.get.GetRequestBuilder.doExecute(GetRequestBuilder.java:201) ~[elasticsearch-1.7.0.jar:na]
        at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91) ~[elasticsearch-1.7.0.jar:na]
        at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65) ~[elasticsearch-1.7.0.jar:na]
        at alien4cloud.dao.ESGenericIdDAO.findById(ESGenericIdDAO.java:64) ~[alien4cloud-common-1.1.0-SM8.jar:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_71]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_71]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_71]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) ~[spring-aop-4.1.4.RELEASE.jar:4.1.4.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.1.4.RELEASE.jar:4.1.4.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.1.4.RELEASE.jar:4.1.4.RELEASE]
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) ~[spring-aop-4.1.4.RELEASE.jar:4.1.4.RELEASE]
        at alien4cloud.dao.ESExceptionHandlingAspect.handleException(ESExceptionHandlingAspect.java:26) ~[alien4cloud-common-1.1.0-SM8.jar:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_71]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_71]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_71]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) [spring-aop-4.1.4.RELEASE.jar:4.1.4.RELEASE]
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) [spring-aop-4.1.4.RELEASE.jar:4.1.4.RELEASE]
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) [spring-aop-4.1.4.RELEASE.jar:4.1.4.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168) [spring-aop-4.1.4.RELEASE.jar:4.1.4.RELEASE]
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) [spring-aop-4.1.4.RELEASE.jar:4.1.4.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.1.4.RELEASE.jar:4.1.4.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) [spring-aop-4.1.4.RELEASE.jar:4.1.4.RELEASE]
        at com.sun.proxy.$Proxy171.findById(Unknown Source) [na:na]
        at alien4cloud.csar.services.CsarService.getIfExists(CsarService.java:66) [alien4cloud-core-1.1.0-SM8.jar:na]
        at alien4cloud.tosca.ArchiveIndexer.importArchive(ArchiveIndexer.java:62) [alien4cloud-core-1.1.0-SM8.jar:na]
        at alien4cloud.tosca.ArchiveUploadService.upload(ArchiveUploadService.java:80) [alien4cloud-core-1.1.0-SM8.jar:na]
        at io.cloudsoft.tosca.a4c.platform.Alien4CloudToscaPlatform.upload(Alien4CloudToscaPlatform.java:126) [brooklyn-tosca-transformer-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at io.cloudsoft.tosca.a4c.platform.Alien4CloudToscaPlatform.loadTypesFromUrl(Alien4CloudToscaPlatform.java:120) [brooklyn-tosca-transformer-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at io.cloudsoft.tosca.a4c.platform.Alien4CloudToscaPlatform.loadNormativeTypes(Alien4CloudToscaPlatform.java:78) [brooklyn-tosca-transformer-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at io.cloudsoft.tosca.a4c.brooklyn.ToscaPlanToSpecTransformer.initialiseAlien(ToscaPlanToSpecTransformer.java:106) [brooklyn-tosca-transformer-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at io.cloudsoft.tosca.a4c.brooklyn.ToscaPlanToSpecTransformer.setManagementContext(ToscaPlanToSpecTransformer.java:93) [brooklyn-tosca-transformer-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.plan.PlanToSpecFactory.all(PlanToSpecFactory.java:98) [brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.plan.PlanToSpecFactory.attemptWithLoaders(PlanToSpecFactory.java:117) [brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.catalog.internal.BasicBrooklynCatalog.internalCreateSpecLegacy(BasicBrooklynCatalog.java:322) [brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.catalog.internal.BasicBrooklynCatalog.createSpec(BasicBrooklynCatalog.java:304) [brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.cli.Main$LaunchCommand.confirmCatalog(Main.java:641) [brooklyn-cli-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.cli.Main$LaunchCommand$1.apply(Main.java:429) [brooklyn-cli-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.cli.Main$LaunchCommand$1.apply(Main.java:419) [brooklyn-cli-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.catalog.internal.CatalogInitialization.populateViaCallbacks(CatalogInitialization.java:377) [brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.catalog.internal.CatalogInitialization.populateCatalogImpl(CatalogInitialization.java:236) [brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.catalog.internal.CatalogInitialization.populateCatalog(CatalogInitialization.java:198) [brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.mgmt.rebind.RebindIteration.rebuildCatalog(RebindIteration.java:422) [brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.mgmt.rebind.RebindIteration.doRun(RebindIteration.java:238) [brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.mgmt.rebind.InitialFullRebindIteration.doRun(InitialFullRebindIteration.java:69) [brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.mgmt.rebind.RebindIteration.run(RebindIteration.java:265) [brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl.rebindImpl(RebindManagerImpl.java:552) [brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl$3.call(RebindManagerImpl.java:502) [brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl$3.call(RebindManagerImpl.java:500) [brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:499) [brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
2016-01-22 17:32:42,903 ERROR o.a.b.c.c.i.CatalogInitialization [brooklyn-execmanager-JGwCF365-0]: Error loading catalog item 'org.apache.brooklyn.core.catalog.internal.CatalogItemDo[CatalogEntityItemDto[vanilla-bash-server:1.0.0/
Server with Launch Script (bash)]]': alien4cloud.exception.IndexingServiceException: Error when calling <alien4cloud.dao.IGenericIdDAO> <findById>
@sjcorbett
Copy link
Contributor

@aledsage What parameters did you use when starting the server? I don't think this is to do with rebind. I think there is an occasional initialisation race - we should be waiting for Alien to be ready, or Alien should be waiting for Elasticsearch to be ready.

@robertgmoss
Copy link
Contributor

This appears to be a problem with the way elasticsearch is used. It can also be resolved by deleting the generated runtime folder before restarting. Needs further investigation. I sometimes get this when running tests in my IDE.

@sjcorbett
Copy link
Contributor

@aledsage As @robertgmoss alluded, we think Elasticsearch's data directory is corrupted. More questions:

  • What actions did you take with the first server? Did you deploy any blueprints?
  • What did you do to terminate the first server?
  • Are there any exceptions in the log from this first termination?
  • Were you running several Brooklyn servers (that included brooklyn-tosca) at once?

@aledsage
Copy link
Member Author

@sjcorbett @robertgmoss :

  • in the first server, I deployed some apps but none that used tosca.
  • kill <pid> to terminate the server.
  • exceptions: yes, I just found the exception below earlier in the log (during the previous run of the server).
  • only one server at a time, I believe.
2016-01-22 16:41:09,551 WARN  o.a.b.r.t.CatalogTransformer [brooklyn-jetty-server-8081-qtp1376941075-1228]: Unable to create spec for org.apache.brooklyn.core.catalog.internal.CatalogItemDo[CatalogEntityItemDto[c
luster:1.0.0/null]]: alien4cloud.exception.IndexingServiceException: Indexing service is in an inconsistent state, the super element [[tosca.relationships.Root]] is not found for element [tosca.relationships.Depe
ndsOn:1.0.0.wd06-SNAPSHOT]
alien4cloud.exception.IndexingServiceException: Indexing service is in an inconsistent state, the super element [[tosca.relationships.Root]] is not found for element [tosca.relationships.DependsOn:1.0.0.wd06-SNAPSHOT]
        at alien4cloud.component.CSARRepositoryIndexerService.indexInheritableElement(CSARRepositoryIndexerService.java:117) ~[alien4cloud-core-1.1.0-SM8.jar:na]
        at alien4cloud.component.CSARRepositoryIndexerService.indexInheritableElements(CSARRepositoryIndexerService.java:94) ~[alien4cloud-core-1.1.0-SM8.jar:na]
        at alien4cloud.tosca.ArchiveIndexer.performIndexing(ArchiveIndexer.java:184) ~[alien4cloud-core-1.1.0-SM8.jar:na]
        at alien4cloud.tosca.ArchiveIndexer.indexArchive(ArchiveIndexer.java:151) ~[alien4cloud-core-1.1.0-SM8.jar:na]
        at alien4cloud.tosca.ArchiveIndexer.importArchive(ArchiveIndexer.java:80) ~[alien4cloud-core-1.1.0-SM8.jar:na]
        at alien4cloud.tosca.ArchiveUploadService.upload(ArchiveUploadService.java:80) ~[alien4cloud-core-1.1.0-SM8.jar:na]
        at io.cloudsoft.tosca.a4c.platform.Alien4CloudToscaPlatform.upload(Alien4CloudToscaPlatform.java:126) ~[brooklyn-tosca-transformer-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at io.cloudsoft.tosca.a4c.platform.Alien4CloudToscaPlatform.loadTypesFromUrl(Alien4CloudToscaPlatform.java:120) ~[brooklyn-tosca-transformer-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at io.cloudsoft.tosca.a4c.platform.Alien4CloudToscaPlatform.loadNormativeTypes(Alien4CloudToscaPlatform.java:78) ~[brooklyn-tosca-transformer-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at io.cloudsoft.tosca.a4c.brooklyn.ToscaPlanToSpecTransformer.initialiseAlien(ToscaPlanToSpecTransformer.java:106) ~[brooklyn-tosca-transformer-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at io.cloudsoft.tosca.a4c.brooklyn.ToscaPlanToSpecTransformer.setManagementContext(ToscaPlanToSpecTransformer.java:93) ~[brooklyn-tosca-transformer-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.plan.PlanToSpecFactory.all(PlanToSpecFactory.java:98) ~[brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.plan.PlanToSpecFactory.attemptWithLoaders(PlanToSpecFactory.java:117) ~[brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.catalog.internal.BasicBrooklynCatalog.internalCreateSpecLegacy(BasicBrooklynCatalog.java:322) ~[brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.core.catalog.internal.BasicBrooklynCatalog.createSpec(BasicBrooklynCatalog.java:304) ~[brooklyn-core-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.rest.transform.CatalogTransformer.catalogEntitySummary(CatalogTransformer.java:73) [brooklyn-rest-server-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.rest.transform.CatalogTransformer.catalogItemSummary(CatalogTransformer.java:110) [brooklyn-rest-server-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.rest.resources.CatalogResource$1.apply(CatalogResource.java:93) [brooklyn-rest-server-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.rest.resources.CatalogResource$1.apply(CatalogResource.java:90) [brooklyn-rest-server-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at com.google.common.collect.Lists$TransformingRandomAccessList$1.transform(Lists.java:582) [guava-17.0.jar:na]
        at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48) [guava-17.0.jar:na]
        at org.apache.brooklyn.rest.resources.CatalogResource.castList(CatalogResource.java:507) [brooklyn-rest-server-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.apache.brooklyn.rest.resources.CatalogResource.listEntities(CatalogResource.java:238) [brooklyn-rest-server-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_71]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_71]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_71]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) [jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) [jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100) [jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.jar:1.19]
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.jar:1.19]
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.jar:1.19]
        at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:927) [jersey-servlet-1.19.jar:1.19]
        at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875) [jersey-servlet-1.19.jar:1.19]
        at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829) [jersey-servlet-1.19.jar:1.19]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.apache.brooklyn.rest.filter.SwaggerFilter.doFilter(SwaggerFilter.java:69) [brooklyn-rest-server-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.apache.brooklyn.rest.filter.HaMasterCheckFilter.doFilter(HaMasterCheckFilter.java:101) [brooklyn-rest-server-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.apache.brooklyn.rest.filter.LoggingFilter.doFilter(LoggingFilter.java:89) [brooklyn-rest-server-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.apache.brooklyn.rest.filter.BrooklynPropertiesSecurityFilter.doFilter(BrooklynPropertiesSecurityFilter.java:122) [brooklyn-rest-server-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.apache.brooklyn.rest.filter.RequestTaggingFilter.doFilter(RequestTaggingFilter.java:49) [brooklyn-rest-server-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) [jetty-security-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) [jetty-server-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) [jetty-server-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) [jetty-server-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.13.v20150730.jar:9.2.13.v20150730]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.13.v20150730.jar:9.2.13.v20150730]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]

@robertgmoss
Copy link
Contributor

@aledsage I have been trying to replicate the error but am unable to. I suspect that your ES data folder was corrupted somehow, but cannot find a cause. Are you able to reproduce the error after deleting the runtime folder?

@sjcorbett
Copy link
Contributor

I'm also having difficulty reproducing this.

The two stacktraces are interesting. Both involve the initialisation of A4C:

at io.cloudsoft.tosca.a4c.brooklyn.ToscaPlanToSpecTransformer.initialiseAlien(ToscaPlanToSpecTransformer.java:106) [brooklyn-tosca-transformer-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]

The source of the first (in this thread, the second time Brooklyn was started) is Brooklyn's start-up:

at org.apache.brooklyn.cli.Main$LaunchCommand.confirmCatalog(Main.java:641) [brooklyn-cli-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]

But the source of the latter is the API:

at org.apache.brooklyn.rest.resources.CatalogResource.listEntities(CatalogResource.java:238) [brooklyn-rest-server-0.9.0-20160121.1603.jar:0.9.0-20160121.1603]

The case coming from the API surprises me but it should (!) be conincidental. Only one thread can initialise A4C at once. To reach the initialiseAlien call a thread must have the ToscaPlanToSpecTransformer class lock.

I'm going to take a look at the A4C code that does the indexing: https://github.com/alien4cloud/alien4cloud/blob/ca638b77f42dd0d65186edd1e23897a6e382b747/alien4cloud-core/src/main/java/alien4cloud/component/CSARRepositoryIndexerService.java#L92-92.

@sjcorbett
Copy link
Contributor

@aledsage Was anybody else running brooklyn-tosca on the same network as you at the same time?

@sjcorbett
Copy link
Contributor

I've posted the following to Alien4Cloud's public Slack channel:

To follow up our question, the faulty run has this surprising exception in its logs:

alien4cloud.exception.IndexingServiceException: Indexing service is in an inconsistent state, the super element [[tosca.relationships.Root]] is not found for element [tosca.relationships.DependsOn:1.0.0.wd06-SNAPSHOT]
        at alien4cloud.component.CSARRepositoryIndexerService.indexInheritableElement(CSARRepositoryIndexerService.java:117) ~[alien4cloud-core-1.1.0-SM8.jar:na]
        at alien4cloud.component.CSARRepositoryIndexerService.indexInheritableElements(CSARRepositoryIndexerService.java:94) ~[alien4cloud-core-1.1.0-SM8.jar:na]
        at alien4cloud.tosca.ArchiveIndexer.performIndexing(ArchiveIndexer.java:184) ~[alien4cloud-core-1.1.0-SM8.jar:na]

This was thrown when importing normative types (from https://github.com/alien4cloud/tosca-normative-types/archive/master.zip) the first time the server was run.

Presumably either IndexedModelUtils.orderByDerivedFromHierarchy is returning the wrong order for the normative relationships (I think unlikely), something went wrong with CSARRepositoryIndexerService.indexInheritableElement when adding tosca.relationships.Root (but that didn't throw an exception), or alienDAO.customFind is unable to find the Root relationship.

Any thoughts? I'm waiting to find out whether there were other Elasticsearch processes on the same network. I'm told that there were no others on the same host.

@aledsage
Copy link
Member Author

@sjcorbett no-one else was using brooklyn-tosca on the same network.

Did this error happen when using a4c code to parse the Tosca, or when trying to talk to a separate a4c?

Is it possible for us to code more defensively for this? For example, what is the ideal behaviour if ToscaPlanToSpecTransformer fails to initialise? Should it throw an exception that causes the server to report it failed to startup (i.e. process is running, but it reported failure)? Thinking about it more, we can't really hide this error - it means that subsequent use of Tosca blueprints will fail so it's good to tell the user early!

What about any automated cleanup? @Graeme-Miller suggested that clearing out some corrupted directory of a4c/Elasticsearch could fix the problem. It would be good to document those steps (or perhaps even to do them automatically on restart?!). If this is just using a4c code to parse the Tosca, then presumably we could make assumptions about it being a dedicated Elasticsearch instance, and presumably it doesn't contain any persisted state that needs preserved between runs? (But we don't want to slow-down the server startup too much either).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@aledsage @sjcorbett @robertgmoss