Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Method Throttling #295

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
e94b945
Revert "[HotFix] Allow disabling automatic text analysis of all vocab…
lukaskabc Aug 15, 2024
5860074
Debounce aspect
lukaskabc Aug 16, 2024
d2e8ebb
Throttle aspect with spring aop
lukaskabc Aug 20, 2024
34ef7f1
Throttle aspect & tests
lukaskabc Aug 21, 2024
aa29ec4
support for SpEL in groups
lukaskabc Aug 21, 2024
92f5a08
explicitly handle transactions
lukaskabc Aug 22, 2024
d2c3033
disable sync term saver
lukaskabc Aug 22, 2024
c55b2c7
async mvc
lukaskabc Aug 23, 2024
f3c30f3
occurrence saving in parallel
lukaskabc Aug 23, 2024
0500052
fix tests
lukaskabc Aug 23, 2024
d2a2e1c
use parallel occurrence saving only for files
lukaskabc Aug 26, 2024
f3d2de0
validation caching
lukaskabc Aug 27, 2024
d6a6651
fix tests
lukaskabc Sep 3, 2024
5b727e7
suppress AsyncRequestNotUsableException from logging
lukaskabc Sep 4, 2024
861ef2d
merge
lukaskabc Sep 4, 2024
f497c35
optimize imports
lukaskabc Sep 4, 2024
d27ec23
validation cache
lukaskabc Sep 4, 2024
c8bf2fa
[Performance #287] Sending messages to session, validation throttling
lukaskabc Sep 5, 2024
41a9546
[Performance #287] Resolve invalid thread synchronization
lukaskabc Sep 5, 2024
8529372
[Performance #287] Trigger validation on vocabulary modification even…
lukaskabc Sep 5, 2024
2abe439
[Performance #287] Introduce base class for Vocabulary related event
lukaskabc Sep 5, 2024
5ec9251
[Performance #287] Docs for throttle related classes
lukaskabc Sep 5, 2024
697d569
[Performance #287] Docs for throttle related classes, tests descripti…
lukaskabc Sep 6, 2024
4412d0b
[Performance #287] Adds async tests for ThrottledFuture running
lukaskabc Sep 6, 2024
c0e1dbd
[Performance #287] test for calling web socket exception handler
lukaskabc Sep 9, 2024
bcc6036
optimize imports
lukaskabc Sep 9, 2024
2f6ab23
[Performance #287] remove async mvc
lukaskabc Sep 9, 2024
5be5e36
[Performance #287] prevent concurrent execution of throttled tasks wi…
lukaskabc Sep 9, 2024
665cb57
[Performance #287] adds tests for throttled future
lukaskabc Sep 9, 2024
c6e89d2
[Performance #287] remove vocabulary content modified event on vocabu…
lukaskabc Sep 9, 2024
e694c8d
[Performance #285] publishing text analysis to websocket
lukaskabc Sep 9, 2024
3d7a7d1
[Performance #285] fix text analysis tests
lukaskabc Sep 10, 2024
8422d07
[Performance #285] remove trace time logging and improve throttled fu…
lukaskabc Sep 10, 2024
bd9f5ee
[Performance #285] rename term service tests for better description
lukaskabc Sep 10, 2024
98b2236
[Performance #285] remove unused class MockedFuture
lukaskabc Sep 10, 2024
c469e78
optimize imports
lukaskabc Sep 10, 2024
a6366d7
code cleanup
lukaskabc Sep 10, 2024
d15a3c1
[Performance #285] Implement API for retrieval of long-running tasks …
lukaskabc Sep 10, 2024
9bc022c
[Performance #285] Fix test error due to new controller missing depen…
lukaskabc Sep 11, 2024
3acb751
[Performance #285] Sending current running tasks to a client on subsc…
lukaskabc Sep 11, 2024
b90036e
[Performance #285] add uuid to long running task status logging
lukaskabc Sep 11, 2024
da44485
[Performance #285] change when notification about running task change…
lukaskabc Sep 11, 2024
85d2e4b
[Performance #285] reset client when all long-running tasks completed
lukaskabc Sep 11, 2024
ace3e16
[Performance #285] add description for name in Throttle annotation
lukaskabc Sep 11, 2024
a0a7304
[Performance #285] remove entity re-fetch
lukaskabc Sep 11, 2024
5f7c719
[Performance #285] notify about long-running task change on future ca…
lukaskabc Sep 11, 2024
66299f4
[Performance #285] add comment to calling get on throttled future
lukaskabc Sep 11, 2024
e0cf305
[Performance #285] remove unnecessary stubbing for removed method call
lukaskabc Sep 11, 2024
ca68c6a
[Performance #285] send long running tasks to client as whole map to …
lukaskabc Sep 12, 2024
1ab3e77
[Performance #285] remove string builders, refetch entities in thrott…
lukaskabc Sep 12, 2024
8d82c67
[PR #295] rename vocabulary events and replace jetbrains annotations
lukaskabc Sep 12, 2024
715f3d7
[Performance #285] fix event ordering in long-running tasks registry
lukaskabc Sep 12, 2024
d922ccc
[PR #295] make defensive copy instead of unmodifiable view
lukaskabc Sep 12, 2024
34d398e
[Performance #285] change long running task registry handling
lukaskabc Sep 12, 2024
4328401
[PR #295] rename validator method to reflect its purpose
lukaskabc Sep 12, 2024
2f45c2f
[Performance #285] fix exceptional future competition
lukaskabc Sep 12, 2024
7c8e89f
[PR #295] rollback VocabularyController changes
lukaskabc Sep 12, 2024
d512acf
[PR #295] merge SynchronousTermOccurrenceSaver with TermOccurrenceSav…
lukaskabc Sep 12, 2024
73818fb
[Performance #285] fix faulty test and ensure that the task is null a…
lukaskabc Sep 12, 2024
7ed714b
[PR #295] remove parallel stream
lukaskabc Sep 12, 2024
95af91a
[PR #295] optimize imports
lukaskabc Sep 12, 2024
be576d1
[PR #295] add Javadoc and prevent useless lambda redefinition in loop
lukaskabc Sep 12, 2024
5974704
[PR #295] revert unsuccessfully attempt to optimize memory usage, res…
lukaskabc Sep 12, 2024
6df9653
[PR #295] move throttle constants to configuration
lukaskabc Sep 12, 2024
d73883c
[Fix] Minor transactional behavior fix.
ledsoft Sep 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added doc/throttle-debounce.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
17 changes: 17 additions & 0 deletions src/main/java/cz/cvut/kbss/termit/config/AppConfig.java
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,17 @@

import cz.cvut.kbss.termit.util.AsyncExceptionHandler;
import org.springframework.aop.interceptor.AsyncUncaughtExceptionHandler;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.EnableAspectJAutoProxy;
import org.springframework.context.annotation.EnableMBeanExport;
import org.springframework.context.annotation.ImportResource;
import org.springframework.context.annotation.aspectj.EnableSpringConfigured;
import org.springframework.retry.annotation.EnableRetry;
import org.springframework.scheduling.annotation.AsyncConfigurer;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler;

@Configuration
@EnableMBeanExport
Expand All @@ -35,10 +38,24 @@
@EnableAsync
@EnableScheduling
@EnableRetry
@ImportResource("classpath*:spring-aop.xml")
public class AppConfig implements AsyncConfigurer {

@Override
public AsyncUncaughtExceptionHandler getAsyncUncaughtExceptionHandler() {
return new AsyncExceptionHandler();
}

/**
* This thread pool is responsible for executing long-running tasks in the application.
*/
@Bean(destroyMethod = "destroy")
public ThreadPoolTaskScheduler longRunningTaskScheduler(cz.cvut.kbss.termit.util.Configuration config) {
ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler();
threadPoolTaskScheduler.setPoolSize(config.getAsyncThreadCount());
threadPoolTaskScheduler.setThreadNamePrefix("TermItScheduler-");
threadPoolTaskScheduler.setWaitForTasksToCompleteOnShutdown(true);
threadPoolTaskScheduler.setRemoveOnCancelPolicy(true);
return threadPoolTaskScheduler;
}
}
44 changes: 23 additions & 21 deletions src/main/java/cz/cvut/kbss/termit/config/WebAppConfig.java
Original file line number Diff line number Diff line change
Expand Up @@ -62,19 +62,13 @@
@Configuration
public class WebAppConfig implements WebMvcConfigurer {

private final cz.cvut.kbss.termit.util.Configuration.Repository config;
private final cz.cvut.kbss.termit.util.Configuration config;

@Value("${application.version:development}")
private String version;

public WebAppConfig(cz.cvut.kbss.termit.util.Configuration config) {
this.config = config.getRepository();
}

@Bean(name = "objectMapper")
@Primary
public ObjectMapper objectMapper() {
return createJsonObjectMapper();
this.config = config;
}

/**
Expand All @@ -99,11 +93,6 @@ public static ObjectMapper createJsonObjectMapper() {
return objectMapper;
}

@Bean(name = "jsonLdMapper")
public ObjectMapper jsonLdObjectMapper() {
return createJsonLdObjectMapper();
}

/**
* Creates an {@link ObjectMapper} for processing JSON-LD using the JB4JSON-LD library.
* <p>
Expand All @@ -119,9 +108,21 @@ public static ObjectMapper createJsonLdObjectMapper() {
jsonLdModule.configure(cz.cvut.kbss.jsonld.ConfigParam.SCAN_PACKAGE, "cz.cvut.kbss.termit");
jsonLdModule.configure(SerializationConstants.FORM, SerializationConstants.FORM_COMPACT_WITH_CONTEXT);
mapper.registerModule(jsonLdModule);
mapper.registerModule(new JavaTimeModule());
return mapper;
}

@Bean(name = "objectMapper")
@Primary
public ObjectMapper objectMapper() {
return createJsonObjectMapper();
}

@Bean(name = "jsonLdMapper")
public ObjectMapper jsonLdObjectMapper() {
return createJsonLdObjectMapper();
}

/**
* Register the proxy for SPARQL endpoint.
*
Expand All @@ -133,10 +134,11 @@ public ServletWrappingController sparqlEndpointController() throws Exception {
controller.setServletClass(AdjustedUriTemplateProxyServlet.class);
controller.setBeanName("sparqlEndpointProxyServlet");
final Properties p = new Properties();
p.setProperty("targetUri", config.getUrl());
final cz.cvut.kbss.termit.util.Configuration.Repository repository = config.getRepository();
p.setProperty("targetUri", repository.getUrl());
p.setProperty("log", "false");
p.setProperty(ConfigParam.REPO_USERNAME.toString(), config.getUsername() != null ? config.getUsername() : "");
p.setProperty(ConfigParam.REPO_PASSWORD.toString(), config.getPassword() != null ? config.getPassword() : "");
p.setProperty(ConfigParam.REPO_USERNAME.toString(), repository.getUsername() != null ? repository.getUsername() : "");
p.setProperty(ConfigParam.REPO_PASSWORD.toString(), repository.getPassword() != null ? repository.getPassword() : "");
controller.setInitParameters(p);
controller.afterPropertiesSet();
return controller;
Expand All @@ -147,7 +149,7 @@ public SimpleUrlHandlerMapping sparqlQueryControllerMapping() throws Exception {
SimpleUrlHandlerMapping mapping = new SimpleUrlHandlerMapping();
mapping.setOrder(0);
final Map<String, Object> urlMap = Collections.singletonMap(Constants.REST_MAPPING_PATH + "/query",
sparqlEndpointController());
sparqlEndpointController());
mapping.setUrlMap(urlMap);
return mapping;
}
Expand Down Expand Up @@ -193,10 +195,10 @@ public FilterRegistrationBean<DiagnosticsContextFilter> mdcFilter() {
@Bean
public OpenAPI customOpenAPI() {
return new OpenAPI().components(new Components().addSecuritySchemes("bearer-key",
new SecurityScheme().type(
SecurityScheme.Type.HTTP)
.scheme("bearer")
.bearerFormat("JWT")))
new SecurityScheme().type(
SecurityScheme.Type.HTTP)
.scheme("bearer")
.bearerFormat("JWT")))
.info(new Info().title("TermIt REST API").description("TermIt REST API definition.")
.version(version));
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,17 @@
import cz.cvut.kbss.termit.util.Constants;
import cz.cvut.kbss.termit.websocket.handler.StompExceptionHandler;
import cz.cvut.kbss.termit.websocket.handler.WebSocketExceptionHandler;
import cz.cvut.kbss.termit.websocket.handler.WebSocketMessageWithHeadersValueHandler;
import org.jetbrains.annotations.NotNull;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Lazy;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.lang.NonNull;
import org.springframework.messaging.Message;
import org.springframework.messaging.converter.MappingJackson2MessageConverter;
import org.springframework.messaging.converter.MessageConverter;
import org.springframework.messaging.converter.StringMessageConverter;
import org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolver;
import org.springframework.messaging.handler.invocation.HandlerMethodReturnValueHandler;
import org.springframework.messaging.simp.SimpMessagingTemplate;
import org.springframework.messaging.simp.config.ChannelRegistration;
import org.springframework.messaging.simp.config.MessageBrokerRegistry;
Expand Down Expand Up @@ -88,17 +86,12 @@ public void addArgumentResolvers(List<HandlerMethodArgumentResolver> argumentRes
* @see <a href="https://github.com/spring-projects/spring-security/blob/6.3.x/config/src/main/java/org/springframework/security/config/annotation/web/socket/WebSocketMessageBrokerSecurityConfiguration.java#L97">Spring security source</a>
*/
@Override
public void configureClientInboundChannel(@NotNull ChannelRegistration registration) {
public void configureClientInboundChannel(@NonNull ChannelRegistration registration) {
AuthorizationChannelInterceptor interceptor = new AuthorizationChannelInterceptor(messageAuthorizationManager);
interceptor.setAuthorizationEventPublisher(new SpringAuthorizationEventPublisher(context));
registration.interceptors(webSocketJwtAuthorizationInterceptor, new SecurityContextChannelInterceptor(), interceptor);
}

@Override
public void addReturnValueHandlers(List<HandlerMethodReturnValueHandler> returnValueHandlers) {
returnValueHandlers.add(new WebSocketMessageWithHeadersValueHandler(simpMessagingTemplate));
}

@Override
public void registerStompEndpoints(StompEndpointRegistry registry) {
registry.addEndpoint("/ws").setAllowedOrigins(allowedOrigins.split(","));
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
package cz.cvut.kbss.termit.event;

import cz.cvut.kbss.termit.model.resource.File;
import org.springframework.lang.NonNull;

import java.net.URI;

/**
* Indicates that text analysis of a file was finished
*/
public class FileTextAnalysisFinishedEvent extends VocabularyEvent {

private final URI fileUri;

public FileTextAnalysisFinishedEvent(Object source, @NonNull File file) {
super(source, file.getDocument().getVocabulary());
this.fileUri = file.getUri();
}

public URI getFileUri() {
return fileUri;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
package cz.cvut.kbss.termit.event;

import cz.cvut.kbss.termit.util.longrunning.LongRunningTaskStatus;
import org.springframework.context.ApplicationEvent;
import org.springframework.lang.NonNull;

/**
* Indicates a status change of a long-running task.
*/
public class LongRunningTaskChangedEvent extends ApplicationEvent {

private final LongRunningTaskStatus status;

public LongRunningTaskChangedEvent(@NonNull Object source, final @NonNull LongRunningTaskStatus status) {
super(source);
this.status = status;
}

public @NonNull LongRunningTaskStatus getStatus() {
return status;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
package cz.cvut.kbss.termit.event;

import cz.cvut.kbss.termit.model.AbstractTerm;
import org.springframework.lang.NonNull;

import java.net.URI;

/**
* Indicates that a text analysis of a term definition was finished
*/
public class TermDefinitionTextAnalysisFinishedEvent extends VocabularyEvent {
private final URI termUri;

public TermDefinitionTextAnalysisFinishedEvent(@NonNull Object source, @NonNull AbstractTerm term) {
super(source, term.getVocabulary());
this.termUri = term.getUri();
}

public URI getTermUri() {
return termUri;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
*/
package cz.cvut.kbss.termit.event;

import org.springframework.context.ApplicationEvent;
import org.springframework.lang.NonNull;

import java.net.URI;

Expand All @@ -26,16 +26,9 @@
* <p>
* This typically means a term is added, removed or modified. Modification of vocabulary metadata themselves is not considered here.
*/
public class VocabularyContentModified extends ApplicationEvent {
public class VocabularyContentModifiedEvent extends VocabularyEvent {

private final URI vocabularyIri;

public VocabularyContentModified(Object source, URI vocabularyIri) {
super(source);
this.vocabularyIri = vocabularyIri;
}

public URI getVocabularyIri() {
return vocabularyIri;
public VocabularyContentModifiedEvent(@NonNull Object source, @NonNull URI vocabularyIri) {
super(source, vocabularyIri);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,16 @@
*/
package cz.cvut.kbss.termit.event;

import org.springframework.context.ApplicationEvent;
import org.springframework.lang.NonNull;

import java.net.URI;

/**
* Indicates that a vocabulary has been created.
*/
public class VocabularyCreatedEvent extends ApplicationEvent {
public class VocabularyCreatedEvent extends VocabularyEvent {

public VocabularyCreatedEvent(Object source) {
super(source);
public VocabularyCreatedEvent(@NonNull Object source, @NonNull URI vocabularyIri) {
super(source, vocabularyIri);
}
}
28 changes: 28 additions & 0 deletions src/main/java/cz/cvut/kbss/termit/event/VocabularyEvent.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
package cz.cvut.kbss.termit.event;

import org.springframework.context.ApplicationEvent;
import org.springframework.lang.NonNull;

import java.net.URI;
import java.util.Objects;

/**
* Base class for vocabulary related events
*/
public abstract class VocabularyEvent extends ApplicationEvent {
protected final URI vocabularyIri;

protected VocabularyEvent(@NonNull Object source, @NonNull URI vocabularyIri) {
super(source);
Objects.requireNonNull(vocabularyIri);
this.vocabularyIri = vocabularyIri;
}

/**
* The identifier of the vocabulary to which this event is bound
* @return vocabulary IRI
*/
public URI getVocabularyIri() {
return vocabularyIri;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
package cz.cvut.kbss.termit.event;

import cz.cvut.kbss.termit.model.validation.ValidationResult;
import org.springframework.lang.NonNull;

import java.net.URI;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;

/**
* Indicates that validation for a set of vocabularies was finished.
*/
public class VocabularyValidationFinishedEvent extends VocabularyEvent {

/**
* Vocabulary closure of {@link #vocabularyIri}.
* IRIs of vocabularies that are imported by {@link #vocabularyIri} and were part of the validation.
*/
private final List<URI> vocabularyIris;

private final List<ValidationResult> validationResults;

/**
* @param source the source of the event
* @param originVocabularyIri Vocabulary closure of {@link #vocabularyIri}.
* @param vocabularyIris IRI of the vocabulary on which the validation was triggered.
* @param validationResults results of the validation
*/
public VocabularyValidationFinishedEvent(@NonNull Object source, @NonNull URI originVocabularyIri,
@NonNull Collection<URI> vocabularyIris,
@NonNull List<ValidationResult> validationResults) {
super(source, originVocabularyIri);
// defensive copy
this.vocabularyIris = new ArrayList<>(vocabularyIris);
this.validationResults = new ArrayList<>(validationResults);
}

@NonNull
public List<URI> getVocabularyIris() {
return Collections.unmodifiableList(vocabularyIris);
}

@NonNull
public List<ValidationResult> getValidationResults() {
return Collections.unmodifiableList(validationResults);
}
}
Loading