Skip to content
This repository has been archived by the owner on Apr 22, 2020. It is now read-only.

Use worker cleanup task instead of direct jedis cleanup #89

Merged
merged 14 commits into from
Jun 26, 2019
Merged

Use worker cleanup task instead of direct jedis cleanup #89

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
70332de
Use worker task instead of jedis cleanup
rajatparida86 Jan 7, 2019
395fd41
Use appropriate UUID, Constructor injection, Formatting
rajatparida86 Jan 8, 2019
694d3e1
Separate interface method for batch entity cleanup
rajatparida86 Jan 8, 2019
cd656a9
Batch entity cleanup
rajatparida86 Jan 8, 2019
3ef0d03
Add timelimit to redis payload
rajatparida86 Jan 8, 2019
9341fb7
Only cleanup if there is something to cleanup
rajatparida86 Jan 8, 2019
de70fb8
Use TR as the prefix of task ID
rajatparida86 Jan 8, 2019
d0fd7e0
Review comments
rajatparida86 Jan 9, 2019
4228868
Only execute batch cleanup in remote schedulers
rajatparida86 Jan 23, 2019
886ba7b
Log entity counts
rajatparida86 Jan 23, 2019
6f5c0a7
Differentiate between Global and Local scheduler
rajatparida86 Jan 23, 2019
ea90f38
unit tests for cleanup and global scheduler
rajatparida86 Jan 23, 2019
8537001
Fix test method name to camel case
rajatparida86 Apr 12, 2019
5802283
Fix camel case for missed method
rajatparida86 Apr 12, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions src/main/java/de/zalando/zmon/scheduler/ng/CommandSerializer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,7 @@

import io.opentracing.Tracer;

import java.util.ArrayList;
import java.util.Collection;
import java.util.Date;
import java.util.List;
import java.util.*;

/**
* Created by jmussler on 30.06.16.
Expand All @@ -28,6 +25,19 @@ public String expiresTime(long interval) {
return LocalDateFormatter.get().format(exp);
}

public byte[] writeCleanUp(Set<String> removedIds){
CeleryBody body = new CeleryBody();

body.task= "cleanup";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know that we don't have a linter for java projects yet, but what do you think if we'll try to follow some basic formatting rules, common for most languages? Like spaces before and after = and + for example.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed

body.expires="";

//TODO: This is not unique yet
body.id="entity-CU:"+System.currentTimeMillis();

body.kwargs.put("cleanup_entities", removedIds);
return writer.asCeleryTask(body);
}

public byte[] writeTrialRun(Entity entity, TrialRunRequest request) {
CeleryBody body = new CeleryBody();

Expand Down
6 changes: 6 additions & 0 deletions src/main/java/de/zalando/zmon/scheduler/ng/entities/EntityRepository.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import de.zalando.zmon.scheduler.ng.scheduler.Scheduler;
import io.opentracing.Tracer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
Expand All @@ -24,6 +25,9 @@
@Component
public class EntityRepository extends CachedRepository<String, EntityAdapterRegistry, Entity> {

@Autowired
Scheduler scheduler;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Best practice is to use constructor injection over field or setter injection (e.g. https://www.vojtechruzicka.com/field-dependency-injection-considered-harmful/)


private static final Logger LOG = LoggerFactory.getLogger(EntityRepository.class);

private List<Map<String, String>> baseFilter = null;
Expand Down Expand Up @@ -192,6 +196,8 @@ public synchronized void fill() {
}
}

scheduler.scheduleEntityCleanUp(removedIds);

for (String k : changedFilterProperties) {
for (EntityChangeListener l : currentListeners) {
l.notifyEntityChange(this, oldUnfiltered.get(k), unfilteredEntities.get(k));
Expand Down
11 changes: 6 additions & 5 deletions src/main/java/de/zalando/zmon/scheduler/ng/scheduler/Scheduler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,7 @@
import org.springframework.scheduling.concurrent.CustomizableThreadFactory;
import redis.clients.jedis.Jedis;

import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.*;
import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

Expand Down Expand Up @@ -236,4 +232,9 @@ public void scheduleTrialRun(TrialRunRequest request) {
service.schedule(new TrialRunCleanupTask(request.id, schedulerConfig), 300, TimeUnit.SECONDS);
}
}

public void scheduleEntityCleanUp(Set<String> removedIds){
byte [] command = taskSerializer.writeCleanUp(removedIds);
queueSelector.execute(null, command);
}
}