Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Proxy Arch (ActiveMQ) #769

Open
wants to merge 271 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
271 commits
Select commit Hold shift + click to select a range
934893e
More tests passing
laullon Jul 7, 2022
87be49a
bug
laullon Jul 8, 2022
3434ab0
Multitenant, only 17 test failing, and few more things
laullon Jul 10, 2022
f6f394f
Multitenant, only 17 test failing, and few more things
laullon Jul 10, 2022
eaa2115
Multitenant
laullon Jul 11, 2022
63747d3
Buffers shutdown
laullon Jul 11, 2022
eaffe81
Logs working, and "more" unit test "passing"
laullon Jul 12, 2022
3b6a556
bug
laullon Jul 13, 2022
e09ed23
received and delivered stats on the log
laullon Jul 13, 2022
056bbc3
restore chain test
laullon Jul 13, 2022
5ff2a0a
some metrics
laullon Jul 13, 2022
7c94529
Fix SourceTagSubmissionTask Failing Tests (#770)
joannatk Jul 14, 2022
c956c8a
Merge branch 'dev' into dev-proxy2
laullon Jul 16, 2022
c8dff23
queue even metrics, lees noisy log, some bugs
laullon Jul 16, 2022
f3bdfed
Passing 12 more unit tests in PushAgentTest (#771)
joannatk Jul 18, 2022
b75e40d
Bridge, better stats, and more tests
laullon Jul 21, 2022
79213a9
External SQS buffer, some config changes, cleanup
laullon Jul 22, 2022
8075f18
initial setup
laullon Jul 27, 2022
6cd6d6f
build
laullon Jul 27, 2022
bb37b08
Update vib-container.yml
laullon Jul 27, 2022
f729ade
Update vib-container.yml
laullon Jul 27, 2022
3b2905d
use make file to build
laullon Jul 27, 2022
5ddee63
bug
laullon Jul 27, 2022
28f4215
bug
laullon Jul 27, 2022
828d011
make build on vib-container
laullon Jul 27, 2022
fcfab12
only verify
laullon Jul 27, 2022
fabeef9
image tag
laullon Jul 27, 2022
61fbfb7
DOCKER_TAG
laullon Jul 27, 2022
65a292d
DOCKER_TAG
laullon Jul 27, 2022
3f86a79
repository
laullon Jul 27, 2022
a8358d8
container-image-lint
laullon Jul 27, 2022
d88b47f
context
laullon Jul 27, 2022
cd4d33a
Update vib-container.json
laullon Jul 27, 2022
940d8fa
container-image-package
laullon Jul 27, 2022
5ee3caa
Merge branch 'laullon/VIB' of github.com:wavefrontHQ/wavefront-proxy …
laullon Jul 27, 2022
2b5fab8
Docker push
laullon Jul 27, 2022
8cff529
verify using docker
laullon Jul 27, 2022
21d28f6
DOCKER_USER
laullon Jul 27, 2022
9ffe52e
no lint
laullon Jul 27, 2022
157a8cd
VIB_ENV_
laullon Jul 27, 2022
983c5a0
oci://
laullon Jul 27, 2022
412340f
local helm
laullon Jul 27, 2022
d509edb
tests
laullon Jul 27, 2022
4e8e491
path
laullon Jul 27, 2022
45ed420
url
laullon Jul 27, 2022
c1c007f
SHA_ARCHIVE
laullon Jul 27, 2022
e4f9cbd
helm chart
laullon Jul 27, 2022
22ad311
path
laullon Jul 27, 2022
2f88528
helm path
laullon Jul 27, 2022
d23d10a
no package
laullon Jul 27, 2022
4c14091
back
laullon Jul 27, 2022
ef08262
name
laullon Jul 27, 2022
3aaaa16
SpanHandlerImpl.spanLogsHandler bug
laullon Jul 27, 2022
46fcec0
new helm
laullon Jul 27, 2022
6338054
workload
laullon Jul 27, 2022
3f24c44
two deployments
laullon Jul 27, 2022
14939e3
deploy-edge-deployment
laullon Jul 28, 2022
01e25bf
goss and debian
laullon Jul 28, 2022
9343dc5
debian
laullon Jul 28, 2022
401233f
debian 1
laullon Jul 28, 2022
ad752a2
yaml
laullon Jul 28, 2022
8664c1e
1000
laullon Jul 28, 2022
29a69e4
644
laullon Jul 28, 2022
35861ff
deploy-wf-proxy-buffer-tests
laullon Jul 28, 2022
d1fd267
goss
laullon Jul 28, 2022
a78b6b1
url
laullon Jul 28, 2022
b5bf70f
ls
laullon Jul 28, 2022
4212ace
cypress
laullon Jul 28, 2022
1acdf73
endpoint
laullon Jul 28, 2022
9aafecc
timeout
laullon Jul 28, 2022
02bc63a
linux tests
laullon Jul 28, 2022
1feb311
revert tests
laullon Jul 28, 2022
fc01fab
cypress
laullon Jul 28, 2022
8ab9612
bug
laullon Jul 28, 2022
bf0c7ef
Merge branch 'dev' into dev-proxy2
laullon Sep 5, 2022
a43c83f
proxy-cheking test
laullon Sep 5, 2022
dde3223
Merge branch 'dev' into laullon/VIB
laullon Sep 5, 2022
0d47e35
curl test
laullon Sep 6, 2022
50095bc
extenmsion
laullon Sep 6, 2022
4974f95
goss on deploy-chain-deployment
laullon Sep 6, 2022
1f4f326
bash on goss
laullon Sep 6, 2022
2b20408
sleep
laullon Sep 6, 2022
7d9c9c9
correct goss test
laullon Sep 6, 2022
7093c4f
timeout
laullon Sep 6, 2022
43db1d3
Update vib-chain-checking.json
laullon Sep 8, 2022
661762f
livenessProbe
laullon Sep 8, 2022
85b66c3
some debug
laullon Sep 9, 2022
6a972e3
Update vib-chain-checking.json
laullon Sep 9, 2022
9a0a862
no fix id
laullon Sep 13, 2022
a6c0de0
test_tenant
laullon Sep 13, 2022
2f6f265
clean
laullon Sep 13, 2022
84a000b
bug
laullon Sep 13, 2022
6605d39
goss
laullon Sep 13, 2022
2184e3d
revet goss
laullon Sep 13, 2022
a762be9
name change
laullon Sep 13, 2022
52630b1
helm names
laullon Sep 13, 2022
7acac68
new runtime_parameters
laullon Sep 13, 2022
2b39685
pushListener
laullon Sep 13, 2022
f1cb821
Update deployment.yaml
laullon Sep 13, 2022
396de90
some changes
laullon Sep 13, 2022
cc8fa21
new tests
laullon Sep 14, 2022
31a4ae3
test_metrics
laullon Sep 14, 2022
c85d304
Merge branch 'laullon/VIB' into dev-proxy2
laullon Sep 14, 2022
609fc8d
Java-lib
laullon Sep 14, 2022
1f0e31c
commented not working tests
laullon Sep 15, 2022
d23a5ee
docker bug
laullon Sep 15, 2022
13303a6
make build-jar
laullon Sep 15, 2022
d989d5e
testEndToEndLogs bug
laullon Sep 15, 2022
f6c23ea
VIB tests based on branch name
laullon Sep 16, 2022
0bbbf37
TestQueue entityType
laullon Sep 19, 2022
a011802
All tests passing
laullon Sep 19, 2022
ae1db4d
disk-buffer inital test
laullon Sep 19, 2022
e38006e
truncate implementation and test
laullon Sep 20, 2022
7987bcc
bug
laullon Sep 20, 2022
1fb5c00
test bug
laullon Sep 20, 2022
b806e98
some todos
laullon Sep 20, 2022
99a6138
bug
laullon Sep 20, 2022
70bbe1b
disck buffer test
laullon Sep 20, 2022
85f5fe7
bug
laullon Sep 20, 2022
f21e340
better buffer test
laullon Sep 21, 2022
d5d72ae
Merge branch 'dev' into dev-proxy2
laullon Sep 21, 2022
bde51a3
refactor AbstractSenderTask and remove some unused code and wierd con…
laullon Sep 22, 2022
895174e
opentel-app-docker-build
laullon Sep 22, 2022
4add49f
file
laullon Sep 22, 2022
52f1454
context
laullon Sep 22, 2022
1568cd2
checkout
laullon Sep 22, 2022
73fec5c
vib files
laullon Sep 22, 2022
1f0b93c
done
laullon Sep 23, 2022
ac71d6d
bug
laullon Sep 23, 2022
bd28bac
Merge branch 'master' into dev-proxy2
laullon Sep 26, 2022
d0fce08
some clean up
laullon Sep 26, 2022
a1c29c1
no more EmbeddedActiveMQ
laullon Sep 26, 2022
c023512
bug
laullon Sep 26, 2022
5462787
buffer-lock test
laullon Sep 26, 2022
054f364
buffer-lock test
laullon Sep 26, 2022
d7d4313
added https proxy tp chain test
laullon Sep 27, 2022
bd9cfb9
bug
laullon Sep 27, 2022
5eca42e
no trivy on chain test
laullon Sep 27, 2022
8612a00
logs test and docker-multi-arch
laullon Sep 29, 2022
94c5117
revert docker-multi-arch
laullon Sep 29, 2022
bedd577
pushback
laullon Sep 29, 2022
d6241ed
log wf data and docker-multi-arch
laullon Sep 30, 2022
48cbf21
jobs names
laullon Oct 3, 2022
09dbcf8
Merge branch 'dev' into dev-proxy2
laullon Oct 3, 2022
1c81670
cleanup undoc functionality
laullon Oct 3, 2022
6967a43
cleanup
laullon Oct 3, 2022
ed09691
cleanup
laullon Oct 3, 2022
0111737
new docker repo
laullon Oct 3, 2022
87cdade
DOCKER_OWNER
laullon Oct 3, 2022
34c7cfa
Merge branch 'dev' into dev-proxy2
laullon Oct 3, 2022
b04c504
new docker names
laullon Oct 3, 2022
833656f
beta action
laullon Oct 3, 2022
3a208df
beta 12.0
laullon Oct 3, 2022
135bccb
Update beta-tag.yml
laullon Oct 3, 2022
917943b
Update beta-tag.yml
laullon Oct 3, 2022
d383aaf
more cleanup
laullon Oct 3, 2022
31b5547
Memory Buffer data dir bug
laullon Oct 4, 2022
1639a53
formating
laullon Oct 4, 2022
92d7b28
solved log error messages (slf4j and jboss)
laullon Oct 4, 2022
19702c3
format
laullon Oct 4, 2022
60d3ec2
Merge branch 'dev' into dev-proxy2
laullon Oct 4, 2022
7a92569
image name error
laullon Oct 4, 2022
8ddc5e3
Merge branch 'dev-proxy2' into beta-12.0
laullon Oct 4, 2022
097168c
Update vib-container.yml
laullon Oct 5, 2022
54e7089
logs fixed
laullon Oct 5, 2022
ee3d127
msgExpirationTime bug
laullon Oct 5, 2022
dccc6ad
linux pkg
laullon Oct 6, 2022
2b70f49
Merge branch 'dev' into dev-proxy2
laullon Oct 10, 2022
f0e7d9d
rename
laullon Oct 10, 2022
157409d
Merge branch 'dev-proxy2' into beta-12.1
laullon Oct 10, 2022
efb8e76
12.1
laullon Oct 10, 2022
78c0156
Linux bug
laullon Oct 11, 2022
155214d
logs retored
laullon Oct 13, 2022
135107c
Merge branch 'beta-12.1' into dev-proxy2
laullon Oct 13, 2022
2328c11
better error control
laullon Oct 17, 2022
ee2875e
Senders priority
laullon Oct 21, 2022
0ddbc50
Merge branch 'dev' into dev-proxy2
laullon Oct 24, 2022
8065591
Update validation.yml
laullon Oct 27, 2022
990bc22
Update vib-deb-package-debian.json
laullon Oct 27, 2022
7cd2403
Update vib-deb-package-debian.json
laullon Oct 28, 2022
afd8fbb
Update vib-deb-package-debian.json
laullon Oct 28, 2022
927fab5
Update goss.yaml
laullon Oct 28, 2022
52f0c85
test deb coy
laullon Oct 28, 2022
101e024
java-lib
laullon Oct 28, 2022
aae252d
deb install
laullon Oct 29, 2022
d47b436
Merge branch 'dev' into dev-proxy2
laullon Oct 29, 2022
f54cd7d
no PC
laullon Oct 29, 2022
ef4f1eb
find
laullon Oct 29, 2022
29a1228
linux test
laullon Oct 31, 2022
dcc19c5
bug
laullon Oct 31, 2022
56e23d8
Memory and stats log
laullon Nov 3, 2022
0ebada2
Update goss.yaml
laullon Nov 3, 2022
c5fb959
Update goss.yaml
laullon Nov 3, 2022
5a289a2
no url
laullon Nov 4, 2022
617bbf9
artifact
laullon Nov 4, 2022
bfa5b2c
Merge branch 'dev' into dev-proxy2
laullon Nov 4, 2022
5c34391
bug
laullon Nov 4, 2022
2e7b7e6
URL
laullon Nov 4, 2022
c5d8a1f
vars
laullon Nov 4, 2022
f37e29a
github.run_id
laullon Nov 4, 2022
6cb88d8
build-linux
laullon Nov 4, 2022
2bb8205
test
laullon Nov 4, 2022
e411b75
Download
laullon Nov 4, 2022
898fb7f
linux file permisions
laullon Nov 14, 2022
8d3b623
Merge branch 'dev' into dev-proxy2
laullon Nov 14, 2022
9202844
improved linux package
laullon Nov 15, 2022
ef4c26a
tests-debian url
laullon Nov 15, 2022
2e952f8
debian test resources
laullon Nov 15, 2022
ff3e4ad
bug
laullon Nov 15, 2022
4983e7f
stress-test to run on remote if need it
laullon Nov 17, 2022
bae7fe2
activemq improvements
laullon Nov 17, 2022
3f75d1e
java.lang.NullPointerException: null
laullon Nov 17, 2022
6468eca
Merge branch 'dev-proxy2' into beta-12.1
laullon Nov 17, 2022
5858e7e
linux/arm64
laullon Nov 17, 2022
3b4f515
reconect
laullon Nov 22, 2022
472d3a4
zip and upload
laullon Nov 23, 2022
b99bfaf
zip and credetials
laullon Nov 23, 2022
baf2835
tests debian package S3
laullon Nov 23, 2022
f134a08
bug
laullon Nov 23, 2022
7df5060
deb_script.zip
laullon Nov 23, 2022
c86a783
correct path
laullon Nov 23, 2022
5d061d9
correct zip path
laullon Nov 23, 2022
aa69bf1
env
laullon Nov 23, 2022
58b36df
checkout
laullon Nov 23, 2022
00bdd16
test path
laullon Nov 23, 2022
faa266c
/tmp/extracted/
laullon Nov 23, 2022
c0e138b
pastebin
laullon Nov 23, 2022
9c5edc2
few things
laullon Nov 23, 2022
6e90093
bug
laullon Nov 23, 2022
541f03f
more tests
laullon Nov 23, 2022
fe59b20
Merge branch 'dev' into dev-proxy2
laullon Nov 24, 2022
a1f5c20
review comments
laullon Nov 29, 2022
5403328
received.bytes and delivered.bytes metrics
laullon Nov 29, 2022
1295995
new ActiveMQ version, and some cleanup
laullon Nov 29, 2022
345e8ac
new version number
laullon Nov 29, 2022
5801811
Use SLF4J as logger
laullon Nov 29, 2022
3416b68
bug
laullon Nov 29, 2022
c530bf4
Merge branch 'dev' into dev-proxy2
laullon Dec 5, 2022
f3f6f92
batch limit control for items and bytes (bytes only on logs for now)
laullon Dec 7, 2022
055d9f4
exporter and midBuffer items
laullon Dec 13, 2022
5058f4f
Merge branch 'dev-proxy2' into beta-12.5
laullon Dec 13, 2022
242f809
unit test changes and build with java lib
laullon Dec 14, 2022
3a3b014
Merge branch 'beta-12.5' into dev-proxy2
laullon Dec 14, 2022
5dd6d0a
Update maven_java-lib.yml
laullon Dec 14, 2022
09a99f8
Update maven_java-lib.yml
laullon Dec 14, 2022
cc45182
largemessages
laullon Dec 15, 2022
40e5ebc
Merge branch 'beta-12.5' into dev-proxy2
laullon Dec 22, 2022
66a9cae
Disk gzip
laullon Jan 16, 2023
5524d45
Merge branch 'dev' into dev-proxy2
laullon May 10, 2023
41db73e
code reformat
laullon May 22, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
some todos
  • Loading branch information
laullon committed Sep 20, 2022
commit b806e98de0729e41325e079d807e0d8afd570e43
Original file line number Diff line number Diff line change
Expand Up @@ -193,12 +193,10 @@ public void doSendPoints(String queue, List<String> points) throws ActiveMQAddre
ClientProducer producer = session.createProducer(queue);
return new Pair<>(session, producer);
} catch (Exception e) {
e.printStackTrace();
throw new RuntimeException(e);
}
return null;
});

// TODO: check if session still valid
ClientSession session = mqCtx._1;
ClientProducer producer = mqCtx._2;
try {
Expand All @@ -214,8 +212,10 @@ public void doSendPoints(String queue, List<String> points) throws ActiveMQAddre
} catch (ActiveMQObjectClosedException e) {
log.log(Level.FINE, "connection close: " + e.getMessage());
producers.remove(queue);
sendPoints(queue, points);
laullon marked this conversation as resolved.
Show resolved Hide resolved
} catch (Exception e) {
log.log(Level.SEVERE, "error", e);
throw new RuntimeException(e);
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@ public static List<Buffer> registerNewQueueIfNeedIt(QueueInfo queue) {
buffers.add(external);
}

// TODO: move this to queue/queueInfo creation
registeredQueues.put(queue.getName(), true);
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,17 @@
import com.wavefront.agent.core.queues.QueueStats;
import com.wavefront.common.NamedThreadFactory;
import com.wavefront.common.logger.MessageDedupingLogger;
import org.apache.activemq.artemis.api.core.ActiveMQAddressFullException;
import org.apache.activemq.artemis.api.core.management.QueueControl;
import org.apache.activemq.artemis.core.settings.impl.AddressSettings;

import java.util.ArrayList;
import java.util.Map;
import java.util.concurrent.*;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.apache.activemq.artemis.api.core.ActiveMQAddressFullException;
import org.apache.activemq.artemis.api.core.management.QueueControl;
import org.apache.activemq.artemis.core.settings.impl.AddressSettings;

import static com.wavefront.agent.ProxyContext.entityPropertiesFactoryMap;

public class MemoryBuffer extends ActiveMQBuffer {
private static final Logger log = Logger.getLogger(MemoryBuffer.class.getCanonicalName());
Expand Down Expand Up @@ -78,12 +81,14 @@ public void sendPoint(QueueInfo queue, String strPoint) {

@Override
public void registerNewQueueInfo(QueueInfo queue) {
// TODO: int interval =
// entityPropsFactoryMap.get(tenantName).get(entityType).getPushFlushInterval();

int interval =
entityPropertiesFactoryMap
.get(queue.getTenant())
.get(queue.getEntityType())
.getPushFlushInterval();
super.registerNewQueueInfo(queue);
for (int i = 0; i < queue.getNumberThreads(); i++) {
executor.scheduleAtFixedRate(new sender(queue), 1, 1, TimeUnit.SECONDS);
executor.scheduleAtFixedRate(new sender(queue), interval, interval, TimeUnit.MILLISECONDS);
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ abstract class AbstractDataSubmissionTask<T extends DataSubmissionTask<T>>
* @param queue port/handle
* @param timeProvider time provider (in millis)
*/
// TODO: remove this
AbstractDataSubmissionTask(
EntityProperties properties,
QueueInfo queue,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,10 @@ public class InteractivePreprocessorTester implements InteractiveTester {

private final ReportableEntityHandlerFactory factory =
new ReportableEntityHandlerFactory() {
@SuppressWarnings("unchecked")
@Override
public <T> ReportableEntityHandler<T> getHandler(String handler, QueueInfo queue) {
public ReportableEntityHandler getHandler(String handler, QueueInfo queue) {
if (queue.getEntityType() == ReportableEntityType.TRACE) {
return (ReportableEntityHandler<T>)
new ReportableEntityHandler<Span>() {
return new ReportableEntityHandler<Span>() {
@Override
public void report(Span reportSpan) {
System.out.println(SPAN_SERIALIZER.apply(reportSpan));
Expand Down Expand Up @@ -72,8 +70,7 @@ public void reject(@Nonnull String t, @Nullable String message) {
public void shutdown() {}
};
}
return (ReportableEntityHandler<T>)
new ReportableEntityHandler<ReportPoint>() {
return new ReportableEntityHandler<ReportPoint>() {
@Override
public void report(ReportPoint reportPoint) {
System.out.println(ReportPointSerializer.pointToString(reportPoint));
Expand Down
17 changes: 13 additions & 4 deletions tests/disk-buffer/helm/scripts/test_metrics.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,10 @@ ID=${PROXY_ID:=$(cat "/tmp/id")}

truncate_buffer(){
curl \
--silent -X 'GET' \
"${WAVEFRONT_URL}v2/proxy/${ID}" \
-H 'accept: application/json' \
-X 'PUT' \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer ${WAVEFRONT_TOKEN}" \
"${WAVEFRONT_URL}v2/proxy/${ID}" \
-d '{"shutdown":false ,"truncate":true}'

wait_buffer_have_points 0
Expand All @@ -34,7 +34,7 @@ wait_buffer_have_points(){
DONE=false
while [ !"${DONE}" ]
do
sleep 2
sleep 15
v=$(get_buffer_points)
echo "${v}"
if [ "${v}" -eq "${1}" ]
Expand All @@ -48,5 +48,14 @@ truncate_buffer

METRICNAME_A="test.gh.buffer-disk.${RANDOM}${RANDOM}"
curl http://localhost:2878 -X POST -d "${METRICNAME_A} ${RANDOM} source=github_proxy_action"
curl http://localhost:2878 -X POST -d "${METRICNAME_A} ${RANDOM} source=github_proxy_action"
curl http://localhost:2878 -X POST -d "${METRICNAME_A} ${RANDOM} source=github_proxy_action"
curl http://localhost:2878 -X POST -d "${METRICNAME_A} ${RANDOM} source=github_proxy_action"
curl http://localhost:2878 -X POST -d "${METRICNAME_A} ${RANDOM} source=github_proxy_action"
curl http://localhost:2878 -X POST -d "${METRICNAME_A} ${RANDOM} source=github_proxy_action"
curl http://localhost:2878 -X POST -d "${METRICNAME_A} ${RANDOM} source=github_proxy_action"
curl http://localhost:2878 -X POST -d "${METRICNAME_A} ${RANDOM} source=github_proxy_action"
curl http://localhost:2878 -X POST -d "${METRICNAME_A} ${RANDOM} source=github_proxy_action"
curl http://localhost:2878 -X POST -d "${METRICNAME_A} ${RANDOM} source=github_proxy_action"

wait_buffer_have_points 10