Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Collector can't write to influxdb #14

Open
drolfe opened this issue Oct 7, 2021 · 9 comments
Open

Collector can't write to influxdb #14

drolfe opened this issue Oct 7, 2021 · 9 comments
Assignees
@javadmohebbi
Labels
bug Something isn't working

Comments

@drolfe
Copy link

drolfe commented Oct 7, 2021

Hi, I seem to have the below issue

Logs from collector:

2021/10/07 00:31:19 influxdb2client E! Write error: Post "http://127.0.0.1:8086/api/v2/write?bucket=nfCollector&org=OPENINTELLIGENCE&precision=ns": dial tcp 127.0.0.1:8086: connect: connection refused Batch kept for retrying 2021/10/07 00:31:24 influxdb2client E! Write error: Post "http://127.0.0.1:8086/api/v2/write?bucket=nfCollector&org=OPENINTELLIGENCE&precision=ns": dial tcp 127.0.0.1:8086: connect: connection refused Batch kept for retrying 2021/10/07 00:31:29 influxdb2client E! Write error: Post "http://127.0.0.1:8086/api/v2/write?bucket=nfCollector&org=OPENINTELLIGENCE&precision=ns": dial tcp 127.0.0.1:8086: connect: connection refused Batch kept for retrying 2021/10/07 00:31:34 influxdb2client E! Write error: Post "http://127.0.0.1:8086/api/v2/write?bucket=nfCollector&org=OPENINTELLIGENCE&precision=ns": dial tcp 127.0.0.1:8086: connect: connection refused Batch kept for retrying 2021/10/07 00:31:39 influxdb2client E! Write error: Post "http://127.0.0.1:8086/api/v2/write?bucket=nfCollector&org=OPENINTELLIGENCE&precision=ns": dial tcp 127.0.0.1:8086: connect: connection refused Batch kept for retrying 2021/10/07 00:31:44 influxdb2client E! Write error: Post "http://127.0.0.1:8086/api/v2/write?bucket=nfCollector&org=OPENINTELLIGENCE&precision=ns": dial tcp 127.0.0.1:8086: connect: connection refused Batch kept for retrying 2021/10/07 00:31:49 influxdb2client E! Write error: Post "http://127.0.0.1:8086/api/v2/write?bucket=nfCollector&org=OPENINTELLIGENCE&precision=ns": dial tcp 127.0.0.1:8086: connect: connection refused Batch kept for retrying ^C

Logs from influxdb:

ts=2021-10-07T00:31:30.505617Z lvl=info msg=Starting log_id=0X22RUUl000 service=telemetry interval=8h ts=2021-10-07T00:31:30.505654Z lvl=info msg=Listening log_id=0X22RUUl000 service=tcp-listener transport=http addr=:8086 port=8086 ts=2021-10-07T00:31:59.158305Z lvl=info msg=Unauthorized log_id=0X22RUUl000 error="authorization not found" ts=2021-10-07T00:31:59.162553Z lvl=info msg=Unauthorized log_id=0X22RUUl000 error="authorization not found" ts=2021-10-07T00:31:59.229866Z lvl=info msg=Unauthorized log_id=0X22RUUl000 error="authorization not found" ts=2021-10-07T00:31:59.230787Z lvl=info msg=Unauthorized log_id=0X22RUUl000 error="authorization not found" ts=2021-10-07T00:31:59.231473Z lvl=info msg=Unauthorized log_id=0X22RUUl000 error="authorization not found" ts=2021-10-07T00:31:59.243734Z lvl=info msg=Unauthorized log_id=0X22RUUl000 error="authorization not found" ts=2021-10-07T00:31:59.256626Z lvl=info msg=Unauthorized log_id=0X22RUUl000 error="authorization not found" ts=2021-10-07T00:31:59.259733Z lvl=info msg=Unauthorized log_id=0X22RUUl000 error="authorization not found" ts=2021-10-07T00:31:59.261795Z lvl=info msg=Unauthorized log_id=0X22RUUl000 error="authorization not found" ts=2021-10-07T00:31:59.263596Z lvl=info msg=Unauthorized log_id=0X22RUUl000 error="authorization not found"

Any Help would be great

  • Ubuntu21.04
  • Nothing has been changed from the defaults
  • Netflow V5 is being received by the collector

Regards, Daniel
@ghost
Copy link

ghost commented Oct 14, 2021

This rather looks like the token required for influxdb2 is either missing or incorrect, or the organisation is in error.
error="authorization not found" suggests it is the first one I mentioned.

@MeneerHeijpaal
Copy link

I have exactly the same problem with a clean install from this repository. Could it be that the token is incorrect?

@ghost
Copy link

ghost commented Oct 18, 2021

I have been migrating to influxdb2, and everytime I get an authorizatiion error its the token, or the organisation set incorrectly.
Once rectified, all smiles :)

@ghost ghost mentioned this issue Oct 18, 2021
Closed
@javadmohebbi
Copy link
Owner

Hi all,

Thank you for the feedback. It is confirmed that the reported issue has been seen on many devices. I will update you in on hour.

@javadmohebbi
Copy link
Owner

@drolfe @aimdev @NVQXE23I
Hi again,

The issue has been resolved.
To fix it do the following steps:

rm $HOME/oi24 -Rfv
wget https://download.openintelligence24.com/latest.sh
chmod +x latest.sh
./latest.sh

Let script download the rest of things and the solution should be worked now. Please read this guide for more information about installation.

Sorry for this issue. Hope you enjoy the solution.

@javadmohebbi javadmohebbi added the bug Something isn't working label Oct 19, 2021
@javadmohebbi javadmohebbi self-assigned this Oct 19, 2021
@izaerax
Copy link

izaerax commented Oct 26, 2021

Hi everybody,

I have some issue writing data on influxdb.

There are the logs from the collector:
2021/10/26 10:07:49 influxdb2client E! Write error: invalid: unable to parse 'detail,device=172.18.0.1,proto=UDP,sASN=NA,shost=,sport=5353/UDP,scountryLong=Invalid_IP_address.,scountryShort=Invalid_IP_address.,sregion=Invalid_IP_address.,scity=Invalid_IP_address.,dASN=NA,dhost=,dport=5353/UDP,dcountryLong=Invalid_IP_address.,dcountryShort=Invalid_IP_address.,dregion=Invalid_IP_address.,dcity=Invalid_IP_address. bytes=2645u,packets=27u 1635242869897724652': missing tag value

unable to parse 'detail,device=172.18.0.1,proto=UDP,sASN=NA,shost=,sport=5353/UDP,scountryLong=Invalid_IP_address.,scountryShort=Invalid_IP_address.,sregion=Invalid_IP_address.,scity=Invalid_IP_address.,dASN=NA,dhost=,dport=5353/UDP,dcountryLong=Invalid_IP_address.,dcountryShort=Invalid_IP_address.,dregion=Invalid_IP_address.,dcity=Invalid_IP_address. bytes=2163u,packets=22u 1635242869898129052': missing tag value

2021/10/26 10:07:58 influxdb2client E! Write error: invalid: unable to parse 'srcHost,device=172.18.0.1,ASN=NA,host=,countryLong=Invalid_IP_address.,countryShort=Invalid_IP_address.,region=Invalid_IP_address.,city=Invalid_IP_address. bytes=2645u,packets=27u 1635242878877495352': missing tag value

unable to parse 'srcDnsLookup,device=172.18.0.1,host=,domain=NA-,countryLong=Invalid_IP_address.,countryShort=Invalid_IP_address.,region=Invalid_IP_address.,city=Invalid_IP_address. bytes=2645u,packets=27u 1635242878877495352': missing tag value

unable to parse 'srcHost,device=172.18.0.1,ASN=NA,host=,countryLong=Invalid_IP_address.,countryShort=Invalid_IP_address.,region=Invalid_IP_address.,city=Invalid_IP_address. bytes=2163u,packets=22u 1635242878878464552': missing tag value

unable to parse 'srcDnsLookup,device=172.18.0.1,host=,domain=NA-,countryLong=Invalid_IP_address.,countryShort=Invalid_IP_address.,region=Invalid_IP_address.,city=Invalid_IP_address. bytes=2163u,packets=22u 1635242878878464552': missing tag value

2021/10/26 10:08:00 influxdb2client E! Write error: invalid: unable to parse 'dstHost,device=172.18.0.1,ASN=NA,host=,countryLong=Invalid_IP_address.,countryShort=Invalid_IP_address.,region=Invalid_IP_address.,city=Invalid_IP_address. bytes=2645u,packets=27u 1635242880390892552': missing tag value

unable to parse 'dstDnsLookup,device=172.18.0.1,host=,domain=NA-,countryLong=Invalid_IP_address.,countryShort=Invalid_IP_address.,region=Invalid_IP_address.,city=Invalid_IP_address. bytes=2645u,packets=27u 1635242880390892552': missing tag value

unable to parse 'dstHost,device=172.18.0.1,ASN=NA,host=,countryLong=Invalid_IP_address.,countryShort=Invalid_IP_address.,region=Invalid_IP_address.,city=Invalid_IP_address. bytes=2163u,packets=22u 1635242880391850052': missing tag value

I generate the flows with nProbe (https://github.com/ntop/nProbe) using the command below:
nprobe -n localhost:6859 -V9 -i eth0 -b 1 -t 5

I'm running nprobe from Windows WSL on ubuntu 18.04

Thanks

@javadmohebbi
Copy link
Owner

javadmohebbi commented Oct 26, 2021 •
edited
Loading

Hi @izaerax

Please share the following information for further investigation:

ls -la $HOME/oi24/nfcollector/vendors/ip2location/db
cat $HOME/oi24/nfcollector/docker-compose.yml
cat $HOME/oi24/nfcollector/etc/ip2location.yml

Also to check our solution can decode the flow traffic. please run the following command for gathering some logs. This command will run a special version of nfcollector which logs decoded netflow packets to a log file /tmp/nfcollector-dump.log

cd $HOME/oi24/nfcollector
docker-compose down
wget -O /tmp/nfcollector-logger https://github.com/javadmohebbi/goNfCollector/raw/main/build/linux/amd64/nfcollector-logger
chmod +x /tmp/nfcollector-logger
sudo /tmp/nfcollector-logger

Wait for some minute and to stop this app, press CTRL + C
After that, log file will be /tmp/nfcollector-dump.log.
*** This file might include some confidential info about IPs. Due to security reason, you can find & replace your confidential information***

Please send me back the /tmp/nfcollector-dump.log file.

@izaerax
Copy link

izaerax commented Oct 26, 2021

thanks for your reply @javadmohebbi

those are the results:

ls -la $HOME/oi24/nfcollector/vendors/ip2location/db:
drwxr-xr-x 5 root root 4096 Oct 26 11:57 .
drwxr-xr-x 4 root root 4096 Oct 26 11:57 ..
drwxr-xr-x 2 root root 4096 Oct 26 11:57 IP2LOCATION-LITE-ASN.IPV6.CSV
drwxr-xr-x 2 root root 4096 Oct 26 11:57 IP2LOCATION-LITE-DB11.IPV6.BIN
drwxr-xr-x 2 root root 4096 Oct 26 11:57 IP2PROXY-LITE-PX10.IPV6.CSV

cat $HOME/oi24/nfcollector/docker-compose.yml:

version: "3"
    
services:
    
        # Influx DB
        influxdb:
    
            image: influxdb:2.0.7
    
            restart: always
    
            volumes:
                - /root/oi24/nfcollector/vendors/influxdb:/var/lib/influxdb2
    
            ports:
                - "8086:8086"
                - "8082:8082"
                - "8089:8089"
    
            networks:
                - tick-graf
    
            restart: on-failure
    
        # Grafna
        grafana:
    
            image: grafana/grafana
    
            restart: always
    
            volumes:
                - /root/oi24/nfcollector/vendors/grafana:/var/lib/grafana
    
            ports:
                - "3000:3000"
    
            networks:
                - tick-graf
    
    
        nfcolector:
    
            image: javadmohebbi/gonfcollector
    
            restart: always
    
            # network_mode: host
    
            ports:
                - "6859:6859/udp"
    
            volumes:
                - /root/oi24/nfcollector/bin:/opt/nfcollector/bin
                # - /root/oi24/nfcollector/etc:/opt/nfcollector/etc
                - /root/oi24/nfcollector/var:/opt/nfcollector/var
                - /root/oi24/nfcollector/vendors:/opt/nfcollector/vendors
    
    
            networks:
                - tick-graf
    
            environment:
               - NFC_DEBUG=true
               # total number of cpu nfcollector could use
               - NFC_CPU_NUM=0
               - NFC_LISTEN_ADDRESS=0.0.0.0
               - NFC_LISTEN_PORT=6859
               - NFC_INFLUXDB_HOST=influxdb
               - NFC_INFLUXDB_PORT=8086
               - NFC_INFLUXDB_TOKEN=5vqt0q0b4g_lZwNgp7-8GgPq5Nxf3YY37xbVZP_ypeK_G3dwdNlTrAkcKN_Q6QzbmG-Th96lT_65Kp0j2UD1HA==
               - NFC_INFLUXDB_BUCKET=nfCollector
               - NFC_INFLUXDB_ORG=OPENINTELLIGENCE
               - NFC_IP_REPTATION_IPSUM=/opt/nfcollector/vendors/ipsum/ipsum.txt
               - NFC_IP2L_ASN=/opt/nfcollector/vendors/ip2location/db/IP2LOCATION-LITE-ASN.IPV6.CSV/IP2LOCATION-LITE-ASN.IPV6.CSV
               - NFC_IP2L_IP=/opt/nfcollector/vendors/ip2location/db/IP2LOCATION-LITE-DB11.IPV6.BIN/IP2LOCATION-LITE-DB11.IPV6.BIN
               - NFC_IP2L_PROXY=/opt/nfcollector/vendors/ip2location/db/IP2PROXY-LITE-PX10.IPV6.CSV/IP2PROXY-LITE-PX10.IPV6.CSV
               - NFC_IP2L_LOCAL=/opt/nfcollector/vendors/ip2location/local-db/local.csv
    
networks:
    tick-graf:
        external: true

cat $HOME/oi24/nfcollector/etc/ip2location.yml:
# IP2Location configuration

# ASN database (for future usage)
asn: /root/oi24/nfcollector/vendors/ip2location/db/IP2LOCATION-LITE-ASN.IPV6.CSV/IP2LOCATION-LITE-ASN.IPV6.CSV

# IP Location database (ONLY *.BIN is supported)
ip: /root/oi24/nfcollector/vendors/ip2location/db/IP2LOCATION-LITE-DB11.IPV6.BIN/IP2LOCATION-LITE-DB11.IPV6.BIN

# Proxy database (for future usage)
proxy: /root/oi24/nfcollector/vendors/ip2location/db/IP2PROXY-LITE-PX10.IPV6.CSV/IP2PROXY-LITE-PX10.IPV6.CSV

# LOCAL CSV DB
local: /root/oi24/nfcollector/vendors/ip2location/local-db/local.csv

/tmp/nfcollector-dump.log:
root@DESKTOP-ODCU1S8:~/oi24/nfcollector# sudo /tmp/nfcollector-logger
2021/10/26 12:58:54 Reading config from "/opt/oi24/netflow-collector/etc/collector.yml"
2021/10/26 12:58:54 Configuration /opt/oi24/netflow-collector/etc/.collector not found in path yml

I launched the wget -O /tmp/nfcollector-logger https://github.com/javadmohebbi/goNfCollector/raw/main/build/linux/amd64/nfcollector
instead of
wget -O /tmp/nfcollector-logger https://github.com/javadmohebbi/goNfCollector/raw/main/build/linux/amd64/nfcollector-logger

@javadmohebbi
Copy link
Owner

javadmohebbi commented Oct 27, 2021 •
edited
Loading

Hi @izaerax

I have just published the new version of the solution right now v0.0.1-30b which might solve your problem. I have already tested it with nprobe using nprobe docker image docker run -it -v $(pwd)/nprobe.license:/etc/nprobe.license --net=host ntop/nprobe:stable -i wlp3s0 -n 192.168.1.102:6859 -V9 our solution could collect, store and visualize flow traffic without any problem.

Please update your docker pull javadmohebbi/gonfcollector:latest to get the latest version or to run the solution again from scratch, do the following:

rm -Rfv $HOME/oi24
wget https://download.openintelligence24.com/latest.sh
chmod +x latest.sh -v 
./latest

When the latest.shcompleted its job, run these commands to run the solution.

cd $HOME/oi24/nfcollector
docker-compose up -d

@javadmohebbi javadmohebbi linked a pull request Oct 27, 2021 that will close this issue
Fix the influxDB configuration #1
Merged
@javadmohebbi javadmohebbi removed a link to a pull request Oct 27, 2021
Fix the influxDB configuration #1
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@javadmohebbi javadmohebbi

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@drolfe @javadmohebbi @izaerax @MeneerHeijpaal