Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

URlhaus Malware Patrol LinuxMalwareDetec InterServer, never updated #398

Open
ghost opened this issue Jul 24, 2021 · 7 comments
Open

URlhaus Malware Patrol LinuxMalwareDetec InterServer, never updated #398

ghost opened this issue Jul 24, 2021 · 7 comments

Comments

@ghost
Copy link

ghost commented Jul 24, 2021

--------------------- ClamUnofficial-update Begin ------------------------

jul 23 12:00:02 Preparing Databases
jul 23 12:00:03 Removing unused file: /var/lib/clamav/spam_marketing.ndb
jul 23 12:00:04 vie jul 23 12:00:04 CEST 2021 - Pausing database file updates for 294 seconds...
jul 23 12:04:59 vie jul 23 12:04:59 CEST 2021 - Pause complete, checking for new database files...
jul 23 12:04:59 Sanesecurity Database & GPG Signature File Updates
jul 23 12:04:59 Checking for Sanesecurity updates...
jul 23 12:05:01 Sanesecurity mirror site used: jessie.fonant.com 95.217.37.104
jul 23 12:05:02 Testing updated Sanesecurity database file: blurl.ndb
jul 23 12:05:02 Clamscan reports Sanesecurity blurl.ndb database integrity tested good
jul 23 12:05:03 Successfully updated Sanesecurity production database file: blurl.ndb
jul 23 12:05:03 Testing updated Sanesecurity database file: junk.ndb
jul 23 12:05:04 Clamscan reports Sanesecurity junk.ndb database integrity tested good
jul 23 12:05:04 Successfully updated Sanesecurity production database file: junk.ndb
jul 23 12:05:04 Testing updated Sanesecurity database file: jurlbl.ndb
jul 23 12:05:04 Clamscan reports Sanesecurity jurlbl.ndb database integrity tested good
jul 23 12:05:04 Successfully updated Sanesecurity production database file: jurlbl.ndb
jul 23 12:05:04 Testing updated Sanesecurity database file: rogue.hdb
jul 23 12:05:04 Clamscan reports Sanesecurity rogue.hdb database integrity tested good
jul 23 12:05:04 Successfully updated Sanesecurity production database file: rogue.hdb
jul 23 12:05:05 Testing updated Sanesecurity database file: jurlbla.ndb
jul 23 12:05:05 Clamscan reports Sanesecurity jurlbla.ndb database integrity tested good
jul 23 12:05:05 Successfully updated Sanesecurity production database file: jurlbla.ndb
jul 23 12:05:05 Testing updated Sanesecurity database file: phishtank.ndb
jul 23 12:05:05 Clamscan reports Sanesecurity phishtank.ndb database integrity tested good
jul 23 12:05:05 Successfully updated Sanesecurity production database file: phishtank.ndb
jul 23 12:05:05 Testing updated Sanesecurity database file: porcupine.hsb
jul 23 12:05:05 Clamscan reports Sanesecurity porcupine.hsb database integrity tested good
jul 23 12:05:05 Successfully updated Sanesecurity production database file: porcupine.hsb
jul 23 12:05:05 Testing updated Sanesecurity database file: porcupine.ndb
jul 23 12:05:06 Clamscan reports Sanesecurity porcupine.ndb database integrity tested good
jul 23 12:05:06 Successfully updated Sanesecurity production database file: porcupine.ndb
jul 23 12:05:06 LinuxMalwareDetect Database File Updates
jul 23 12:05:06 Checking for LinuxMalwareDetect updates...
jul 23 12:05:08 No LinuxMalwareDetect database file updates
jul 23 12:05:08 interserver Database File Updates
jul 23 12:05:08 Checking for interserver updates...
jul 23 12:05:08 Checking for updated interServer database file: whitelist.fp
jul 23 12:05:11 No updated interServer whitelist.fp database file
jul 23 12:05:11 Checking for updated interServer database file: interserver256.hdb
jul 23 12:05:17 No updated interServer interserver256.hdb database file
jul 23 12:05:17 Checking for updated interServer database file: interservertopline.db
jul 23 12:05:24 No updated interServer interservertopline.db database file
jul 23 12:05:24 No interServer database file updates
jul 23 12:05:24 Removing disabled Malware Expert Database files
jul 23 12:05:24 MalwarePatrol Database File Updates
jul 23 12:05:24 Checking for MalwarePatrol updates...
jul 23 12:05:24 Checking for updated MalwarePatrol database file: malwarepatrol.db
jul 23 12:05:31 No updated MalwarePatrol malwarepatrol.db database file
jul 23 12:05:31 No MalwarePatrol database file updates
jul 23 12:05:31 URLhaus Database File Updates
jul 23 12:05:31 Checking for urlhaus updates...
jul 23 12:05:31 Checking for updated urlhaus database file:
jul 23 12:05:38 WARNING: Failed connection to https://urlhaus.abuse.ch/downloads - SKIPPED urlhaus update
jul 23 12:05:38 No updated urlhaus database file
jul 23 12:05:38 No urlhaus database file updates
jul 23 12:05:38 Removing disabled yararulesproject Database files
jul 23 12:05:39 Update(s) detected, reloading ClamAV databases
jul 23 12:05:39 ClamAV databases reloading
jul 23 12:05:39 Issue tracker : https://github.com/extremeshok/clamav-unofficial-sigs/issues
jul 23 12:05:51 Powered By https://eXtremeSHOK.com

---------------------- ClamUnofficial-update End -------------------------

URlhaus not is: https://urlhaus.abuse.ch/downloads; Is: https://urlhaus.abuse.ch/downloads/urlhaus.ndb

@perplexityjeff
Copy link
Contributor

Hi @cotelo, could you please specify what version you are running?

@ghost
Copy link
Author

ghost commented Aug 1, 2021

Hello.

ClamUnofficial 7.2.5
config_version="97"
Ubuntu 16.04

Thank you very much.

@perplexityjeff
Copy link
Contributor

perplexityjeff commented Aug 2, 2021

Hi @cotelo, the error just displays the URL host part of the master config not the full URL. The full URL the script tries is the URL you mentioned.

Could you try and wget or curl 'https://urlhaus.abuse.ch/downloads/urlhaus.ndb' to a temporary place to check if you are able to download the file without the use of the script? There is a good chance that there is a connection issue that is not coming from the script.

Also could you run the update script with --force.

Let me know how it all goes.

Thank you

@jengels
Copy link

jengels commented Oct 8, 2021

Hi @perplexityjeff, I am also experiencing this problem on:

  • CentOS 7 + "clamav-unofficial-sigs-7.2.4-1.el7.noarch" + config_version="97"

and on:

  • CentOS 7 + latest version of "clamav-unofficial-sigs.sh" (master branch) + config_version="97"

I've checked that the urlhaus database is accessible and can be successfully downloaded:
wget "https://urlhaus.abuse.ch/downloads/urlhaus.ndb"
--2021-10-08 13:10:42-- https://urlhaus.abuse.ch/downloads/urlhaus.ndb
Resolving urlhaus.abuse.ch (urlhaus.abuse.ch)... 151.101.114.49
Connecting to urlhaus.abuse.ch (urlhaus.abuse.ch)|151.101.114.49|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1258385 (1.2M)
Saving to: ‘urlhaus.ndb’
...
2021-10-08 13:10:42 (12.8 MB/s) - ‘urlhaus.ndb’ saved [1258385/1258385]

But my log looks like:
grep WARNING /var/log/clamav-unofficial-sigs/clamav-unofficial-sigs.log | tail -n5
Oct 08 11:19:57 WARNING: Failed connection to https://urlhaus.abuse.ch/downloads - SKIPPED urlhaus urlhaus.ndb update
Oct 08 12:31:57 WARNING: Failed connection to https://urlhaus.abuse.ch/downloads - SKIPPED urlhaus urlhaus.ndb update
Oct 08 12:38:44 WARNING: Failed connection to https://urlhaus.abuse.ch/downloads - SKIPPED urlhaus urlhaus.ndb update
Oct 08 12:48:14 WARNING: Failed connection to https://urlhaus.abuse.ch/downloads - SKIPPED urlhaus urlhaus.ndb update
Oct 08 12:53:38 WARNING: Failed connection to https://urlhaus.abuse.ch/downloads - SKIPPED urlhaus urlhaus.ndb update

I think I've also found a possible(?) bug (variable "$work_dir_urlhaust" instead of "$work_dir_urlhaus") in the script "clamav-unofficial-sigs.sh":

grep -n work_dir_urlhaust clamav-unofficial-sigs.sh
1923:if [ -z "$work_dir_urlhaust" ] ; then

But even after correcting this "possible bug" the script still fails to download the urlhaus.ndb (even after removing everything inside the caching directory):
rm -rf /var/lib/clamav-unofficial-sigs/*

In the production environment I'm still running an older version: CentOS 7 + clamav-unofficial-sigs-7.0.1-5.el7.noarch + config_version="91". There everything seems to be OK. The Database exists and gets updated as expected:

ls -l /var/lib/clamav/urlhaus.ndb
-rw-r--r-- 1 clamupdate clamupdate 1252317 Oct 8 13:15 /var/lib/clamav/urlhaus.ndb

grep urlhaus /var/log/clamav-unofficial-sigs/clamav-unofficial-sigs.log | tail -n10
Oct 08 12:15:18 Checking for urlhaus updates...
Oct 08 12:15:18 Checking for updated urlhaus database file: urlhaus.ndb
Oct 08 12:15:18 Testing updated urlhaus database file: urlhaus.ndb
Oct 08 12:15:18 Clamscan reports urlhaus urlhaus.ndb database integrity tested good
Oct 08 12:15:18 Successfully updated urlhaus production database file: urlhaus.ndb
Oct 08 13:16:41 Checking for urlhaus updates...
Oct 08 13:16:41 Checking for updated urlhaus database file: urlhaus.ndb
Oct 08 13:16:41 Testing updated urlhaus database file: urlhaus.ndb
Oct 08 13:16:41 Clamscan reports urlhaus urlhaus.ndb database integrity tested good
Oct 08 13:16:41 Successfully updated urlhaus production database file: urlhaus.ndb

So this seems to be a problem which only affects the newer version(s)?

@perplexityjeff
Copy link
Contributor

perplexityjeff commented Oct 26, 2021

@jengels In previous versions there was support for urlhaus but because of a typo it was not used. I don't know when that was introduced, but I had the same issues as you guys.

See the issue here
#385, #386

I attempted that to fix here
#390

Related
#400

Currently it is merged into 'dev' and the original developer still needs to give the 'go' and push the fix as an actual update.

I hope that if you change the script that you are able to fix it at least until an official update is available.

I am myself currently looking into https://github.com/rseichter/fangfrisch for our production environment instead of this script because sadly it takes a while for these bugs to get fixed. I have full respect for the original developer of this script and understand that it is not his full time priority but it does at least for us users take some time for these bugs to get fixed.

@jengels
Copy link

jengels commented Oct 26, 2021

@perplexityjeff, thanks for the feedback. The problem got fixed by applying patches #390 and #386 locally. Hopefully the patches will be included in the next release...

@perplexityjeff
Copy link
Contributor

@jengels No problem at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants