AIL Framework version 3.1 with new crawling with cookies, Telegram username correlation, new external feeders (e.g. Twitter) and many improvements

AIL framework v3.1 released

AIL framework v3.1 has been released including many new features such as:

• Crawling website with a set of cookies collected from a browser (allowing to crawl website password protected or alike)
• An extraction module to find Telegram users
• New correlation engine for username starting with Telegram
• Improve timeout on various modules
• New JSON importer to be used with new external feeder. A first Twitter feeder and url-feeder is available to feed specific tweets and discovered urls in AIL.

The AIL project moved into a dedicated project to allow new contributions and project within the AIL project scope. If you want to join us and contribute new modules or specific feeders, don't hesitate to contact us.

Complete changelog

• [update] add update v3.1 + install crawler python requirements by
  default. [Terrtia]
• [UI correlation] add username correlation card. [Terrtia]
• [UI correlation] add username correlation graph. [Terrtia]
• [UI correlation] add username correlation: daterange pages. [Terrtia]
• [telegram module] add new tag: telegram invite code. [Terrtia]
• [telegram backend] add username correlation + save invite hash.
  [Terrtia]
• [importer] add map twitter id - item id, add parents link between
  twitter and url extracted. [Terrtia]
• [travis install] shallow clone. fetch tags + commit id. [Terrtia]
• [travis install] shallow clone. fetch tags. [Terrtia]
• [install] debug travis. [Terrtia]
• [README] update gitter link. [Terrtia]
• [import er url_extract] add item twitter parent. [Terrtia]
• [importer url_extract] fix item id. [Terrtia]
• [importer] add url_extract importer. [Terrtia]
• [API json import] take list as imput. [Terrtia]
• [core import] add AIL JSON import format, API endpoint added (AIL
  feeders) [Terrtia]
• [update doc] update doc install + logo + fix updater. [Terrtia]
• [Updater] change default branch origin. [Terrtia]
• [Updater] change default branch origin. [Terrtia]
• [update Readme logo + links] [Terrtia]
• [add new logo] [Terrtia]
• [AIL logo UI] [Terrtia]
• [doc] add cookiejar screenshot. [Terrtia]
• [Crawler] default docker memory usage. [Terrtia]
• [MISP export] export domain as domain-crawled object. [Terrtia]
• [domain explorer UI] add shortcut button: misp export + correlation.
  [Terrtia]
• [crawler] edit cookie and cookiejar + add cookie to cookiejar + fix
  screenshot duplicate. [Terrtia]
• [Crawler core + UI] crawler lua: handle retry + fix cookie loader and
  selector. [Terrtia]
• [cookiejar UI] add cookiejar + show all. [Terrtia]
• [crawler cookies] use cookiejar. [Terrtia]
• [crawler - cookies] add/show/select cookies. [Terrtia]
• [crawler] add cookies list by user/global, save cookies from file +
  dict(name, value), TODO: API + handle errors. [Terrtia]
• [crawler] bypass login: use cookie provided by user and accept cookie
  from server + refractor. [Terrtia]
• [Splash Crawler] use cookies to bypass login. [Terrtia]
• [UI] bump jquery to 3.4.1. [Terrtia]
• [import_dir] fix is_gzip test, use magic number. [Terrtia]
• [MISP Importer] files: handle missing sha1/sha256 attributes + fix
  Items and Screenshots dir. [Terrtia]
• [Updater] relauch updater on change (git pull) [Terrtia]

Fixes

• [Mail module] replace signal by multiprocessing
  (https://docs.python.org/3.4/library/signal.html#execution-of-python-
  signal-handlers) [Terrtia]
• [Mail] debug signal. [Terrtia]
• [Mail] debug. [Terrtia]
• [Mail module] remove test time.wait. [Terrtia]
• [Mail module] debug signal timeout. [Terrtia]
• [Mail module] debug signal timeout. [Terrtia]
• [telegram module] typo. [Terrtia]
• [telegram module] fix tagging. [Terrtia]
• [telegram module] check username length. [Terrtia]
• [telegram module] remove debug. [Terrtia]
• [Mails] regex timeout. [Terrtia]
• [Mails] change module output. [Terrtia]
• [Mails] remove print + test. [Terrtia]
• [Mails] refactor Mail module. [Terrtia]
• [urlextract importer] fix parent map + replace set JSON queue by list.
  [Terrtia]
• [import urlextract parent] fix typo. [Terrtia]
• [travis install] shallow clone. fetch missing commit id. [Terrtia]
• [Mails regex timeout] reduce default timeout. [Terrtia]
• [Mails dns resolver] update timeout exception. [Terrtia]
• [Mails] typo. [Terrtia]
• [Mails] import. [Terrtia]
• [Mails] import. [Terrtia]
• [Mails] add regex timeout. [Terrtia]
• [Credential] add regex timeout. [Terrtia]
• [Credential] add regex timeout. [Terrtia]
• [urlexport importer] fix item name + redis config. [Terrtia]
• [urlexport importer] item_id, force str type. [Terrtia]
• [urlexport importer] add missing import. [Terrtia]
• [urlextract importer] class name. [Terrtia]
• [importer] fix typo. [Terrtia]
• [Global] extend ungzip error catching, catch invalid compressed file.
  [Terrtia]
• [json import API] remove list input. [Terrtia]
• [install create default user] default passwd file: add missing new
  line. [Terrtia]
• [installer] get last git version. [Terrtia]
• [Readme travis badge] [Terrtia]
• [Readme travis badge] [Terrtia]
• [Readme logo size] [Terrtia]
• [show item UI] return 404. [Terrtia]
• [Cookiejar UI] fix typo. [Terrtia]
• [Flask session cookie name] add uuid to cookie name. [Terrtia]
• [crawler] typo. [Terrtia]
• [Crawler splash ResponseNeverReceived] add retry. [Terrtia]
• [crawler] error catcher. [Terrtia]
• [MISP export UI] fix input: name + value overwrite. [Terrtia]
• [update thirdparty] update taxonomies. [Terrtia]
• [crawler] cleanup. [Terrtia]
• [PgpDump parser] remove header comment (rfc4880) + remove empty lines.
  [Terrtia]
• [Pgp Dump] remove tool version. [Terrtia]
• [ZMQ Feeder] performance: replace zmq recv NOBLOCK by Poller.
  [Terrtia]
• [pgpdump] fix subtype save. [Terrtia]
• [Updater] force updater update. [Terrtia]
• [Updater] fix current_tag parser. [Terrtia]
• [import_dir] remove special characters. [Terrtia]
• [import_dir] remove dir whitespaces #475. [Terrtia]

Other

• Merge pull request #492 from sunil3590/master. [Thirion Aurélien]

  crawler_time -> crawler_delta

• Crawler_time -> crawler_delta. [Sunil D S]

• Chg [telegram + correlation] new module: telegram (username + login
  code + join_chat) + add simple_correlation backend. [Terrtia]

• Chg [telegram + correlation] new module: telegram (username + login
  code + join_chat) + add simple_correlation backend. [Terrtia]

• Merge pull request #487 from CIRCL/crawler_v2. [Thirion Aurélien]

  fix: [crawler] error catcher

• Merge pull request #486 from CIRCL/crawler_v2. [Thirion Aurélien]

  Crawler v2 - Add cookiejar - use cookie to bypass login form

• Merge branch 'master' into crawler_v2. [Terrtia]

AIL Framework version 3.0 with full MISP format export/import and small improvements

AIL Framework version 3.0 with full MISP format export and import has been released. AIL users can now export a set of selected items as a MISP event including objects (items, decoded, screenshot, pgp...), correlations and metadata. There is an import to allow analyst to keep a specific set of analysis in AIL for moving across AIL instances. This release also includes support for authentication of the SMTP server (thanks to Mike Peters for the contribution). Multiple bugs fixed and small improvements.

AIL Framework version 2.9 released with a critical security fix (CVE-2020-8545) and minor bugs fixed

AIL Framework version 2.9 released with a critical security fix (CVE-2020-8545) and minor bugs fixed

This release of AIL includes a major security fix for CVE-2020-8545. The security vulnerability was in the handler of the global feed which could allow malicious feed providers to overwrite and potentially, execute Python code in the environment. This release also includes various bugs fixed. We urge users to upgrade as soon as possible.

Changes

• [domain explorer] domains explorer v2, filter domains by daterange.
  [Terrtia]

Fix

• [IPAddress] catch empty config error. [Terrtia]
• [Global: already saved filename] save updated + filter duplicated
  items. [Terrtia]
• [Global: filename provided by all feeders] avoid path tranversal.
  [Terrtia]
• [Domain explorer UI] fix daterange pagination links. [Terrtia]
• [Tag core] check if item_date type is an integer. [Terrtia]

AIL Framework version 2.8 released with a domain screenshot browser and many bugs fixed

AIL Framework version 2.8 released with a domain screenshot browser and many bugs fixed.

There is also a "Practical Darkweb and criminal Blockchain monitoring using AIL (Framework for Analysis of Information Leaks) - free Training/Workshop" hosted by CIRCL (Luxembourg) which will take place the 20th Feb 2020. Registration link.

Next release

In the next release of AIL, only Python 3.6 and upper will be supported.

Changes

• [domain core + UI] add domain explorer v1. [Terrtia]
• [test api] increase import timeout. [Terrtia]
• [UI + core] tag decoded items, fix: #455. [Terrtia]
• [UI item tags search] refractor: search item tag by object, use new
  functions. [Terrtia]
• [UI tags] add + delete image (screenshot) tags. [Terrtia]
• [UI tags] add + delete image (screenshot) tags. [Terrtia]

Fix

• [MISP export] force pymisp version. [Terrtia]
• [tag core] typo. [Terrtia]
• [tag core] item date type. [Terrtia]
• [Tag core] add tag, update tag last seen. [Terrtia]
• [Flask server + cookie session] chg default cookie name (also use port
  number) + add Flask port number to config. [Terrtia]
• [expand btc adress] filter empty addr fields. [Terrtia]
• [UI navbar] chg icon decoded tags. [Terrtia]
• [UI show item min] fix empty modal. [Terrtia]
• Change name popper.js-1.14.3 to popper-core-1.14.3. [mangelft]
• [UI] screenshot url. [Terrtia]

Other

• Merge pull request #453 from mangelft/master. [Thirion Aurélien]

  fix: change name popper.js-1.14.3 to popper-core-1.14.3

• Merge pull request #450 from CIRCL/tags_v2. [Thirion Aurélien]

  Tags v2

AIL Framework version 2.7 released with an improved tagging system

AIL Framework version 2.7 released with an improved tagging system

AIL Framework version 2.7 released includes a major refactoring of the tagging system. Tagging speed has been significantly improved. Tags can now be used with items, domains and images and added in the objects. A new feature to search by tags has been also added. Multiple bugs were fixed.

Detailed change logs (v2.7 (2020-01-13))

Changes

• [UI domain] add input: show domain by name. [Terrtia]
• [tags blueprint] clean code. [Terrtia]
• [update v2.7] sort domain full_onion_up and full_regular_up. [Terrtia]
• [UI tags] search domains by tags. [Terrtia]
• [core + UI] search domain by tags. [Terrtia]
• [Update v2.7] add update v2.7 scripts. [Terrtia]
• [tags UI] edit object tags (delete tags) [Terrtia]
• [Tag core] objects tagging, Part 2/2 TODO: UI tags domain + screenshot
  • object rename paste=>item. [Terrtia]
• [Tag core] objects tagging, Part 1/2 TODO UI (tags) + rename
  paste=>item. [Terrtia]
• [Tag core] objects tagging, Part 1/2. [Terrtia]
• [README] remove top terms. [Terrtia]
• [correlation graph UI] add json error handler + add loading status.
  [Terrtia]
• [UI correlation graph + UI domain] correlation screenshot: show img in
  toolip + show hash in ShowDomain TODO: pixelate images. [Terrtia]

Fix

• [UI tags] fix domain links. [Terrtia]
• [UI term] remove deprecated trending charts, fix #446 #447. [Terrtia]
• [Crawler] typo. [Terrtia]
• [UI decoded item] sort mimetype. [Terrtia]
• [Crawler] fix screenshot-domain typo. [Terrtia]
• [Crawler] fix screenshot-domain typo. [Terrtia]
• [Crawler] fix screenshot-domain map. [Terrtia]
• [UI showDomain] fix screenshot accordeon. [Terrtia]

Other

• Merge pull request #449 from CIRCL/tags_v2. [Alexandre Dulaunoy]

  Tags v2 - Tagging system refractoring

• Merge branch 'master' into tags_v2. [Terrtia]

• Update README.md. [Thirion Aurélien]

AIL Framework version 2.6 released with improved correlations (hover information, screenshot hash correlation), API improvements and various fixes

AIL Framework version 2.6 released with improved correlations (hover information, screenshot hash correlation), API improvements and various fixes. Thanks to the enforce project for the feedback during the training. New features were based on constructive remarks from the users.

New and Improvements

• [slides] source code added. [Alexandre Dulaunoy]
• [screenshot correlation + v2.6] add screenshot-domain correlation + v2.6 update. [Terrtia]
• [API] get domain min metadata (first up, last up) + get crawled domain by daterange and status. [Terrtia]
• [Domain + Date] get domain up range + get date days and months by daterange. [Terrtia]
• [Domain] get all/by month domains up. [Terrtia]
• [API] get domain metadata (minimal) [Terrtia]
• [UI correlation graph] tooltip: show domain tags. [Terrtia]
• [UI correlation graph] popover: add loading status + chg css. [Terrtia]
• [correlation UI] add basic popover. [Terrtia]
• [slide] update slide. [Terrtia]
• [pgpdump] add debug. [Terrtia]
• Linked TOR installation instruction in the README. [Sami Mokaddem]
• [Onion] add discovery queue. [Terrtia]
• [Showpaste] check if tags are safe (img) + fix domain link. [Terrtia]
• [crawler dashboard UI] add UP/Down domains url. [Terrtia]

Bugs fixed

• [UI correlation graph] typo. [Terrtia]
• [UI showDomain] fix down domain history. [Terrtia]
• [Domain] domain was up. [Terrtia]
• [Domain] is_domain_up. [Terrtia]
• Typo. [Terrtia]
• [Update] force manual update, fix #443. [Terrtia]
• [UI] fix show paste modal. [Terrtia]
• [screenshot canevas + domain link] fix item domain link + screenshot
  canevas: chg colors and icons for unsafe tags. [Terrtia]
• [PgpDump] catch bs4 error. [Terrtia]
• Placed Tor installation instruction in the installation section. [Sami Mokaddem]
• [Update v2.4] fix empty set. [Terrtia]
• [Update v2.4] fix empty set. [Terrtia]
• [Item lib] fix import. [Terrtia]
• [Paste submit] fix tags unpack. [Terrtia]
• [Show Domain UI] fix screenshot link, fix #431. [Terrtia]
• [Update] filter invalid tags. [Terrtia]

AIL Framework version 2.5 released with improved correlation and experimental support for MISP modules

AIL Framework version 2.5 released.

AIL Framework version 2.5 released with correlation and experimental support for MISP modules. The correlation is now improved to add correlation between PGP, cryptocurrencies, pastes, decoded values against any items in AIL framework. The correlation interface has been redesigned to allow filtering per type (pastes, crawled) and also limit the number of correlations. The MISP modules is still very early but the objective is gain from all MISP modules expansion within AIL. We introduced an experimental module to automatically expand BTC transactions from addresses seen in AIL and pivot to new correlations. New roles were added (read_only and users without) to improve the profile of the various AIL users.

The improvements were designed with some requirements from the ENFORCE project to better support law-enforcement usage.

AIL Framework version 2.4 released with improved crawled domain correlation (cryptocurrency, pgp keys, decoded...)

AIL version 2.4 released

AIL version 2.4 has been released including the following new features:

• Improved crawled domain correlation to correlate such domain via cryptocurrency addresses, PGP keys UID, decoded hash content
• Screenshot of crawled item can be selected from the UI
• Crawled domain and port are now properly supported
• Tagging functionality added to domain crawled
• Configuration files have been moved to a coherent directory
• Documentation of the code improved
• Various bugs fixed and small improvements

AIL Framework version 2.3 released with improved cryptocurrencies detection, SQLi, reconnaissance tools

AIL Framework version 2.3 released with improved cryptocurrencies detection, SQLi and detection of network reconnaissance tools output. Many bugs fixed and small improvements were performed.

Changes

• [Cryptocurrency + Tools] launch by default + remove old Bitcoin module. [Terrtia]
• [Keys module] detect public key. [Terrtia]
• [Tools detection] add tool detection module. [Terrtia]
• [Cryptocurrency, RegexTracker] update cryptocurrency list + fix: RegexTracker typo. [Terrtia]
• [Cryptocurrency] add private_key entry + fix dash regex. [Terrtia]
• [Cryptocurrency] add new Cryptocurrency module. [Terrtia]
• [Tracker] add optional description field. [Terrtia]

Fix

- [Tool] fix searchsploit regex. [Terrtia]
- [Tools] typo. [Terrtia]
- [Tools] typo. [Terrtia]
- [Tools] fix loop. [Terrtia]
- [url_prefix] add root blueprint, fix:#403. [Terrtia]
- [TermTracker] fix performance: disable token stats. [Terrtia]
- [SQL module] fix typo. [Terrtia]

# Other

- Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia]
- Merge pull request #417 from andurin/master. [Alexandre Dulaunoy]

  Fix pybgpranking dependency in requirements
- Fix pybgpranking dependency in requirements. [Hendrik]

  Relates #334
- Merge branch 'master' of https://github.com/CIRCL/AIL-framework.
  [Terrtia]
- Merge pull request #404 from WimpyMan/master. [Thirion Aurélien]
- LAUNCH.py: Added execution of script IPAddress.py. [Bastien Schils]
- IPAddress.py: use ipaddress module. [Bastien Schils]

  Improved readability, maintainability and use of standard module
- Modules.cfg: Minor: Added \n to seperate sections. [WimpyMan]
- Config.cfg.sample: Improved example for IP module. [WimpyMan]

  By default, the list of networks to monitor is now empty.
  The previous value is now given as example.
- Added: IP matching module. [Bastien Schils]
- Merge pull request #411 from krial057/patch-1. [Alexandre Dulaunoy]

  Fixed some typos
- Fixed some typos. [krial057]

  Fixed some typos in the readme
- Merge pull request #408 from stamparm/master. [Thirion Aurélien]

  Adding more tools
- Adding more tools. [Miroslav Stampar]
- Merge pull request #407 from stamparm/patch-1. [Thirion Aurélien]

  Covering special cases (on pastebin)
- Covering special cases (on pastebin) [Miroslav Stampar]

  There is no need for checking `()` in case of (e.g.) Litecoin and Dash as those are also additionally checked with Bitcoin address verifier
- Merge pull request #406 from stamparm/master. [Thirion Aurélien]

  Adding tool regexes
- Adding tool regexes. [Miroslav Stampar]
- Merge branch 'master' of https://github.com/CIRCL/AIL-framework.
  [Terrtia]
- Merge pull request #405 from stamparm/master. [Thirion Aurélien]

  Enforcing Base58 check on Litecoin and Dash addresses
- Enforcing Base58 check on Litecoin and Dash addresses. [Miroslav
  Stampar]
- Merge branch 'master' of https://github.com/CIRCL/AIL-framework.
  [Terrtia]
- Merge pull request #401 from stamparm/master. [Thirion Aurélien]

  Improvement of crypto-address regexes (lesser FPs)
- Improvement of crypto-address regexes (lesser FPs) [Miroslav Stampar]
- Merge pull request #398 from stamparm/master. [Thirion Aurélien]

  Implementation for different cryptocurrencies
- Implementation for different cryptocurrencies. [Miroslav Stampar]
- Merge pull request #396 from stamparm/master. [Thirion Aurélien]

  Improvement of SQLi detection
- Improvement of SQLi detection. [Miroslav Stampar]

AIL Framework version 2.2 released with refactoring of term tracking

AIL Framework version 2.2 released with refactoring of term tracking

Changes

• [API] add Tracker documentation. [Terrtia]
• [Tracker] add more info. [Terrtia]
• [update] add v2.2 update + fix default update + fix Empty
  Item.get_tags() + add new LAUNCHER options. [Terrtia]
• [tracker] add missing btn. [Terrtia]
• [trackers] filter trackers list by type + minor fix. [Terrtia]
• [UI sparklines] sparklines: fix + factory. [Terrtia]
• [merge] master. [Terrtia]
• [UI term tracker] refractor term management: trackers list + show
  trackers + add new trackers. [Terrtia]
• [api] add endpoint: get tracked item_id by uuid and daterange.
  [Terrtia]
• [api] add endpoint: delete tracker term (regex/set/word) [Terrtia]
• [Term Tracker] refractor term tracker word/set/regex modules + remove
  old modules. [Terrtia]
• [Term tracker] add term tracker module (word + set) + API: add new
  term to track (word + set + regex) [Terrtia]
• [term] refractor + add new tracked word/set. [Terrtia]
• [README] add link to API documentation. [Thirion Aurélien]
• [tests API] use argv api key. [Terrtia]
• [api] add new endpoints: get bitcoin/pgp name/pgp keys/pgp mail
  metadata + items list. [Terrtia]

Fixes

• [sparkline] datatable drawing. [Terrtia]
• [d3 graph] fix script path. [Terrtia]
• Add missing file. [Terrtia]
• [d3 js plugin] [Terrtia]
• [template] add trackers. [Terrtia]
• [Term Tracker module] chg module flow. [Terrtia]
• [BankAccount] fix #385. [Terrtia]
• [API doc] get bitcoin metadata + list of items. [Terrtia]
• [UI caching] avoid domain archive caching. [Terrtia]
• Typo. [Terrtia]
• [UI caching] fix: #373 avoid screenshot caching. [Terrtia]

(free) Trainings

• Prague, Thursday, 19 Sep 2019 09:00 : https://en.xing-events.com/CLTDKUU.html
• Luxembourg, Monday, 23 Sep 2019 10:00 : https://en.xing-events.com/JDVIRXW.html