AIL Framework version 3.6 released with new features (such as YARA retrohunt), significant performance improvements, refactoring of the modules and many bugs were fixed. This version includes a new advanced to perform YARA retrohunt on the whole AIL dataset. YARA retrohunts can be started and stopped live, sources can be defined and also the period where to search retroactively for. Tags can be also applied to reclassify information collected in AIL. The speed of YARA retrohunt depends of the hardware used, SSD devices are strongly recommended. New tool such as export of crypto currencies correlation has been added to allow the use of data from AIL in other tools for analysing cryptocurrency transactions.

Many bugs were also fixed in this release. Don’t forget to update.

A huge thanks to all the contributors and especially @osagit for the numerous contributions.

Changelog

v3.6 (2021-07-14)

Changes

• Chg: [README] add Olivier Sagit @osagit to contributors list. [Terrtia]

• [requirements] minimal version + remove old packages. [Terrtia]

• Add Retro Hunt. [Terrtia]

• [UI trackers] add/edit/remove tracker source/target #43 #102. [Terrtia]

• [Trackers regex + yara] filter by item source. [Terrtia]

• [Tracker term] track terms by sources. [Terrtia]

• [test] add new workflow. [Thirion Aurélien]

• [requirements] minimal version + remove old packages. [Terrtia]

• [modules + tests] fix modules + test modules on samples. [Terrtia]

• [update v3.6 + installer] fix faup install + add v3.6 update. [Terrtia]

• [Launcher] update modules directory. [Terrtia]

• [SQLInjectionDetection LibInjection modules] add module class. [Terrtia]

• [Urls (Web) module] fix regex + rename. [Terrtia]

• [Tracker_Regex module] create module class + mv module. [Terrtia]

• [Tracker_Yara module] create module class. [Terrtia]

• [modules] create new modules repository + small fixs. [Terrtia]

• [Keys] add test. [Terrtia]

• [modules] add tests: CreditCard, Global, DomClassifier. [Terrtia]

• [Categ] tests + docs. [Terrtia]

• [ApiKey] refactor module + tests. [Terrtia]

• [launcher + modules] add module tests (Onion module) [Terrtia]

• [AIL items + Onion] create AIL item objects + Onion module refactor. [Terrtia]

• [tools extract_cryptocurrency] filter by correlation objects + errors messages. [Terrtia]

• [tools extract_cryptocurrency] new input: list of addresses to extract. [Terrtia]

• [tools] add 2 new tools: extract cryptocurrency correlation by type + trigger manual update. [Terrtia]

• [modules] set log level to critcal on unexpected errors. [Terrtia]

Fix

• [Term tracker] fix fct args. [Terrtia]

• [crawler] fix new crawled item id. [Terrtia]

• [Retro Humt UI] clarify tags input. [Terrtia]

• URLs www word to match. [osagit]

  There is 4 ‘w’ chars instead of 3

• [crawlers] get_all_splash return type. [Terrtia]

• [Splash_Manager errors] catch invalid response. [Terrtia]

• [Trackers UI] fix sources logo + tracker metadata. [Terrtia]

• Change module name Web to Urls. [lpwm9803]

• [Splash_Manager errors] catch invalid response. [Terrtia]

• [github workflows] fix test. [Terrtia]

• [github workflows] clone depth. [Thirion Aurélien]

• [installer] pyfaup install. [Terrtia]

• [installer] pyfaup install. [Terrtia]

• [UI crawler dashboard] block_languages_search: fix domains_types #110. [Terrtia]

• [UI crawler dashboard] block_domains_name_search: fix domains_types #110. [Terrtia]

• [modules] print + save traceback in logs. [Terrtia]

• [paste_submit] restrict source characters. [Terrtia]

• [submit_paste] restrict source name. [Terrtia]

• [tools extract_cryptocurrency] argparse flags. [Terrtia]

• [tags] invalid tags. [Terrtia]

• Replace redis init with generic ConfigLoader. [osagit]

  StrictRedis() replaced by ConfigLoader.get_redis_conn()

• Debug() string takes 1 positional argument. [osagit]

• FILE_ALLOWED_EXTENSIONS without quotes. [osagit]

• Stuck queues and submit paste. [Olivier SAGIT]

• Name pystemon feeder in feeder monitor dashboard. [Olivier SAGIT]

• [tools extract cryptocurrency] correlation type. [Terrtia]

• [Updater] don’t check if modified config files (redis and ardb config) [Terrtia]

• [Web module] resolver, change log level. [Terrtia]

• [WebStats] typo. [Terrtia]

• [Indexer] debug messages. [Terrtia]

• [WebStats] typo. [Terrtia]

• [terms tracker] refresh Tracked terms. [Terrtia]

• [redis cache] remove old paste_name db. [Terrtia]

• [crawler] typo: splash restart. [Terrtia]

Other

• Merge branch ‘master’ of https://github.com/ail-project/ail-framework into dev. [Terrtia]

• Merge pull request #115 from My-WAF/master. [Thirion Aurélien]

  Remove Block Copy Git Directory

• Update .dockerignore. [VNC Company]

• Merge branch ‘dev’ of https://github.com/ail-project/ail-framework into dev. [Terrtia]

• Merge pull request #116 from osagit/patch-1. [Thirion Aurélien]

  fix: URLs www word to match

• Merge. [Terrtia]

• Merge branch ‘dev’ of https://github.com/ail-project/ail-framework into dev. [Terrtia]

• Merge pull request #114 from osagit/dev. [Thirion Aurélien]

  fix: Change module name Web to Urls

• Merge branch ‘master’ into dev. [Terrtia]

• Merge branch ‘master’ of https://github.com/CIRCL/AIL-framework. [Terrtia]

• Merge pull request #559 from lesleyxyz/patch-1. [Thirion Aurélien]

  Submit paste contents to TheHive

• Submit file contents to TheHive. [Lesley De Keyser]

• Merge pull request #103 from osagit/scriptsenhance. [Thirion Aurélien]

  fix: stuck queues and submit paste

• Merge branch ‘dev’ into scriptsenhance. [Thirion Aurélien]

• Merge branch ‘master’ of https://github.com/ail-project/ail-framework. [Terrtia]

• Perf: use defined compiled regex. [osagit]

  re.compile(regex) definition was not used use compile_regex.findAll() directely instead of re.findall(regex)

• Log message split error + perf. [osagit]

  fix: log message split errors perf: string affected at start doc: comments

• Merge pull request #101 from osagit/namedfeeder. [Alexandre Dulaunoy]

  fix: name pystemon feeder name in feeder monitor dashboard

• Merge pull request #97 from osagit/abstract-module. [Thirion Aurélien]

  feat: module factorization

• Feat: module factorization. [lpwm9803]