AIL Project Release v5.9 - 2024-10-18

We are glad to announce the release of AIL v5.9, packed with numerous updates and fixes that enhance the performance and features of the framework. This release focuses on improvements in qrcode handling, dom-hash support, title processing, and crawler operations, among other critical updates.

dom-hash is a structural fingerprint of the HTML’s Document Object Model (DOM) originally developed by CERT.PL.

Software such as LookyLoo or MISP have implemented the algorithm, the AIL framework now support the dom-hash algorithm to cluster and group similar page structure.

What’s New in v5.9

Changes

• [qrcodes] Daterange search: Sort qrcode by content. [terrtia]
• [dom-hash] Add dom-hash object to compute dom-hash for domains and crawled items. [terrtia]
• [CEDetector] Tag domains. [terrtia]
• [CEDetector] Add detect message functionality. [terrtia]
• [titles] Setup Titles queues and integrate CEDetector module. [terrtia]
• [tools] Add reprocess option for Titles + CEDetector. [terrtia]
• [crawler] Disable unsafe title auto-tagging. [terrtia]
• [module] General debugging and improvements. [terrtia]
• [module] Add CEDetector module. [terrtia]
• [crawler] Tag domain by vanity. [terrtia]
• [crawler] Crawl list URLs: Filter duplicates. [terrtia]
• [crawler] Submit free text of URLs to crawl. [terrtia]
• [onion module] Filter out Onion v2 domains. [terrtia]
• [show domain] Display title content and fix item tags. [terrtia]
• [crawler] Update TOR user agent for more efficient crawling. [terrtia]
• [message image] Show extracted QR codes in messages. [terrtia]
• [domain lookup] Extract domains from URL input. [terrtia]
• [api] Rename domain lookup function. [terrtia]

Fixes

• [module] Fixed a typo in the module. [terrtia]
• [tag] Tag delete: Fixed update for first/last seen times. [terrtia]
• [show domain] Fixed the correlation button when correlation count is zero. [terrtia]
• [crawler] Filter lookup tags for better accuracy. [terrtia]
• [crawler] Filter lookup for parent + domain daterange. [terrtia]

We hope these updates enhance your experience with the AIL framework. Stay tuned for future updates as we continue to improve and expand AIL’s capabilities.

Funding

MISP-LEA, a collaborative endeavor between Shadowserver and CIRCL, is a 24-month initiative funded by the European Union. The project’s central aim is to establish operational and enduring MISP and AIL instances dedicated specifically to law enforcement agencies. This setup will facilitate a smoother exchange of evidence between law enforcement agencies and improve the onset of collaborative investigations. For this purpose, the system will ingest data from Shadowserver’s ransomware and C2 infrastructure tracking.

Law enforcement agencies willing to discover and leverage the MISP-LEA platform can apply on the misp-lea.org website.

Stay Connected:

• Website: ail-project.org
• GitHub: github.com/ail-project/ail-framework
• Mastodon @ail_project
• Twitter/X/N*zi platform: @AIL_Project

Thank you for your continued support. We look forward to your feedback!