Version 5.1 (2023-06-26) includes several changes, fixes, and updates. The changes include fixing gzipped pastes in the pystemon importer, showing a message when the maximum number of nodes is reached in the correlation graph, and adding the ability to auto tag crawled domains. Additionally, new features were added such as pagination for title searches, the ability to search title IDs and contents, and the inclusion of a favicon object.

Several fixes were implemented, including resolving issues with base64 encoding in the pystemon importer, maintaining the same capture UUID for already crawled domains in the crawler, and handling empty queues in the IPAdress module. Other fixes addressed issues with title searches returning empty results, incomplete responses in the crawler, and errors related to user tokens and deletion.

Various improvements were made to different modules and objects, such as the addition of a new cookie-name object along with its correlation, enhancements to importers, improvements to the HOWTO guide, and updates to correlation graphs and statistics. The Phone module was also updated to filter invalid phone numbers and display extracted information in the user interface.

In addition to the changes and fixes, there were updates to the MISP export, domains explorer, daterange object, tracker module, and various other components. The README.md file and CI badge were corrected, the installer was fixed for YARA and pycld3 installations, and tests were updated and replaced.

Lastly, there were some general updates, including merging changes from the old CIRCL/AIL-framework repository (the official repository is ail-project/ail-framework and incorporating a pull request related to email categorization.

Overall, version 5.1 introduced new features, addressed several issues, and included various updates and improvements to different parts of the system.

Detailed Change Log

v5.1 (2023-06-26)

Changes

• [pystemon importer] fix gzipped pastes. [Terrtia]

• [correlation graph] show message if max_nodes reached + fix cookie-name sparkline. [Terrtia]

• [crawler] auto tag crawled domains. [Terrtia]

• [correlation] add an option to remove max number of nodes if max_node == 0. [Terrtia]

• [object cookie-name] add new cookie-name object + correlation. [Terrtia]

• [title search] add pagination. [Terrtia]

• [titles] add title IDs and contents search. [Terrtia]

• [favicon object] add favicon object. [Terrtia]

• [sow item] show item investigations. [Terrtia]

• [kvrocks migration] mv update/v.50. [Terrtia]

• [redis] update minimal version. [Terrtia]

• [doc] add AIL v5.0 + objects + Importers + sync. [Terrtia]

• [correlation] filter blank screenshots. [Terrtia]

• [importers] improve abstract class and logs. [Terrtia]

• [domains explorer] unsafe tag default image. [Terrtia]

• [README.md] update. [Terrtia]

• [HOWTO] improve HOWTO. [Terrtia]

• [correlation graph] update node legend. [Terrtia]

• [correlation graph] select correlation depth. [Terrtia]

• [correlation] correlation graph: filter title objects. [Terrtia]

• [correlation] add direct correlation stats. [Terrtia]

• [new title object] add new title object + correlation on page title. [Terrtia]

• [Phone module] Filter Invalid Phone numbers + UI Show extracted. [Terrtia]

• [importers] add Dir/File Importer. [Terrtia]

Fix

• [pystemon importer] fix base64 encoding. [Terrtia]

• [crawler] same capture uuid if a domain is already crawled. [Terrtia]

• [IPAdress module] empty queue if no IP ranges provided. [Terrtia]

• [retro hunt] fix object tag queue + decoded content. [Terrtia]

• [daterange object] fix objects by date. [Terrtia]

• [title] fix title search empty result. [Terrtia]

• [crawler] fix incomplete response. [Terrtia]

• [user] fix get user token #163. [Terrtia]

• [user] fix user delete #163. [Terrtia]

• [MISP export] fix ail object first/last seen + obj logger. [Terrtia]

• [MISP export] fix empty event. [Terrtia]

• [d4] change enable d4. [Terrtia]

• [kvrocks migration] [Terrtia]

• [objects] fix investigation + ail2ail + screenshot MISP export. [Terrtia]

• [domains explorer] None screeenshot. [Terrtia]

• [show domains] fix down domains. [Terrtia]

• [domains explorer] domain screeenshot. [Terrtia]

• [domains explorer] fix empty screenshots. [Terrtia]

• [correlation] fix tagging nb nodes. [Terrtia]

• [README.md] fix CI badge. [Terrtia]

• [README.md] fix logo. [Terrtia]

• [module.cfg] fix templateModule example. [Terrtia]

• [module extractor] fix tracker extractor. [Terrtia]

• [tracker] fix tracker delete. [Terrtia]

• [tracker] fix webhook. [Terrtia]

• [crawler] fix undefined capture status. [Terrtia]

• [correlation btc info] catch btc txs error. [Terrtia]

• [Phone module] Filter Invalid Phone numbers. [Terrtia]

• [phone] fix phone module. [Terrtia]

• [domain search] fix template domain types filter. [Terrtia]

• [domain search] fix template domain types filter. [Terrtia]

• [MISP auto export] fix module input message. [Terrtia]

• [tests] replace unmaintened nose by nose2. [Terrtia]

• [tests] fix tests. [Terrtia]

• [instaler] fix yara and pycld3 install. [Terrtia]

• [tests] github workflow. [Terrtia]

• [tests] github workflow. [Terrtia]

• [flask] remove old import. [Terrtia]

Other

• Merge github.com:CIRCL/AIL-framework. [Terrtia]

• Merge pull request #592 from shadow2033/patch-2. [Thirion Aurélien]

  Update Categ Mail

• Update Mail. [shadow2033]

  ///English added (inbox; zoho)

  ///Russian добавлен (inbox; zoho)