AIL Framework version 4.0 has been released including a major new feature to allow synchronisation to other AIL instance(s).

The new synchronisation mechanism allow the sync from one AIL instance to another AIL using a standard WebSocket using AIL JSON protocol. The synchronisation allows to filter and sync specific collected items including crawled items or specific tagged items matching defined rules. This feature can be very useful to limit the scope of analysis in specific fields or resource intensive activity. This sync can be also used to share filtered streams with other partners.

A new functionality has been added to trigger a webhook when a tracker is matched in AIL. This is in addition to email notification. The webhook can be used to trigger additional pipelines in AIL.

Additional API endpoints were added such get_item_sources get_check_item_source and get_default_yara_rule_content.

Thanks to the numerous external contributors such as Olivier Sagit and Tony Jabbour. A special thank for the webhook and API developed by Tony Jabbour from CSIRT POST Cyberforce in Luxembourg.

The first version of the synchronisation protocol has been developed in the scope of the JTAN (Joint Threat Analysis Network), a CEF co-funded project (2020-EU-IA-0260).

Many bugs were fixed in this release and many small improvements were added.

Detailed changelog is available on https://www.ail-project.org/ChangeLog.

Changelog

v4.0 (2021-12-01)

Changes

• [sync UI] disable pull. [Terrtia]

• [sync UI] dashboard, show nb of imported items + launch/kill ail servers when a queue is subscribed/unsubscribed. [Terrtia]

• [ail sync UI] restarr/launch/kill sync connections + show sync mode api/pull/push. [Terrtia]

• [ail sync server] add server controller + list connected clients ail_uuid->sync_modes. [Terrtia]

• [ail sync ui] copy to clipboard ail_uuid, ail server key. [Terrtia]

• [ail sync] edit ail_servers/sync_queues + fix logs. [Terrtia]

• [api] rename endpoints. [Terrtia]

• [ail sync] add sync api (ping, version) + UI/client error handler. [Terrtia]

• [doc] GI Badge. [Steve Clement]

• [v4.0 AIL SYNC / AIL 2 AIL] SYNC Manager + fixs + views. [Terrtia]

• [crawler] add auto crawler functions. [Terrtia]

Fix

• [sync client] don’t launch client if ail server not linked with a sync queue. [Terrtia]

• [sync server] remove hardcoded host. [Terrtia]

• [sync server] host. [Terrtia]

• [sync client] fix websockets client connect for python >= 3.8. [Terrtia]

• [ail sync] fix refresh_ail_instance_connection. [Terrtia]

• [ail sync] fix refresh_ail_instance_connection. [Terrtia]

• [ail sync] server + client: resend object in queue on ConnectionClosedError. [Terrtia]

• [crawler] add comment. [Terrtia]

• [UI ail sync] fix nav. [Terrtia]

• [UI ail sync] add missing ail icon. [Terrtia]

• [doc] Remove Travis. [Steve Clement]

• [py] Minor python dependency change. [Steve Clement]

• Inherit AbstractModule to prevent stuck queues. [osagit]

  regex compiled only at start, not in the loop no duplicate warning string comments

• Error message contains http protocol twice. [osagit]

  Error Can’t connect to AIL Splash Manager, http://https://localhost:7001/

Other

• Merge pull request #130 from TonyJabbour/master. [Thirion Aurélien]

  New restAPIs

• Merge branch ‘dev’ into master. [Thirion Aurélien]

• Merge branch ‘master’ of github.com:ail-project/ail-framework. [Terrtia]

• Merge branch ‘master’ of github.com:ail-project/ail-framework. [Alexandre Dulaunoy]

• Merge pull request #569 from SteveClement/master. [Steve Clement]

• Chg_ [AIL 2 AIL] add backend. [Terrtia]

• Merge branch ‘master’ of github.com:ail-project/ail-framework. [Terrtia]

• Add tracker fixed api function replaced it with internal function. [TonyJabbour]

• Added get_tracker_metadata_api Removed unnecessarily parentheses. [TonyJabbour]

• New API Endpoint: Fixed get_item_content_encoded_text Added get_item_sources Added get_check_item_source Added get_default_yara_rule_content. [TonyJabbour]

• Removed unnecessarily parentheses. [TonyJabbour]

• New API Endpoint: Return Item Content in base64 in non JSON format. [TonyJabbour]

• Merge remote-tracking branch ‘origin/master’ [TonyJabbour]

• Merge branch ‘ail-project:master’ into master. [Tony]

• Merge pull request #129 from TonyJabbour/master. [Thirion Aurélien]

  Webhook implementation

• Unnecessarily parenthesis removed. [TonyJabbour]

• Base64 Problem. [TonyJabbour]

• Base64 Problem. [TonyJabbour]

• Type fixed. [TonyJabbour]

• Type fixed. [TonyJabbour]

• Error Fixed. [TonyJabbour]

• Add new API endpoint that return only content encoded in base64. [TonyJabbour]

• Webhook unnecessarily line removed Removed unnecessarily parentheses. [TonyJabbour]

• Type fixed. [TonyJabbour]

• Added a try/catch to handle exceptions Replaced the raise to send message to redis_logger. [TonyJabbour]

• Fixed tracker_metadata. [TonyJabbour]

• Typo Fixed. [TonyJabbour]

• Fixed UI Datatable in showTracker Page Removed “Advanced Search” text from menu misleading button. [TonyJabbour]

• Fixed Webhook integration with Trackers. [TonyJabbour]

• -Fixed “get_tracker_metedata” typo -Typo Fixed. [TonyJabbour]

• -Fixed “description” arg -Typo Fixed. [TonyJabbour]

• -Fixed the 500 error issue when installing new instance of ail when adding new trackers -Fixed missing arguments -Typo Fixed. [TonyJabbour]

• Fix webhook. [TonyJabbour]

• Remove dict from Trackers. [TonyJabbour]

• Add webhook post support in yara and regex trackers. [TonyJabbour]

• Fix get_term_webhook. [TonyJabbour]

• Add some changes for webhook. [TonyJabbour]

• Add initial support for Webhook in Term Trackers. [TonyJabbour]

• Fix spelling issue in Webhook. [TonyJabbour]

• Add more support for Webhook URL. [TonyJabbour]

• Add initial UI support for Webhook in tracker. [TonyJabbour]

• Merge pull request #127 from osagit/patch-3. [Thirion Aurélien]

  fix: inherit AbstractModule to prevent stuck queues

• Merge pull request #126 from osagit/patch-2. [Thirion Aurélien]

  fix: error message contains http protocol twice