Skip to content

Commit

Permalink
merge master
Browse files Browse the repository at this point in the history
  • Loading branch information
@Terrtia
Terrtia committed May 17, 2023
2 parents 5b6a7f1 + 20fad97 commit 175963e
Show file tree
Hide file tree
Showing 4 changed files with 89 additions and 89 deletions.
69 changes: 24 additions & 45 deletions HOWTO.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -78,21 +78,6 @@ If you want to add a new processing or analysis module in AIL, follow these simp
2. Use [./bin/template.py](./bin/template.py) as a sample module and create a new file in bin/ with the module name used in the modules.cfg configuration.


How to create a new webpage
---------------------------

If you want to add a new webpage for a module in AIL, follow these simple steps:

1. Launch [./var/www/create_new_web_module.py](./var/www/create_new_web_module.py) and enter the name to use for your webpage (Usually, your newly created python module).

2. A template and flask skeleton has been created for your new webpage in [./var/www/modules/](./var/www/modules/)

3. Edit the created html files under the template folder as well as the Flask_* python script so that they fit your needs.

4. You can change the order of your module in the top navigation header in the file [./var/www/templates/header_base.html](./var/www/templates/header_base.html)

5. You can ignore module, and so, not display them in the top navigation header by adding the module name in the file [./var/www/templates/ignored_modules.txt](./var/www/templates/ignored_modules.txt)

How to contribute a module
--------------------------

Expand All @@ -104,30 +89,6 @@ To contribute your module, feel free to pull your contribution.
Additional information
======================

Manage modules: ModulesInformationV2.py
---------------------------------------

You can do a lots of things easily with the [./bin/ModulesInformationV2](./bin/ModulesInformationV2) script:

- Monitor the health of other modules
- Monitor the ressources comsumption of other modules
- Start one or more modules
- Kill running modules
- Restart automatically stuck modules
- Show the paste currently processed by a module

### Navigation

You can navigate into the interface by using arrow keys. In order to perform an action on a selected module, you can either press <ENTER> or <SPACE> to show the dialog box.

To change list, you can press the <TAB> key.

Also, you can quickly stop or start modules by clicking on the ``<K>`` or ``<S>`` symbol respectively. These are located in the _Action_ column.

Finally, you can quit this program by pressing either ``<q>`` or ``<C-c>``.



Crawler
---------------------

Expand All @@ -145,6 +106,7 @@ In AIL, you can crawl websites and Tor hidden services. Don't forget to review t
1. Lacus URL:
In the webinterface, go to ``Crawlers>Settings`` and click on the Edit button


![Splash Manager Config](./doc/screenshots/lacus_config.png?raw=true "AIL Lacus Config")

![Splash Manager Config](./doc/screenshots/lacus_config_edit.png?raw=true "AIL Lacus Config")
Expand All @@ -156,9 +118,26 @@ Choose the number of crawlers you want to launch
![Splash Manager Nb Crawlers Config](./doc/screenshots/crawler_nb_captures_edit.png?raw=true "AIL Lacus Nb Crawlers Config")



#### Old updates

##### Python 3 Upgrade

To upgrade from an existing AIL installation, you have to launch [python3_upgrade.sh](./python3_upgrade.sh), this script will delete and create a new virtual environment. The script **will upgrade the packages but won't keep your previous data** (neverthless the data is copied into a directory called `old`). If you install from scratch, you don't require to launch the [python3_upgrade.sh](./python3_upgrade.sh).
Kvrocks Migration
---------------------
**Important Note:
We are currently working on a [migration script](https://github.com/ail-project/ail-framework/blob/master/bin/DB_KVROCKS_MIGRATION.py) to facilitate the migration to Kvrocks.
Once this script is ready, AIL version 5.0 will be released.**

Please note that the current version of this migration script only supports migrating the database on the same server.
(If you plan to migrate to another server, we will provide additional instructions in this section once the migration script is completed)

To migrate your database to Kvrocks:
1. Launch ARDB and Kvrocks
2. Pull from remote
```
git checkout master
git pull
```
3. Launch the migration script:
```
git checkout master
git pull
cd bin/
./DB_KVROCKS_MIGRATION.py
```
38 changes: 26 additions & 12 deletions bin/Update.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,8 @@ def auto_update_enabled(cfg):

# check if files are modify locally
def check_if_files_modified():
process = subprocess.run(['git', 'ls-files', '-m'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# return True
process = subprocess.run(['git', 'ls-files' ,'-m'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if process.returncode == 0:
modified_files = process.stdout
if modified_files:
Expand All @@ -61,6 +62,7 @@ def check_if_files_modified():
sys.exit(1)

def repo_is_fork():
# return False
print('Check if this repository is a fork:')
process = subprocess.run(['git', 'remote', '-v'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

Expand Down Expand Up @@ -234,9 +236,9 @@ def get_git_upper_tags_remote(current_tag, is_fork):
except ValueError:
continue

if float(current_tag) < 4.2:
if float(current_tag) < 5.0:
# add tag with last commit
if float(current_tag_val) <= float(tag_val) < float(4.2):
if float(current_tag_val) <= float(tag_val) < float(5.0):
dict_tags_commit[tag] = commit
else:
# add tag with last commit
Expand Down Expand Up @@ -270,13 +272,28 @@ def update_ail(current_tag, list_upper_tags_remote, current_version_path, is_for

update_submodules()

print('{}git pull:{}'.format(TERMINAL_YELLOW, TERMINAL_DEFAULT))
process = subprocess.run(['git', 'pull'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
temp_current_tag = current_tag.replace('v', '')
if temp_current_tag.count('.') > 1:
temp_current_tag = temp_current_tag.rsplit('.', 1)
temp_current_tag = ''.join(temp_current_tag)

if process.returncode == 0:
output = process.stdout.decode()
print(output)
if float(temp_current_tag) < 5.0:
roll_back_update('2c65194b94dab95df9b8da19c88d65239f398355')
pulled = True
else:
print('{}git pull:{}'.format(TERMINAL_YELLOW, TERMINAL_DEFAULT))
process = subprocess.run(['git', 'pull'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if process.returncode == 0:
output = process.stdout.decode()
print(output)
pulled = True
else:
print('{}{}{}'.format(TERMINAL_RED, process.stderr.decode(), TERMINAL_DEFAULT))
aborting_update()
pulled = False
sys.exit(1)

if pulled:
# CHECK IF UPDATER Update
if float(os.stat(UPDATER_FILENAME).st_mtime) > UPDATER_LAST_MODIFICATION:
# request updater relaunch
Expand Down Expand Up @@ -321,10 +338,7 @@ def update_ail(current_tag, list_upper_tags_remote, current_version_path, is_for
print(f'{TERMINAL_YELLOW}**************** AIL Successfully Updated *****************{TERMINAL_DEFAULT}')
print()
sys.exit(0)
else:
print('{}{}{}'.format(TERMINAL_RED, process.stderr.decode(), TERMINAL_DEFAULT))
aborting_update()
sys.exit(1)

else:
print('{}{}{}'.format(TERMINAL_RED, process.stderr.decode(), TERMINAL_DEFAULT))
aborting_update()
Expand Down
4 changes: 1 addition & 3 deletions var/www/modules/dashboard/templates/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -74,9 +74,7 @@
{%endif%}

<!-- TODO: Add users messages -->
<!-- {%if update_note%}
{% include 'dashboard/update_modal.html' %}
{%endif%} -->
{# {% include 'dashboard/update_modal.html' %}#}

<div class="row my-2">

Expand Down
67 changes: 38 additions & 29 deletions var/www/templates/dashboard/update_modal.html
View file
Original file line number Diff line number Diff line change
@@ -1,40 +1,49 @@
<div class="modal fade" id="update_modal" tabindex="-1" role="dialog" aria-labelledby="exampleModalLabel" aria-hidden="true">
<div class="modal-dialog modal-lg" role="document">
<div class="modal-content">
<div class="modal-header bg-secondary text-white">
<h5 class="modal-title" id="exampleModalLabel">Update Note: v3.5 - Splash Manager</h5>
<button type="button" class="close" data-dismiss="modal" aria-label="Close">
<span aria-hidden="true">&times;</span>
</button>
</div>
<div class="modal-body">
<div class="alert alert-danger text-danger" role="alert">All Splash Crawlers have been removed from the core.</div>
AIL is using a new Crawler manager to start and launch dockers and tor/web crawlers
<div class="modal-dialog modal-lg" role="document">
<div class="modal-content">
<div class="modal-header bg-secondary text-white">
<h5 class="modal-title" id="exampleModalLabel">Update Note: <b>v5.0 - Kvrocks Migration</b></h5>
<button type="button" class="close" data-dismiss="modal" aria-label="Close">
<span aria-hidden="true">&times;</span>
</button>
</div>
<div class="modal-body">
<div>
In order to improve the performance and reliability of AIL, we have migrated our database from ARDB to <a href="https://github.com/apache/incubator-kvrocks">Kvrocks</a>,
and the AIL crawler has been upgraded to <a href="https://github.com/ail-project/lacus">Lacus</a>.
</div>
<hr>

<ul class="list-group my-3">
<li class="list-group-item active">Splash Manager Features:</li>
<li class="list-group-item">Install and run Splash crawlers on another server</li>
<li class="list-group-item">Handle proxies (Web and tor)</li>
<li class="list-group-item">Launch/Kill Splash Dockers</li>
<li class="list-group-item">Restart crawlers on crash</li>
</ul>
<div class="alert alert-danger text-danger" role="alert">Kvrocks DB Migration.</div>
<div>
As a result of this change, it is necessary to manually update your AIL installation.
Instructions on how to perform the update can be found <a href="https://github.com/ail-project/ail-framework/blob/master/HOWTO.md#kvrocks-migration">here</a>.
</div>

<div class="d-flex justify-content-center">
<a class="btn btn-info" href="https://github.com/ail-project/ail-splash-manager" role="button">
<i class="fab fa-github"></i> Install and Configure AIL-Splash-Manager
</a>
</div>
<br><br>

<div class="alert alert-danger text-danger" role="alert">AIL Crawlers Migration: Lacus</div>
If you wish to crawl Tor or website domains, you need to install Lacus.

<div class="d-flex justify-content-center">
<a class="btn btn-info" href="https://github.com/ail-project/lacus" role="button">
<i class="fab fa-github"></i> Install and Configure Lacus
</a>
</div>
How to configure the AIL crawler:
<a href="https://github.com/ail-project/ail-framework/blob/master/HOWTO.md#crawler">HOWTO</a>
<br><br><br>
If you encounter any difficulties, you can open an <a href="https://github.com/ail-project/ail-framework/issues">issue</a>

</div>
<div class="modal-footer">
<button type="button" class="btn btn-secondary" data-dismiss="modal">Close</button>
</div>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-secondary" data-dismiss="modal">Close</button>
</div>
</div>
</div>
</div>
</div>



<script>
$(window).on('load', function() {
$('#update_modal').modal('show');
Expand Down

0 comments on commit 175963e

Please sign in to comment.