Skip to content

Commit

Permalink
standalone version documentation bump alephdata#214
Browse files Browse the repository at this point in the history
  • Loading branch information
Your Name committed Jan 5, 2023
1 parent 8106fa9 commit 5a27fda
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 0 deletions.
8 changes: 8 additions & 0 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,5 +78,13 @@ When you're working on your crawlers, it's not convenient to rebuild your Docker
* Run `pip install memorious`. If your crawlers use Python extensions, you'll need to run `pip install` in your crawlers directory as well
* Run `memorious list` to list your crawlers and `memorious run your-crawler` to run a crawler.

For multi-threaded mode:

* Add `export MEMORIOUS_DATASTORE_URI=postgresql://user:pass@localhost/database` and `export REDIS_URL=redis://localhost:6379` in env.sh
* Create database in PostgreSQL and create table: `CREATE TABLE memorious_tags ( key CHARACTER VARYING, value JSONB, timestamp TIMESTAMP)`
* Run `memorious run book_scraper --threads 10`

You can also run `cd memorious;python3 cli.py run bookscraper --threads 10`

*Note: In development mode Memorious uses a single threaded worker (because FakeRedis is single threaded). So task execution concurrency is limited and the worker executes stages in a crawler's pipeline linearly one after another.*

1 change: 1 addition & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
"stringcase",
"flask",
"babel",
"sqlalchemy == 1.4.46",
],
entry_points={
"console_scripts": ["memorious = memorious.cli:main"],
Expand Down

0 comments on commit 5a27fda

Please sign in to comment.