-
Notifications
You must be signed in to change notification settings - Fork 67
/
QUIRKS
65 lines (46 loc) · 1.93 KB
/
QUIRKS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
## Quirks
This library aims to be compatible with the old ScraperWiki Classic library,
which was itself quite quirky.
It's not an exact drop in replacement, this document describes the differences.
## Differences
### Datastore differences
The local `scraperwiki.sqlite` is powered by
[SQLAlchemy](http://sqlalchemy.org), so some things
work a bit differently.
Data is stored to a local sqlite database named `scraperwiki.sqlite`.
Bizarre table and column names are supported.
`scraperwiki.sqlite.execute` will return an empty list of keys on an
empty select statement result.
`scraperwiki.sqlite.attach` downloads the whole datastore from ScraperWiki, the first time it runs; it then uses the cached database
### Other Differences
## Status of implementation
In general, features that have not been implemented raise a `NotImplementedError`.
### Datastore
`scraperwiki.sqlite` may implement some less explored features
of its API slightly differently from ScraperWiki.
### Utils
`scraperwiki.utils` is implemented, as well as the following functions.
* `scraperwiki.log`
* `scraperwiki.scrape`
* `scraperwiki.pdftoxml`
### Deprecated
These submodules are deprecated and thus will not be implemented.
* `scraperwiki.apiwrapper`
* `scraperwiki.datastore`
* `scraperwiki.jsqlite`
* `scraperwiki.metadata`
* `scraperwiki.newsql`
* `scraperwiki.sqlite.attach`
* `scraperwiki.utils.swimport`
* `scraperwiki.geo`
* `scraperwiki.log`
verbose parameter to various functions is deprecated
### Specs
Here are some ScraperWiki scrapers that demonstrate the non-local library's quirks.
https://scraperwiki.com/scrapers/scraperwiki-python/
https://scraperwiki.com/scrapers/cast/
https://scraperwiki.com/scrapers/things_happen_when_you_do_not_commit/
https://scraperwiki.com/scrapers/what_does_show_tables_return/
https://scraperwiki.com/scrapers/on_conflict/
https://scraperwiki.com/scrapers/spaces_in_table_names/
https://scraperwiki.com/scrapers/spaces_in_table_names_1/