Skip to content

Commit

Permalink
build: switch to sqlcipher3 and update Windows install commands/CLI s…
Browse files Browse the repository at this point in the history 
…cript
  • Loading branch information
@dylanljones
dylanljones committed Oct 3, 2023
1 parent 50f2b7d commit 2b0a4f3
Show file tree
Hide file tree
Showing 3 changed files with 36 additions and 243 deletions.
51 changes: 18 additions & 33 deletions INSTALLATION.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -58,68 +58,53 @@ Unlocking the new Rekordbox 6 `master.db` database file requires [SQLCipher][sql
To compile the amalgamation files, follow [this tutorial](http://www.jerryrw.com/howtocompile.php).


6. **Clone [pysqlcipher3] into any directory**
6. **Clone [sqlcipher3] into any directory**

````commandline
git clone https://github.com/rigglemania/pysqlcipher3
git clone https://github.com/coleifer/sqlcipher3
````


7. **Create directory ``...\pysqlcipher3\amalgamation``**
7. **Copy amalgamation files to the `sqlcipher3` directory**

Copy files ``sqlite3.c`` and ``sqlite3.h`` from the amalgamation directory from step 5
to the root of the ``sqlcipher3`` directory from step 6.


8. **Create directory ``...\pysqlcipher3\src\python3\sqlcipher``**

Copy files ``sqlite3.c``, ``sqlite3.h`` and ``sqlite3ext.h`` from the amalgamation directory from step 5


9. **Modify the ``...\pysqlcipher3\setup.py`` script (optional, see [this](https://stackoverflow.com/questions/65345077/unable-to-build-sqlcipher3-on-windows) discussion)**
8. **Modify the ``sqlcipher3/setup.py`` script (optional, see [this](https://stackoverflow.com/questions/65345077/unable-to-build-sqlcipher3-on-windows) discussion)**

If building the amalgamation fails, modify the ``setup.py`` script:
- Change
````python
def quote_argument(arg):
quote = '"' if sys.platform != 'win32' else '\\"'
return quote + arg + quote
````
to
````python
def quote_argument(arg):
quote = '"'
return quote + arg + quote
````

- The library names of OpenSSL have changed in version 1.1.0. If you are using a newer version, you might have to change the library name from
````python
ext.extra_link_args.append("libeay32.lib")
openssl_libname = os.environ.get('OPENSSL_LIBNAME') or 'libeay32.lib'
````
to
````python
ext.extra_link_args.append("<NAME>")
openssl_libname = os.environ.get('OPENSSL_LIBNAME') or '<NAME>'
````
where ``<NAME>`` is something like ``libcrypto.lib`` (depending on your version).


10. **Build using the amalgamation**
9. **Build using the amalgamation**

``cd`` into the ``pysqlcipher3`` directory and run
``cd`` into the ``sqlcipher3`` directory and run
````commandline
python setup.py build_amalgamation
python setup.py build_static build
````


11. **Install ``pysqlcipher3``**
10. **Install ``sqlcipher3``**

````commandline
python setup.py install
````

You now should have a working ``pysqlcipher3`` installation! The directory of the
cloned ``pysqlcipher3`` repo can be deleted after installing the package.
You now should have a working ``sqlcipher3`` installation! The directory of the
cloned ``sqlcipher3`` repo can be deleted after installing the package.

Steps 5-11 can be automated using the CLI of ``pyrekordbox``:
Steps 5-10 can be automated using the CLI of ``pyrekordbox``:
````commandline
> python3 -m pyrekordbox install-sqlcipher --help
usage: pyrekordbox install-sqlcipher [-h] [-t TMPDIR] [-l CRYPTOLIB] [-q] [-b]
Expand All @@ -128,13 +113,13 @@ usage: pyrekordbox install-sqlcipher [-h] [-t TMPDIR] [-l CRYPTOLIB] [-q] [-b]
-t TMPDIR, --tmpdir TMPDIR
Path for storing temporary data (default: '.tmp')
-l CRYPTOLIB, --cryptolib CRYPTOLIB
-q, --fixquote Don't fix the quotes in the pysqlcipher3 setup.py script
-b, --buildonly Don't install pysqlcipher3, only build the amalgamation
The name of the OpenSSl crypto libary (default: 'libcrypto.lib')
-b, --buildonly Don't install sqlcipher3, only build the amalgamation
````

After the installation SQLCipher-databases can be unlocked via the `pysqlcipher3` package:
````python
from pysqlcipher3 import dbapi2 as sqlite3
from sqlcipher3 import dbapi2 as sqlite3

conn = sqlite3.connect('test.db')
c = conn.cursor()
Expand Down Expand Up @@ -167,7 +152,7 @@ Make sure the `C_INCLUDE` and `LIBRARY_PATH` point to the installed SQLCipher pa

[VS]: https://visualstudio.microsoft.com/de/vs/community/
[OpenSSL]: https://slproweb.com/products/Win32OpenSSL.html
[pysqlcipher3]: https://github.com/rigglemania/pysqlcipher3
[sqlcipher3]: https://github.com/coleifer/sqlcipher3
[Pypi]: https://pypi.org/project/pyrekordbox/
[GitHub]: https://github.com/dylanljones/pyrekordbox
[sqlcipher]: https://www.zetetic.net/sqlcipher/open-source/
176 changes: 1 addition & 175 deletions docs/source/installation.md
View file
Original file line number Diff line number Diff line change
@@ -1,177 +1,3 @@
# Installation
```{include} ../../INSTALLATION.md
Pyrekordbox is available on [PyPI]:

```sh
$ pip install pyrekordbox
```

Alternatively, it can be installed via [GitHub]:

```shell
$ pip install git+https://github.com/dylanljones/pyrekordbox.git@VERSION
```

where `VERSION` is a branch, tag or release. The project can also be cloned/forked
and installed via

```shell
$ python setup.py install
```

## Installing SQLCipher

Unlocking the new Rekordbox 6 `master.db` database file requires [SQLCipher][sqlcipher].

### Windows


1. **Install [Visual Studio Community Edition][VS]**

Make sure to select all the GCC options (VC++, C++, etc) in the installation process.


2. **Install a prebuilt [OpenSSL binary][OpenSSL]**

Choose the latest Win32/Win64 version.


3. **Confirm that the `OPENSSL_CONF` environment variable is set properly in environment variables**

This should not be root openssl path (ex: `C:/Program Files/openssl-Win64`),
but instead should be the path to the config file, for example:
- 32-bit: ``C:/Program Files (x86)/openssl-Win32/bin/openssl.cfg``
- 64-bit: ``C:/Program Files/openssl-Win64/bin/openssl.cfg``


4. **Copy the openssl folder to the VC include directory (ex: `C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/include`)**

The openssl folder is
- 32-bit: `C:/Program Files (x86)/OpenSSL-Win32/include/openssl`
- 64-bit: `C:/Program Files/OpenSSL-Win64/include/openssl`

Confirm the following path exists `../../VC/include/openssl/aes.h`


5. **Download / compile the SQLCipher 3 amalgamation files**

Pre-built SQLCipher 3 amalgamation files can be downloaded from [this repo](https://github.com/geekbrother/sqlcipher-amalgamation).
To compile the amalgamation files, follow [this tutorial](http://www.jerryrw.com/howtocompile.php).


6. **Clone [pysqlcipher3] into any directory**

````shell
git clone https://github.com/rigglemania/pysqlcipher3
````


7. **Create directory ``...\pysqlcipher3\amalgamation``**

Copy files ``sqlite3.c`` and ``sqlite3.h`` from the amalgamation directory from step 5 to the new `amalgamation` directory.


8. **Create directory ``...\pysqlcipher3\src\python3\sqlcipher``**

Copy files ``sqlite3.c``, ``sqlite3.h`` and ``sqlite3ext.h`` from the amalgamation directory from step 5 to the new `sqlcipher` directory.


9. **Modify the ``...\pysqlcipher3\setup.py`` script (optional, see [this](https://stackoverflow.com/questions/65345077/unable-to-build-sqlcipher3-on-windows) discussion)**

If building the amalgamation fails, modify the ``setup.py`` script:
- Change
````python
def quote_argument(arg):
quote = '"' if sys.platform != 'win32' else '\\"'
return quote + arg + quote
````
to
````python
def quote_argument(arg):
quote = '"'
return quote + arg + quote
````

- The library names of OpenSSL have changed in version 1.1.0. If you are using a newer version, you might have to change the library name from
````python
ext.extra_link_args.append("libeay32.lib")
````
to
````python
ext.extra_link_args.append("<NAME>")
````
where ``<NAME>`` is something like ``libcrypto.lib`` (depending on your version).


10. **Build using the amalgamation**

``cd`` into the ``pysqlcipher3`` directory and run
````shell
python setup.py build_amalgamation
````


11. **Install ``pysqlcipher3``**

In the same directory, run
````shell
python setup.py install
````

You now should have a working ``pysqlcipher3`` installation! The directory of the
cloned ``pysqlcipher3`` repo can be deleted after installing the package.

Steps 5-11 can be automated using the CLI of ``pyrekordbox``:
````commandline
> python3 -m pyrekordbox install-sqlcipher --help

usage: pyrekordbox install-sqlcipher [-h] [-t TMPDIR] [-l CRYPTOLIB] [-q] [-b]

-h, --help show this help message and exit
-t TMPDIR, --tmpdir TMPDIR
Path for storing temporary data (default: '.tmp')
-l CRYPTOLIB, --cryptolib CRYPTOLIB
-q, --fixquote Don't fix the quotes in the pysqlcipher3 setup.py script
-b, --buildonly Don't install pysqlcipher3, only build the amalgamation
````

After the installation SQLCipher-databases can be unlocked via the `pysqlcipher3` package:
````python
from pysqlcipher3 import dbapi2 as sqlite3

conn = sqlite3.connect('test.db')
c = conn.cursor()
c.execute("PRAGMA key='password'")
````


### MacOS

For MacOS follow these steps:

1) Install [Homebrew](https://brew.sh) if you do not have it on your machine.
2) Install SQLCipher with `brew install SQLCipher`.
3) With the python environment you are using to run pyrekordbox active execute the following:
```shell
git clone https://github.com/rigglemania/pysqlcipher3
cd pysqlcipher3
SQLCIPHER_PATH=$(brew info sqlcipher | awk 'NR==4 {print $1; exit}'); C_INCLUDE_PATH="$SQLCIPHER_PATH"/include LIBRARY_PATH="$SQLCIPHER_PATH"/lib python setup.py build
SQLCIPHER_PATH=$(brew info sqlcipher | awk 'NR==4 {print $1; exit}'); C_INCLUDE_PATH="$SQLCIPHER_PATH"/include LIBRARY_PATH="$SQLCIPHER_PATH"/lib python setup.py install
```
Make sure the `C_INCLUDE` and `LIBRARY_PATH` point to the installed SQLCipher path. It may differ on your machine.


### References

- [Sqlcipher](https://www.zetetic.net/sqlcipher/open-source/)
- [https://stackoverflow.com/questions/33618565/how-to-build-sql-cipher-python-binding-for-windows](https://stackoverflow.com/questions/33618565/how-to-build-sql-cipher-python-binding-for-windows)
- [https://github.com/Monogi/pysqlcipher3_install_win10](https://github.com/Monogi/pysqlcipher3_install_win10)



[VS]: https://visualstudio.microsoft.com/de/vs/community/
[OpenSSL]: https://slproweb.com/products/Win32OpenSSL.html
[pysqlcipher3]: https://github.com/rigglemania/pysqlcipher3
[Pypi]: https://pypi.org/project/pyrekordbox/
[GitHub]: https://github.com/dylanljones/pyrekordbox
[sqlcipher]: https://www.zetetic.net/sqlcipher/open-source/
Loading

0 comments on commit 2b0a4f3

Please sign in to comment.