migrate-data.md

Migrate Hit Data from Filesystem to PostgreSQL

The Node.js MVP saves data to the filesystem because that was the simplest way of storing data without having to manage a database. See: hits-nodejs/lib/db_filesystem.js#L26-L68 #81

• Log-in to the Linode instance and inspect how many directories have been created (this is the number of people - GitHub usernames - using the hits badge)

ssh [email protected]

• Count the number of log files in the /logs directory:

ls | wc -l
787

• zip the data on Linode instance to get it off the instance - because I will be running the Elixir script on my localhost. https://unix.stackexchange.com/questions/93139/zip-an-entire-folder-using-gzip

cd hits
tar -zcvf logs.tar.gz logs/

• download the data archive to localhost

https://stackoverflow.com/questions/9427553/how-to-download-a-file-from-server-using-ssh

scp [email protected]:hits/logs.tar.gz ./logs.tar.gz

• unzip data on localhost

tar -zxvf logs.tar.gz

Note: the zip function did not do what I needed, so I decided to attempt a recursive remote copy using scp:

scp -r [email protected]:/root/hits/logs /Users/n/code/hits

see: https://stackoverflow.com/questions/11304895/copy-remote-folder-to-local-using-scp

But that failed half way through because my Mac went to sleep ... So I decided to try rsync instead:

rsync -vh [email protected]:/root/hits/logs/* /Users/n/code/hits
// -vh means verbose & human-readable respectively

That works fine because it's only 489mb But attempting to get the agents is another story:

rsync -vh [email protected]:/root/hits/logs/agents/* /Users/n/code/hits/logs/agents

error:

bash: /usr/bin/rsync: Argument list too long

https://www.tecmint.com/rsync-local-remote-f ile-synchronization-commands/

• write the script
  • insert data into hits_dev PostgreSQL on localhost

Path.wildcard("logs/*.log")

• run the script

MIX_ENV=dev mix run priv/migrate-data.exs

• export the data from hits_dev PostgreSQL on localhost

• load the data on the remote server