Change to Twitter archive format.

[garrettc]

It looks like Twitter have changed the format or their archives (possibly related to #227 ?):

$ ./manage.py import_twitter_tweets --path=/Users/garrettc/temp/twitter-2022-02-06/

CommandError: Expected to find a directory at '/Users/garrettc/temp/twitter-2022-02-06/data/js/tweets' containing JSON files

<archive_root>/data now contains a whole bunch of js files, with a single tweet.js file that contains your public timeline and a corresponding tweet_media directory with the associated media.

As a quick test in the vain hope that they'd maybe only changed the directory structure I moved tweet.js into a js/tweets/ directory and re-ran the command, but it bails out with:

Traceback (most recent call last):
  File "/Users/garrettc/sandbox/polytechnic/env/lib/python3.9/site-packages/ditto/twitter/ingest.py", line 102, in _get_data_from_file
    tweets_data = json.loads("".join(lines))
  File "/opt/homebrew/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/opt/homebrew/Cellar/[email protected]/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 25 column 4 (char 676)

So I'm guessing there are more fundemental changes afoot here. I don't have an example of an old archive to narrow down what else has changed.

The "new" tweet.js structure is as follows:

window.YTD.tweet.part0 = [
  {
    "tweet" : {
      "retweeted" : false,
      "source" : "<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>",
      "entities" : {
        "hashtags" : [ ],
        "symbols" : [ ],
        "user_mentions" : [ ],
        "urls" : [ ]
      },
      "display_text_range" : [
        "0",
        "68"
      ],
      "favorite_count" : "0",
      "id_str" : "200873",
      "truncated" : false,
      "retweet_count" : "0",
      "id" : "200873",
      "created_at" : "Fri Nov 24 21:02:53 +0000 2006",
      "favorited" : false,
      "full_text" : "Some random string of words",
      "lang" : "en"
    }
  }
]

(On the upsidea, the Pinboard and Last.fm imports went perfectly. I'm having some issues with Flickr, but I think that's to do with the amount of photos I have. I'm doing some more investigating and then I'll report back.)

[philgyford]

Thanks - I'll have a look.

I did have an issue with importing my Flickr photos when running it on a Digital Ocean 512MB Droplet although it worked OK on my MacBook. (#148) If the issue is something like that I guess that adding an option to continue an import from whatever point it failed might be the simplest work around (assuming I have no idea how to fix the memory issue). But we can continue that discussion on that issue if necessary.

Glad to hear Pinboard and Last.fm went OK!

[philgyford]

The twitter archive format is remarkably different from what it was when I wrote the code. Here's a screenshot of the old download:

And here's the new one (I think that tweet.js file contains all a user's tweets; mine is 25.6MB):

The old month-dated .js files used to start like this:

Grailbird.data.tweets_2006_11 = 
 [ {
  "source" : "\u003Ca href=\"http:\/\/twitter.com\" rel=\"nofollow\"\u003ETwitter Web Client\u003C\/a\u003E",
  "entities" : {
    "user_mentions" : [ ],
    "media" : [ ],
    "hashtags" : [ ],
    "urls" : [ ]
  },
  "geo" : { },
  "id_str" : "467973",
  "text" : "Woke up one minute before the alarm, and now feel unusually awake.",
  "id" : 467973,
  "created_at" : "2006-11-30 00:00:00 +0000",
  "user" : {
    "name" : "Phil Gyford",
    "screen_name" : "philgyford",
    "protected" : false,
    "id_str" : "12552",
    "profile_image_url_https" : "https:\/\/pbs.twimg.com\/profile_images\/1167616130\/james_200208_300x300_normal.jpg",
    "id" : 12552,
    "verified" : false
  }
},

And the new tweet.js file starts like this:

window.YTD.tweet.part0 = [
  {
    "tweet" : {
      "retweeted" : false,
      "source" : "<a href=\"http://www.cloudhopper.com/\" rel=\"nofollow\">Twitter SMS</a>",
      "entities" : {
        "hashtags" : [ ],
        "symbols" : [ ],
        "user_mentions" : [ ],
        "urls" : [ ]
      },
      "display_text_range" : [
        "0",
        "35"
      ],
      "favorite_count" : "0",
      "id_str" : "71534",
      "truncated" : false,
      "retweet_count" : "0",
      "id" : "71534",
      "created_at" : "Fri Nov 17 16:47:58 +0000 2006",
      "favorited" : false,
      "full_text" : "Supermarket so bright hurt my eyes.",
      "lang" : "en"
    }
  },

Anyway, just a very quick cursory look so far.

[philgyford]

@garrettc OK, hopefully import_twitter_tweets will work with the new format now!

During all this I've realised that your Twitter dev account will need to apply for "Extended access" in order access the v1.1 API, which is what Ditto uses, for new Apps. I've no idea how long that takes to happen.

Let me know how the import goes!

[garrettc]

I just put in my request for elevated access, and they approved it straight away! (I must have a kind face), so I'll start testing the new importer.

I've started work on the Flickr importer improvements, I had to take a bit of time to get my head around ArgumentParser and mutually exclusive arguments, but I think I've got something that might not destroy the universe. More on that later.

[garrettc]

Worked perfectly!

(env) [garrettc - 12:45] polytechnic $ python manage.py import_twitter_tweets --path=/Users/garrettc/temp/twitter-2022-02-06
Imported 32011 tweets from 1 file, and 1043 media files
(env) [garrettc - 12:50] polytechnic $

[philgyford]

Excellent!