Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Infinite Status: 'JOB_STATE_FAILED' loop + duplicate tweets on relaunch #109

Open
cleterrier opened this issue Dec 17, 2024 · 7 comments
Open

Infinite Status: 'JOB_STATE_FAILED' loop + duplicate tweets on relaunch #109

cleterrier opened this issue Dec 17, 2024 · 7 comments

Comments

@cleterrier
Copy link

cleterrier commented Dec 17, 2024 •
edited
Loading

Hi,

I'm using version 0.14.1, I have the following .env (under Intel MacOS):

BLUESKY_USERNAME=christletgo.bsky.social
BLUESKY_PASSWORD=**password**
ARCHIVE_FOLDER=**path**
VIDEO_UPLOAD_RETRIES=10
TWITTER_HANDLES=christlet
IGNORE_VIDEO_ERRORS=1
MIN_DATE=2018-10-25T23:00:00.000Z

and use the npm run start_log command.

The log states at start:

Import started at 2024-12-17T07:54:56.821Z
Simulate is OFF
Import Reply is ON
Min Date is 2018-10-25T23:00:00.000Z
Max Date is OFF
API Delay is 2500ms
Archive Folder is **path**
Bluesky Username is christletgo.bsky.social
Ignore video errors? true
Video upload retries: 10
Strip Embed Image Metadata is false

Despite the VIDEO_UPLOAD_RETRIES=10 and IGNORE_VIDEO_ERRORS=1, within every batch of post creation (below the daily limit), I usually get into an infinite Status: JOB_STATE_FAILED loop in the log, which corresponds to an XRPCError in the Terminal (sorry I don't have the exact error on hand, but it is video-related). Here is how it starts in the log for a given tweet:

Parse tweet id '1598793299002892305'
 Created at 2022-12-02T21:36:41.000Z
 Full text 'Room upgrade with a view (at CNRS price!), already on a bike, I'm all set! https://t.co/kBmaGnfjNO'
 JobId: ctg0l3qga6cs72oec0c0
 Upload video, attempt: 1
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED

and it keeps going until I stop it.

Moreover, if I stop and relaunch the import (quitting terminal, relaunching it, restart with npm run start_log), it restarts the import but duplicates the posts. See for example this post duplicated five times over the last days: https://bsky.app/search?q=from%3A%40christletgo.bsky.social+%23BonneMere

I now have an account that is full of duplicates so I will have to nuke it and start fresh for ~30K tweets. Before doing that, I'd like to be sure that the Status: JOB_STATE_FAILED problem is solved. Thank you!
@cleterrier cleterrier changed the title Infinite Status: 'JOB_STATE_FAILED loop' + duplicate tweets on relaunch Infinite Status: 'JOB_STATE_FAILED' loop + duplicate tweets on relaunch Dec 17, 2024
@marcomaroni-github
Copy link
Owner

Hi @cleterrier please could you try with version 0.14.0?

@cleterrier
Copy link
Author

cleterrier commented Dec 17, 2024 •
edited
Loading

Hi @marcomaroni-github, I was using 0.12.0 when I started the import (about a month ago) and I had the same problem with both the infinite JOB_STATE_FAILED loop and the duplicate tweets creation. I updated to 0.14.1 on Dec 15 (transferring the .env and tweets_mapping.json files to the new install), hoping to resolve the problems, but to no avail. I just launch import with 0.14.0 in a similar way (after transferring the env and json files), will report what happens. For now it seems to be still creating duplicate posts (which might be logical if each time the script is killed by quitting the terminal the json file is not updates?), let's see if the JOB_STATE_FAILED loop appears.

@cleterrier
Copy link
Author

0.14.0 stopped on the following error:

/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/xrpc/dist/xrpc-client.js:82
                    throw new types_1.XRPCInvalidResponseError(methodNsid, e, resBody);
                          ^

XRPCInvalidResponseError: The server gave an invalid response and may be out of date.
    at AtpAgent.call (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/xrpc/dist/xrpc-client.js:82:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async main (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/app.js:844:70) {
  error: 'Invalid Response',
  headers: undefined,
  success: false,
  status: 2,
  lexiconNsid: 'app.bsky.video.getJobStatus',
  validationError: ValidationError: Output/jobStatus/progress can not be greater than 100
      at integer (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validators/primitives.js:126:24)
      at validate (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validators/complex.js:13:45)
      at validateOneOf (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validators/complex.js:167:15)
      at object (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validators/complex.js:109:31)
      at validate (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validators/complex.js:23:20)
      at validateOneOf (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validators/complex.js:167:15)
      at object (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validators/complex.js:109:31)
      at validateOneOf (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validators/complex.js:166:15)
      at assertValidOneOf (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validation.js:44:45)
      at assertValidXrpcOutput (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validation.js:32:16),
  responseBody: {
    jobStatus: {
      did: 'did:plc:og2kvr2eqd6so7ixfgjagwxc',
      jobId: 'ctgm0vi0ours72v0dr2g',
      progress: 104,
      state: 'JOB_STATE_ENCODING'
    }
  },
  [cause]: ValidationError: Output/jobStatus/progress can not be greater than 100
      at integer (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validators/primitives.js:126:24)
      at validate (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validators/complex.js:13:45)
      at validateOneOf (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validators/complex.js:167:15)
      at object (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validators/complex.js:109:31)
      at validate (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validators/complex.js:23:20)
      at validateOneOf (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validators/complex.js:167:15)
      at object (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validators/complex.js:109:31)
      at validateOneOf (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validators/complex.js:166:15)
      at assertValidOneOf (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validation.js:44:45)
      at assertValidXrpcOutput (/Users/christo/Desktop/X/twitter-to-bluesky-0.14.0/node_modules/@atproto/lexicon/dist/validation.js:32:16)
}

Node.js v22.11.0

I am relaunching it

@cleterrier
Copy link
Author

cleterrier commented Dec 17, 2024 •
edited
Loading

After relaunch, I hit the JOB_STATE_FAILED error on another tweet with 0.14.0. Nothing seems to appear in the Terminal, full log is below. Not sure why it says 0.12.0 in the first line, I unzipped 0.14.0, did npm install and npm run start_log in that folder, I don't think the fact that I had prior install and run in folders containing 0.12.0 and 0.14.1 should matter?

> [email protected] start_log
> npx tsc && node app.js >> import.log

(node:9457) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
 Video job status: 'already_exists'. Video will not be posted.
Error fetching embed URL card: Timeout
Error fetching embed URL card: request to https://events.tum.de/frontend/index.php?sub=183/ failed, reason: getaddrinfo ENOTFOUND events.tum.de
 Video job status: 'already_exists'. Video will not be posted.
 Video job status: 'already_exists'. Video will not be posted.
 Video job status: 'already_exists'. Video will not be posted.
Error fetching embed URL card: request to https://carrieres.cnrs.fr/ failed, reason: unable to verify the first certificate
 Video job status: 'already_exists'. Video will not be posted.
 Video job status: 'already_exists'. Video will not be posted.
 Video job status: 'already_exists'. Video will not be posted.
Error fetching embed URL card: HTTP error: 429 Too Many Requests
 Video job status: 'already_exists'. Video will not be posted.
 Video job status: 'already_exists'. Video will not be posted.
 Video job status: 'already_exists'. Video will not be posted.
Error fetching embed URL card: HTTP error: 403 Forbidden
 Video job status: 'already_exists'. Video will not be posted.
Error fetching embed URL card: HTTP error: 403 Forbidden
 Video job status: 'video duration (74.07 sec) is longer than the maximum allowed duration (60 sec)'. Video will not be posted.
Error fetching embed URL card: HTTP error: 404 Not Found
Error fetching embed URL card: HTTP error: 403 Forbidden
Error fetching embed URL card: HTTP error: 403 Forbidden
Error fetching embed URL card: HTTP error: 403 Forbidden
Error fetching embed URL card: HTTP error: 403 Forbidden
Error fetching embed URL card: HTTP error: 403 Forbidden
Error fetching embed URL card: HTTP error: 403 Forbidden

The import.log says for the problematic tweet:

Parse tweet id '1700232001502969984'
 Created at 2023-09-08T19:37:52.000Z
 Full text 'EM of the brain at macro scale https://t.co/NZbvPgzv5s'
 JobId: ctgmpea0ours72v0e2ng
 Upload video, attempt: 1
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED 
  Status: JOB_STATE_FAILED

and so on.

@cleterrier
Copy link
Author

cleterrier commented Dec 18, 2024 •
edited
Loading

btw I found why the log says 0.12 even for the 0.14.0 and 0.14.1 version, it's hardcoded here:

twitter-to-bluesky/package.json

Line 3 in 591500f

"version": "0.12.0",

@marcomaroni-github
Copy link
Owner

btw I found why the log says 0.12 even for the 0.14.0 and 0.14.1 version, it's hardcoded here:

twitter-to-bluesky/package.json

Line 3 in 591500f

"version": "0.12.0",

Yes right I forgot to change the version in the package.json

@cleterrier
Copy link
Author

Anything I can do to address the bug itself (JOB_STATE_FAILED loop even with a VIDEO_UPLOAD_RETRIES parameter)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cleterrier @marcomaroni-github