429 error

[joshjohanning]

Guessing southwest changed something with their API?

user@linuxvm:~$                        Caught error from revenue or international trip
Caught error from revenue or international trip
Caught error from revenue or international trip
INFO:root:itinerary original total price: 27941
{
    "code": 429999999,

    "message": "Error.",

    "messageKey": "ERROR",

    "httpStatusCode": "BAD_REQUEST",

    "requestId": "",

    "infoList": []
   }
Traceback (most recent call last):
  File "/app/southwestalerts/app.py", line 89, in <module>
    check_for_price_drops(user.username, user.password, user.email)
  File "/app/southwestalerts/app.py", line 39, in check_for_price_drops
    destination_airport
  File "/app/southwestalerts/southwest.py", line 62, in get_available_flights
    return self._session.get(url)
  File "/app/southwestalerts/southwest.py", line 80, in get
    return self._parsed_response(resp, success_codes=success_codes)
  File "/app/southwestalerts/southwest.py", line 101, in _parsed_response
    raise Exception('Invalid status code received. Expected {}. Received {}.'.format(success_codes, response.status_code))
Exception: Invalid status code received. Expected [200]. Received 429.

[rogergzou]

Also was playing around with the API and have no idea how to fix. I think the request headers have changed. For example, I think the API_KEY has changed from l7xx12ebcbc825eb480faa276e7f192d98d1 to l7xx0a43088fe6254712b10787646d1b298e, based on looking into network requests made on mobile.southwest.com.

[infinite-monkeys]

Changing the API key around makes it fail in a different way. The mobile southwest site now uses a /v2 oauth-endpoint. There's some headers that seem to be required, that I'm not sure where they come from: X-5Ku220Jw-a (which is really giant), X-5Ku220Jw-b, X-5Ku220Jw-c, X-5Ku220Jw-d, X-5Ku220Jw-uniqueStateKey... removing these from the request my browser made gives a BAD_REQUEST response.

The current mobile site seems to be "non-standard" HTML with no real 'buttons' and 'links' but naviagation areas that are tappable/clickable. This might make it harder to write a different sort of scraper.

[Fffrank]

https://github.com/byalextran/luvwrapper seems to have gotten around this. I'm not much of a programmer but it appears they are submitting a JSON request rather than an HTML request and then parsing the results?

[rogergzou]

luvwrapper doesn't run into problems because it never tries to hit the Flights for X to Y on Z date endpoint. From my reading of https://github.com/byalextran/luvwrapper/blob/master/lib/luvwrapper.rb#L72, it makes the same GET and POST style requests this app uses. Correct me if I'm wrong.

Perhaps it might be easier to even write a beautiful soup / selenium webdriver based scraper.

[infinite-monkeys]

luvwrapper likely does similar things as this software, and based on its last updated dates, I wouldn't be surprised if that also was broken by this change. Re: selenium based scraper, I was thinking along those lines, hence my mention of "non-standard HTML" -- this might be part of some anti-scraping code to keep the OTAs and such out, in which case it would also be more difficult for us.

[hoopsbwc34]

Selenium isn't working now either... seems they are checking for selenium related variables or something and it fails to process the request

[hildebrau]

So, catch me up. Does this script work at all for anyone? Is there a new solution for doing personal price checks (other than manually)? I guess this container I was running stopped working a long time ago. ;(

Thanks!

[joshjohanning]

@hildebrau , no, believe it is giving 400 errors to everyone since Southwest changed some of their api's.

[Fffrank]

I paid a reddit developer to update this script and have it working. Going to wait a few days and make sure it still works (we are also working on adding functionality to it -- international itens are def not going to work but it should be able to scan for flights paid with dollars.

@xur17 are you alive and/or still interested in continuing development?

[hoopsbwc34]

I paid a reddit developer to update this script and have it working. Going to wait a few days and make sure it still works (we are also working on adding functionality to it -- international itens are def not going to work but it should be able to scan for flights paid with dollars.

@xur17 are you alive and/or still interested in continuing development?

Can you point us to your fork when you get the changes merged in? I expect many others will be interested in how this problem was solved. Thanks!

[hildebrau]

Nice! Looking forward to trying it out!

…

On Wed, Jul 17, 2019, 6:28 PM hoopsbwc34 ***@***.***> wrote: I paid a reddit developer to update this script and have it working. Going to wait a few days and make sure it still works (we are also working on adding functionality to it -- international itens are def not going to work but it should be able to scan for flights paid with dollars. @xur17 <https://github.com/xur17> are you alive and/or still interested in continuing development? Can you point us to your fork when you get the changes merged in? I expect many others will be interested in how this problem was solved. Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16?email_source=notifications&email_token=AAUHSPJCSORK7XYVAPX4WV3P76TJFA5CNFSM4G4WZAGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2G34GQ#issuecomment-512605722>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUHSPJM3ZVNPTHT54CUNETP76TJFANCNFSM4G4WZAGA> .

[pyro2927]

Seems like Southwest is really cracking down on this stuff. I was digging into the web interface to see if there is a way to mimic their authentication API calls on the web, but it appears it too is executing a crap ton of Javascript to create secret request headers. Anyone able to decipher swa-common.js?

[Fffrank]

http://jsnice.org/ does a fine job of deobfuscating this code (not that it makes any sense to me.) The headers that are required are not being computed or generated in code -- they appear to be static -- but they are being changed every few days. I still haven't figured out where they are coming from but I am getting closer to converting this script to use pyppeteer to launch headless chrome to pull the required headers at the beginning of each session.

[Fffrank]

Alright everyone, my script is now running in python (I'll try to find time to test if the docker library can be built, will probably have to add some chromium dependencies but we will see.)

https://github.com/Fffrank/southwest-alerts

Anyone care to test it out? Still doesn't work for international fares but does a much better job picking up paid fares in addition to award fares. I've tested it with several bookings both with and w/o companion's booked and it seems to work. Feel free to drop me feedback over on that repo.

[Fffrank]

Docker now works correctly, too -- docker pull fffrank/southwest-alerts

[hildebrau]

@Fffrank , I just tried it out. It looks super promising, but it is throwing some tracebacks.
Any clues of things I should try?

docker@hoth:~/docker-server$ docker logs southwest-alerts
[I:pyppeteer.launcher] Browser listening on: ws://127.0.0.1:59997/devtools/browser/933e6dad-0dad-46e8-a2a4-28c60b62e38f
[I:pyppeteer.launcher] terminate chrome process...
INFO:root:Processing: XXXXXB
INFO:root:itinerary original total price: 16956
INFO:root:fare type 0 is sold out
INFO:root:Price increase of 27426 Points detected for flight XXXXXB from AUS to DCA on 2019-11-23
INFO:root:Processing: XXXXXC
INFO:root:itinerary original total price: $184
INFO:root:Price increase of 0 Dollars detected for flight XXXXXC from BWI to MSP on 2019-11-27
INFO:root:Processing: XXXXXD
INFO:root:itinerary original total price: $657
Traceback (most recent call last):
File "/app/southwestalerts/app.py", line 175, in
check_for_price_drops(user.username, user.password, user.email, user.headers)
File "/app/southwestalerts/app.py", line 129, in check_for_price_drops
for faretype, fare in enumerate(matching_flight['fares']):
TypeError: 'NoneType' object is not iterable
[I:pyppeteer.launcher] Browser listening on: ws://127.0.0.1:57775/devtools/browser/d8c44512-df28-40a2-953e-d3ca7e8f41d3
[I:pyppeteer.launcher] terminate chrome process...
INFO:root:Processing: XXXXXB
INFO:root:itinerary original total price: 16956
INFO:root:fare type 0 is sold out
INFO:root:Price increase of 27426 Points detected for flight XXXXXB from AUS to DCA on 2019-11-23
INFO:root:Processing: XXXXXC
INFO:root:itinerary original total price: $184
INFO:root:Price increase of 0 Dollars detected for flight XXXXXC from BWI to MSP on 2019-11-27
INFO:root:Processing: XXXXXD
INFO:root:itinerary original total price: $657
Traceback (most recent call last):
File "/app/southwestalerts/app.py", line 175, in
check_for_price_drops(user.username, user.password, user.email, user.headers)
File "/app/southwestalerts/app.py", line 129, in check_for_price_drops
for faretype, fare in enumerate(matching_flight['fares']):
TypeError: 'NoneType' object is not iterable

In this case XXXXXD is a flight from AUS to HNL on 12/20, and all available flights for that route and day are in the SOLD OUT status. That may require some extra logic that the script is mishandling.

[hildebrau]

I'm also seeing some 429 responses, which I think means we're hitting their site too quickly. The line numbers may be off a bit, as I've been adding some debug logging.

INFO:root:itinerary original total price: 16956
{
"code": 429999999,

"message": "Error.",

"messageKey": "ERROR",

"httpStatusCode": "BAD_REQUEST",

"requestId": "",

"infoList": []

}
Traceback (most recent call last):
File "/app/southwestalerts/app.py", line 175, in
check_for_price_drops(user.username, user.password, user.email, user.headers)
File "/app/southwestalerts/app.py", line 92, in check_for_price_drops
destination_airport
File "/app/southwestalerts/southwest.py", line 72, in get_available_flights
return self._session.get(url)
File "/app/southwestalerts/southwest.py", line 102, in get
return self._parsed_response(resp, success_codes=success_codes)
File "/app/southwestalerts/southwest.py", line 148, in _parsed_response
'Invalid status code received. Expected {}. Received {}.'.format(success_codes, response.status_code))
Exception: Invalid status code received. Expected [200]. Received 429.

[Fffrank]

429 has generally meant there's a header missing (I've tried adding in 10 second delays and it still errors out occasionally.) I haven't figured out exactly what's causing it because it's not repeatable on my end.

The first issue is indeed because the flight is sold out across all booking buckets. Should be an easy fix (but a difficult test on my end since I don't have a booking with that problem (and it's hard to create one that's sold out!) I've tried to fix it -- want to try pulling and running fffrank/southwest-alerts-test via docker

[hildebrau]

I ran it, and I'm getting a larger output. I feel like it's having a different problem. Here is the error I saw. I'll try it a few more times afterwards, so keep scrolling.

docker@hoth:~/docker-server$ docker logs -f southwest-alerts
[I:pyppeteer.launcher] Browser listening on: ws://127.0.0.1:51662/devtools/browser/a93b2385-e8d7-4941-84ba-776278f02260
[I:pyppeteer.connection] connection closed
Traceback (most recent call last):
File "/app/southwestalerts/app.py", line 180, in
user.headers = loop.run_until_complete(login_get_headers(mobile_url, user.username, user.password))
File "/usr/local/lib/python3.5/asyncio/base_events.py", line 466, in run_until_complete
return future.result()
File "/usr/local/lib/python3.5/asyncio/futures.py", line 293, in result
raise self._exception
File "/usr/local/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/app/southwestalerts/app.py", line 38, in login_get_headers
await page.waitForSelector(selector)
File "/usr/local/lib/python3.5/site-packages/pyppeteer/frame_manager.py", line 552, in await
result = (yield from self.promise)
File "/usr/local/lib/python3.5/asyncio/futures.py", line 380, in iter
yield self # This tells Task to wait for completion.
File "/usr/local/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
future.result()
File "/usr/local/lib/python3.5/asyncio/futures.py", line 293, in result
raise self._exception
File "/usr/local/lib/python3.5/site-packages/pyppeteer/frame_manager.py", line 573, in rerun
success = (await context.evaluateHandle(waitForPredicatePageFunction, self._predicateBody, self._polling, self._timeout, *self._args))
File "/usr/local/lib/python3.5/site-packages/pyppeteer/execution_context.py", line 89, in evaluateHandle
_rewriteError(e)
File "/usr/local/lib/python3.5/site-packages/pyppeteer/execution_context.py", line 217, in _rewriteError
raise error
File "/usr/local/lib/python3.5/site-packages/pyppeteer/execution_context.py", line 86, in evaluateHandle
'userGesture': True,
File "/usr/local/lib/python3.5/asyncio/futures.py", line 380, in iter
yield self # This tells Task to wait for completion.
File "/usr/local/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
future.result()
File "/usr/local/lib/python3.5/asyncio/futures.py", line 293, in result
raise self._exception
pyppeteer.errors.NetworkError: Protocol error Runtime.callFunctionOn: Target closed.
[I:pyppeteer.launcher] terminate chrome process...
ERROR:asyncio:Future exception was never retrieved
future: <Future finished exception=ConnectionClosed('WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason',)>
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/websockets/protocol.py", line 674, in transfer_data
message = yield from self.read_message()
File "/usr/local/lib/python3.5/site-packages/websockets/protocol.py", line 742, in read_message
frame = yield from self.read_data_frame(max_size=self.max_size)
File "/usr/local/lib/python3.5/site-packages/websockets/protocol.py", line 815, in read_data_frame
frame = yield from self.read_frame(max_size)
File "/usr/local/lib/python3.5/site-packages/websockets/protocol.py", line 884, in read_frame
extensions=self.extensions,
File "/usr/local/lib/python3.5/site-packages/websockets/framing.py", line 99, in read
data = yield from reader(2)
File "/usr/local/lib/python3.5/asyncio/streams.py", line 666, in readexactly
raise IncompleteReadError(incomplete, n)
asyncio.streams.IncompleteReadError: 0 bytes read on a total of 2 expected bytes

The above exception was the direct cause of the following exception:

websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason

Yep. I ran it again, and it ended in this way. I didn't notice any different output w/ your -test version. Did I do something wrong?

...[snip]...
INFO:root:itinerary original total price: $657
Traceback (most recent call last):
File "/app/southwestalerts/app.py", line 181, in
check_for_price_drops(user.username, user.password, user.email, user.headers)
File "/app/southwestalerts/app.py", line 132, in check_for_price_drops
for faretype, fare in enumerate(matching_flight['fares']):
TypeError: 'NoneType' object is not iterable

I understand the difficulty in testing for a sold out test case. I'm happy to run whatever you throw at me to debug this one to death. It'll be on my account until Dec 20th. ;) Obviously sooner the better on a fix though!
Thank you!

[Fffrank]

The first error happens when the browser fails to login. Sometimes southwest just stalls and spins the loading icon on the login page (and I haven't figured out a way to detect/retry this.)

The second error seems exactly the same as before which is puzzling. I can add a debug line that will dump the relevant dictionary/variables and you can send them to me (you may just need to spend some time to scrub your personal info.) Would that work?

I probably won't have any time to work on this until Monday so it's good thing you're not in a huge hurry. ;)

[hildebrau]

That will work. I tried to log some debug info; but failed when I kept hitting the 429 error in the same docker container session after a while. I felt like 429 translated to "you're hitting us too often, so chill out". I ran bash inside the container, and installed vim via apt-get so I could edit. But once I saved changes, it seemed that I had to run the script within that container. If I started the container with docker again, it didn't seem to see my changes. I'm still a docker newb though; so I'm not sure what to expect in that scenario.

Anyway, I'm happy to help and happy to have help. I can certainly scrub the output as needed.

[Fffrank]

Right -- docker containers are built on "images" and this one doesn't have any external volumes (so outside of logging, it's really kind of on an island.) You're braver than I am if you're using vim, though -- I'm a medium newb at docker and python and open up vim and can't figure out how to exit fast enough.

[hildebrau]

Oh. hah. vim is an appendage to me. I don't have to even think when I'm using it. But I get where you're coming from. hah.

[Fffrank]

@hildebrau I updated the fffrank/southwest-alerts-test image to log many more details. If you want to put it in a pastebin or a gist I'd be happy to look at it and see if I can fix your error.

[hildebrau]

@Fffrank sorry for the delay. I finally had some time to try this one out. Debug output is here: https://gist.github.com/hildebrau/8e9e897f64979b05df40d6fce459a717

I fear it's crapping out before a useful debug output on that last record locator. But you tell me.

[Fffrank]

@hildebrau You're correct but you leaving the URL at the bottom seemed like a big help. I've updated the script (kind of shooting in the dark here) But feel free to do a new docker pull and see if I've fixed it. ;)

[hildebrau]

@Fffrank here is the new debug. Definitely closer!

https://gist.github.com/hildebrau/579827321a4a77dd937fe184e46d6a43

[Fffrank]

@hildebrau I'm going to consider that fixed except for my crappy python syntax! I'm confident that I can easily fix the logging error. Harder is why it looped back to the first confirmation number a second time..... Your paste isn't two executions, is it??

[hildebrau]

It's one execution.. but I think it found another sold out flight perhaps.. (hawaii over Christmas time..)
let me know if you have a fix to try for the python exceptions. I did just save $48 & 3400 points on my HNL->OGG leg for two people. Thanks!

[Fffrank]

@hildrebrau Give it another shot. Fingers crossed that fixes it!

And if anyone else can find anywhere on the southwest site or via API where it lists how many points/dollars you've paid for an international flight that is one feature that I would LOVE to add (but haven't found anywhere except the original booking email -- but I could add a command line feature to manually include that flight w/ the confirmation number. I'm thinking on it.)

[hildebrau]

@Fffrank it got rid of the nasty parse errors! this time it ran through my wife's account too, and it failed on something. Maybe it just failed to get a proper API key on that run through?

https://gist.github.com/hildebrau/dfea0175fb305a0b568cd572a75c3277
Maybe you can add a check for a DEBUG environment variable that we can add to our docker environment to get it to dump more debug like the first test build you did for me.

I'd steer clear of command line options since we run this as a docker container mostly.. environment variables perhaps? I don't have any SWA international flights to test with. Sorry! Nor the points to buy one. ;(

[Fffrank]

@hildebrau Cool! I pushed those testing changes into the master branch so you can switch back to the main docker if you'd like. I'll experiment with some logic to make sure that the headers are properly generated before it moves on. That's def the problem with your second run (I have that issue about 5% of the time but it seems to come and go.) Shouldn't be that hard of a fix but it is difficult to test since it's not easily repeatable.

[hildebrau]

@Fffrank, could you tell me how I can enable debug messages that could help determine why my flights that are booked through my work's travel agency software are showing as $239 drop. The reservations can't be changed via the Southwest website. It shows a message about having to call in to make a change.

Ever since I added my rapid rewards number to these itineraries, I get alerts from this script about huge price drops.

[Weizilla]

@Fffrank does your script work for just checking prices? As in, first log in then call get_available_flights_dollars for various parameters without having any trips booked with the account. I tried running it that way both in and out of docker but got error code 429 every time.

[Fffrank]

@hildebrau As I'm sure you've noticed the script is broken again and I can't seem to get it to work. If you'd like a go at debugging it, it's probably easiest at this point if you install pycharms and set breakpoints to try and debug. It has a great way you can mark/watch variables. I had never used it before this project and somehow made it work (however briefly.)

@Weizilla Does not have that functionality but would be relatively easy to implement (if I had the desire to do so and if the script was currently functional.)

[Weizilla]

@Fffrank Sounds good. I wasn't looking for whole app to handle that use case, just confirming that the relevant parts in southwest.py could technically do it (or used to before all the 429 errors). Thanks!

[KevinLinghu]

Still keep running into 429 error. Just submitted a new issue here: #17 (comment)

[Fffrank]

If anyone is lurking here -- I have updated my test branch and it's currently working again for me. Feel free to try it out and let me know? Would like one more confirmation before I merge it and push a new docker image.

[hildebrau]

I'm lurking. I was going to try to test it out; but I don't have any scheduled flights. My companion pass expired, so I think I'm done having a ton of SWA flights scheduled. ;( I will give it a shot to see if it can at least log in and see that I have no flights to search for in a clean fashion. I need to wait for my bad password attempts to expire, though.

[hildebrau]

@Fffrank , I tested it and it threw some errors.. mind you, I don't have any flights reserved at the moment.

https://pastebin.com/5FiRP9GW
Thanks for keeping at this! Much appreciated!

[hoopsbwc34]

Update. I think as long as I wait for the login cookie to expire it runs fine. Trying to run the script back to back brings 429 errors immediately. I still get random 429 errors but it’s once in every 5 or 10 times I run it now. Might be worth some retry logic in the future on 429 rather than killing the script. Also added texting through twilio and config file based setting of variables instead of using environment variables (just a personal preference). Thanks!

…

On Sat, Jan 11, 2020 at 6:00 PM hildebrau - ***@***.*** ***@***.***> wrote: @Fffrank <https://github.com/Fffrank> , I tested it and it threw some errors.. mind you, I don't have any flights reserved at the moment. https://pastebin.com/5FiRP9GW Thanks for keeping at this! Much appreciated! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#16?email_source=notifications&email_token=ABZ4Z3Q2WG4XHDGMW57DFPTQ5JTTRA5CNFSM4G4WZAGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIWOYZA#issuecomment-573369444>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZ4Z3RKANI6ADVBU4J52ULQ5JTTRANCNFSM4G4WZAGA> .