Unable to Connect to ETrade API

[pmiller1414]

from utils.list_accounts import get_account_list

DevTools listening on ws://127.0.0.1:57601/devtools/browser/d4f396ab-ebc0-47a3-982f-9100639aa686
Exception details: Message: no such element: Unable to locate element: {"method":"css selector","selector":"#sendOTPCodeBtn"}
(Session info: chrome-headless-shell=120.0.6099.225)
Stacktrace:
GetHandleVerifier [0x00CF6EE3+174339]
(No symbol) [0x00C20A51]
(No symbol) [0x00936FF6]
(No symbol) [0x00969876]
(No symbol) [0x00969C2C]
(No symbol) [0x0099BD42]
(No symbol) [0x00987054]
(No symbol) [0x0099A104]
(No symbol) [0x00986DA6]
(No symbol) [0x00961034]
(No symbol) [0x00961F8D]
GetHandleVerifier [0x00D94B1C+820540]
sqlite3_dbdata_init [0x00E553EE+653550]
sqlite3_dbdata_init [0x00E54E09+652041]
sqlite3_dbdata_init [0x00E497CC+605388]
sqlite3_dbdata_init [0x00E55D9B+656027]
(No symbol) [0x00C2FE6C]
(No symbol) [0x00C283B8]
(No symbol) [0x00C284DD]
(No symbol) [0x00C15818]
BaseThreadInitThunk [0x760E7BA9+25]
RtlInitializeExceptionChain [0x773EBD2B+107]
RtlClearBits [0x773EBCAF+191]

ConnectionError: Trying again with chrome.

DevTools listening on ws://127.0.0.1:57721/devtools/browser/c1357f15-735a-4c0b-a3e4-dd9fcf47622d
Exception details: Message: no such element: Unable to locate element: {"method":"css selector","selector":"#sendOTPCodeBtn"}
(Session info: chrome-headless-shell=120.0.6099.225)
Stacktrace:
GetHandleVerifier [0x00CF6EE3+174339]
(No symbol) [0x00C20A51]
(No symbol) [0x00936FF6]
(No symbol) [0x00969876]
(No symbol) [0x00969C2C]
(No symbol) [0x0099BD42]
(No symbol) [0x00987054]
(No symbol) [0x0099A104]
(No symbol) [0x00986DA6]
(No symbol) [0x00961034]
(No symbol) [0x00961F8D]
GetHandleVerifier [0x00D94B1C+820540]
sqlite3_dbdata_init [0x00E553EE+653550]
sqlite3_dbdata_init [0x00E54E09+652041]
sqlite3_dbdata_init [0x00E497CC+605388]
sqlite3_dbdata_init [0x00E55D9B+656027]
(No symbol) [0x00C2FE6C]
(No symbol) [0x00C283B8]
(No symbol) [0x00C284DD]
(No symbol) [0x00C15818]
BaseThreadInitThunk [0x760E7BA9+25]
RtlInitializeExceptionChain [0x773EBD2B+107]
RtlClearBits [0x773EBCAF+191]

ConnectionError: Trying again with chrome.

DevTools listening on ws://127.0.0.1:57868/devtools/browser/1cdb7243-af4d-476d-900d-5b1958b40b91
Exception details: Message: no such element: Unable to locate element: {"method":"css selector","selector":"#sendOTPCodeBtn"}
(Session info: chrome-headless-shell=120.0.6099.225)
Stacktrace:
GetHandleVerifier [0x00CF6EE3+174339]
(No symbol) [0x00C20A51]
(No symbol) [0x00936FF6]
(No symbol) [0x00969876]
(No symbol) [0x00969C2C]
(No symbol) [0x0099BD42]
(No symbol) [0x00987054]
(No symbol) [0x0099A104]
(No symbol) [0x00986DA6]
(No symbol) [0x00961034]
(No symbol) [0x00961F8D]
GetHandleVerifier [0x00D94B1C+820540]
sqlite3_dbdata_init [0x00E553EE+653550]
sqlite3_dbdata_init [0x00E54E09+652041]
sqlite3_dbdata_init [0x00E497CC+605388]
sqlite3_dbdata_init [0x00E55D9B+656027]
(No symbol) [0x00C2FE6C]
(No symbol) [0x00C283B8]
(No symbol) [0x00C284DD]
(No symbol) [0x00C15818]
BaseThreadInitThunk [0x760E7BA9+25]
RtlInitializeExceptionChain [0x773EBD2B+107]
RtlClearBits [0x773EBCAF+191]

ConnectionError: Trying again with chrome.
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\user\OneDrive\Documents\Python\etradebot\utils\list_accounts.py", line 16, in
etrade = ETrade(consumer_key, consumer_secret, web_username, web_password, account_id, etrade_cookie)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\user\OneDrive\Documents\Python\etradebot\etrade\etrade.py", line 46, in init
self.accounts, self.orders, self.market = self.auth.etrade_login()
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\user\OneDrive\Documents\Python\etradebot\authentication\authentication.py", line 172, in etrade_login
raise Exception("Failed to connect to Etrade API after multiple retries")
Exception: Failed to connect to Etrade API after multiple retries

Please see above for the executed code. I am currently attempting to get the account list from ETrade, but am getting this error. Not sure if this is because it is being run in headless mode or something else.

[nathanramoscfa]

Looks like the webdriver operating the headless browser reaches a webpage looking for CSS element "sendOTPCodeBtn" on the webpage and is not finding it. To pinpoint why, we need to see the webpage, and run ETradeBot in headless=False mode so we can see the browser.

1. Make sure you're running etradebot==3.0.5 as that is the latest working version I am using. Also, make sure the dependencies shown in requirements.txt and setup.py are installed in your Python environment.
2. You can try to repeat the error by setting headless=False in a Juypter notebook <-- like this, and run it. It should open a browser which will stop if ETradeBot can't find a specific element on the webpage.

What webpage does the browser stop at when you encounter your error? That would help troubleshoot. If no error occurs, the browser would just close when it's finished and you'd see the Jupyter notebook populate with API call responses.

Report back with what you encounter. If it's a new page element that I haven't encountered, then that could be something else, but I am currently able to automatically log in with ETradeBot.

[pmiller1414]

Thanks for the suggestions, I ran it in a jupyter notebook like you suggested and it popped up the login screen and logged into my account with no issues. It then just stayed on that browser screen while the same error persisted (the one I posted in my original comment). I am using the most current version of etradebot and all the dependencies are install my my venv.

Not sure if this could be causing anything, but my cookie is in this format (dict) in keyring {'name':'SWH','value':'my_actual_cookie_string','domain':'.etrade.com','secure':True,'httpOnly':True} - if I put just the actual cookie string in here, I get a login error.

One possible thing I'm thinking could be an issue is that I have yet to fully fund my account, but assumed I didn't need this if I was accessing the sandbox environment. Do you know if the Sandbox environment uses my actual account balance to place fake trades or if it is similar to the Alpaca Sandbox that provides an arbitrary amount (100k) of fake money in a separate account to place trades?

[nathanramoscfa]

1. The sendOTPCodeBtn element ETradeBot is looking for on the screen isn't being found because likely the browser is on a webpage where the element doesn't exist. sendOTPCodeBtn on screen looks like a simple button you click to have E-Trade send you a two-factor authentication code to your phone. Since ETradeBot can't find it, the browser is likely on some other page. It would be helpful if you could post a censored screenshot of the browser at the point where it stops and you encounter the error, so we can see what page is being displayed.

2. Your cookie appears correct. It should be passed to ETradeBot as a dict. If it was able to log in successfully, then I don't think the cookie is the problem.

3. Try setting dev=False in the notebook to run it in live mode. As long as preview=True, no trades will be placed when you run the notebook. I haven't really built that much sandbox environment functionality into ETradeBot, and so it may not work. The way I test out ETradeBot is like I've mentioned, making sure 'preview=True' so no actual trades are placed, but, it simulates as if it were placing trades, where you'd see "Previewing Trades..." in the output. That's how I have been testing running ETradeBot, in live mode, without placing any actual trades.

[pmiller1414]

So I tried testing it again today and it turns out the browser was never getting to the OTP screen, it was just going from the username and password screen and going straight to being logged in (skipping the OTP screen and throwing the sendOTPCodeBtn error). Not sure why this was happening before, but it seems to have resolved itself.

However, when continuing the testing by running the example strategy, I ran into the below error:

Looks like historical_prices has a default value of None in model.py, so that's why the error is occurring. I tried setting historical_prices = api.get_historical_prices() in the call to the model function, but got the same error. I saw this both when using jupyter notebook as well as using a batch file as described in the documentation.

I attempted to see what would happen if I just ran a simple custom strategy, so I wouldn't run into the historical prices issue, but it came back with an error that it could not find any values in any columns in my portfolio which makes sense (I only have cash in it currently, no positions in any stocks yet). Also, the cash has not fully cleared yet, so that could be throwing things off - I will attempt this again once the cash has cleared and I have at least one position in my portfolio.

Lastly, I wanted to run the configuration of my main_bat file by you since I am using a venv and not conda - I'm reasonably sure this is correct, but wanted to double check:

[nathanramoscfa]

I just made a commit to fix the AttributeError you encountered. Please try again.

Unfortunately, when I try to run it again, when the bot tries to log in, I get this error:

Exception details: Message: no such element: Unable to locate element: {"method":"css selector","selector":"#sendOTPCodeBtn"}
  (Session info: chrome=121.0.6167.87)
Stacktrace:
	GetHandleVerifier [0x00CB1673+52979]
	(No symbol) [0x00C37961]
	(No symbol) [0x00B1DD3D]
	(No symbol) [0x00B55FBB]
	(No symbol) [0x00B560FB]
	(No symbol) [0x00B8CF92]
	(No symbol) [0x00B74534]
	(No symbol) [0x00B8B3FE]
	(No symbol) [0x00B74286]
	(No symbol) [0x00B4C063]
	(No symbol) [0x00B4CECD]
	GetHandleVerifier [0x00FC8D83+3294723]
	GetHandleVerifier [0x01006CC2+3548482]
	GetHandleVerifier [0x01001C9C+3527964]
	GetHandleVerifier [0x00D4870E+671630]
	(No symbol) [0x00C41EB4]
	(No symbol) [0x00C3D808]
	(No symbol) [0x00C3D92D]
	(No symbol) [0x00C2F7E0]
	BaseThreadInitThunk [0x75AD7BA9+25]
	RtlInitializeExceptionChain [0x77C9BD2B+107]
	RtlClearBits [0x77C9BCAF+191]

When I run in headless=False mode, the webpage displayed shows:

Instead, the web page should be displaying a verifier code that the bot can retrieve. As such, if no verifier code is provided, then ETradeBot is not able to authenticate, and will not work. You can see that ETradeBot is able to successfully log in, but after logging in, instead of showing a verifier code, it shows the web page above. I don't know why E-Trade server is not showing the correct web page, and this is out of my control.

It seems like every few weeks or months, E-Trade changes their authentication process, or what gets displayed when ETradeBot tries to log in. I have spent many months writing code to handle every change they make, and every error I encounter, but E-Trade keeps changing things up. E-Trade is not friendly to automated authentication, they want you to manually log in. This is a huge problem for algorithmic trading strategies. As such, I am increasingly convinced E-Trade doesn't want algorithmic traders on their platform. I have already asked E-Trade customer service, API desk, if they support automated authentication for trading bots, and they flat out said "No."

It's becoming increasingly clear to me, that E-Trade may not be the best platform for algorithmic trading. When I have to spend more time writing code to just be able to automatically log in, and that code breaks every few weeks to months, because E-Trade changes something on their website, then I have to start thinking about other platforms that are more friendly to algorithmic traders. As such, I am considering stopping development of ETradeBot, and developing new trading bot software for a more friendly platform. Candidates include: Alpaca, Interactive Brokers, and TD Ameritrade, which according to ChatGPT4, allow for automated authentication.

[pmiller1414]

This was the exact issue I was originally seeing - it was logging me in, but taking me to the page from your screenshot. I am now able to login with no issues since it now always goes to the web page that displays the verifier code, so it is odd that the server is now displaying this web page without going to the page with the verifier code.

I tried again with your fix for model.py, but am still seeing the same error around NoneType. I tried with a custom strategy instead of the example strategy and got a bit further. I funded my account and bought one stock which fixed the error that said it my portfolio was empty, but later ran into a ValueError.

I'm not going to pursue a fix for this (although it seems rather straightforward), so no need to respond to this.

Seeing that you have constantly ran into errors with ETrade, I may try IB instead. I originally was going to pursue TDAmeritrade, but due to the acquisition by Schwab, their API has taken a backseat and any information on it is scarce. The new Schwab API was supposed to be up in Q3/Q4 of 2023, but there are no updates on their website.

I built a trading bot on Alpaca last year and it was fairly straightforward to build and access since that platform is geared towards algorithmic traders. It runs automatically every day without issue, but you are only able to have one trading account, so having more than one strategy running at a time AND being able to track its performance in a simple way is an issue.

Thanks for all your help with this, really appreciate it!

[nathanramoscfa]

@pmiller1414 I just deprecated this project to work on a trading bot that will work on Alpaca. If interested, please become of a follower of GitHub account.

[obscurerichard]

@nathanramoscfa I can understand where you are coming from deprecating this, given the struggles with authentication.

I've been poking around this project for a few weeks now and wondering what, if anything, you would do about the seemingly intractable authentication problems.

I've also been mulling over a potential authentication workaround. What if we had a service that would refresh the token when needed by sending a notification to the user, so that they could do the login authentication dance that E*Trade wants? The service could store the oauth_token and oauth_token_secret (with appropriate security measures) so that an authorized client program could retrieve them. The service would periodically refresh the tokens as much as it could.

I've actually started a proof of concept project that aims to implement this solution. If this works out, it would likely require daily interactive re-authentication but I'm hopeful that it would hold otherwise.

Any feedback you have about this notion would be valuable to me.

[nathanramoscfa]

@obscurerichard What I have come to realize is, any workaround is just lots of time and effort spent, for a solution that you may be lucky will work for a few weeks. I created an authentication workaround that could sign in, pass two-factor authentication, etc., but then E-Trade changes something server-side, and breaks my code. I then have to spend hours fixing, it works for a few weeks, then E-Trade changes something again server-side, breaks my code. I just got tired of this cycle. There are platforms out there that are far more developer-friendly, like Alpaca. The E-Trade brand is ironic, because it implies they are technologically at the forefront, but in reality, it a name from the 1990s when trading online was at the technological forefront. Frankly, they just don't care about developers, it's not their focus. They seem focused on retail investors who trade with mouse clicks in browser. Nothing against them, but developers should probably look at other platforms.

[obscurerichard]

It is very disappointing to hear that this has been your experience.

As I've been learning more about this space it's clear that E*TRADE is not the only brokerage that has a disappointing API implementation, or that have super-stringent security measures unfriendly to algorithmic traders.

Take Interactive Brokers APIs for example, the OAuth 1.0 docs show it is _way+ more complex than E*TRADE - or you have to establish a session with one of two supported Windows desktop trading programs (TWS or IB Gateway) before you make any calls to their APIs. You can have only one session enabled for trading.

Pivoting to Alpaca or another more developer-friendly platform seems like a good decision from an ease of development perspective, I mean look how easy their authentication documentation is by comparison!

I just worry a bit about Alpaca's relative novelty especially since other firms such as Quantopian tried innovative approaches and then folded. It's hard to peek at the innards of a private company, but I used my LinkedIn Premium subscription to peek at the number of people that say they are employed there - and it's shrunk 23% in the past 2 years. No, these numbers are not precise or accurate but it's a worrying trend for what should be a growth operation. So like many other companies they probably overreached during the pandemic times.

E*TRADE is part of Morgan Stanley and I have way more confidence in their long-term stability and the safety of customer accounts, even if the API has some major warts.

It's just so frustrating that this is not better at ETRADE. I think I'm going to press ahead with my proof of concept for a while longer as it is a good learning exercise for me, and if this works, it would be a boon for anyone who uses ETRADE and wants to have more flexibility about using their APIs.

[nathanramoscfa]

@obscurerichard I looked at Interactive Brokers, and, decided on Alpaca because, like you said, very simple authentication and $0 commissions, whereas Interactive Brokers charges commissions, which can quickly add up. I don't think E-Trade is usable for automated trading until they provide an automated authentication solution. I tried asked their customer service, who routed me to whoever answers API questions, if they had an automated authentication solution, and they flat out said no. At that point, I knew it was a waste of my time to spend any more time on this project. I learned a lot from it, and am applying it my next trading bot project which is in the works, and use Alpaca.

Please follow my GitHub for future updates. I have followed yours!

[danchev]

For anyone facing the same issue, I wanted to clarify that E-Trade indeed supports automatic authorization. For more details, check out the discussion in the pyetrade repository.

jessecooper/pyetrade#104