• fixed deprecated Google Messages handler code
• added support for /activate and /deactivate routes
• updated handling of the IB Key promo
• running ibeam_starter.py with -s flag causes IBeam to get into an infinite loop to allow only starting the Gateway within the Docker image
• fixed logs not having 'outputs' directory
• added support for the authentication initialise endpoint auth/ssodh/init
• added additional errors if account or password are not provided
• increased IBEAM_MAX_STATUS_CHECK_RETRIES from 15 to 120 and fixed var.all_variables
• Added Dockerfile armv7

Minor Changes

• added fix to imports when running in standalone
• added better verbosity for env vars that need to be defined
• specified outputs_dir parameter in TwoFaHandlers as a kwarg to fix #165

Breaking changes:

• We expanded customisability for different versions of website found. Each env var that specifies the DOM element looked for, requires a [TYPE]@@[IDENTIFIER] format, eg.: CSS_SELECTOR@@.btn.btn-lg.btn-primary for the primary submit button. I concluded that such system is necessary when I realised that the 2FA forms have a very different layout for different users. Now it can be customised even more. If you used custom env vars for DOM elements: due to the change in the env var DOM elements processing, you'll need to specify the [TYPE]@@[IDENTIFIER]. This will most likely mean you'll need to add [TYPE]@@ to your existing env vars that specify custom DOM elements if you had any.

Major changes:

Login Strategies:

• IBeam now supports various login logic flows - we call them strategies. These can be selected by setting IBEAM_AUTHENTICATION_STRATEGY env var as A or B. It is possible that this is a temporary feature that will be removed once a clear winning strategy is found.
• Currently used login logic is called strategy 'A'
• New strategy 'B' was added, and is now used by default, based on @migoohao's suggestions in #146. The new logic is described in detail in #147. This is done in an attempt to simplify the login logic and rely on reauthenticate more than on full re-login. Many thanks for suggesting this logic flow @migoohao 👏 I also want to give kudos to any other users who mentioned using reauthenticate in the past. See: #147

Authentication rework:

• reworking the authenticate.py. Now it is broken down to step-by-step functions, and hopefully has a much better readability and maintainability. It's functionality has been moved to login_handler.py
• removing reading environment variables throughout the app. Now all env vars are read in ibeam_starter.py and then everything is passed as arguments
• decoupling things from GatewayClient even more. Now we have a bunch of handlers that are responsible for different things

Minor changes:

• updated information about ignoring gateway log display
• added support for UI scaling
• passing current chrome WebDriver to TwoFaHandler.get_two_fa_code()
• 2fa submitting action is now handled by pressing RETURN key instead of looking for the submit element, as the button is not always visible
• added call to validate on each maintenance
• added GCP Secrets Manager as one of the secrets source
• added handling of 503 service unavailable
• fixed boolean env var parsing
• fixed process starting and killing, ensuring we kill all processes
• added more verbosity to 'Gateway running and authenticated' by attaching * session_id and server_name added support for GET and POST requests according to IBKR docs
• logging to sys.stdout instead of default sys.stder, sending logs below WARNING to stdout and equal and above to stderr

Setup changes:

• changed env var defaults: IBEAM_REQUEST_RETRIES from 1 to 2 IBEAM_MAX_REAUTHENTICATE_RETRIES from 5 to 3
• post-authentication treating not status.running or not status.session as a reason to kill and restart too

Documentation:

• updated README with GPC Secret Manager
• Mention IBC for TWS/Gateway

Dependencies:

• updated the Gateway to 24 Apr 2023
• updated selenium to 4.12.*
• added pillow to requirements.txt to support cropping and manipulating webpage screenshots

• added small fix to timeout exception handling
• added missing Gateway files
• added incremental presubmit buffer
• clearing input fields before typing into them to prevent inputting incorrect/duplicated text
• added call to reauthenticate after a wait upon failed authentication before logging out and restarting
• added call to validate before tickling

• updated requirements.txt to most recent minor versions - this should address the Timeout value connect was <object object at 0x7fac7de54590>, but it must be an int, float or None. error from #140
• changed python:3.11.2-slim-buster to python:3.11.3-slim-bullseye
• improved logging of timeout errors and verbosity on creating DOM elements' list

• updated Python to 3:11-2 from 3.7.7
• updated Gateway to July 2022

• added IBKR website version checking for receiving various types of IBKR website
• add kubernetes health check APIs
• added NotificationResend2FAHandler for handling the 2FA as phone notification
• added handling for various types of 2FA, handled by TWO_FA_SELECT_TARGET
• added capability to read secrets from the container filesystem, to support using Docker Swarm secrets

• added handling of 500 Internal Server Error - expected on server restart
• saving screenshot and releasing google chrome driver upon google-msg-handler exceptions
• fixed google_msg_handler after Google updated the classes
• fixed ports that were open to other people on local network by specifying 127.0.0.1 in docker-compose
• whitelist 10.* ips for kubernetes
• changed most .debug statements to .info statements as to avoid the default run behaviour to be with DEBUG logs

Major:

• Switched from chrome to chromium webdriver
• Added ARM64 support (fantastic contribution from @maksimstojkovic in #31 👏)
  • Tested on AMD64 Windows PC and ARM64 Ubuntu 20.04.2 Server 64-bit RPi4
  • Currently does not support ARMHF due to cryptography rust dependencies
  • Added multi-platform build instructions
• Added reauthentication
  • Not killing gateway after logout, instead now parsing tickle data for 'competing' flag and reauthenticating if found
• Reordered Docker layers to reduce consecutive build times
• Made 'Invalid username password combination' count towards MAX_FAILED_AUTH
• Added critical failure and shutdown on no 2FA handler provided when it's needed
• Added retries to Google2faHandler marking the message as read
• Added handling of ibkey promo (fantastic contribution from @JackD111 in #69 🥳)

Misc:

• Added log when found a custom conf.yaml
• Made logging format customisable through LOG_FORMAT env var
• Removed unused invoke build requirements
• Added a warning about 'Open https://localhost:5000 to login'
• Changed BlockingScheduler to BackgroundScheduler and support for keyboard interrupt

Thank you to everyone who contributed in the discussions, reported problems, contacted IBKR and helped build this tool 👍 Great job everyone!

• refactored and broken down the gateway_client to:

  • http_handler - for HTTP requests to the Gateway
  • authenticate - for authenticating using the Gateway's webpage
  • inputs_handler - for handling Inputs Directory
  • var - for environment variables
  • process_utils - for starting the gateway

  As a result gateway_client is more of a logic manager that utilises these various modules/classes

• added 2FA handling - executed through TwoFaHandler child classes

  • User can specify which handler to use by setting IBEAM_TWO_FA_HANDLER env var. Accepted values are 'GOOGLE_MSG', 'EXTERNAL_REQUEST' and 'CUSTOM_HANDLER'. These correspond to the following:
  • TwoFaHandlers available:
    • GoogleMessageTwoFaHandler - using Google Messages, written originally by @gmcmicken - I added some small tweaks
    • ExternalRequestTwoFaHandler - calling an external URL for the 2FA code
    • custom handler - allowing users to provide a custom handler through the Inputs Directory

• tweaked some logging messages to be more accurate and moved logs' logic to a separate module

• updated the authentication logic to make it more reliable and provide more failsafes

• added support/verify_connection.py script for testing the connection and certificates

• updated the wikipedia to reflect and document all of these changes

• Added handling for '[Errno 111] Connection refused' - treating it as Gateway running but not serving yet.

• added IBEAM_REQUEST_RETRIES env var used during maintenance to determine how many times should IBeam attempt to communicate with the Gateway before attempting reauthentication
• added IBEAM_REQUEST_TIMEOUT env var defining the timeout for requests to the Gateway in seconds.
• added IBEAM_OAUTH_TIMEOUT env var defining the timeout for login.
• now uses tickle to verify authentication and extend sessions, instead of validate
• now checks for authentication status apart from successful requests
• deprecated -l/--validate standalone argument
• added -c / --check standalone argument to check if Gateway's session is authenticated