ARM64 docker build #31

maksimstojkovic · 2021-09-03T06:24:34Z

Consolidated dockerfile into fewer layers
Switched from chrome to chromium webdriver
Updated script file execution permissions
Basic authetication tested successfully
Tested on AMD64 Windows PC and ARM64 Ubuntu 20.04.2 Server 64-bit RPi4
Currently does not support ARMHF due to cryptography rust dependencies
Using CMD in Dockerfile as run.sh currently does not handle arguments

* Switched from chrome to chromium webdriver * Updated script file execution permissions * Basic authetication tested successfully * Tested on AMD64 Windows PC and ARM64 Ubuntu 20.04.2 Server 64-bit RPi4 * Currently does not support ARMHF due to cryptography rust dependencies

maksimstojkovic · 2021-09-03T06:27:43Z

Build addresses issue #17.
Image not yet tested on M1 ARM platform as discussed in issue #29.

maksimstojkovic · 2021-09-03T07:18:37Z

An image of the build to Docker Hub as maksimstojkovic/ibeam-test:latest.
I will take it down once the pull request is reviewed and/or merged.

docker pull maksimstojkovic/ibeam-test:latest

Voyz · 2021-09-06T07:59:21Z

Hey @maksimstojkovic what a fantastic contribution! 👏🎉

We can certainly get it merged in and I'm happy you figured out the way to do it. Let me address a few things first:

You say that this image work on both x64 and ARM64? I just want to double check, as if this is the case we could replace the original with this one instead - this would be amazing.
Can your image utilise SSL successfully? I remember seeing some SSL issues on ARM64 build I attempted.
Can you share the commands you use to build and run this image?
Did you need to use the docker's multi-arch build?
I notice you dropped psutil install from the Dockerfile. If I remember correctly this was required for the cryptography module. Can you confirm if that's the case, and if so whether providing IBEAM_PASSWORD and IBEAM_KEY functionality works?
This image seems to be working on M1 ARM according to this comment from Authentication Process Failed #29 - great news!

Apart from these questions, I just want to compliment you on tidying up and simplifying the Dockerfile. I've been meaning to do so at some point, so it's great to see such contribution to IBeam - thank you! 👏

maksimstojkovic · 2021-09-06T09:53:09Z

Hi @Voyz,

Glad to hear the changes are of use. Below are answers to your questions:

Yes, the image appears to be functioning as expected on both x64 (amd64) and ARM64 (arm64) platforms. The image is able to successfully authenticate on both platforms, and can respond to API requests (I used Postman to perform GET and POST requests).
As far as I can tell, it is able to use SSL successfully. No major errors appear to be thrown. At the end of this comment I have pasted the verbose logs output from runs on both the x64 and ARM64 platforms during testing, for reference. Please let me know if I am missing something related to SSL.
To build and push the image to Docker Hub I used:

docker buildx build --platform linux/amd64,linux/arm64 -t maksimstojkovic/ibeam-test:latest --push .

Yes, the steps used in that article are more or less the same as the steps I used to build the image. A general overview of my procedure is available in this build workflow from another image I made.
I found that psutil was listed in requirements.txt so I removed it from the Dockerfile to prevent confusion in the future. For reference, the logs at the end of this comment are from runs which use the IBEAM_PASSWORD and IBEAM_KEY variables in env.list, so the functionality appears to work as expected.

One notable observation was that sometimes when running the image on an ARM64 RPi 4 2GB, the image will randomly hang for 10 minutes during startup when trying to launch the auth webpage (see logs below). I'm not entirely sure what the cause of this issue is, but after the 10 minutes, the image seems to resume normal operation. I haven't had much time to look into it too closely.

Let me know what you think.

x64 (amd64) Logs

2021-09-06 08:40:10,054|I| ############ Starting IBeam version 0.3.0 ############
2021-09-06 08:40:10,057|D| {'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '/srv/outputs', 'GATEWAY_DIR': '/srv/clientportal.gw', 'CHROME_DRIVER_PATH': '/usr/bin/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'DEBUG', 'LOG_TO_FILE': True, 'REQUEST_RETRIES': 1, 'REQUEST_TIMEOUT': 15, 'GATEWAY_BASE_URL': 'https://localhost:5000', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_USER': '/v1/api/one/user', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'USER_NAME_EL_ID': 'user_name', 'PASSWORD_EL_ID': 'password', 'SUBMIT_EL_ID': 'submitForm', 'ERROR_EL_ID': 'ERRORMSG', 'SUCCESS_EL_TEXT': 'Client login succeeds', 'OAUTH_TIMEOUT': 15, 'PAGE_LOAD_TIMEOUT': 15, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 5, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'TWO_FA_EL_ID': 'twofactbase', 'TWO_FA_INPUT_EL_ID': 'chlginput', 'TWO_FA_HANDLER': None, 'STRICT_TWO_FA_CODE': True}
2021-09-06 08:40:10,058|I| Gateway not found, starting new one...
2021-09-06 08:40:10,058|D| Starting Linux process with params: ['bash', 'bin/run.sh', 'root/conf.yaml']
running
 runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/*
 verticle     :
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/srv/clientportal.gw/build/lib/runtime/netty-common-4.1.15.Final.jar) 
to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
 -> mount demo on /demo
Java Version: 11.0.12
****************************************************
version: ed4af2592e9dd4a784d5403843bd18292fd441ea Fri, 9 Nov 2018 13:23:18 -0500
****************************************************
This is a Beta release of the Client Portal Gateway
for any issues, please contact [email protected]
and include a copy of your logs
****************************************************
https://www.interactivebrokers.com/api/doc.html
****************************************************
Open https://localhost:5000 to login
App demo is available after you login under: https://localhost:5000/demo/
2021-09-06 08:40:11,063|I| Gateway started with pid: 13
2021-09-06 08:40:11,064|D| HTTPS (unverified) request to: https://localhost:5000
2021-09-06 08:40:12,467|D| Gateway connection established
2021-09-06 08:40:12,467|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-09-06 08:40:12,701|I| No active sessions, logging in...
2021-09-06 08:40:12,701|D| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2021-09-06 08:40:18,467|D| Gateway auth webpage loaded
2021-09-06 08:40:18,467|D| Login attempt number 1
2021-09-06 08:40:23,657|D| Submitting the form
2021-09-06 08:40:25,271|D| Webpage displayed "Client login succeeds"
2021-09-06 08:40:27,272|D| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7f374f040f10> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="2d3225493e5e7d83217ce5935b5d466b")>
2021-09-06 08:40:27,333|I| Authentication process succeeded
2021-09-06 08:40:30,336|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-09-06 08:40:30,587|I| Gateway running and authenticated.
2021-09-06 08:40:30,892|I| ############ Starting IBeam version 0.3.0 ############
2021-09-06 08:40:30,895|D| {'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '/srv/outputs', 'GATEWAY_DIR': '/srv/clientportal.gw', 'CHROME_DRIVER_PATH': '/usr/bin/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'DEBUG', 'LOG_TO_FILE': True, 'REQUEST_RETRIES': 1, 'REQUEST_TIMEOUT': 15, 'GATEWAY_BASE_URL': 'https://localhost:5000', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_USER': '/v1/api/one/user', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'USER_NAME_EL_ID': 'user_name', 'PASSWORD_EL_ID': 'password', 'SUBMIT_EL_ID': 'submitForm', 'ERROR_EL_ID': 'ERRORMSG', 'SUCCESS_EL_TEXT': 'Client login succeeds', 'OAUTH_TIMEOUT': 15, 'PAGE_LOAD_TIMEOUT': 15, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 5, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'TWO_FA_EL_ID': 'twofactbase', 'TWO_FA_INPUT_EL_ID': 'chlginput', 'TWO_FA_HANDLER': None, 'STRICT_TWO_FA_CODE': True}
2021-09-06 08:40:30,944|I| Starting maintenance with interval 60 seconds
2021-09-06 08:41:30,945|D| Maintenance
2021-09-06 08:41:30,947|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-09-06 08:41:31,136|I| Gateway running and authenticated
2021-09-06 08:42:30,946|D| Maintenance
2021-09-06 08:42:30,947|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-09-06 08:42:31,209|I| Gateway running and authenticated
2021-09-06 08:43:30,945|D| Maintenance
2021-09-06 08:43:30,946|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-09-06 08:43:31,154|I| Gateway running and authenticated

ARM64 (arm64) Logs

2021-09-06 09:02:20,664|I| ############ Starting IBeam version 0.3.0 ############
2021-09-06 09:02:20,672|D| {'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '/srv/outputs', 'GATEWAY_DIR': '/srv/clientportal.gw', 'CHROME_DRIVER_PATH': '/usr/bin/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'DEBUG', 'LOG_TO_FILE': True, 'REQUEST_RETRIES': 1, 'REQUEST_TIMEOUT': 15, 'GATEWAY_BASE_URL': 'https://localhost:5000', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_USER': '/v1/api/one/user', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'USER_NAME_EL_ID': 'user_name', 'PASSWORD_EL_ID': 'password', 'SUBMIT_EL_ID': 'submitForm', 'ERROR_EL_ID': 'ERRORMSG', 'SUCCESS_EL_TEXT': 'Client login succeeds', 'OAUTH_TIMEOUT': 15, 'PAGE_LOAD_TIMEOUT': 15, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 5, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'TWO_FA_EL_ID': 'twofactbase', 'TWO_FA_INPUT_EL_ID': 'chlginput', 'TWO_FA_HANDLER': None, 'STRICT_TWO_FA_CODE': True}
2021-09-06 09:02:20,675|I| Gateway not found, starting new one...
2021-09-06 09:02:20,676|D| Starting Linux process with params: ['bash', 'bin/run.sh', 'root/conf.yaml']
running
 runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/*
 verticle     :
2021-09-06 09:02:21,688|I| Gateway started with pid: 12
2021-09-06 09:02:21,690|D| HTTPS (unverified) request to: https://localhost:5000
2021-09-06 09:02:21,699|D| Cannot ping Gateway. Retrying for another 20 seconds
2021-09-06 09:02:22,701|D| HTTPS (unverified) request to: https://localhost:5000
2021-09-06 09:02:22,711|D| Cannot ping Gateway. Retrying for another 19 seconds
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/srv/clientportal.gw/build/lib/runtime/netty-common-4.1.15.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2021-09-06 09:02:23,713|D| HTTPS (unverified) request to: https://localhost:5000
2021-09-06 09:02:23,718|D| Cannot ping Gateway. Retrying for another 18 seconds
 -> mount demo on /demo
Java Version: 11.0.12
****************************************************
version: ed4af2592e9dd4a784d5403843bd18292fd441ea Fri, 9 Nov 2018 13:23:18 -0500
****************************************************
This is a Beta release of the Client Portal Gateway
for any issues, please contact [email protected]
and include a copy of your logs
****************************************************
https://www.interactivebrokers.com/api/doc.html
****************************************************
Open https://localhost:5000 to login
App demo is available after you login under: https://localhost:5000/demo/
2021-09-06 09:02:24,723|D| HTTPS (unverified) request to: https://localhost:5000
2021-09-06 09:02:24,739|D| Cannot ping Gateway. Retrying for another 17 seconds
2021-09-06 09:02:25,744|D| HTTPS (unverified) request to: https://localhost:5000
2021-09-06 09:02:28,866|D| Gateway connection established
2021-09-06 09:02:28,866|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-09-06 09:02:29,254|I| No active sessions, logging in...
2021-09-06 09:02:29,255|D| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2021-09-06 09:02:37,911|D| Gateway auth webpage loaded
2021-09-06 09:02:37,912|D| Login attempt number 1
2021-09-06 09:02:43,644|D| Submitting the form
2021-09-06 09:02:45,520|D| Webpage displayed "Client login succeeds"
2021-09-06 09:02:47,523|D| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0xffff9584c190> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="abd737c18665be2fc2a5c13f50963bd7")>
2021-09-06 09:02:47,798|I| Authentication process succeeded
2021-09-06 09:02:50,801|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-09-06 09:02:51,057|I| Gateway running and authenticated.
2021-09-06 09:02:52,330|I| ############ Starting IBeam version 0.3.0 ############
2021-09-06 09:02:52,338|D| {'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '/srv/outputs', 'GATEWAY_DIR': '/srv/clientportal.gw', 'CHROME_DRIVER_PATH': '/usr/bin/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'DEBUG', 'LOG_TO_FILE': True, 'REQUEST_RETRIES': 1, 'REQUEST_TIMEOUT': 15, 'GATEWAY_BASE_URL': 'https://localhost:5000', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_USER': '/v1/api/one/user', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'USER_NAME_EL_ID': 'user_name', 'PASSWORD_EL_ID': 'password', 'SUBMIT_EL_ID': 'submitForm', 'ERROR_EL_ID': 'ERRORMSG', 'SUCCESS_EL_TEXT': 'Client login succeeds', 'OAUTH_TIMEOUT': 15, 'PAGE_LOAD_TIMEOUT': 15, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 5, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'TWO_FA_EL_ID': 'twofactbase', 'TWO_FA_INPUT_EL_ID': 'chlginput', 'TWO_FA_HANDLER': None, 'STRICT_TWO_FA_CODE': True}
2021-09-06 09:02:52,438|I| Starting maintenance with interval 60 seconds
2021-09-06 09:03:52,439|D| Maintenance
2021-09-06 09:03:52,449|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-09-06 09:03:52,697|I| Gateway running and authenticated
2021-09-06 09:04:52,442|D| Maintenance
2021-09-06 09:04:52,449|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-09-06 09:04:52,672|I| Gateway running and authenticated

ARM64 (arm64) Logs with WebDriver Error

2021-09-06 10:30:23,607|I| ############ Starting IBeam version 0.3.0 ############
2021-09-06 10:30:23,615|D| {'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '/srv/outputs', 'GATEWAY_DIR': '/srv/clientportal.gw', 'CHROME_DRIVER_PATH': '/usr/bin/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'DEBUG', 'LOG_TO_FILE': True, 'REQUEST_RETRIES': 1, 'REQUEST_TIMEOUT': 15, 'GATEWAY_BASE_URL': 'https://localhost:5000', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_USER': '/v1/api/one/user', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'USER_NAME_EL_ID': 'user_name', 'PASSWORD_EL_ID': 'password', 'SUBMIT_EL_ID': 'submitForm', 'ERROR_EL_ID': 'ERRORMSG', 'SUCCESS_EL_TEXT': 'Client login succeeds', 'OAUTH_TIMEOUT': 15, 'PAGE_LOAD_TIMEOUT': 15, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 5, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'TWO_FA_EL_ID': 'twofactbase', 'TWO_FA_INPUT_EL_ID': 'chlginput', 'TWO_FA_HANDLER': None, 'STRICT_TWO_FA_CODE': True}
2021-09-06 10:30:23,618|I| Gateway not found, starting new one...
2021-09-06 10:30:23,618|D| Starting Linux process with params: ['bash', 'bin/run.sh', 'root/conf.yaml']
running
 runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/*
 verticle     :
2021-09-06 10:30:24,631|I| Gateway started with pid: 12
2021-09-06 10:30:24,632|D| HTTPS (unverified) request to: https://localhost:5000
2021-09-06 10:30:24,641|D| Cannot ping Gateway. Retrying for another 20 seconds
2021-09-06 10:30:25,643|D| HTTPS (unverified) request to: https://localhost:5000
2021-09-06 10:30:25,649|D| Cannot ping Gateway. Retrying for another 19 seconds
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/srv/clientportal.gw/build/lib/runtime/netty-common-4.1.15.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2021-09-06 10:30:26,653|D| HTTPS (unverified) request to: https://localhost:5000
2021-09-06 10:30:26,658|D| Cannot ping Gateway. Retrying for another 18 seconds
 -> mount demo on /demo
Java Version: 11.0.12
****************************************************
version: ed4af2592e9dd4a784d5403843bd18292fd441ea Fri, 9 Nov 2018 13:23:18 -0500
****************************************************
This is a Beta release of the Client Portal Gateway
for any issues, please contact [email protected]
and include a copy of your logs
****************************************************
https://www.interactivebrokers.com/api/doc.html
****************************************************
Open https://localhost:5000 to login
App demo is available after you login under: https://localhost:5000/demo/
2021-09-06 10:30:27,661|D| HTTPS (unverified) request to: https://localhost:5000
2021-09-06 10:30:27,673|D| Cannot ping Gateway. Retrying for another 17 seconds
2021-09-06 10:30:28,675|D| HTTPS (unverified) request to: https://localhost:5000
2021-09-06 10:30:31,622|D| Gateway connection established
2021-09-06 10:30:31,623|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-09-06 10:30:32,080|I| No active sessions, logging in...
2021-09-06 10:30:32,081|D| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2021-09-06 10:40:36,284|E| Error encountered during authentication
Traceback (most recent call last):
  File "/srv/ibeam/src/authenticate.py", line 151, in authenticate_gateway
    driver = start_driver(base_url, driver_path)
  File "/srv/ibeam/src/authenticate.py", line 286, in start_driver
    raise e
  File "/srv/ibeam/src/authenticate.py", line 274, in start_driver
    driver = new_chrome_driver(driver_path)
  File "/srv/ibeam/src/authenticate.py", line 48, in new_chrome_driver
    driver = webdriver.Chrome(driver_path, options=options)
  File "/opt/venv/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
    desired_capabilities=desired_capabilities)
  File "/opt/venv/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/opt/venv/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/opt/venv/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/opt/venv/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from timeout: Timed out receiving message from renderer: 600.000
  (Session info: headless chrome=89.0.4389.114)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/srv/ibeam/src/authenticate.py", line 254, in authenticate_gateway
    raise RuntimeError('Error encountered during authentication') from e
RuntimeError: Error encountered during authentication
2021-09-06 10:40:36,290|D| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0xffff9f4af750> | Driver: None
2021-09-06 10:40:36,373|I| Authentication process failed
2021-09-06 10:40:37,535|I| ############ Starting IBeam version 0.3.0 ############
2021-09-06 10:40:37,543|D| {'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '/srv/outputs', 'GATEWAY_DIR': '/srv/clientportal.gw', 'CHROME_DRIVER_PATH': '/usr/bin/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'DEBUG', 'LOG_TO_FILE': True, 'REQUEST_RETRIES': 1, 'REQUEST_TIMEOUT': 15, 'GATEWAY_BASE_URL': 'https://localhost:5000', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_USER': '/v1/api/one/user', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'USER_NAME_EL_ID': 'user_name', 'PASSWORD_EL_ID': 'password', 'SUBMIT_EL_ID': 'submitForm', 'ERROR_EL_ID': 'ERRORMSG', 'SUCCESS_EL_TEXT': 'Client login succeeds', 'OAUTH_TIMEOUT': 15, 'PAGE_LOAD_TIMEOUT': 15, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 5, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'TWO_FA_EL_ID': 'twofactbase', 'TWO_FA_INPUT_EL_ID': 'chlginput', 'TWO_FA_HANDLER': None, 'STRICT_TWO_FA_CODE': True}
2021-09-06 10:40:37,643|I| Starting maintenance with interval 60 seconds
2021-09-06 10:41:37,644|D| Maintenance
2021-09-06 10:41:37,652|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-09-06 10:41:38,642|I| No active sessions, logging in...
2021-09-06 10:41:38,643|D| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2021-09-06 10:41:45,010|D| Gateway auth webpage loaded
2021-09-06 10:41:45,010|D| Login attempt number 1
2021-09-06 10:41:50,606|D| Submitting the form
2021-09-06 10:41:52,573|D| Webpage displayed "Client login succeeds"
2021-09-06 10:41:54,576|D| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0xffff83e26910> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="1c04eb2eea70409d2ce77ccaab75bc9b")>
2021-09-06 10:41:54,705|I| Authentication process succeeded
2021-09-06 10:41:57,709|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-09-06 10:41:57,973|I| Gateway running and authenticated
2021-09-06 10:42:37,646|D| Maintenance
2021-09-06 10:42:37,652|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-09-06 10:42:38,255|I| Gateway running and authenticated
2021-09-06 10:43:37,644|D| Maintenance
2021-09-06 10:43:37,651|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-09-06 10:43:38,041|I| Gateway running and authenticated

* Fixes SessionNotCreatedException error observed on ARM64 platform

maksimstojkovic · 2021-09-06T14:50:52Z

Hi @Voyz,

I've done some further investigation and testing and believe the WebDriver error previously mentioned should now be fixed with the latest commit.

I have also pushed the updated image to Docker Hub for inspection and testing.

maksimstojkovic

The following was used as a reference for solving the issue:
https://stackoverflow.com/questions/48450594/selenium-timed-out-receiving-message-from-renderer

Voyz · 2021-09-07T09:37:29Z

Great work @maksimstojkovic 👏 Thanks for answering these questions I had. Let me address some of your responses

As far as I can tell, it is able to use SSL successfully.

Have you used it with self-generated SSL certificates though? Have a look at this doc on TLS certificates for more context. From your logs I can see that the requests are (unverified) which indicates that IBeam doesn't have a valid SSL certificate:

2021-09-06 08:40:11,064|D| HTTPS (unverified) request to: https://localhost:5000

No worries if not, I just want to understand your answer.

I found that psutil was listed in requirements.txt

Damn, you're very much correct, sorry. I've extracted it as it caused some build issues and I found having it as a separate layer was easier to figure out what was wrong. I'm nevertheless surprised you don't need to install gcc for it to install correctly, but I guess it's good news either way.

One notable observation was that sometimes when running the image on an ARM64 RPi 4 2GB, the image will randomly hang for 10 minutes during startup when trying to launch the auth webpage (see logs below)

This isn't new, we've seen this behaviour before, although I thought we managed to eliminate it. You can read about fixing this issue on the current latest image starting with the comment:

#12 (comment)

I'm happy to hear your additions have fixed the issue on the machines you're testing it on.

Give me some time to test your image out, let it run for a while and if things will be in order we should be able to integrate it. Keep in touch if you run into further issues or have other observations.

Once again, fantastic job on this mate 👍

maksimstojkovic · 2021-09-07T10:18:01Z

Hi @Voyz,

Yes, you're correct. I had not installed my own SSL certificates, but assumed that it was working as intended as the logs did not display any critical issues related to SSL when using the certificates generated by the gateway.

Regarding gcc, I switched it to build-essential in the Dockerfile as I thought it was more of an all-in-one solution for building dependencies, and it is removed before the layer is completed. As far as I can tell, using either in the Dockerfile will achieve the same outcome, albeit in the future build-essential will simplify adding dependencies, but at the cost of a slower first build. There shouldn't be any difference once the first build of the layer is cached.

Will keep you posted if I encounter anything interesting.

Voyz · 2021-09-14T09:26:09Z

@maksimstojkovic I've managed to successfully run your image on my development machine as well as on a cloud instance - all seems to be in order 👏 I'm going to let it run for a bit longer to give it a chance to break up a bit, but things are looking good. Will keep you posted 👍

Voyz · 2021-10-04T14:00:46Z

@maksimstojkovic I've had your image running for the past several weeks and I've observed no discrepancies when compared to the current latest. Good job 👍

I've tried rebuilding the image using my source and the updated Dockerfile - could you give it a shot and see if it works for you? docker pull voyz/ibeam:0.4.0-rc1

Also, changed the base of this PR to v0.4.0 as I'd like to release it together with the most recent updates to IBeam.

maksimstojkovic · 2021-10-06T14:23:26Z

Hi @Voyz,

That's awesome! I just had a look at the image on DockerHub and it seems like it was only built for amd64.

To setup multi-platform builds for both amd64 and arm64, run the following:

# Build docker buildx from source
export DOCKER_BUILDKIT=1
docker build --platform=local -o . git://github.com/docker/buildx
mkdir -p ~/.docker/cli-plugins
mv buildx ~/.docker/cli-plugins/docker-buildx

# Install qemu-user-static container
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker buildx create --name builder --driver docker-container --use
docker buildx inspect --bootstrap

Note: This only needs to be executed once on the machine running the builds.

To build the image for testing on your local machine, run:

docker buildx build -t voyz/ibeam:0.4.0-rc1 --load .

To build a multi-platform image and push to DockerHub, run:

docker buildx build --platform linux/amd64,linux/arm64 -t voyz/ibeam:0.4.0-rc1 --push .

Note: This command will not work when trying to load the image onto a local machine, and will only work when pushing to a repository.

Voyz · 2021-10-06T18:48:24Z

Snap, my bad @maksimstojkovic thanks for catching it! 😅 Thanks for outlining the process in so much detail, that's very useful. I'll rebuild and update the image 👍

Voyz · 2021-10-07T11:41:25Z

hey @maksimstojkovic I've just built and pushed a new image that should support arm: docker pull voyz/ibeam:0.4.0-rc2 Thanks for a very clear instructions on how to do it - all went smoothly 👍 Let me know if you're able to run it

codegamc · 2021-10-17T08:43:46Z

Hey @maksimstojkovic - would this build also work on ARMv7 Pis? I think the standard Pi OS from the Pi foundation is ARMv7 not ARM64

maksimstojkovic · 2021-10-17T10:19:09Z

Hi @Voyz,

I managed to pull the image on a RPI 4 running 64-bit Ubuntu 20.04 Server, and it runs without any exec format errors.

However, I'm getting the logs below which show the gateway repeatedly failing to recognise that authentication has occurred, and I am of what the cause is. The same problem occurs when running the 0.4.0-rc2 image on an amd64 machine, so I don't think its a processor platform-specific problem. For reference, I tried running the Client Portal on my amd64 machine and it authenticates normally.

If I recall correctly, a similar/the same problem has been previously observed in an issue but a reliable solution has not yet beeb found, right?

Note: The logs below are from the voyz/ibeam:0.4.0-rc2 image, not voyz/ibeam:0.4.0-rc1 as the first line suggests.

ARM64 (arm64) IBeam Version 0.4.0-rc2 Logs

2021-10-17 10:13:00,416|I| ############ Starting IBeam version 0.4.0-rc1 ############
2021-10-17 10:13:00,419|D| {'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '/srv/outputs', 'GATEWAY_DIR': '/srv/clientportal.gw', 'CHROME_DRIVER_PATH': '/usr/bin/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'DEBUG', 'LOG_TO_FILE': True, 'REQUEST_RETRIES': 1, 'REQUEST_TIMEOUT': 15, 'RESTART_FAILED_SESSIONS': True, 'GATEWAY_BASE_URL': 'https://localhost:5000', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_USER': '/v1/api/one/user', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'ROUTE_LOGOUT': '/v1/api/logout', 'USER_NAME_EL_ID': 'user_name', 'PASSWORD_EL_ID': 'password', 'SUBMIT_EL_ID': 'submitForm', 'ERROR_EL_ID': 'ERRORMSG', 'SUCCESS_EL_TEXT': 'Client login succeeds', 'OAUTH_TIMEOUT': 15, 'PAGE_LOAD_TIMEOUT': 15, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 5, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'TWO_FA_EL_ID': 'twofactbase', 'TWO_FA_INPUT_EL_ID': 'chlginput', 'TWO_FA_HANDLER': None, 'STRICT_TWO_FA_CODE': True}
2021-10-17 10:13:00,423|I| Gateway not found, starting new one...
2021-10-17 10:13:00,424|I| Note that the Gateway log below may display "Open https://localhost:5000 to login" - ignore this command.
2021-10-17 10:13:00,428|D| Starting Linux process with params: ['bash', 'bin/run.sh', 'root/conf.yaml']
running
 runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/*
 verticle     :
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/srv/clientportal.gw/build/lib/runtime/netty-common-4.1.15.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
 -> mount demo on /demo
Java Version: 11.0.12
****************************************************
version: ed4af2592e9dd4a784d5403843bd18292fd441ea Fri, 9 Nov 2018 13:23:18 -0500
****************************************************
This is a Beta release of the Client Portal Gateway
for any issues, please contact [email protected]
and include a copy of your logs
****************************************************
https://www.interactivebrokers.com/api/doc.html
****************************************************
Open https://localhost:5000 to login
App demo is available after you login under: https://localhost:5000/demo/
2021-10-17 10:13:01,431|I| Gateway started with pid: 13
2021-10-17 10:13:01,431|D| HTTPS (unverified) request to: https://localhost:5000
2021-10-17 10:13:03,560|D| Gateway connection established
2021-10-17 10:13:03,560|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-10-17 10:13:03,823|I| No active sessions, logging in...
2021-10-17 10:13:03,823|D| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2021-10-17 10:13:09,111|D| Gateway auth webpage loaded
2021-10-17 10:13:09,112|D| Login attempt number 1
2021-10-17 10:13:14,238|D| Submitting the form
2021-10-17 10:13:15,635|D| Webpage displayed "Client login succeeds"
2021-10-17 10:13:17,638|D| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7f6397960f90> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="4ea80e1c856d260ea31180e277e885c7")>
2021-10-17 10:13:17,706|I| Authentication process succeeded
2021-10-17 10:13:20,710|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-10-17 10:13:20,913|E| Gateway session active but not authenticated
2021-10-17 10:13:20,914|I| Logging out and restarting the Gateway
2021-10-17 10:13:20,914|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/logout
2021-10-17 10:13:21,127|I| Gateway logout successful
2021-10-17 10:13:22,131|I| Gateway shutdown successful
2021-10-17 10:13:22,161|I| Starting maintenance with interval 60 seconds


2021-10-17 10:14:22,162|D| Maintenance
2021-10-17 10:14:22,163|I| Gateway not found, starting new one...
2021-10-17 10:14:22,164|I| Note that the Gateway log below may display "Open https://localhost:5000 to login" - ignore this command.
2021-10-17 10:14:22,164|D| Starting Linux process with params: ['bash', 'bin/run.sh', 'root/conf.yaml']
running  
 runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/*
 verticle     : 
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/srv/clientportal.gw/build/lib/runtime/netty-common-4.1.15.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
 -> mount demo on /demo
Java Version: 11.0.12
****************************************************
version: ed4af2592e9dd4a784d5403843bd18292fd441ea Fri, 9 Nov 2018 13:23:18 -0500
****************************************************
This is a Beta release of the Client Portal Gateway
for any issues, please contact [email protected]
and include a copy of your logs
****************************************************
https://www.interactivebrokers.com/api/doc.html
****************************************************
Open https://localhost:5000 to login
App demo is available after you login under: https://localhost:5000/demo/
2021-10-17 10:14:23,167|I| Gateway started with pid: 130
2021-10-17 10:14:23,167|D| HTTPS (unverified) request to: https://localhost:5000
2021-10-17 10:14:25,556|D| Gateway connection established
2021-10-17 10:14:25,556|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-10-17 10:14:25,832|I| No active sessions, logging in...
2021-10-17 10:14:25,832|D| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2021-10-17 10:14:30,272|D| Gateway auth webpage loaded
2021-10-17 10:14:30,273|D| Login attempt number 1
2021-10-17 10:14:35,418|D| Submitting the form
2021-10-17 10:14:37,305|D| Webpage displayed "Client login succeeds"
2021-10-17 10:14:39,308|D| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7f639789ca90> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="b8e26b485b1656a67cfce0b3a878ea8e")>
2021-10-17 10:14:39,410|I| Authentication process succeeded
2021-10-17 10:14:42,413|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/tickle
2021-10-17 10:14:42,607|E| Gateway session active but not authenticated
2021-10-17 10:14:42,608|I| Logging out and restarting the Gateway
2021-10-17 10:14:42,608|D| HTTPS (unverified) request to: https://localhost:5000/v1/api/logout
2021-10-17 10:14:42,789|I| Gateway logout successful
2021-10-17 10:14:43,792|I| Gateway shutdown successful

maksimstojkovic · 2021-10-17T10:26:26Z

Hey @maksimstojkovic - would this build also work on ARMv7 Pis? I think the standard Pi OS from the Pi foundation is ARMv7 not ARM64

Hey @codegamc,

From what I've tested, there seem to be dependency issues with the armv7/armhf platform related to rust and cryptography. The build error logs suggest that these dependencies are dropping support for armv7, likely for the purpose of enforcing stronger encryption.

There may be a workaround which you could look into, but I'm unsure how long it would remain viable given how processor technology is evolving.

maksimstojkovic · 2021-10-17T11:19:14Z

Hey @Voyz,

I just tried running it again and it seems to be working as expected. Still not sure of the cause, but it appears to be intermittent and unrelated to Ibeam.

Otherwise the multiplatform image appears to be working as expected. 👍

Voyz · 2021-10-19T06:59:44Z

hey @maksimstojkovic fantastic, thanks for trying it out - super happy to hear that it's working for you as expected!

The 'the gateway repeatedly failing to recognise that authentication has occurred' is the common problem that users are experiencing with the Gateway, something which the new logout/restart logic is trying to overcome - you can see it in your logs. It's been reported that doing so can help with the issue, so we're trying it out. You can read more about it here #19

In either case, I'd say that the multiimage build you proposed, as well as the Dockerfile changes are working and ready to be merged. Fantastic work man, thank you for taking the initiative and proposing this contribution, as I'm sure it will benefit a lot of users who use arm.

Just before we go ahead with merging this PR, could I ask you for outlining the buildx process in the CONTRIBUTING.md? I can copy-paste the guideline you gave me in this PR, but I thought as you clearly know more about this than I do, you may be the better person to write it up. Probably this segment will need to be expanded upon:

Thanks once again! 😊

* Removed unused `invoke` build requirements

maksimstojkovic · 2021-10-21T02:16:00Z

Hey @Voyz,

The CONTRIBUTING.md guidelines have now been updated as requested.

I removed the invoke requirements from the repo as the client gateway is already available in the copy_cache directory, and chromium is now installed during Docker builds.

Let me know if any changes are required.

Voyz

Hey @maksimstojkovic sorry for taking so long to review this - crunch time at work. Fantastic job on editing the CONTRIBUTING.md 😊👏 I've left a few suggestions, let me know what you think 👍

CONTRIBUTING.md

Voyz · 2021-11-03T10:30:43Z

CONTRIBUTING.md

+To build a multi-platform image and push to a Docker repository, run:
+
+```shell
+docker buildx build --platform linux/amd64,linux/arm64 -t <repo-username>/ibeam:<tag> --push .


Great stuff 👍 I'd just drop the --push flag from this command. Firstly, I don't think others would have permissions to push into voyz/ibeam and secondly we'd want the process to involve some testing between building and pushing the image.

I agree that they won't be pushing to voyz/ibeam, but without this command, the file doesn't really explain how to create a multi-platform build which can be used for testing. Hence, the format is for a generic repo-username.

The --load flag only builds the image for the building machine's own platform, and will cause exec errors on other platforms.

Also, wouldn't testing come after the build, possibly in a different section?

Hey yeah, so I think we agree here, right? Build, then test, then push. So that would be

docker buildx build --platform linux/amd64,linux/arm64 -t <repo-username>/ibeam:<tag> .

Would calling it without --push not work?

I believe we agree.

My understanding is that with neither --load nor --push the image is left in the build cache and is not visible, regardless of whether it is multi-platform or not.

When --load is used, the image for the building platform is loaded, and visible when running docker images. Note that this is only compatible from the platform of the building machine.

When --push is used, the multi-platform image is pushed to a repository, and any compatible platform will automatically pull the correct version using docker pull <image>:<tag>

Ah, I understand what you mean now. Damn, is there any way around it? Can't it be pushed later?

Hmm, maybe there's been some miscommunication.

If you wanted a build, test, push workflow (and this is referring to the general user who intends to push to their own repo), you would build using --load, run any tests required (assuming the tests are run on the building machine), then build using --push to deploy to an image repository.

If the tests had to be run on a different platform (e.g. build on amd64 and test on arm64), the order would be build with --push, pull the image from the repository using the testing machine, and then run tests. Alternatively they could build directly on the testing platform, as described above. From testing this is a lot slower on things like a Raspberry Pi 4, may be less of an issue on a Mac.

Right that makes sense, thanks for explaining it in more detail. As such, I'd be reluctant to suggest build -> push -> pull -> test as the default development cycle.

Maybe we can outline two workflows: first one for development, where we recommend to use --load, and then later when ready to push the one with --push?

hey @maksimstojkovic what do you recon about the above idea?

@Voyz I've separated the build workflows in my latest commit, please have a look.

Voyz · 2021-11-03T10:35:30Z

CONTRIBUTING.md

@@ -147,29 +147,61 @@ from the main (upstream) repository:
    git pull --ff upstream master
    ```

-## <a name="building-docker"></a>Building a Docker image
-In order to build a Docker image of IBeam, you need to first ensure both Chrome Driver and CPW Gateway are available in `./copy_cache` directory under:


I'd probably opt for keeping this information as it was - although happy to discuss it if you feel otherwise - as it is essential to completing a build, and may not be straightforward to derive without having it pointed out specifically, as we never mention /copy_cache folder and its purpose elsewhere. Also, you reason that:

I removed the invoke requirements from the repo as the client gateway is already available in the copy_cache directory, and chromium is now installed during Docker builds.

While it's true for the client gateway that it's included in the repo and that chromium is installed during builds, the chrome driver still is provided through copy_cache and its copying can be automated by running the invoke as a pre-build task. Thoughts?

Currently the build does not copy any chrome-driver files from the copy_cache (and I don't think it's feasible to have it setup that way due to to possible incompatibilities during multi-platform builds). However replacing the clientportal.gw files in the copy_cache may be of use, so I guess adding a comment explaining how it is copied during builds may be a good idea. What do you think?

You're right! Dang, I missed it in my review of the new Dockerfile, thanks for pointing it out. Where does the chromedriver come from then? Or does IBeam not use it anymore?

DEBIAN_FRONTEND=noninteractive apt-get install -y default-jre dbus-x11 xfonts-base xfonts-100dpi
xfonts-75dpi xfonts-cyrillic xfonts-scalable xorg xvfb gtk2-engines-pixbuf nano curl iputils-ping
chromium chromium-driver build-essential && \

It is installed using the apt-get package manager on line 28.

right thanks! And don't we need to point to it with env variables? I can see this in the doc for WebDriver:

executable_path - path to the executable. If the default is used it assumes the executable is in the $PATH

I guess we'd need to remove driver_path from this instantiation then so that it goes with default:

driver = webdriver.Chrome(driver_path, options=options)

If you follow that we could deprecate driver_path in all functions and logic stack that call new_chrome_driver, as well as deprecate the CHROME_DRIVER_PATH = os.environ.get('IBEAM_CHROME_DRIVER_PATH')

Right?

If you run the following, you can see it's available in the $PATH because you can call it directly without needing to specify the absolute path.

docker run --rm voyz/ibeam:<tag> which chromedriver

EDIT: Make sure you use the tag for the build.

If I'm following correctly, then yes, it should be possible to deprecate those variables because calling chromedriver directly from bash inside the image runs the chromium-driver installed.

Right, that's fantastic, this will simplify the code even further. Great job putting it in 👏 Would you like to remove the driver_path from code and the CHROME_DRIVER_PATH env var? I'm equally well happy to do it myself after we merge this in as I know it may not appear straightforward to you at a glance.

I think it's probably best that you handle this removal, I'm only really familiar with the docker build procedures. I would recommend giving the image a test after doing so. Given that multi-platform builds work and all the dependencies get installed correctly, if it works on amd64, it should also work the same on arm64.

Ok, great 👍 I'll take care of it and test it again later. Nice one 👏

Voyz

Great changes @maksimstojkovic 👍 I think we've got this one pretty much nailed down, happy to go ahead with merging this in 👏

Just a future note - I noticed you've added the docker-compose.yml instructions to README, which we'll also need to add into the Wiki page once v0.4.0 is released.

Thanks for your hard work Maksim and appreciate all the patience in getting this PR reviewed 👍

maksimstojkovic · 2021-12-06T10:05:03Z

My pleasure 😊

maksimstojkovic marked this pull request as draft September 3, 2021 06:25

maksimstojkovic marked this pull request as ready for review September 3, 2021 06:25

maksimstojkovic mentioned this pull request Sep 3, 2021

Linux arm64 support #17

Closed

mattspinach mentioned this pull request Sep 4, 2021

Authentication Process Failed #29

Closed

maksimstojkovic added 2 commits September 6, 2021 22:47

Reordered layers to reduce consecutive build times

5373bd1

Updated chromedriver options to fix exception

da9bd97

* Fixes SessionNotCreatedException error observed on ARM64 platform

maksimstojkovic commented Sep 6, 2021

View reviewed changes

Voyz changed the base branch from master to v0.4.0 October 4, 2021 11:38

Voyz self-requested a review October 4, 2021 12:41

Voyz self-assigned this Oct 4, 2021

Added multi-platform build instructions

ab4a7f3

* Removed unused `invoke` build requirements

Voyz requested changes Nov 3, 2021

View reviewed changes

maksimstojkovic added 2 commits November 5, 2021 01:05

Added build instruction subheadings

91506e6

Split docker build workflow docs

037a51c

Voyz approved these changes Dec 6, 2021

View reviewed changes

Voyz merged commit 9ab9f87 into Voyz:v0.4.0 Dec 9, 2021

ARM64 docker build #31

ARM64 docker build #31

Conversation

maksimstojkovic commented Sep 3, 2021

maksimstojkovic commented Sep 3, 2021

maksimstojkovic commented Sep 3, 2021

Voyz commented Sep 6, 2021

maksimstojkovic commented Sep 6, 2021 • edited Loading

maksimstojkovic commented Sep 6, 2021 • edited Loading

maksimstojkovic left a comment • edited Loading

Choose a reason for hiding this comment

Voyz commented Sep 7, 2021

maksimstojkovic commented Sep 7, 2021

Voyz commented Sep 14, 2021

Voyz commented Oct 4, 2021

maksimstojkovic commented Oct 6, 2021 • edited Loading

Voyz commented Oct 6, 2021

Voyz commented Oct 7, 2021

codegamc commented Oct 17, 2021

maksimstojkovic commented Oct 17, 2021 • edited Loading

maksimstojkovic commented Oct 17, 2021

maksimstojkovic commented Oct 17, 2021

Voyz commented Oct 19, 2021

maksimstojkovic commented Oct 21, 2021 • edited Loading

Voyz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maksimstojkovic Nov 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maksimstojkovic Nov 4, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maksimstojkovic Nov 4, 2021 • edited Loading

Choose a reason for hiding this comment

maksimstojkovic Nov 4, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Voyz left a comment

Choose a reason for hiding this comment

maksimstojkovic commented Dec 6, 2021

maksimstojkovic commented Sep 6, 2021 •

edited

Loading

maksimstojkovic commented Sep 6, 2021 •

edited

Loading

maksimstojkovic left a comment •

edited

Loading

maksimstojkovic commented Oct 6, 2021 •

edited

Loading

maksimstojkovic commented Oct 17, 2021 •

edited

Loading

maksimstojkovic commented Oct 21, 2021 •

edited

Loading

maksimstojkovic Nov 12, 2021 •

edited

Loading

maksimstojkovic Nov 4, 2021 •

edited

Loading

maksimstojkovic Nov 4, 2021 •

edited

Loading

maksimstojkovic Nov 4, 2021 •

edited

Loading