Category: What I’m Up To

The ChromeDriver 103 bug

If you had an application or script that uses Selenium and ChromeDriver to control or automate instances of Chrome or Chromium, it’s probably not working right now. Instead, you’re probably seeing error messages that look like this:

Message: unknown error: cannot determine loading status
from unknown error: unexpected command response
  (Session info: chrome=103.0.5060.53)
Stacktrace:
0   chromedriver                        0x000000010fb6f079 chromedriver + 4444281
1   chromedriver                        0x000000010fafb403 chromedriver + 3970051
2   chromedriver                        0x000000010f796038 chromedriver + 409656

...and on it goes...

The culprit is ChromeDriver 103, and I wrote about it a couple of days ago in a post titled Why your Selenium / ChromeDriver / Chrome setup stopped working.

ChromeDriver 103 — or more accurately, ChromeDriver 103.0.5060.53 — works specifically with Chrome/Chromium 103.0.5060.53. If you regularly update Chrome or Chromium and use ChromeDriverManager to keep ChromeDriver’s version in sync with the browser, you’ve probably updated to ChromeDriver 103.0.5060.53, which has a causes commands to ChromeDriver to sometimes fail.

The fix

Luckily, the bug has been fixed in ChromeDriver 104, which works specifically with Chrome 104. This means that if you update to Chrome and ChromeDriver 104, your Selenium / ChromeDriver / Chrome setup will work again.

The challenge is that Chrome 104 is still in beta. As I’m writing this, the Google Chrome Beta site is hosting an installer for Chrome 104, or more accurately, Chrome 104.0.5112.20. The application has the name Google Chrome Beta and is considered a separate app from Google Chrome. This means that you can have the current release and the beta version installed on your machine at the same time.

Once you’ve installed Chrome Beta, you just need to make your Selenium/ChromeDriver application or script use the appropriate version of ChromeDriver. Here’s how I did it in Transmogrifier, my Python script that helps me assemble the Tampa Bay Tech Events list:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager


// 1
option = Options()
option.binary_location='/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta'

// 2
driver = webdriver.Chrome(service=Service(ChromeDriverManager(version='104.0.5112.20').install()), options=option)

Here are some explanation that match the number comments in the code above:

1. These lines create an instance of the Options class, which we’re using to specify the directory where the Chrome beta app can be found. We’ll use the instance, options, when we create driver, the object that controls the Chrome beta app.
2. This constructor creates a WebDriver.Chrome object which will control the Chrome beta app. The service parameter specifies that the driver should be an instance of ChromeDriver 104, and that it should be installed if it isn’t already present on the system. The options parameter specifies that ChromeDriver should drive the version of Chrome located at the directory path specified in the option object: the Chrome beta app.

Once that’s done, I can proceed with my script. Here’s a simplified version of the start of my Transmogrifier script:

from bs4 import BeautifulSoup

// Load the web page whose URL is `url` into
// the browser under script control
driver.get(url)

// Execute any scripts on the page so that
// we can scrape the final rendered version
html = driver.execute_script('return document.body.innerHTML')

// And now it’s time to scrape!
soup = BeautifulSoup(html, 'html.parser')
// ...the rest of the code goes here...

Hope this helps!

Update: You’ll also want to see this follow-up article — Fix the ChromeDriver 103 bug with ChromeDriver 104.

If you run applications or scripts that use Selenium to control instances of Chrome via ChromeDriver, you may find that they no longer work, and instead provide you with error messages that look like this:

Message: unknown error: cannot determine loading status
from unknown error: unexpected command response
  (Session info: chrome=103.0.5060.53)
Stacktrace:
0   chromedriver                        0x000000010fb6f079 chromedriver + 4444281
1   chromedriver                        0x000000010fafb403 chromedriver + 3970051
2   chromedriver                        0x000000010f796038 chromedriver + 409656
3   chromedriver                        0x000000010f7833c8 chromedriver + 332744
4   chromedriver                        0x000000010f782ac7 chromedriver + 330439
5   chromedriver                        0x000000010f782047 chromedriver + 327751
6   chromedriver                        0x000000010f780f16 chromedriver + 323350
7   chromedriver                        0x000000010f78144c chromedriver + 324684
8   chromedriver                        0x000000010f78e3bf chromedriver + 377791
9   chromedriver                        0x000000010f78ef22 chromedriver + 380706
10  chromedriver                        0x000000010f79d5b3 chromedriver + 439731
11  chromedriver                        0x000000010f7a147a chromedriver + 455802
12  chromedriver                        0x000000010f78177e chromedriver + 325502
13  chromedriver                        0x000000010f79d1fa chromedriver + 438778
14  chromedriver                        0x000000010f7fc62d chromedriver + 828973
15  chromedriver                        0x000000010f7e9683 chromedriver + 751235
16  chromedriver                        0x000000010f7bfa45 chromedriver + 580165
17  chromedriver                        0x000000010f7c0a95 chromedriver + 584341
18  chromedriver                        0x000000010fb4055d chromedriver + 4253021
19  chromedriver                        0x000000010fb453a1 chromedriver + 4273057
20  chromedriver                        0x000000010fb4a16f chromedriver + 4292975
21  chromedriver                        0x000000010fb45dea chromedriver + 4275690
22  chromedriver                        0x000000010fb1f54f chromedriver + 4117839
23  chromedriver                        0x000000010fb5fed8 chromedriver + 4382424
24  chromedriver                        0x000000010fb6005f chromedriver + 4382815
25  chromedriver                        0x000000010fb768d5 chromedriver + 4475093
26  libsystem_pthread.dylib             0x00007ff81931a4e1 _pthread_start + 125
27  libsystem_pthread.dylib             0x00007ff819315f6b thread_start + 15

It turns out that there’s a bug in version 103 of ChromeDriver, which works specifically with version 103 of Chrome. This bug causes commands to ChromeDriver, such as its get() method, which points the browser to a specific URL, to sometimes fail.

The quick solution

While this bug exists, the best workaround — and one that I’m using at the moment — is to do the following:

1. Uninstall version 103 of Chrome.
2. Install version 102 of Chrome.
3. Install version 102 of ChromeDriver.
4. Disable Chrome’s auto-update and don’t update Chrome.

How I encountered the bug

This happened to me on Wednesday. Earlier that day, I saw the “Update” button on Chrome change from green to yellow…

…and my conditioned-by-security-training response was to click it, updating Chrome to version 103.

Later that evening, I started assembling the weekly list of tech, entrepreneur, and nerd events for the Tampa Bay area. You know, this one:

When I started putting this list together back in early 2017, I did so manually by doing a lot of copying and pasting from Meetup and EventBrite pages. However, as the tech scene in Tampa grew, what used to be an hour’s work on a Saturday afternoon starting growing to consume more and more of that afternoon. I’d watch entire feature-length films in the background while I put them together. It became clear to me that it was time to add some automation to the process.

These days, I put together the list with the help of “The Transmogrifier,” my name for a collection of Python scripts inside a Jupyter Notebook. Given a set of URLs for Meetup and Eventbrite pages, it scrapes their pages for the following information:

• The name of the group organizing the event
• The name of the event
• The time of the event

In the beginning, scraping Meetup was simply a matter of having Python make a GET request to a Meetup page, and then use BeautifulSoup to scrape its contents. But Meetup is a jealous and angry site, and they really, really, really hate scraping. So they employ all manner of countermeasures, and I have accepted the fact that as long as I put together the Tampa Bay Tech Events list, I will continually be playing a “move / counter-move” game with them.

One of Meetup’s more recent tricks was to serve an intermediate page that would not be complete until some JavaScript within that page executed within the browser upon loading. This means that the web page isn’t complete until you load the page into a browser, and only a browser. GETting the page programmatically won’t execute the page’s JavaScript.

Luckily, I’d heard of this trick before, and decided that I could use Selenium and ChromeDriver so that the Transmogrifier would take control of a Chrome instance, use it to download Meetup pages — which would then execute their JavaScript to create the final page. Once that was done, the Transmogrifier could then read the HTML of that final page via the browser under its control, which it could scrape.

Creating an instance of Chrome that would be under the Transmogrifier’s control is easy:

# Python

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

The single line of code after all the import statements does the following:

• It launches an instance of Chrome that can be programmatically controlled.
• It has ChromeDriver check to see if it is compatible with the Chrome instance. If not, it installs the compatible version of ChromeDriver.

This is where my problem began. ChromeDrive saw that I’d updated to Chrome 103, so it updated itself to version 103.

Here’s the code where the bug became apparent:

print(f"Processing Meetup URL: {url}")
driver.get(url)

About one time out of three, this code would do what it was supposed to: print a message to the console, and then make Chrome load the page at the given URL.

But two out of three times, it would end up with this error:

Message: unknown error: cannot determine loading status
from unknown error: unexpected command response
  (Session info: chrome=103.0.5060.53)
Stacktrace:
0   chromedriver                        0x000000010fb6f079 chromedriver + 4444281
1   chromedriver                        0x000000010fafb403 chromedriver + 3970051
2   chromedriver                        0x000000010f796038 chromedriver + 409656

(...and the stacktrace goes on from here...)

This happens when executing several driver.get(url) calls in a row, which is what the Transmogrifier does. It’s executing driver.get(url) for many URLs in rapid succession. When this happens, there are many times when Chrome is processing a new ChromeDriver command request after a previous ChromeDriver session (a previous web page fetch) has already concluded and detached. In this case, Chrome responds with a “session not found” error. ChromeDriver gets this error while waiting for another valid command to complete, causing that command to fail. (You can find out more here.)

In the end, my solution was to downgrade to Chrome 102, use ChromeDriver 102, and keep an eye open for Chrome/ChromeDriver updates.