スポンサーリンク
はじめに
ChromebookでWEBブラウジングを自動化(WEBスクレイピング)したので、その方法をまとめる。
Linux環境でPython+Seleniumで実現した。
システム環境
Acer Chromebook Spin 311 CP311-3H-A14N/E
Google Chrome OS 94.0.4606.114
環境セットアップ
Linux環境の有効化
ChomebookにLinuxをインストール(有効化)したを参照。
今回使用するPythonはデフォルトでLinux環境にインストール済みである。
testuser@penguin:~$ python3 --version
Python 3.7.3
testuser@penguin:~$ Chromiumのインストール
ChromebookにはデフォルトでGoogle Chromeがインストールされているが、これは使えない。
有効化したLinux環境に、別途ブラウザ「Chromium」をインストールする必要がある。
testuser@penguin:~$ sudo apt install chromium
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
chromium-common chromium-sandbox fonts-liberation libaacs0 libatomic1 libavformat58 libbdplus0
libbluray2 libchromaprint1 libevent-2.1-6 libgme0 libimobiledevice6 libjsoncpp1 libminizip1
libmpg123-0 libopenmpt0 libplist3 libre2-5 libssh-gcrypt-4 libu2f-udev libupower-glib3 libusbmuxd4
libvorbisfile3 libwebpdemux2 libxslt1.1 notification-daemon upower usbmuxd
Suggested packages:
chromium-l10n chromium-shell chromium-driver libbluray-bdj libusbmuxd-tools
The following NEW packages will be installed:
chromium chromium-common chromium-sandbox fonts-liberation libaacs0 libatomic1 libavformat58
libbdplus0 libbluray2 libchromaprint1 libevent-2.1-6 libgme0 libimobiledevice6 libjsoncpp1
libminizip1 libmpg123-0 libopenmpt0 libplist3 libre2-5 libssh-gcrypt-4 libu2f-udev libupower-glib3
libusbmuxd4 libvorbisfile3 libwebpdemux2 libxslt1.1 notification-daemon upower usbmuxd
0 upgraded, 29 newly installed, 0 to remove and 0 not upgraded.
Need to get 54.6 MB of archives.
After this operation, 195 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 https://deb.debian.org/debian buster/main arm64 libatomic1 arm64 8.3.0-6 [8,820 B]
Get:2 https://security.debian.org/debian-security buster/updates/main arm64 libavformat58 arm64 7:4.1.8-0+deb10u1 [961 kB]
Get:3 https://deb.debian.org/debian buster/main arm64 libbluray2 arm64 1:1.1.0-1 [132 kB]
Get:4 https://deb.debian.org/debian buster/main arm64 libchromaprint1 arm64 1.4.3-3 [38.3 kB]
Get:5 https://deb.debian.org/debian buster/main arm64 libgme0 arm64 0.6.2-1 [116 kB]
Get:6 https://deb.debian.org/debian buster/main arm64 libmpg123-0 arm64 1.25.10-2 [127 kB]
Get:7 https://deb.debian.org/debian buster/main arm64 libvorbisfile3 arm64 1.3.6-2 [24.8 kB]
Get:8 https://deb.debian.org/debian buster/main arm64 libopenmpt0 arm64 0.4.3-1+deb10u1 [567 kB]
Get:9 https://deb.debian.org/debian buster/main arm64 libssh-gcrypt-4 arm64 0.8.7-1+deb10u1 [190 kB]
Get:10 https://deb.debian.org/debian buster/main arm64 libevent-2.1-6 arm64 2.1.8-stable-4 [150 kB]
Get:11 https://deb.debian.org/debian buster/main arm64 libjsoncpp1 arm64 1.7.4-3 [71.0 kB]
Get:12 https://deb.debian.org/debian buster/main arm64 libminizip1 arm64 1.1-8+b1 [17.4 kB]
Get:13 https://deb.debian.org/debian buster/main arm64 libre2-5 arm64 20190101+dfsg-2 [144 kB]
Get:14 https://deb.debian.org/debian buster/main arm64 libwebpdemux2 arm64 0.6.1-2+deb10u1 [87.5 kB]
Get:15 https://deb.debian.org/debian buster/main arm64 libxslt1.1 arm64 1.1.32-2.2~deb10u1 [225 kB]
Get:16 https://deb.debian.org/debian buster/main arm64 chromium-common arm64 89.0.4389.114-1~deb10u1 [1,280 kB]
Get:17 https://deb.debian.org/debian buster/main arm64 chromium arm64 89.0.4389.114-1~deb10u1 [49.0 MB]
Get:18 https://deb.debian.org/debian buster/main arm64 chromium-sandbox arm64 89.0.4389.114-1~deb10u1 [146 kB]
Get:19 https://deb.debian.org/debian buster/main arm64 fonts-liberation all 1:1.07.4-9 [827 kB]
Get:20 https://deb.debian.org/debian buster/main arm64 libaacs0 arm64 0.9.0-2 [48.2 kB]
Get:21 https://deb.debian.org/debian buster/main arm64 libbdplus0 arm64 0.1.2-3 [39.5 kB]
Get:22 https://deb.debian.org/debian buster/main arm64 libplist3 arm64 2.0.1~git20190104.3f96731-1 [31.4 kB]
Get:23 https://deb.debian.org/debian buster/main arm64 libusbmuxd4 arm64 1.1.0~git20181007.07a493a-1 [15.2 kB]
Get:24 https://deb.debian.org/debian buster/main arm64 libimobiledevice6 arm64 1.2.1~git20181030.92c5462-2+deb10u1 [65.7 kB]
Get:25 https://deb.debian.org/debian buster/main arm64 libu2f-udev all 1.1.9-1 [7,344 B]
Get:26 https://deb.debian.org/debian buster/main arm64 libupower-glib3 arm64 0.99.10-1 [49.4 kB]
Get:27 https://deb.debian.org/debian buster/main arm64 notification-daemon arm64 3.20.0-4 [59.3 kB]
Get:28 https://deb.debian.org/debian buster/main arm64 upower arm64 0.99.10-1 [111 kB]
Get:29 https://deb.debian.org/debian buster/main arm64 usbmuxd arm64 1.1.1~git20181007.f838cf6-1 [37.1 kB]
Fetched 54.6 MB in 16s (3,316 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libatomic1:arm64.
(Reading database ... 34830 files and directories currently installed.)
Preparing to unpack .../00-libatomic1_8.3.0-6_arm64.deb ...
Unpacking libatomic1:arm64 (8.3.0-6) ...
Selecting previously unselected package libbluray2:arm64.
Preparing to unpack .../01-libbluray2_1%3a1.1.0-1_arm64.deb ...
Unpacking libbluray2:arm64 (1:1.1.0-1) ...
Selecting previously unselected package libchromaprint1:arm64.
Preparing to unpack .../02-libchromaprint1_1.4.3-3_arm64.deb ...
Unpacking libchromaprint1:arm64 (1.4.3-3) ...
Selecting previously unselected package libgme0:arm64.
Preparing to unpack .../03-libgme0_0.6.2-1_arm64.deb ...
Unpacking libgme0:arm64 (0.6.2-1) ...
Selecting previously unselected package libmpg123-0:arm64.
Preparing to unpack .../04-libmpg123-0_1.25.10-2_arm64.deb ...
Unpacking libmpg123-0:arm64 (1.25.10-2) ...
Selecting previously unselected package libvorbisfile3:arm64.
Preparing to unpack .../05-libvorbisfile3_1.3.6-2_arm64.deb ...
Unpacking libvorbisfile3:arm64 (1.3.6-2) ...
Selecting previously unselected package libopenmpt0:arm64.
Preparing to unpack .../06-libopenmpt0_0.4.3-1+deb10u1_arm64.deb ...
Unpacking libopenmpt0:arm64 (0.4.3-1+deb10u1) ...
Selecting previously unselected package libssh-gcrypt-4:arm64.
Preparing to unpack .../07-libssh-gcrypt-4_0.8.7-1+deb10u1_arm64.deb ...
Unpacking libssh-gcrypt-4:arm64 (0.8.7-1+deb10u1) ...
Selecting previously unselected package libavformat58:arm64.
Preparing to unpack .../08-libavformat58_7%3a4.1.8-0+deb10u1_arm64.deb ...
Unpacking libavformat58:arm64 (7:4.1.8-0+deb10u1) ...
Selecting previously unselected package libevent-2.1-6:arm64.
Preparing to unpack .../09-libevent-2.1-6_2.1.8-stable-4_arm64.deb ...
Unpacking libevent-2.1-6:arm64 (2.1.8-stable-4) ...
Selecting previously unselected package libjsoncpp1:arm64.
Preparing to unpack .../10-libjsoncpp1_1.7.4-3_arm64.deb ...
Unpacking libjsoncpp1:arm64 (1.7.4-3) ...
Selecting previously unselected package libminizip1:arm64.
Preparing to unpack .../11-libminizip1_1.1-8+b1_arm64.deb ...
Unpacking libminizip1:arm64 (1.1-8+b1) ...
Selecting previously unselected package libre2-5:arm64.
Preparing to unpack .../12-libre2-5_20190101+dfsg-2_arm64.deb ...
Unpacking libre2-5:arm64 (20190101+dfsg-2) ...
Selecting previously unselected package libwebpdemux2:arm64.
Preparing to unpack .../13-libwebpdemux2_0.6.1-2+deb10u1_arm64.deb ...
Unpacking libwebpdemux2:arm64 (0.6.1-2+deb10u1) ...
Selecting previously unselected package libxslt1.1:arm64.
Preparing to unpack .../14-libxslt1.1_1.1.32-2.2~deb10u1_arm64.deb ...
Unpacking libxslt1.1:arm64 (1.1.32-2.2~deb10u1) ...
Selecting previously unselected package chromium-common.
Preparing to unpack .../15-chromium-common_89.0.4389.114-1~deb10u1_arm64.deb ...
Unpacking chromium-common (89.0.4389.114-1~deb10u1) ...
Selecting previously unselected package chromium.
Preparing to unpack .../16-chromium_89.0.4389.114-1~deb10u1_arm64.deb ...
Unpacking chromium (89.0.4389.114-1~deb10u1) ...
Selecting previously unselected package chromium-sandbox.
Preparing to unpack .../17-chromium-sandbox_89.0.4389.114-1~deb10u1_arm64.deb ...
Unpacking chromium-sandbox (89.0.4389.114-1~deb10u1) ...
Selecting previously unselected package fonts-liberation.
Preparing to unpack .../18-fonts-liberation_1%3a1.07.4-9_all.deb ...
Unpacking fonts-liberation (1:1.07.4-9) ...
Selecting previously unselected package libaacs0:arm64.
Preparing to unpack .../19-libaacs0_0.9.0-2_arm64.deb ...
Unpacking libaacs0:arm64 (0.9.0-2) ...
Selecting previously unselected package libbdplus0:arm64.
Preparing to unpack .../20-libbdplus0_0.1.2-3_arm64.deb ...
Unpacking libbdplus0:arm64 (0.1.2-3) ...
Selecting previously unselected package libplist3:arm64.
Preparing to unpack .../21-libplist3_2.0.1~git20190104.3f96731-1_arm64.deb ...
Unpacking libplist3:arm64 (2.0.1~git20190104.3f96731-1) ...
Selecting previously unselected package libusbmuxd4:arm64.
Preparing to unpack .../22-libusbmuxd4_1.1.0~git20181007.07a493a-1_arm64.deb ...
Unpacking libusbmuxd4:arm64 (1.1.0~git20181007.07a493a-1) ...
Selecting previously unselected package libimobiledevice6:arm64.
Preparing to unpack .../23-libimobiledevice6_1.2.1~git20181030.92c5462-2+deb10u1_arm64.deb ...
Unpacking libimobiledevice6:arm64 (1.2.1~git20181030.92c5462-2+deb10u1) ...
Selecting previously unselected package libu2f-udev.
Preparing to unpack .../24-libu2f-udev_1.1.9-1_all.deb ...
Unpacking libu2f-udev (1.1.9-1) ...
Selecting previously unselected package libupower-glib3:arm64.
Preparing to unpack .../25-libupower-glib3_0.99.10-1_arm64.deb ...
Unpacking libupower-glib3:arm64 (0.99.10-1) ...
Selecting previously unselected package notification-daemon.
Preparing to unpack .../26-notification-daemon_3.20.0-4_arm64.deb ...
Unpacking notification-daemon (3.20.0-4) ...
Selecting previously unselected package upower.
Preparing to unpack .../27-upower_0.99.10-1_arm64.deb ...
Unpacking upower (0.99.10-1) ...
Selecting previously unselected package usbmuxd.
Preparing to unpack .../28-usbmuxd_1.1.1~git20181007.f838cf6-1_arm64.deb ...
Unpacking usbmuxd (1.1.1~git20181007.f838cf6-1) ...
Setting up libgme0:arm64 (0.6.2-1) ...
Setting up libchromaprint1:arm64 (1.4.3-3) ...
Setting up libssh-gcrypt-4:arm64 (0.8.7-1+deb10u1) ...
Setting up libmpg123-0:arm64 (1.25.10-2) ...
Setting up libplist3:arm64 (2.0.1~git20190104.3f96731-1) ...
Setting up libminizip1:arm64 (1.1-8+b1) ...
Setting up libu2f-udev (1.1.9-1) ...
Setting up libwebpdemux2:arm64 (0.6.1-2+deb10u1) ...
Setting up chromium-sandbox (89.0.4389.114-1~deb10u1) ...
Setting up notification-daemon (3.20.0-4) ...
Setting up libvorbisfile3:arm64 (1.3.6-2) ...
Setting up libatomic1:arm64 (8.3.0-6) ...
Setting up libusbmuxd4:arm64 (1.1.0~git20181007.07a493a-1) ...
Setting up libaacs0:arm64 (0.9.0-2) ...
Setting up libevent-2.1-6:arm64 (2.1.8-stable-4) ...
Setting up libbdplus0:arm64 (0.1.2-3) ...
Setting up libxslt1.1:arm64 (1.1.32-2.2~deb10u1) ...
Setting up fonts-liberation (1:1.07.4-9) ...
Setting up libupower-glib3:arm64 (0.99.10-1) ...
Setting up libre2-5:arm64 (20190101+dfsg-2) ...
Setting up chromium-common (89.0.4389.114-1~deb10u1) ...
Setting up libbluray2:arm64 (1:1.1.0-1) ...
Setting up libimobiledevice6:arm64 (1.2.1~git20181030.92c5462-2+deb10u1) ...
Setting up libopenmpt0:arm64 (0.4.3-1+deb10u1) ...
Setting up libjsoncpp1:arm64 (1.7.4-3) ...
Setting up libavformat58:arm64 (7:4.1.8-0+deb10u1) ...
Setting up upower (0.99.10-1) ...
upower.service is a disabled or a static unit, not starting it.
Setting up usbmuxd (1.1.1~git20181007.f838cf6-1) ...
Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory
Adding system user `usbmux' (UID 108) ...
Adding new user `usbmux' (UID 108) with group `plugdev' ...
Not creating home directory `/var/lib/usbmux'.
usbmuxd.service is a disabled or a static unit, not starting it.
Setting up chromium (89.0.4389.114-1~deb10u1) ...
Processing triggers for desktop-file-utils (0.23-4) ...
Processing triggers for mime-support (3.62) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for libc-bin (2.28-10) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for dbus (1.12.20-0+deb10u1) ...
Processing triggers for fontconfig (2.13.1-2) ...
testuser@penguin:~$ Chromiumインストールが完了すると、アプリ一覧の「Linuxアプリ」内に、「Chromium ウェブ・ブラウザ」が出てくる。
chromium-driverのインストール
続いてchromium-driverをインストールする。
testuser@penguin:~$ sudo apt install chromium-driver
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
chromium-driver
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 4,157 kB of archives.
After this operation, 12.3 MB of additional disk space will be used.
Get:1 https://deb.debian.org/debian buster/main arm64 chromium-driver arm64 89.0.4389.114-1~deb10u1 [4,157 kB]
Fetched 4,157 kB in 3s (1,472 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package chromium-driver.
(Reading database ... 35171 files and directories currently installed.)
Preparing to unpack .../chromium-driver_89.0.4389.114-1~deb10u1_arm64.deb ...
Unpacking chromium-driver (89.0.4389.114-1~deb10u1) ...
Setting up chromium-driver (89.0.4389.114-1~deb10u1) ...
testuser@penguin:~$ Seleniumのインストール
続いてSeleniumをインストールする。
testuser@penguin:~$ sudo apt install python3-selenium
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
python3-six python3-urllib3
Suggested packages:
firefoxdriver python3-cryptography python3-idna python3-openssl python3-socks
The following NEW packages will be installed:
python3-selenium python3-six python3-urllib3
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 206 kB of archives.
After this operation, 1,038 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 https://deb.debian.org/debian buster/main arm64 python3-six all 1.12.0-1 [15.7 kB]
Get:2 https://deb.debian.org/debian buster/main arm64 python3-urllib3 all 1.24.1-1 [97.1 kB]
Get:3 https://deb.debian.org/debian buster/main arm64 python3-selenium all 3.14.1+dfsg1-1 [93.6 kB]
Fetched 206 kB in 1s (298 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package python3-six.
(Reading database ... 35176 files and directories currently installed.)
Preparing to unpack .../python3-six_1.12.0-1_all.deb ...
Unpacking python3-six (1.12.0-1) ...
Selecting previously unselected package python3-urllib3.
Preparing to unpack .../python3-urllib3_1.24.1-1_all.deb ...
Unpacking python3-urllib3 (1.24.1-1) ...
Selecting previously unselected package python3-selenium.
Preparing to unpack .../python3-selenium_3.14.1+dfsg1-1_all.deb ...
Unpacking python3-selenium (3.14.1+dfsg1-1) ...
Setting up python3-six (1.12.0-1) ...
Setting up python3-urllib3 (1.24.1-1) ...
Setting up python3-selenium (3.14.1+dfsg1-1) ...
testuser@penguin:~$ 以上で実行環境の準備が整った。
WEBブラウジング自動化の実行例
実行例として、新規タブを作成して検索キーワードを変えながらGoogle検索するプログラムを作成した。
from selenium import webdriver
import time
driver = webdriver.Chrome()
word1 = 'ラーメン 名店'
word2_list = ['しょうゆ', 'みそ', 'しお', 'とんこつ']
for word2 in word2_list:
# word1とword2を結合し、検索キーワードとする。
keyword = word1 + ' ' + word2
# 新規タブを開く。
driver.execute_script("window.open()")
# 新規タブに移動する。
driver.switch_to.window(driver.window_handles[-1])
# Googleページにアクセスする。
driver.get('https://google.com')
# 検索ボックスに検索キーワードを入力し、検索実行する。
search = driver.find_element_by_name('q')
search.send_keys(keyword)
search.submit()
# 1秒間待機
time.sleep(1)
# 5秒間待機し、ブラウザを閉じる。
time.sleep(5)
driver.quit()Linuxターミナルを起動し、このPythonプログラムを実行する。
testuser@penguin:~$ python3 test.pyすると自動的にChromiumブラウザが起動し、新規タブを作成しながらGoogle検索していく。
検索終了5秒後、自動的にブラウザが閉じられる。
尚、自動操作するとブラウザ上部に「Chrome is being controlled by automated test software.(自動テストソフトソフトウェアによって制御されています)」と表示される。