WSL2上のUbuntu 22.04.3 LTSでSpark and Icebergを試してみます。
まずはDocker/Docker Composeをインストールします。以下のサイト通りに実施すれば使えるようになります。
Install Docker Engine on Ubuntu | Docker Docs
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
続いて以下のサイトにある通りにdocker-compose.ymlを準備し、Spark and Icebergを試す環境を作ります。
Spark and Iceberg Quickstart
docker-compose.yml
version: "3"
services:
spark-iceberg:
image: tabulario/spark-iceberg
container_name: spark-iceberg
build: spark/
networks:
iceberg_net:
depends_on:
- rest
- minio
volumes:
- ./warehouse:/home/iceberg/warehouse
- ./notebooks:/home/iceberg/notebooks/notebooks
environment:
- AWS_ACCESS_KEY_ID=admin
- AWS_SECRET_ACCESS_KEY=password
- AWS_REGION=us-east-1
ports:
- 8888:8888
- 8080:8080
- 10000:10000
- 10001:10001
rest:
image: tabulario/iceberg-rest
container_name: iceberg-rest
networks:
iceberg_net:
ports:
- 8181:8181
environment:
- AWS_ACCESS_KEY_ID=admin
- AWS_SECRET_ACCESS_KEY=password
- AWS_REGION=us-east-1
- CATALOG_WAREHOUSE=s3://warehouse/
- CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
- CATALOG_S3_ENDPOINT=http://minio:9000
minio:
image: minio/minio
container_name: minio
environment:
- MINIO_ROOT_USER=admin
- MINIO_ROOT_PASSWORD=password
- MINIO_DOMAIN=minio
networks:
iceberg_net:
aliases:
- warehouse.minio
ports:
- 9001:9001
- 9000:9000
command: ["server", "/data", "--console-address", ":9001"]
mc:
depends_on:
- minio
image: minio/mc
container_name: mc
networks:
iceberg_net:
environment:
- AWS_ACCESS_KEY_ID=admin
- AWS_SECRET_ACCESS_KEY=password
- AWS_REGION=us-east-1
entrypoint: >
/bin/sh -c "
until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
/usr/bin/mc rm -r --force minio/warehouse;
/usr/bin/mc mb minio/warehouse;
/usr/bin/mc policy set public minio/warehouse;
tail -f /dev/null
"
networks:
iceberg_net:
サイトにはdocker-compose upで起動するように書かれていますが、先ほどインストールしたDocker ComposeはVersion 2なので以下のコマンドで起動します。
> docker compose up
spark-icebergが起動できたらSparkSQLを使ってテーブルを作り、INSERTとSELECTを試してみます。
もう１つコンソールを開いてspark-icebergに接続し、spark-sqlを立ち上げます。
> docker exec -it spark-iceberg spark-sql
以下実行結果
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
24/01/01 09:39:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/01/01 09:39:43 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
Spark Web UI available at http://2157a499f607:4041
Spark master: local[*], Application Id: local-1704101983121
spark-sql ()>
続いてCREATE TABLE/INSERT/SELECTの実行。
以下CREATE TABLE実行結果
spark-sql ()> CREATE TABLE demo.nyc.taxis
> (
> vendor_id bigint,
> trip_id bigint,
> trip_distance float,
> fare_amount double,
> store_and_fwd_flag string
> )
> PARTITIONED BY (vendor_id);
Time taken: 2.788 seconds
以下INSERT実行結果
spark-sql ()> INSERT INTO demo.nyc.taxis
> VALUES (1, 1000371, 1.8, 15.32, 'N'), (2, 1000372, 2.5, 22.15, 'N'), (2, 1000373, 0.9, 9.01, 'N'), (1, 1000374, 8.4, 42.13, 'Y');
Time taken: 4.687 seconds
以下SELECT実行結果
spark-sql ()> SELECT * FROM demo.nyc.taxis;
1 1000371 1.8 15.32 N
1 1000374 8.4 42.13 Y
2 1000372 2.5 22.15 N
2 1000373 0.9 9.01 N
Time taken: 0.896 seconds, Fetched 4 row(s)
