I cannot use --package option on bitnami/spark docker container

Found the solution to itas given in https://github.com/bitnami/bitnami-docker-spark/issues/7what we have to do is create a volume on host mapped to docker path

volumes:  - ./jars_dir:/opt/bitnami/spark/ivy:z

give this path as cache path like this

spark-shell --conf spark.jars.ivy=/opt/bitnami/spark/ivy --confspark.cassandra.connection.host=127.0.0.1 --packagescom.datastax.spark:spark-cassandra-connector_2.12:3.0.0-beta --confspark.sql.extensions=com.datastax.spark.connector.CassandraSparkExtensions

All happened because /opt/bitnami/spark is not writable and we have to mount a volume to bypass that.

The error "java.io.FileNotFoundException: /opt/bitnami/spark/.ivy2/" occured because the location /opt/bitnami/spark/ is not writable. so in order to resolve this issue do modify the master spark service like this.Added user as root and add mounted volume path for required jars.

see the working block of spark service written in docker compose:

spark:image: docker.io/bitnami/spark:3container_name: sparkenvironment:  - SPARK_MODE=master  - SPARK_RPC_AUTHENTICATION_ENABLED=no  - SPARK_RPC_ENCRYPTION_ENABLED=no  - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no  - SPARK_SSL_ENABLED=nouser: rootports:  - '8880:8080'volumes:  - ./spark-defaults.conf:/opt/bitnami/spark/conf/spark-defaults.conf  - ./jars_dir:/opt/bitnami/spark/ivy:z