Run Azure Databricks without Spark cluster
You always have to have a "cluster", but it can be a single node cluster (only with a driver node). Set the number of worker nodes to zero for this configuration. You are not able to run Spark on a driver-only cluster. See the following example using MXNet on a driver-only cluster.
Another option that is available with September 2020 platform release is Single Node Cluster.You can select "Single-Node" from the Cluster Mode to create a single-node cluster with Spark running in local mode.