Spark Connector

The Spark Connector enables querying of data using a Spark Session. This is useful for querying local instances of Spark or Delta. However, the most useful application of this connector is to leverage Spark Connect to enable connecting to a remote Spark Cluster to provide the compute for the query being run from a local machine.

from rtdip_sdk.connectors import SparkConnection

spark_server = "spark_server"
access_token = "my_token"

spark_remote = "sc://{}:443;token={}".format(spark_server, access_token)
connection = SparkConnection(spark_remote=spark_remote)

Replace the access_token with your own authentiction token.

← Previous Next →

Spark Connector

Course Progress