Download
Sparkmagic is a set of tools for interactively working with remote Spark clusters in Jupyter notebooks. Sparkmagic interacts with remote Spark clusters through a REST server. Automatic visualization of SQL queries in the PySpark, Spark and SparkR kernels; use an easy visual interface to interactively construct visualizations, no code required. Ability to capture the output of SQL queries as Pandas dataframes to interact with other Python libraries (e.g. matplotlib). Send local files or dataframes to a remote cluster (e.g. sending pretrained local ML model straight to the Spark cluster) Authenticate to Livy via Basic Access authentication or via Kerberos.
Features
- For running interactive sessions on Yarn
- For running interactive sessions on Yarn or Kubernetes (only PySpark sessions are supported)
- For running interactive sessions on Yarn or Kubernetes
- The Sparkmagic project includes a set of magics for interactively running Spark code in multiple languages
- Run Spark code in multiple languages against any remote Spark cluster through Livy
- Automatic SparkContext (sc) and HiveContext (sqlContext) creation
- Easily execute SparkSQL queries with the %%sql magic
Project Samples
Categories
Operating System KernelsLicense
MIT LicenseFollow sparkmagic
Other Useful Business Software
Build on Google Cloud with $300 in Free Credit
Start your next project with $300 in free Google Cloud credit. Spin up VMs, run containers, query exabytes in BigQuery, or build AI apps with Vertex AI and Gemini. Once your credits are used, keep building with 20+ products with free monthly usage, including Compute Engine, Cloud Storage, GKE, and Cloud Run functions. Sign up to start building right away.
Rate This Project
Login To Rate This Project
User Reviews
Be the first to post a review of sparkmagic!
