Develop Apache Spark Apps with IntelliJ IDEA
 |
 |
|---|---|
This tutorial helps you to start you first Spark program in IntelliJ IDEA.
1) Download and install latest Intellij IDEA community edition
https://www.jetbrains.com/idea/download/#section=windows
2) Start Intellij IDEA
Skip all configurations at startup and don’t install anything. Until you get to the “create new project screen”, pictured below:
3) Install Scala Plugin
On this page go to the Configure => Plugins => Browse Repositories
4) Create and configure a new Scala and Maven Project
- Create a new Maven project
- Choose the JAVA SDK installed on your computer
- Don’t choose any architectural pattern
- Give the name for you environment:
- Give the name for your projects and choose the project directory
- After that, you should see the following project structure
5) Configure libraries for the newly created project
Install Scala SDK
- Go to _File => Project Structure => Libraries => Add new library (+) _and download compatible with your Spark's Scala SDK version
Configure dependencies for Spark App
Once this is done Intellij will create the project and will open the pom.xml.Lets analyze this pom.xml.
In order to find appropriate dependencies go to https://mvnrepository.com/artifact/org.apache.spark and find all needed dependencies, copy the dependencies into pom file. For example, you have to add at least the spark core library:
Open the pom.xml file and click the “pom.xml” tab. Paste in the Spark dependency and then click “import” for dependencies command:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>alina</groupId>
<artifactId>org</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.1.0</version>
</dependency>
</dependencies>
</project>
Start your first simple Spark program
Add new Scala module (Java => New Scala Class) and code a simple spark program:
import org.apache.spark.{SparkConf, SparkContext}
object TestScala {
def main(args: Array[String]): Unit = {
val conf = new SparkConf()
conf.setAppName("Spark Hello World")
conf.setMaster("local[2]")
val sc = new SparkContext(conf)
println(sc) }
}
Code and run the following Scala test object to validate your configuration works. Correct output should look like this:
