An open-source unified analytics engine for analyzing enormous amounts of data is Apache Spark. Spark processes data in memory, making it 100 times quicker than Hadoop. It has the capacity to process a significant volume of data very quickly. In-memory data processing is Spark’s most crucial feature.
Interview questions about Apache Spark
In this blog, we’ll go through the key points and most typical interview questions about Apache Spark, which is one of the interviewer’s favourite topics in big data interviews.
Which spark is it?
An all-purpose in-memory computation engine is Spark. Any storage system, including local storage, HDFS, Amazon S3, and others, can be connected to it. Furthermore, it gives you the flexibility to choose the resource management of your choice.
What does Apache Spark’s RDD mean?
Resilient Distributed Datasets are referred to as RDDs. It is the most crucial component in every spark application. It is unchangeable.
What distinguishes Spark Context from Spark Session?
Spark context
The primary point of access for the spark features is spark Context. It symbolises the link between Spark clusters, which is important for creating RDDs, accumulators, and broadcast variables on the cluster.
Spark Session
Before the release of Spark 2.0, we needed various contexts to access the various Spark functionalities. In contrast, Spark 2.0 has a single entry point called Spark Session. It includes Streaming Context, HiveContext, and SQLContext. There’s no need to make different ones.
What is broadcast variable?
In Spark, read-only data sharing is accomplished through the use of broadcast variables. Without broadcast variables, every time an executor performs a transformation or action, we must ship the data to each one, which can add network cost.
Describe Pair RDD
Key-value pairs are gathered in the Spark Paired RDDs. Two data items make up a key-value pair (KVP). The value is the information that corresponds to the key value, whereas the key is the identification. On RDDs comprising key-value pairs, a few specific operations are allowed.
What distinguishes the RDD cache() and persist() methods?
The strategies for optimization are persistence and caching systems.
Persist():
Results of calculations are stored in its partitions when using the persist() technique. When using Java and Scala, the persistent method will store the data in the JVM. In contrast, when we call the persist method in Python, the data is serialised. The data might be kept on the disc or in memory. Both can be used in tandem.
Cache():
The only difference between it and the persist method is that the cache keeps the calculations’ results at the default storage level.
Describe Spark Core
All spark applications are built on the Spark Core. Memory management, fault recovery, scheduling, job distribution & monitoring, and communication with storage systems are among the functionalities it performs. Application programming interfaces (APIs) created in Java, Scala, Python, and R can be used to access Spark Core.
Describe RDD Lineage.
A graph called RDD Lineage, also known as an RDD operator graph or RDD dependency graph, comprises all of an RDD’s parent RDDs.
What does Spark’s Accumulator shared variables mean?
Read-only shared variables are accumulators. Only an associative and commutative operation may “add” them. They can be utilised to implement counters or sums. Numeric kinds are already supported by Spark’s native accumulators, and you may easily add support for additional types.
What distinguishes an RDD from a Data Frame?
Data Frame:-
The data is kept in tabular form in a data frame. It is a dispersed collection of data organized into columns and rows. Data types like numeric, logical, factor, or character can all be stored in the columns. Larger dataset processing is facilitated by it.
RDD:-
An RDD (Resilient Distributed Dataset) is a group of components dispersed over several cluster nodes. RDDs are fault-tolerant and immutable.