Big data interview Questions

• Spark prevents the unnecessary disk I/O operations by doing processing in main memory of worker nodes where as MapReduce make use of disks to store intermediate results.
• Spark comes with  advanced directed acyclic graph (DAG) that is more optimized and efficient as compared to MapReduce.
•  Apache Spark DAGS are lazily evaluated which means it delays its evaluation till it is absolutely necessary. This is one of the key factors contributing to its speed 

Spark creates a single partition for a single input split. For example,

•  In case of text file one spark partition is approximately equal to one HDFS block. (HDFS Blocks are split based on HDFS Block size where as partitions use line splitting)
• In case of compressed file you would get a single partition for a single file (as compressed text files are not splittable).

In Spark, we can use Cache() and Persist() to save RDD intermediate results.

• In Cache() - Default Storage level is Memory_Only which means that RDD intermediate results are stored on main memory.  If the size of RDD is greater than memory, It will not cache some partition and recompute them next time whenever needed.
• In Persist() - there are various other storage level as well such as   Memory_Only,Memory_and_Disk,Memory_Only_Ser,Memory_and_Disk_Ser,Disk_Only

• Memory_Only - In this persistence level, data is stored in memory so space requirement is high and CPU Computation is low. It is comparatively faster but less resilient.  If the size of RDD is greater than memory, It will not cache some partition and recompute them next time.
• Memory_And_Disk - In this persistence level,  if  RDD size is more than the size of memory, the excess partition are stored  on the disk, and retrieved from disk whenever required. Space requirement is high and CPU Computation is medium. It is more resilient but less faster than Memory_Only

Spark context is the entry gate and  allows spark application to access cluster using resource manager such as Yarn.

• It encapsulates all  Spark configurations.
• It can be used to access services like TaskScheduler, BlockManager, ShuffelManager etc
• It can be used to persist and unpersist RDDs
• It can be used for dynamic allocation of executors: requestExecutors, killExecutors, requestTotalExecutors, getExecutorIds. 
• To register Spark listeners.

• In Checkpointing, the lineage is destroyed and intermediate rdds are physically stored on hdfs.
• Checkpointing data remain intact even if Spark context stops.
• Checkpointing data can be used as an entry point in another driver program.
• Checkpointing is a costly operation because computation is done twice once to cache the result in memory and another to checkpoint it to storage.
• In Persist/Cache - Lineage is not destroyed. If Spark Context stops, the persisted and cached data is also cleared.

 Although you can create multiple Spark context in a single JVM by setting the spark configuration spark.driver.allowMultipleContexts to true. However, Multiple Spark context is discouraged as it can lead to unexpected behaviour

In Spark, repartitioning can be done using two API's : repartition and coleasce.  

• repartition can be used to increase and decrease the number of partitions and cause complete data shuffling and therefore is a costly operation 
• Coleasce can only be used to decrease the number of partitions and do not cause complete data shuffling and is recommended over repartition when partition size is to be reduced.

Speculative Execution is an optimization technique in Spark in which if any tasks is taking too long , Spark automatically launch a parallel task in another worker and Spark accepts the result of worker whichever completes first. Spark speculative excecution is disabled by default and can be enabled by setting spark.speculation property to true.

• Narrow transformations: Narrow transformations are the one that do not cause data shuffling such as Map. Filter , FlatMap etc.

• Wide Transformations:  are the transformations that cause data to be shuffled across the cluster such as groupByKey, Join, reduceByKey etc.

Spark Driver is a program that runs on master node and defines actions and transformations on RDD.

• It creates SparkContext to connect to spark cluster using resource Manager.
• It splits Spark Application into tasks and stages and fed them to DAGScheduler & TaskScheduler.
• It coordinates with workers for overall execution.
• It also hosts Spark Web UI.

Cluster Manager is responsible for managing and allocating resources in a cluster. Spark can run on many cluster managers:

• YARN
• MESOS
• SPARK STANDALONE
• KUBERNETES