How to programmatically Get Number of Cores

Cluster Executor

Cluster executors can also be referred to as cluster workers. If you’re aware of Hadoop MapReduce, you can think of driver as master node that divide jobs into different tasks and allocate those tasks to the data nodes. Executors are responsible for executing tasks and reporting to the driver the result.

You can see the difference between executors and driver, according to AI below.

Getting Number of Cluster Cores Within Spark Application

Note: I will be using Scala

import java.lang.Runtime.getRuntime
val processor = getRuntime.availableProcessors()
* (sparkSn.sparkContext.statusTracker.getExecutorInfos.length 1)

The availableProcessors() is the total number of cores per executor or you can think of it as the number of driver cores since driver and workers have the same number of cores.

The (sparkSn.sparkContext.statusTracker.getExecutorInfos.length 1) is basically the total number of workers minus the driver. You can subtract the driver and only get the number of workers. For example, if you want to do repartition base on the number of cores, you want to use just the executor cores since the executors are responsible for executing the tasks.

Difference Between Executor and Driver


  • Cluster executors are worker nodes within a distributed computing cluster. They are responsible for executing tasks in parallel on the data distributed across the cluster.
  • Each executor typically runs on a separate machine or container within the cluster.
  • Executors are responsible for processing data, running computations, and performing tasks such as map and reduce operations in parallel.
  • They communicate with the driver (the master node) to receive instructions and report the results of their computations.
  • Executors are typically numerous and can scale horizontally based on the size of the cluster and the computational needs of the job.


  • The driver is the central control node of a distributed computing application.
  • It is responsible for coordinating the entire job or application, including distributing tasks to the cluster executors, monitoring progress, and aggregating results.
  • The driver initiates the execution of a distributed job, divides it into tasks, and schedules those tasks to be executed by the cluster executors.
  • It collects and processes the results from the executors, making high-level decisions about the job’s flow and handling failures or retries.
  • In many distributed frameworks, the driver also manages the job’s configuration and can interact with external data sources or storage systems.


In summary, cluster executors are the worker nodes responsible for performing distributed computations in parallel, while the driver is the master node responsible for coordinating the execution of the distributed job or application. Together, they form the core components of a distributed computing system, where the driver manages the overall execution and the executors perform the actual computation.
Spread the love

Leave a comment and use the rating scales to rate this review.?

Leave a Reply

Your email address will not be published. Required fields are marked *

Top Reviews

Adidas Falcon Review – Pros and Cons

Buy NowRead Review

Adidas Forest Grove Review – Pros and Cons

Buy NowRead Review

Adidas NMD R1 Shoes – Reasons to Buy/Not

Buy NowRead Review

Adidas Deerupt Runner Review – Pros and Cons

Buy NowRead Review

Adidas Originals Supercourt Review-Pros and Cons

Buy NowRead Review

Adidas Originals Stan Smith Review

Buy NowRead Review

Adidas Nite Jogger Review – Pros and Cons

Buy NowRead Review

Adidas Ozweego Review- Pros and Cons

Buy NowRead Review

Adidas Cloudfoam Ultimate Review – Pros and Cons

Buy NowRead Review

Adidas Continental 80 Review – Pros and Cons

Buy NowRead Review