In spark documentation, we’ve information about learn how to run spark driver in a separate pod in Shopper Mode Networking. However we haven’t any details about operating the driving force in host system itself as a substitute of operating the driving force in separate pod in a kubernetes cluster. Have the next questions.
- How one can we explicitly configure to run driver in a separate pod or
operating within the host system itself. What’s the profit of getting
driver run in separate pod
I’m going through points with operating within the host system (guess that is the default mode when consumer mode is talked about when operating in k8s and solely executor pictures are offered within the config.) . Driver not ready entry executor with ip deal with to fetch block information. I can see that its as a result of the driving force and executors are in two completely different networks.
I See many blogs to run in consumer mode with spark driver in a separate pod, however not for operating it within the host system itself.
Any options or instructions will probably be vastly useful