pyspark - Why can't I see my spark job on port 4040 even though I see it running in the yarn UI? -
it understanding can see spark jobs @ port 4040 on master node.
i running spark on yarn, , i'm running job through spark submit. when go yarn ui, able see job running
, , when go spark master @ <master ip>:18081
can see spark job picked master in history server.
(and of course not see runing @ <master ip>:18080
because i'm not running in standalone).
however, not able see job running @ port 4040 in yarn ui.
job not fail - hangs forever. , can see workers alive in spark master ui.
this resource utilization shown yarn while running:
how fix show spark job running, or @ least give me feedback on whether or not job progressing?
Comments
Post a Comment