java - Spark Streaming Graceful Shutdown -
i saw there option configure graceful shutdown in 2 places:
- while defining spark conf:
"spark.streaming.stopgracefullyonshutdown", "true".
- when stopping streaming context:
javastreamingcontext scc;
scc.stop(true,true)
what differences between 2 options?
thanks
the first way
sparkconf.set(“spark.streaming.stopgracefullyonshutdown","true")
setting parameter true in spark configuration ensures proper graceful shutdown in new spark version (1.4 onwards) applications. should not use 1st explicit shutdown hook approach or call
ssc.stop
method in driver along parameter . can set parameter, , call methodsssc.start()
,ssc.awaittermination()
. no need callssc.stop
method. otherwise application might hung during shutdown.
the second way scc.stop(true,true)
the
scc.stop()
method’s 1st boolean argument stopping associated spark context while 2nd boolean argument graceful shutdown of streaming context
this has been mention here chandan prakash
hope helps!
Comments
Post a Comment