linux - Collect logs for a function in python using shell script -


i have pyspark script working fine. script fetch data mysql , create hive tables in hdfs.

the pyspark script below.

#!/usr/bin/env python import sys pyspark import sparkcontext, sparkconf pyspark.sql import hivecontext conf = sparkconf() sc = sparkcontext(conf=conf) sqlcontext = hivecontext(sc)  #condition specify exact number of arguments in spark-submit command line if len(sys.argv) != 8:     print "invalid number of args......"     print "usage: spark-submit import.py arguments"     exit() table = sys.argv[1] hivedb = sys.argv[2] domain = sys.argv[3] port=sys.argv[4] mysqldb=sys.argv[5] username=sys.argv[6] password=sys.argv[7]  df = sqlcontext.read.format("jdbc").option("url", "{}:{}/{}".format(domain,port,mysqldb)).option("driver", "com.mysql.jdbc.driver").option("dbtable","{}".format(table)).option("user", "{}".format(username)).option("password", "{}".format(password)).load()  #register dataframe table df.registertemptable("mytemptable")  # create hive table temp table: sqlcontext.sql("create table {}.{} select * mytemptable".format(hivedb,table))  sc.stop()

now pyspark script invoked using shell script. shell script passing table names arguments file.

the shell script below.

#!/bin/bash  source /home/$user/spark/source.sh [ $# -ne 1 ] && { echo "usage : $0 table ";exit 1; }  args_file=$1  timestamp=`date "+%y-%m-%d"` touch /home/$user/logs/${timestamp}.success_log touch /home/$user/logs/${timestamp}.fail_log success_logs=/home/$user/logs/${timestamp}.success_log failed_logs=/home/$user/logs/${timestamp}.fail_log  #function status of job creation function log_status {        status=$1        message=$2        if [ "$status" -ne 0 ];                 echo "`date +\"%y-%m-%d %h:%m:%s\"` [error] $message [status] $status : failed" | tee -a "${failed_logs}"                 #echo "please find attached log file more details"                 exit 1                 else                     echo "`date +\"%y-%m-%d %h:%m:%s\"` [info] $message [status] $status : success" | tee -a "${success_logs}"                 fi } while read -r table ;do    spark-submit --name "${table}" --master "yarn-client" --num-executors 2 --executor-memory 6g  --executor-cores 1 --conf "spark.yarn.executor.memoryoverhead=609" /home/$user/spark/sql_spark.py ${table} ${hivedb} ${domain} ${port} ${mysqldb} ${username} ${password} > /tmp/logging/${table}.log 2>&1   g_status=$?   log_status $g_status "spark job ${table} execution" done < "${args_file}"  echo "************************************************************************************************************************************************************************"

i able collect logs each individual table in args_file using above shell script.

now have more 200 tables in mysql. have modified pyspark script below. have create function itreate on args_file , execute code.

new spark script

#!/usr/bin/env python import sys pyspark import sparkcontext, sparkconf pyspark.sql import hivecontext conf = sparkconf() sc = sparkcontext(conf=conf) sqlcontext = hivecontext(sc)  #condition specify exact number of arguments in spark-submit command line if len(sys.argv) != 8:     print "invalid number of args......"     print "usage: spark-submit import.py arguments"     exit() args_file = sys.argv[1] hivedb = sys.argv[2] domain = sys.argv[3] port=sys.argv[4] mysqldb=sys.argv[5] username=sys.argv[6] password=sys.argv[7]  def testing(table, hivedb, domain, port, mysqldb, username, password):      print "*********************************************************table = {} ***************************".format(table)     df = sqlcontext.read.format("jdbc").option("url", "{}:{}/{}".format(domain,port,mysqldb)).option("driver", "com.mysql.jdbc.driver").option("dbtable","{}".format(table)).option("user", "{}".format(username)).option("password", "{}".format(password)).load()      #register dataframe table     df.registertemptable("mytemptable")      # create hive table temp table:     sqlcontext.sql("create table {}.{} stored parquet select * mytemptable".format(hivedb,table))  input = sc.textfile('/user/xxxxxxx/spark_args/%s' %args_file).collect()  table in input:  testing(table, hivedb, domain, port, mysqldb, username, password)  sc.stop()

now want collect logs individual table in args_file. getting 1 log file has log tables.

how can achieve requirement? or method doing wrong

new shell script:

spark-submit --name "${args_file}" --master "yarn-client" --num-executors 2 --executor-memory 6g  --executor-cores 1 --conf "spark.yarn.executor.memoryoverhead=609" /home/$user/spark/sql_spark.py ${table} ${hivedb} ${domain} ${port} ${mysqldb} ${username} ${password} > /tmp/logging/${args_file}.log 2>&1


Comments

Post a Comment

Popular posts from this blog

android - InAppBilling registering BroadcastReceiver in AndroidManifest -

i'm implementing inapp billing in android app. works well, however, i'm trying decouple broadcast receiver activity manifest. suggestion in android's trivialdrive sample: // important: dynamically register broadcast messages updated purchases. // register receiver here instead of <receiver> in manifest // because call getpurchases() @ startup, therefore can ignore // broadcasts sent while app isn't running. // note: registering listener in activity bad idea, done here // because sample. regardless, receiver must registered after // iabhelper setup, before first call getpurchases(). currently there's class extends broadcastreceiver : public class iabbroadcastreceiver extends broadcastreceiver { /** * intent action receiver should filter for. */ public static final string action = "com.android.vending.billing.purchases_updated"; private final iabbroadcastlistener mlistener; public iabbroadcastreceiver(iabbroadcastlistener listener) { m...
Read more

python Tkinter Capturing keyboard events save as one single string -

is possible store keyboard events 1 string? code below stores 1 char , prints it. card reader or bar code reader, contains collection of character/ string not 1 character @ time. goal save char pressed text variable. from tkinter import * root = tk() def key(event): text= event.char text+= event.char print ("pressed", text) def callback(event): frame.focus_set() print ("clicked at", event.x, event.y) frame = frame(root, width=100, height=100) frame.bind("<key>", key) frame.bind("<button-1>", callback) frame.pack() root.mainloop() currently, creating text variable , everytime key() function gets called, , text stores last character typed. you can define text module level variable , , use module level text inside key function - text = '' def key(event): global text text+= event.char print("pressed", text)
Read more

sql server - Why does Linq-to-SQL add unnecessary COUNT()? -

here tables: create table customer ( cid int primary key, name varchar(32) ) create table ord ( oid int primary key, cid int foreign key references customer, address varchar(20) ) here's linq-to-sql statement: var aaa = c in db.customer select new { c, o = c.ord.tolist() }; and here's inexplicable query generated linq-to-sql: select [t0].[cid] [cid], [t0].[name] [name], [t1].[oid] [oid], [t1].[cid] [cid2], [t1].[address] [address], (select count(*) [dbo].[ord] [t2] [t2].[cid] = [t0].[cid]) [value] [dbo].[customer] [t0] left outer join [dbo].[ord] [t1] on [t1].[cid] = [t0].[cid] order [t0].[cid], [t1].[oid] i want understand how rid of count(*) part. it's entirely uncalled for! linq needs count(*) in order determine number of entries returned o = c.ord.tolist() .
Read more