hadoop - Apache Phoenix 4.7 csvBulkLoad.run() throws ClassNotFoundException -


we running spark streaming job read kafka, convert csv , write hbase. using api csvbulkload run bulk load job. spark job starts fine , converts csv csvbulkload.run() starts new mr job fails exception

error: java.lang.classnotfoundexception: org.apache.commons.csv.csvformat     @ java.net.urlclassloader.findclass(urlclassloader.java:381)     @ java.lang.classloader.loadclass(classloader.java:424)     @ sun.misc.launcher$appclassloader.loadclass(launcher.java:331)     @ java.lang.classloader.loadclass(classloader.java:357)     @ org.apache.phoenix.mapreduce.csvtokeyvaluemapper$csvlineparser.<init>(csvtokeyvaluemapper.java:96)     @ org.apache.phoenix.mapreduce.csvtokeyvaluemapper.setup(csvtokeyvaluemapper.java:69)     @ org.apache.hadoop.mapreduce.mapper.run(mapper.java:143)     @ org.apache.hadoop.mapred.maptask.runnewmapper(maptask.java:787)     @ org.apache.hadoop.mapred.maptask.run(maptask.java:341)     @ org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:170)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:422)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1865)     @ org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:164)

this spark job command

spark-submit --jars $(echo lib/*.jar | tr ' ' ',') --class "com.lam.app.driver" --name "mybulkloader" --master yarn --deploy-mode cluster --driver-class-path $(echo lib/*.jar | tr ' ' ',') --driver-memory 4g --executor-memory 1g --num-executors 2 --executor-cores 2 --files conf.properties pipeline-1.0.jar

earlier used error missing class (java.lang.noclassdeffounderror: com/yammer/metrics/core/metricsregistry) got resolved after including "--driver-class-path $(echo lib/*.jar | tr ' ' ',') ". getting error missing csvformat class.

my lib directory has jar files needed including commons-csv jar still getting error. use maven build project. below code snippet running csvbulkload

csvbulkloadtool csvbulkloadtool = new csvbulkloadtool(); final org.apache.hadoop.conf.configuration conf = hbaseconfiguration.create(); system.out.println("zkquorum " + this.zkquorum); conf.set(hconstants.zookeeper_quorum, this.zkquorum); conf.set("hbase.zookeeper.property.clientport", string.valueof(2181)); conf.set("zookeeper.znode.parent", "/hbase-unsecure");  csvbulkloadtool.setconf(conf); int exitcode = csvbulkloadtool.run(new string[] { "--input", "\"" + this.hdfsinputfile + "\"",     "--table", this.tablename, "--zookeeper", this.zkquorum + ":2181:/hbase-unsecure" }); system.out.println("return code of wdl bulk load execution " + exitcode);

my exitcode -1 always. please let me know if missing in class path

i resolved finally. had missed adding phoenix client jar , issue got resolved after that. hope tip helps come across similar issue


Comments

Post a Comment

Popular posts from this blog

PHP and MySQL WP -

i trying accomplish 2 things here code. writing php directly wp page using insert php plugin. 1) query mysql database , return results 1 column 4 columns. 2) make each result own hyperlink directs user new wp page runs new query on page. i able query show results , turn results dynamic hyperlink, cannot formatting down. right returns result 1 column. here have: [insert_php] $servername = "localhost"; $username = "login"; //edited privacy $password = "password"; //edited privacy $database = "database"; //edited privacy // create connection $conn = new mysqli($servername, $username, $password, $database); // check connection if ($conn->connect_error) { die("connection failed: " . $conn->connect_error); } echo "connected successfully"; //get results database $result = mysqli_query($conn,"select mine mines order mine"); $all_property = array(); //declare array saving property //showing proper...
Read more

android - InAppBilling registering BroadcastReceiver in AndroidManifest -

i'm implementing inapp billing in android app. works well, however, i'm trying decouple broadcast receiver activity manifest. suggestion in android's trivialdrive sample: // important: dynamically register broadcast messages updated purchases. // register receiver here instead of <receiver> in manifest // because call getpurchases() @ startup, therefore can ignore // broadcasts sent while app isn't running. // note: registering listener in activity bad idea, done here // because sample. regardless, receiver must registered after // iabhelper setup, before first call getpurchases(). currently there's class extends broadcastreceiver : public class iabbroadcastreceiver extends broadcastreceiver { /** * intent action receiver should filter for. */ public static final string action = "com.android.vending.billing.purchases_updated"; private final iabbroadcastlistener mlistener; public iabbroadcastreceiver(iabbroadcastlistener listener) { m...
Read more

go - golang pprof for c library code -

i have been trying profile golang application pprof, seems details of c code(linked static library) missed in output result. flame graph from frame graph can see c code time consuming reported within "runtime._externalcode", no details @ all. , sure static library compiled debug enable. so there way make pprof details(like stack trace) of 3rd party c code? or there other golang profiling tool this?
Read more