apache spark - How to format JSON string after conversion from pyspark dataframe -


i have converted data frame json using tojson in pyspark gives me each row json string. want reformat bit

my code given below:

spark=sparksession.builder.config("spark.sql.warehouse.dir", "c:\spark\spark-warehouse").appname("testapp").enablehivesupport().getorcreate() sqlstring="select lflow1.leasetype leasetype, lflow1.status status, lflow1.property property, lflow1.city city, lesflow2.dealtype dealtype, lesflow2.area area, lflow1.did did, lesflow2.mid mid lflow1, lesflow2  lflow1.did = lesflow2.mid"  def querybuilder(sqlval):     df=spark.sql(sqlval)     df.show()     return df  result=querybuilder(sqlstring) resultlist=result.tojson().collect() print(resultlist) print("type of",type(resultlist))

after this, output is:

[     '{"leasetype":"offer lease","status":"fully executed","property":"10230104","city":"edmonton","dealtype":"renewal","area":"2312","did":"79cc3959ffc8403f943ff0e7e93584f8","mid":"79cc3959ffc8403f943ff0e7e93584f8"}',     '{"leasetype":"offer renew","status":"fully executed","property":"1040hami","city":"vancouver","dealtype":"renewal","area":"784","did":"ecf922d0583247c0a4cb591bd4b3843e","mid":"ecf922d0583247c0a4cb591bd4b3843e"}',      '{"leasetype":"offer renew","status":"fully executed","property":"1040hami","city":"vancouver","dealtype":"renewal","area":"2223","did":"ecf922d0583247c0a4cb591bd4b3843e","mid":"ecf922d0583247c0a4cb591bd4b3843e"}',      '{"leasetype":"offer lease","status":"conditional","property":"106portw","city":"toronto","dealtype":"renewal","area":"2212","did":"69c3af0527014fd99d1ccf156c0bce93","mid":"69c3af0527014fd99d1ccf156c0bce93"}',      '{"leasetype":"offer lease","status":"fully executed","property":"106portw","city":"toronto","dealtype":"0","area":"","did":"04aedb01da5d44fead7e1315115c2530","mid":"04aedb01da5d44fead7e1315115c2530"}' ]

but want format json array example: following 2 rows:

[     {         "leasetype": "offer lease",         "status": "fully executed",         "property": "10230104",         "city": "edmonton",         "dealtype": "renewal",         "area": "2312",         "did": "79cc3959ffc8403f943ff0e7e93584f8",         "mid": "79cc3959ffc8403f943ff0e7e93584f8"     },     {         "leasetype": "offer renew",         "status": "fully executed",         "property": "1040hami",         "city": "vancouver",         "dealtype": "renewal",         "area": "784",         "did": "ecf922d0583247c0a4cb591bd4b3843e",         "mid": "ecf922d0583247c0a4cb591bd4b3843e"     } ]

i want omit ' here.

kindly me figure out.

hope helps!

import re import json  resultlist = [     '{"leasetype":"offer lease","status":"fully executed","property":"10230104","city":"edmonton","dealtype":"renewal","area":"2312","did":"79cc3959ffc8403f943ff0e7e93584f8","mid":"79cc3959ffc8403f943ff0e7e93584f8"}',     '{"leasetype":"offer renew","status":"fully executed","property":"1040hami","city":"vancouver","dealtype":"renewal","area":"784","did":"ecf922d0583247c0a4cb591bd4b3843e","mid":"ecf922d0583247c0a4cb591bd4b3843e"}',     '{"leasetype":"offer renew","status":"fully executed","property":"1040hami","city":"vancouver","dealtype":"renewal","area":"2223","did":"ecf922d0583247c0a4cb591bd4b3843e","mid":"ecf922d0583247c0a4cb591bd4b3843e"}',     '{"leasetype":"offer lease","status":"conditional","property":"106portw","city":"toronto","dealtype":"renewal","area":"2212","did":"69c3af0527014fd99d1ccf156c0bce93","mid":"69c3af0527014fd99d1ccf156c0bce93"}',     '{"leasetype":"offer lease","status":"fully executed","property":"106portw","city":"toronto","dealtype":"0","area":"","did":"04aedb01da5d44fead7e1315115c2530","mid":"04aedb01da5d44fead7e1315115c2530"}' ]  data_to_dump = re.sub(r"\'", "", str(resultlist)) json_data= json.dumps(data_to_dump) print json_data

Comments

Post a Comment

Popular posts from this blog

PHP and MySQL WP -

i trying accomplish 2 things here code. writing php directly wp page using insert php plugin. 1) query mysql database , return results 1 column 4 columns. 2) make each result own hyperlink directs user new wp page runs new query on page. i able query show results , turn results dynamic hyperlink, cannot formatting down. right returns result 1 column. here have: [insert_php] $servername = "localhost"; $username = "login"; //edited privacy $password = "password"; //edited privacy $database = "database"; //edited privacy // create connection $conn = new mysqli($servername, $username, $password, $database); // check connection if ($conn->connect_error) { die("connection failed: " . $conn->connect_error); } echo "connected successfully"; //get results database $result = mysqli_query($conn,"select mine mines order mine"); $all_property = array(); //declare array saving property //showing proper...
Read more

android - InAppBilling registering BroadcastReceiver in AndroidManifest -

i'm implementing inapp billing in android app. works well, however, i'm trying decouple broadcast receiver activity manifest. suggestion in android's trivialdrive sample: // important: dynamically register broadcast messages updated purchases. // register receiver here instead of <receiver> in manifest // because call getpurchases() @ startup, therefore can ignore // broadcasts sent while app isn't running. // note: registering listener in activity bad idea, done here // because sample. regardless, receiver must registered after // iabhelper setup, before first call getpurchases(). currently there's class extends broadcastreceiver : public class iabbroadcastreceiver extends broadcastreceiver { /** * intent action receiver should filter for. */ public static final string action = "com.android.vending.billing.purchases_updated"; private final iabbroadcastlistener mlistener; public iabbroadcastreceiver(iabbroadcastlistener listener) { m...
Read more

go - golang pprof for c library code -

i have been trying profile golang application pprof, seems details of c code(linked static library) missed in output result. flame graph from frame graph can see c code time consuming reported within "runtime._externalcode", no details @ all. , sure static library compiled debug enable. so there way make pprof details(like stack trace) of 3rd party c code? or there other golang profiling tool this?
Read more