Spring Cloud DataFlow http polling and deduplication -


i have been reading spring cloud dataflow , related documentation in order produce data ingest solution run in organization's cloud foundry deployment. goal poll http service data, perhaps 3 times per day sake of discussion, , insert/update data in postgresql database. http service seems provide 10s of thousands of records per day.

one point of confusion far best practice in context of dataflow pipeline deduplicating polled records. source data not have timestamp field aid in tracking polling, coarse day-level date field. have no guarantee records not ever updated retroactively. records appear have unique id, can dedup records way, not sure based on documentation how best implement logic in dataflow. far can tell, spring cloud stream starters not provide out-of-the-box. reading spring integration's smart polling, i'm not sure that's meant address concern either.

my intuition create custom processor java component in dataflow stream performs database query determine whether polled records have been inserted, inserts appropriate records target database, or passes them on down stream. querying target database in intermediate step acceptable in stream app? alternatively, implement in spring cloud task batch operation triggers based on schedule.

what best way proceed respect dataflow app? common/best practices achieving deduplication described above in dataflow/stream/task/integration app? should copy setup of starter app or start scratch, because i'll need write custom code? need spring cloud dataflow, because i'm not sure i'll using dsl @ all? apologies questions, being new cloud foundry , these spring projects, it's daunting piece together.

thanks in advance help.

you on right track, given requirements need create custom processor. need keep track of has been inserted in order avoid duplication.

there's nothing preventing writing such processor in stream app, performance may take hit, since each record issue db query.

if order not important, parallelize query process several concurrent messages, in end db still pay price.

another approach use bloomfilter can quite lot on speeding checking inserted records.

you can start cloning starter apps, have poller trigger http client processor fetches data , go through custom code processor , jdbc-sink. stream create time --triger.cron=<cron_expression> | httpclient --httpclient.url-expression=<remote_endpoint> | customprocessor | jdbc

one of advantages of using scdf independently scale custom processor via deployment properties such deployer.customprocessor.count=8


Comments

Post a Comment

Popular posts from this blog

android - InAppBilling registering BroadcastReceiver in AndroidManifest -

i'm implementing inapp billing in android app. works well, however, i'm trying decouple broadcast receiver activity manifest. suggestion in android's trivialdrive sample: // important: dynamically register broadcast messages updated purchases. // register receiver here instead of <receiver> in manifest // because call getpurchases() @ startup, therefore can ignore // broadcasts sent while app isn't running. // note: registering listener in activity bad idea, done here // because sample. regardless, receiver must registered after // iabhelper setup, before first call getpurchases(). currently there's class extends broadcastreceiver : public class iabbroadcastreceiver extends broadcastreceiver { /** * intent action receiver should filter for. */ public static final string action = "com.android.vending.billing.purchases_updated"; private final iabbroadcastlistener mlistener; public iabbroadcastreceiver(iabbroadcastlistener listener) { m...
Read more

python Tkinter Capturing keyboard events save as one single string -

is possible store keyboard events 1 string? code below stores 1 char , prints it. card reader or bar code reader, contains collection of character/ string not 1 character @ time. goal save char pressed text variable. from tkinter import * root = tk() def key(event): text= event.char text+= event.char print ("pressed", text) def callback(event): frame.focus_set() print ("clicked at", event.x, event.y) frame = frame(root, width=100, height=100) frame.bind("<key>", key) frame.bind("<button-1>", callback) frame.pack() root.mainloop() currently, creating text variable , everytime key() function gets called, , text stores last character typed. you can define text module level variable , , use module level text inside key function - text = '' def key(event): global text text+= event.char print("pressed", text)
Read more

sql server - Why does Linq-to-SQL add unnecessary COUNT()? -

here tables: create table customer ( cid int primary key, name varchar(32) ) create table ord ( oid int primary key, cid int foreign key references customer, address varchar(20) ) here's linq-to-sql statement: var aaa = c in db.customer select new { c, o = c.ord.tolist() }; and here's inexplicable query generated linq-to-sql: select [t0].[cid] [cid], [t0].[name] [name], [t1].[oid] [oid], [t1].[cid] [cid2], [t1].[address] [address], (select count(*) [dbo].[ord] [t2] [t2].[cid] = [t0].[cid]) [value] [dbo].[customer] [t0] left outer join [dbo].[ord] [t1] on [t1].[cid] = [t0].[cid] order [t0].[cid], [t1].[oid] i want understand how rid of count(*) part. it's entirely uncalled for! linq needs count(*) in order determine number of entries returned o = c.ord.tolist() .
Read more