go - Create a golang pipeline automatically -
i have golang pipeline code. entities getting input channel. when invoking start() start ranging on input channel , writing output channel.
tee component gets input channel , fans them out several channels.
coffee component has add method merges several input channels single output channel
here example code builds such pipeline. "inchan" entry point of pipeline.
inchan := make(chan common.event, 1000) prefix := []byte("foo_") m := make(map[string][]common.enricher) ippa := enricher.newiptocountryenricher(prefix, geomock{}) uapa := enricher.newparseuaenricher(prefix) m[k] = []common.enricher{ippa, uapa} enricherinst := enricher.newenricher(m, inchan) enricherin := enricherinst.start(1000) teechannels := newtee(ctx, 10000, 2, enricherin) kinesiscoffee.add(k, teechannels[1]) fsconfig := fsw.config{chanlen: config.filer.chanlen, maxlineslimit: config.filer.maxlineslimit, timelimit: config.filer.timelimit, host: "local-test", fileprefix: config.filer.fileprefix, lines: teechannels[0], streamname: k, streamid: cfg.idbyname(k)} fsservice := fsw.new(fsconfig) linesendch := fsservice.start() s3coffee.add(k, linesendch) routemap[k] = inchan
this code chunk unreadable. compose pipeline programatically. note can not create components , pass them builder since components split, components merge , matters when "start" method being invoked (if it's being invoked in wrong order ,we might deadlock.)
i wish simplify process. create sort of function gets description - components connect, @ order , how , when star them
Comments
Post a Comment