python 3.x - Adding documents to gensim model -


i have class wrapping various objects required calculating lsi similarity:

class similarityfiles:      def __init__(self, file_name, tokenized_corpus, stoplist=none):         if stoplist none:             self.filtered_corpus = tokenized_corpus         else:             self.filtered_corpus = []             convo in tokenized_corpus:                 self.filtered_corpus.append([token token in convo if token not in stoplist])         self.dictionary = corpora.dictionary(self.filtered_corpus)         self.corpus = [self.dictionary.doc2bow(text) text in self.filtered_corpus]         self.lsi = models.lsimodel(self.corpus, id2word=self.dictionary, num_topics=100)         self.index = similarities.matrixsimilarity(self.lsi[self.corpus])

i want add function class allow adding documents corpus , updating model accordingly. i've found dictionary.add_documents, , model.add_documents, there 2 things aren't clear me:

  1. when create lsi model, 1 of parameters function receives id2word=dictionary. when updating model, how tell use updated dictionary? unnecessary, or make difference?
  2. how update index? looks documentation if use similarity class, , not matrixsimilarity class, can add documents index, don't see such functionality matrixsimilarity. if understood correctly, matrixsimilarity better if input corpus contains dense vectors (which does, because i'm using lsi model). have change similarity can update index? or, conversely, what's complexity of creating index? if it's insignificant, should create new index updated corpus, follows:

code:

self.dictionary.add_documents(new_docs)    # new_docs after filtering stop words new_corpus = [self.dictionary.doc2bow(text) text in new_docs] self.lsi.add_documents(new_corpus) self.index = similarities.matrixsimilarity(self.lsi[self.corpus])

thanks. :)


Comments

Post a Comment

Popular posts from this blog

PHP and MySQL WP -

i trying accomplish 2 things here code. writing php directly wp page using insert php plugin. 1) query mysql database , return results 1 column 4 columns. 2) make each result own hyperlink directs user new wp page runs new query on page. i able query show results , turn results dynamic hyperlink, cannot formatting down. right returns result 1 column. here have: [insert_php] $servername = "localhost"; $username = "login"; //edited privacy $password = "password"; //edited privacy $database = "database"; //edited privacy // create connection $conn = new mysqli($servername, $username, $password, $database); // check connection if ($conn->connect_error) { die("connection failed: " . $conn->connect_error); } echo "connected successfully"; //get results database $result = mysqli_query($conn,"select mine mines order mine"); $all_property = array(); //declare array saving property //showing proper...
Read more

android - InAppBilling registering BroadcastReceiver in AndroidManifest -

i'm implementing inapp billing in android app. works well, however, i'm trying decouple broadcast receiver activity manifest. suggestion in android's trivialdrive sample: // important: dynamically register broadcast messages updated purchases. // register receiver here instead of <receiver> in manifest // because call getpurchases() @ startup, therefore can ignore // broadcasts sent while app isn't running. // note: registering listener in activity bad idea, done here // because sample. regardless, receiver must registered after // iabhelper setup, before first call getpurchases(). currently there's class extends broadcastreceiver : public class iabbroadcastreceiver extends broadcastreceiver { /** * intent action receiver should filter for. */ public static final string action = "com.android.vending.billing.purchases_updated"; private final iabbroadcastlistener mlistener; public iabbroadcastreceiver(iabbroadcastlistener listener) { m...
Read more

go - golang pprof for c library code -

i have been trying profile golang application pprof, seems details of c code(linked static library) missed in output result. flame graph from frame graph can see c code time consuming reported within "runtime._externalcode", no details @ all. , sure static library compiled debug enable. so there way make pprof details(like stack trace) of 3rd party c code? or there other golang profiling tool this?
Read more