javascript - Text clustering: chosing the k in k means -


after eliminating stop words , applied stemming process in set of documents, applied bisecting k-means in javascript in order cluster set of documents received web pages finding similarity between them.

what should method finding how many cluster should created when having text-based clusters? saw methods such elbow, silhouette, or information criterion approaches, assuming don't have information of clusters create, other methods seem better fit numeric clustering, not on text-based clusters.

can entropy measure in helping me find right number of clusters after applying bisecting k-means in text clustering? or f-measure? mean stop dividing cluster after value reached? large sets of data?

short answer:

you can use termfequency- inversedocument-frequency (tf-idf). emphasises rare words used in single document, , penalizes words when found in documents. if applied pca tfidf on dataset, might use "scree plot" (~ elbow method) find suitable number of clusters.

long example:

the following example of not using kmeans, , example uses few long documents, , has been decided there 2 "clusters" (using principal components , tf-idf, actually), uses real data in creative way:

in phd dissertation documenting "textmining" package tm developed r software, author of tm, ingo feinerer, gives example (chapter 10) how stylometry, cluster/identify 5 books "wizard of oz" series. 1 of these books authorship disputed (there 2 authors in series, thompson , baum, contributions 1 of books unknown).

feinerer chops the documents 500-line chunks build termdocumentmatrix, performs variants of principal component analysis (pca), 1 tfidf, on matrix, , shows visual inspection of pca plots disputed book tends authored thompson. parts might have been written baum.

in plot indicated points inside pinkish wiggly oval (drawn me). green dots chunks book known authorship (t.) , yellow dots book of unknown / disputed authorship. (the points fall close in plot. that's evidence here; it's qualitative, 1 example of many in pdf) enter image description here

the tf-idf pca plot on page 95 looks similar.

i have not given r code because don't know if r, , post getting long, , can read in pdf.

(and don't know implementations of tf-idf in javascript).


Comments

Post a Comment

Popular posts from this blog

android - InAppBilling registering BroadcastReceiver in AndroidManifest -

i'm implementing inapp billing in android app. works well, however, i'm trying decouple broadcast receiver activity manifest. suggestion in android's trivialdrive sample: // important: dynamically register broadcast messages updated purchases. // register receiver here instead of <receiver> in manifest // because call getpurchases() @ startup, therefore can ignore // broadcasts sent while app isn't running. // note: registering listener in activity bad idea, done here // because sample. regardless, receiver must registered after // iabhelper setup, before first call getpurchases(). currently there's class extends broadcastreceiver : public class iabbroadcastreceiver extends broadcastreceiver { /** * intent action receiver should filter for. */ public static final string action = "com.android.vending.billing.purchases_updated"; private final iabbroadcastlistener mlistener; public iabbroadcastreceiver(iabbroadcastlistener listener) { m...
Read more

python Tkinter Capturing keyboard events save as one single string -

is possible store keyboard events 1 string? code below stores 1 char , prints it. card reader or bar code reader, contains collection of character/ string not 1 character @ time. goal save char pressed text variable. from tkinter import * root = tk() def key(event): text= event.char text+= event.char print ("pressed", text) def callback(event): frame.focus_set() print ("clicked at", event.x, event.y) frame = frame(root, width=100, height=100) frame.bind("<key>", key) frame.bind("<button-1>", callback) frame.pack() root.mainloop() currently, creating text variable , everytime key() function gets called, , text stores last character typed. you can define text module level variable , , use module level text inside key function - text = '' def key(event): global text text+= event.char print("pressed", text)
Read more

sql server - Why does Linq-to-SQL add unnecessary COUNT()? -

here tables: create table customer ( cid int primary key, name varchar(32) ) create table ord ( oid int primary key, cid int foreign key references customer, address varchar(20) ) here's linq-to-sql statement: var aaa = c in db.customer select new { c, o = c.ord.tolist() }; and here's inexplicable query generated linq-to-sql: select [t0].[cid] [cid], [t0].[name] [name], [t1].[oid] [oid], [t1].[cid] [cid2], [t1].[address] [address], (select count(*) [dbo].[ord] [t2] [t2].[cid] = [t0].[cid]) [value] [dbo].[customer] [t0] left outer join [dbo].[ord] [t1] on [t1].[cid] = [t0].[cid] order [t0].[cid], [t1].[oid] i want understand how rid of count(*) part. it's entirely uncalled for! linq needs count(*) in order determine number of entries returned o = c.ord.tolist() .
Read more