understanding python memory allocation and free up -


i came across this article python memory allocation.

in page describes memory usage of python , in there there example showing deepcopy of list of integers. did benchmark myself on python 2.7

line #    mem usage    increment   line contents ================================================      4   28.051 mib    0.000 mib   @profile      5                             def function():      6   59.098 mib   31.047 mib       x = list(range(1000000))  # allocate big list      7  107.273 mib   48.176 mib       y = copy.deepcopy(x)      8   99.641 mib   -7.633 mib       del x      9   99.641 mib    0.000 mib       return y

so delete x directly removes x , references integer x right?

doing not either (so difference del x , del x[:]?):

line #    mem usage    increment   line contents ================================================      4   28.047 mib    0.000 mib   @profile      5                             def function():      6   59.094 mib   31.047 mib       x = list(range(1000000))  # allocate big list      7  107.270 mib   48.176 mib       y = copy.deepcopy(x)      8   99.637 mib   -7.633 mib       del x[:]      9   99.637 mib    0.000 mib       return y

and in contrast deepcopy, if use copy, after deletion seems memory restores previous state when x newly created

line #    mem usage    increment   line contents ================================================      4   28.039 mib    0.000 mib   @profile      5                             def function():      6   59.090 mib   31.051 mib       x = list(range(1000000))  # allocate big list      7   66.895 mib    7.805 mib       y = copy.copy(x)      8   59.262 mib   -7.633 mib       del x[:]      9   59.262 mib    0.000 mib       return y

for dict:

line #    mem usage    increment   line contents ================================================      4   28.051 mib    0.000 mib   @profile      5                             def function():      6  100.523 mib   72.473 mib       x = dict((e, e) e in xrange(1000000))      7  183.398 mib   82.875 mib       y = copy.deepcopy(x)      8  135.395 mib  -48.004 mib       del x      9  135.395 mib    0.000 mib       return y

and list of lists (compare list of integers, assume del x or del x[:] removes huge array list on heap?):

line #    mem usage    increment   line contents ================================================      4   28.043 mib    0.000 mib   @profile      5                             def function():      6  107.691 mib   79.648 mib       x = [[] _ in xrange(1000000)]      7  222.312 mib  114.621 mib       y = copy.deepcopy(x)      8  214.680 mib   -7.633 mib       del x[:]      9  214.680 mib    0.000 mib       return y

so want ask:

  1. so if there no way claim memory occupied integers? integer object right? why memory not released @ all? integer cannot claimed? or float , string well? object references well?
  2. why there -7 mb memory? because list, implemented array list, freed heap?
  3. whether list or dict, del x can free data structure (what mean array list structure, or dict structure), integers, objects references can marked free, not returned system?

and how or if there way free underlining lists in x in example?

line #    mem usage    increment   line contents ================================================      4   28.047 mib    0.000 mib   @profile      5                             def function():      6  248.008 mib  219.961 mib       x = [list(range(10)) _ in xrange(1000000)]      7  502.195 mib  254.188 mib       y = copy.deepcopy(x)      8  494.562 mib   -7.633 mib       del x[:]      9  494.562 mib    0.000 mib       return y

tl;dr

del not free variables in c, says no longer need it. happens implementation detail.

rationalle

so happening here del not free memory, tells python done variable. specifically:

7.5. del statement

del_stmt ::= “del” target_list

deletion recursively defined similar way assignment defined. rather spelling out in full details, here hints.

deletion of target list recursively deletes each target, left right.

deletion of name removes binding of name local or global namespace, depending on whether name occurs in global statement in same code block. if name unbound, nameerror exception raised.

deletion of attribute references, subscriptions , slicings passed primary object involved; deletion of slicing in general equivalent assignment of empty slice of right type (but determined sliced object).

note there no mention of freeing memory. instead happens tell python can "whatever wants" memory. in case python implementation (which assume cpython) stores memory later use in memory cache. allows python run faster not needing allocate memory later.

example

consider example, del x , create copy of y again. note amount of memory allocated during second copy smaller during first. because memory re-used. if again, see hardly memory @ allocated during third copy, because python re-using allocated memory:

line #    mem usage    increment   line contents ================================================      4   34.777 mib    0.000 mib   @profile      5                             def function():      6   37.504 mib    2.727 mib       x = [list(range(10)) _ in xrange(10000)]      7   40.773 mib    3.270 mib       y = copy.deepcopy(x)      8   40.773 mib    0.000 mib       del x      9   41.820 mib    1.047 mib       y2 = copy.deepcopy(y)     10   41.820 mib    0.000 mib       del y2     11   41.824 mib    0.004 mib       y3 = copy.deepcopy(y)     12   41.824 mib    0.000 mib       return y

sources

excellent "blog": http://www.evanjones.ca/memoryallocator/

http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm


Comments

Post a Comment

Popular posts from this blog

android - InAppBilling registering BroadcastReceiver in AndroidManifest -

i'm implementing inapp billing in android app. works well, however, i'm trying decouple broadcast receiver activity manifest. suggestion in android's trivialdrive sample: // important: dynamically register broadcast messages updated purchases. // register receiver here instead of <receiver> in manifest // because call getpurchases() @ startup, therefore can ignore // broadcasts sent while app isn't running. // note: registering listener in activity bad idea, done here // because sample. regardless, receiver must registered after // iabhelper setup, before first call getpurchases(). currently there's class extends broadcastreceiver : public class iabbroadcastreceiver extends broadcastreceiver { /** * intent action receiver should filter for. */ public static final string action = "com.android.vending.billing.purchases_updated"; private final iabbroadcastlistener mlistener; public iabbroadcastreceiver(iabbroadcastlistener listener) { m...
Read more

python Tkinter Capturing keyboard events save as one single string -

is possible store keyboard events 1 string? code below stores 1 char , prints it. card reader or bar code reader, contains collection of character/ string not 1 character @ time. goal save char pressed text variable. from tkinter import * root = tk() def key(event): text= event.char text+= event.char print ("pressed", text) def callback(event): frame.focus_set() print ("clicked at", event.x, event.y) frame = frame(root, width=100, height=100) frame.bind("<key>", key) frame.bind("<button-1>", callback) frame.pack() root.mainloop() currently, creating text variable , everytime key() function gets called, , text stores last character typed. you can define text module level variable , , use module level text inside key function - text = '' def key(event): global text text+= event.char print("pressed", text)
Read more

sql server - Why does Linq-to-SQL add unnecessary COUNT()? -

here tables: create table customer ( cid int primary key, name varchar(32) ) create table ord ( oid int primary key, cid int foreign key references customer, address varchar(20) ) here's linq-to-sql statement: var aaa = c in db.customer select new { c, o = c.ord.tolist() }; and here's inexplicable query generated linq-to-sql: select [t0].[cid] [cid], [t0].[name] [name], [t1].[oid] [oid], [t1].[cid] [cid2], [t1].[address] [address], (select count(*) [dbo].[ord] [t2] [t2].[cid] = [t0].[cid]) [value] [dbo].[customer] [t0] left outer join [dbo].[ord] [t1] on [t1].[cid] = [t0].[cid] order [t0].[cid], [t1].[oid] i want understand how rid of count(*) part. it's entirely uncalled for! linq needs count(*) in order determine number of entries returned o = c.ord.tolist() .
Read more