ByteCount on Function

up vote
3
down vote

favorite

I was comparing RAM and CPU efficiency for List, Rule, Association, Function. The following code

n=10^6; 
tbl=Table[2i,i,n]; ByteCount[tbl]
rls=Table[2i->i,i,n]; ByteCount[rls]
asc=Association@Table[2i->i,i,n]; ByteCount[asc]
Do[fnc[2i]=i,i,n]; ByteCount[fnc]
AbsoluteTiming[Position[tbl,2n][[1,1]]]
AbsoluteTiming[2n/.rls]
AbsoluteTiming[asc[2n]]
AbsoluteTiming[fnc[2n]]

gave the following results:

8000144

96000080

128382000

0

0.142235, 1000000

0.805691, 1000000

0.00001, 1000000

4.*10^-6, 1000000

Thus Function is the fastest, but how do I get its real memory requirement?

asked Aug 25 at 17:09

Leon

298111

add a commentÂ |Â

up vote
3
down vote

favorite

I was comparing RAM and CPU efficiency for List, Rule, Association, Function. The following code

n=10^6; 
tbl=Table[2i,i,n]; ByteCount[tbl]
rls=Table[2i->i,i,n]; ByteCount[rls]
asc=Association@Table[2i->i,i,n]; ByteCount[asc]
Do[fnc[2i]=i,i,n]; ByteCount[fnc]
AbsoluteTiming[Position[tbl,2n][[1,1]]]
AbsoluteTiming[2n/.rls]
AbsoluteTiming[asc[2n]]
AbsoluteTiming[fnc[2n]]

gave the following results:

8000144

96000080

128382000

0

0.142235, 1000000

0.805691, 1000000

0.00001, 1000000

4.*10^-6, 1000000

Thus Function is the fastest, but how do I get its real memory requirement?

asked Aug 25 at 17:09

Leon

298111

add a commentÂ |Â

up vote
3
down vote

favorite

I was comparing RAM and CPU efficiency for List, Rule, Association, Function. The following code

n=10^6; 
tbl=Table[2i,i,n]; ByteCount[tbl]
rls=Table[2i->i,i,n]; ByteCount[rls]
asc=Association@Table[2i->i,i,n]; ByteCount[asc]
Do[fnc[2i]=i,i,n]; ByteCount[fnc]
AbsoluteTiming[Position[tbl,2n][[1,1]]]
AbsoluteTiming[2n/.rls]
AbsoluteTiming[asc[2n]]
AbsoluteTiming[fnc[2n]]

gave the following results:

8000144

96000080

128382000

0

0.142235, 1000000

0.805691, 1000000

0.00001, 1000000

4.*10^-6, 1000000

Thus Function is the fastest, but how do I get its real memory requirement?

asked Aug 25 at 17:09

Leon

298111

I was comparing RAM and CPU efficiency for List, Rule, Association, Function. The following code

n=10^6; 
tbl=Table[2i,i,n]; ByteCount[tbl]
rls=Table[2i->i,i,n]; ByteCount[rls]
asc=Association@Table[2i->i,i,n]; ByteCount[asc]
Do[fnc[2i]=i,i,n]; ByteCount[fnc]
AbsoluteTiming[Position[tbl,2n][[1,1]]]
AbsoluteTiming[2n/.rls]
AbsoluteTiming[asc[2n]]
AbsoluteTiming[fnc[2n]]

gave the following results:

8000144

96000080

128382000

0

0.142235, 1000000

0.805691, 1000000

0.00001, 1000000

4.*10^-6, 1000000

Thus Function is the fastest, but how do I get its real memory requirement?

asked Aug 25 at 17:09

Leon

298111

asked Aug 25 at 17:09

Leon

298111

asked Aug 25 at 17:09

Leon

298111

asked Aug 25 at 17:09

Leon

298111

add a commentÂ |Â

2 Answers
2

active

oldest

votes

up vote
3
down vote

DownValues should get close to the actual value I believe.

ByteCount[DownValues[fnc]]

192000080

You could also use MaxMemoryUsed during the construction:

ClearAll[fnc]

MaxMemoryUsed[Do[fnc[2 i] = i, i, 10^6];]

168327840

answered Aug 25 at 18:10

Mr.Wizardâ™¦

227k284631014

DownValues returns way more, than only the right hand side. Is it a fair comparison? I guess all memory structures should be compared by DownValues then.
â€“Â Johu
Aug 25 at 18:17

@Johu I think it's a good first order approximation at least. I did provide a second method that I presume is more accurate.
â€“Â Mr.Wizardâ™¦
Aug 25 at 18:18

add a commentÂ |Â

up vote
3
down vote

In your example the value of the argument is a part of the definition. Value of fnc[2 i], where i is a symbol, is not defined in your MWE.

Total@Table[ByteCount[fnc[2 i]], i, n]

16 000 000

(Edit: note, that my solution only gives only the size of the right hand side and probably underestimates the real memory cost. See the other solution.)

Note also, that your timing measurement lead to misleading results as you apply it to so simple operations. Compare to

n2 = 10;
AbsoluteTiming[Position[tbl, #] & /@ (2 RandomInteger[n, n2])] // First
AbsoluteTiming[Replace[rls] /@ (2 RandomInteger[n, n2]);] // First

2.19423 
1.8028

Both of these approaches are very slow, as they require going through the whole list to fine the element.

These are much much faster:

n2 = 100000;
AbsoluteTiming[fnc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[asc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[Lookup[asc, 2 RandomInteger[n, n2]];] // First

> 0.267781
> 0.226777
> 0.171752

I think it is misleading to call fnc[i] a function, as a function usually is evaluated runtime. In your MWE you save a precomputed value. This technique is referred to as memoization.

When you wonder which one you should use, I would always use what makes sense semantically, because the engineers behind the kernel and native commands have but a lot of effort into finding a balance between all the features one usually needs from a List, Rules, Associations and Symbols. Associations and Symbols are the ones, which require fast random access.

edited Aug 25 at 22:45

answered Aug 25 at 17:15

Johu

2,451826

Hmm, this is very low RAM usage, comparable to Table, yet faster than all of them. Why wouldn't one use Function instead of Table or Association?
â€“Â Leon
Aug 25 at 17:21

Please see the updated answer.
â€“Â Johu
Aug 25 at 17:43

1

Your problem is that foo /@ 2 RandomInteger[n, n2] is parsed as (foo /@ 2) RandomInteger[n, n2].
â€“Â Carl Woll
Aug 25 at 20:42

@CarlWoll oh lord. how stupid.
â€“Â Johu
Aug 25 at 22:36

I fixed it now and adjusted my conclusions. I did think it had to be impossible for the lookup from List to be that fast. Now it all makes sense.
â€“Â Johu
Aug 25 at 22:46

add a commentÂ |Â

Your Answer

StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\$","\$"]]);
);
);
, "mathjax-editing");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "387"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f180642%2fbytecount-on-function%23new-answer', 'question_page');

);

Post as a guest

Name

2 Answers
2

active

oldest

votes

2 Answers
2

active

oldest

votes

up vote
3
down vote

DownValues should get close to the actual value I believe.

ByteCount[DownValues[fnc]]

192000080

You could also use MaxMemoryUsed during the construction:

ClearAll[fnc]

MaxMemoryUsed[Do[fnc[2 i] = i, i, 10^6];]

168327840

answered Aug 25 at 18:10

Mr.Wizardâ™¦

227k284631014

DownValues returns way more, than only the right hand side. Is it a fair comparison? I guess all memory structures should be compared by DownValues then.
â€“Â Johu
Aug 25 at 18:17

@Johu I think it's a good first order approximation at least. I did provide a second method that I presume is more accurate.
â€“Â Mr.Wizardâ™¦
Aug 25 at 18:18

add a commentÂ |Â

up vote
3
down vote

DownValues should get close to the actual value I believe.

ByteCount[DownValues[fnc]]

192000080

You could also use MaxMemoryUsed during the construction:

ClearAll[fnc]

MaxMemoryUsed[Do[fnc[2 i] = i, i, 10^6];]

168327840

answered Aug 25 at 18:10

Mr.Wizardâ™¦

227k284631014

DownValues returns way more, than only the right hand side. Is it a fair comparison? I guess all memory structures should be compared by DownValues then.
â€“Â Johu
Aug 25 at 18:17

@Johu I think it's a good first order approximation at least. I did provide a second method that I presume is more accurate.
â€“Â Mr.Wizardâ™¦
Aug 25 at 18:18

add a commentÂ |Â

up vote
3
down vote

DownValues should get close to the actual value I believe.

ByteCount[DownValues[fnc]]

192000080

You could also use MaxMemoryUsed during the construction:

ClearAll[fnc]

MaxMemoryUsed[Do[fnc[2 i] = i, i, 10^6];]

168327840

answered Aug 25 at 18:10

Mr.Wizardâ™¦

227k284631014

DownValues should get close to the actual value I believe.

ByteCount[DownValues[fnc]]

192000080

You could also use MaxMemoryUsed during the construction:

ClearAll[fnc]

MaxMemoryUsed[Do[fnc[2 i] = i, i, 10^6];]

168327840

answered Aug 25 at 18:10

Mr.Wizardâ™¦

227k284631014

answered Aug 25 at 18:10

Mr.Wizardâ™¦

227k284631014

answered Aug 25 at 18:10

Mr.Wizardâ™¦

227k284631014

answered Aug 25 at 18:10

Mr.Wizardâ™¦

227k284631014

DownValues returns way more, than only the right hand side. Is it a fair comparison? I guess all memory structures should be compared by DownValues then.
â€“Â Johu
Aug 25 at 18:17

@Johu I think it's a good first order approximation at least. I did provide a second method that I presume is more accurate.
â€“Â Mr.Wizardâ™¦
Aug 25 at 18:18

add a commentÂ |Â

DownValues returns way more, than only the right hand side. Is it a fair comparison? I guess all memory structures should be compared by DownValues then.
â€“Â Johu
Aug 25 at 18:17

@Johu I think it's a good first order approximation at least. I did provide a second method that I presume is more accurate.
â€“Â Mr.Wizardâ™¦
Aug 25 at 18:18

DownValues returns way more, than only the right hand side. Is it a fair comparison? I guess all memory structures should be compared by DownValues then.
â€“Â Johu
Aug 25 at 18:17

@Johu I think it's a good first order approximation at least. I did provide a second method that I presume is more accurate.
â€“Â Mr.Wizardâ™¦
Aug 25 at 18:18

add a commentÂ |Â

up vote
3
down vote

In your example the value of the argument is a part of the definition. Value of fnc[2 i], where i is a symbol, is not defined in your MWE.

Total@Table[ByteCount[fnc[2 i]], i, n]

16 000 000

(Edit: note, that my solution only gives only the size of the right hand side and probably underestimates the real memory cost. See the other solution.)

Note also, that your timing measurement lead to misleading results as you apply it to so simple operations. Compare to

n2 = 10;
AbsoluteTiming[Position[tbl, #] & /@ (2 RandomInteger[n, n2])] // First
AbsoluteTiming[Replace[rls] /@ (2 RandomInteger[n, n2]);] // First

2.19423 
1.8028

Both of these approaches are very slow, as they require going through the whole list to fine the element.

These are much much faster:

n2 = 100000;
AbsoluteTiming[fnc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[asc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[Lookup[asc, 2 RandomInteger[n, n2]];] // First

> 0.267781
> 0.226777
> 0.171752

I think it is misleading to call fnc[i] a function, as a function usually is evaluated runtime. In your MWE you save a precomputed value. This technique is referred to as memoization.

edited Aug 25 at 22:45

answered Aug 25 at 17:15

Johu

2,451826

Hmm, this is very low RAM usage, comparable to Table, yet faster than all of them. Why wouldn't one use Function instead of Table or Association?
â€“Â Leon
Aug 25 at 17:21

Please see the updated answer.
â€“Â Johu
Aug 25 at 17:43

1

Your problem is that foo /@ 2 RandomInteger[n, n2] is parsed as (foo /@ 2) RandomInteger[n, n2].
â€“Â Carl Woll
Aug 25 at 20:42

@CarlWoll oh lord. how stupid.
â€“Â Johu
Aug 25 at 22:36

I fixed it now and adjusted my conclusions. I did think it had to be impossible for the lookup from List to be that fast. Now it all makes sense.
â€“Â Johu
Aug 25 at 22:46

add a commentÂ |Â

up vote
3
down vote

In your example the value of the argument is a part of the definition. Value of fnc[2 i], where i is a symbol, is not defined in your MWE.

Total@Table[ByteCount[fnc[2 i]], i, n]

16 000 000

(Edit: note, that my solution only gives only the size of the right hand side and probably underestimates the real memory cost. See the other solution.)

Note also, that your timing measurement lead to misleading results as you apply it to so simple operations. Compare to

n2 = 10;
AbsoluteTiming[Position[tbl, #] & /@ (2 RandomInteger[n, n2])] // First
AbsoluteTiming[Replace[rls] /@ (2 RandomInteger[n, n2]);] // First

2.19423 
1.8028

Both of these approaches are very slow, as they require going through the whole list to fine the element.

These are much much faster:

n2 = 100000;
AbsoluteTiming[fnc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[asc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[Lookup[asc, 2 RandomInteger[n, n2]];] // First

> 0.267781
> 0.226777
> 0.171752

I think it is misleading to call fnc[i] a function, as a function usually is evaluated runtime. In your MWE you save a precomputed value. This technique is referred to as memoization.

edited Aug 25 at 22:45

answered Aug 25 at 17:15

Johu

2,451826

Hmm, this is very low RAM usage, comparable to Table, yet faster than all of them. Why wouldn't one use Function instead of Table or Association?
â€“Â Leon
Aug 25 at 17:21

Please see the updated answer.
â€“Â Johu
Aug 25 at 17:43

1

Your problem is that foo /@ 2 RandomInteger[n, n2] is parsed as (foo /@ 2) RandomInteger[n, n2].
â€“Â Carl Woll
Aug 25 at 20:42

@CarlWoll oh lord. how stupid.
â€“Â Johu
Aug 25 at 22:36

I fixed it now and adjusted my conclusions. I did think it had to be impossible for the lookup from List to be that fast. Now it all makes sense.
â€“Â Johu
Aug 25 at 22:46

add a commentÂ |Â

up vote
3
down vote

In your example the value of the argument is a part of the definition. Value of fnc[2 i], where i is a symbol, is not defined in your MWE.

Total@Table[ByteCount[fnc[2 i]], i, n]

16 000 000

(Edit: note, that my solution only gives only the size of the right hand side and probably underestimates the real memory cost. See the other solution.)

Note also, that your timing measurement lead to misleading results as you apply it to so simple operations. Compare to

n2 = 10;
AbsoluteTiming[Position[tbl, #] & /@ (2 RandomInteger[n, n2])] // First
AbsoluteTiming[Replace[rls] /@ (2 RandomInteger[n, n2]);] // First

2.19423 
1.8028

Both of these approaches are very slow, as they require going through the whole list to fine the element.

These are much much faster:

n2 = 100000;
AbsoluteTiming[fnc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[asc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[Lookup[asc, 2 RandomInteger[n, n2]];] // First

> 0.267781
> 0.226777
> 0.171752

I think it is misleading to call fnc[i] a function, as a function usually is evaluated runtime. In your MWE you save a precomputed value. This technique is referred to as memoization.

edited Aug 25 at 22:45

answered Aug 25 at 17:15

Johu

2,451826

In your example the value of the argument is a part of the definition. Value of fnc[2 i], where i is a symbol, is not defined in your MWE.

Total@Table[ByteCount[fnc[2 i]], i, n]

16 000 000

(Edit: note, that my solution only gives only the size of the right hand side and probably underestimates the real memory cost. See the other solution.)

Note also, that your timing measurement lead to misleading results as you apply it to so simple operations. Compare to

n2 = 10;
AbsoluteTiming[Position[tbl, #] & /@ (2 RandomInteger[n, n2])] // First
AbsoluteTiming[Replace[rls] /@ (2 RandomInteger[n, n2]);] // First

2.19423 
1.8028

Both of these approaches are very slow, as they require going through the whole list to fine the element.

These are much much faster:

n2 = 100000;
AbsoluteTiming[fnc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[asc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[Lookup[asc, 2 RandomInteger[n, n2]];] // First

> 0.267781
> 0.226777
> 0.171752

I think it is misleading to call fnc[i] a function, as a function usually is evaluated runtime. In your MWE you save a precomputed value. This technique is referred to as memoization.

edited Aug 25 at 22:45

answered Aug 25 at 17:15

Johu

2,451826

edited Aug 25 at 22:45

answered Aug 25 at 17:15

Johu

2,451826

answered Aug 25 at 17:15

Johu

2,451826

answered Aug 25 at 17:15

Johu

2,451826

Hmm, this is very low RAM usage, comparable to Table, yet faster than all of them. Why wouldn't one use Function instead of Table or Association?
â€“Â Leon
Aug 25 at 17:21

Please see the updated answer.
â€“Â Johu
Aug 25 at 17:43

1

Your problem is that foo /@ 2 RandomInteger[n, n2] is parsed as (foo /@ 2) RandomInteger[n, n2].
â€“Â Carl Woll
Aug 25 at 20:42

@CarlWoll oh lord. how stupid.
â€“Â Johu
Aug 25 at 22:36

I fixed it now and adjusted my conclusions. I did think it had to be impossible for the lookup from List to be that fast. Now it all makes sense.
â€“Â Johu
Aug 25 at 22:46

add a commentÂ |Â

Hmm, this is very low RAM usage, comparable to Table, yet faster than all of them. Why wouldn't one use Function instead of Table or Association?
â€“Â Leon
Aug 25 at 17:21

Please see the updated answer.
â€“Â Johu
Aug 25 at 17:43

1

Your problem is that foo /@ 2 RandomInteger[n, n2] is parsed as (foo /@ 2) RandomInteger[n, n2].
â€“Â Carl Woll
Aug 25 at 20:42

@CarlWoll oh lord. how stupid.
â€“Â Johu
Aug 25 at 22:36

I fixed it now and adjusted my conclusions. I did think it had to be impossible for the lookup from List to be that fast. Now it all makes sense.
â€“Â Johu
Aug 25 at 22:46

Hmm, this is very low RAM usage, comparable to Table, yet faster than all of them. Why wouldn't one use Function instead of Table or Association?
â€“Â Leon
Aug 25 at 17:21

Please see the updated answer.
â€“Â Johu
Aug 25 at 17:43

Your problem is that foo /@ 2 RandomInteger[n, n2] is parsed as (foo /@ 2) RandomInteger[n, n2].
â€“Â Carl Woll
Aug 25 at 20:42

@CarlWoll oh lord. how stupid.
â€“Â Johu
Aug 25 at 22:36

I fixed it now and adjusted my conclusions. I did think it had to be impossible for the lookup from List to be that fast. Now it all makes sense.
â€“Â Johu
Aug 25 at 22:46

add a commentÂ |Â

draft saved

draft discarded

draft saved

draft discarded

Post as a guest

Name

Search This Blog

Iyfjky