Mixing
ByteCount on Function
Clash Royale CLAN TAG#URR8PPP
up vote
3
down vote
favorite
I was comparing RAM and CPU efficiency for List, Rule, Association, Function. The following code
n=10^6;
tbl=Table[2i,i,n]; ByteCount[tbl]
rls=Table[2i->i,i,n]; ByteCount[rls]
asc=Association@Table[2i->i,i,n]; ByteCount[asc]
Do[fnc[2i]=i,i,n]; ByteCount[fnc]
AbsoluteTiming[Position[tbl,2n][[1,1]]]
AbsoluteTiming[2n/.rls]
AbsoluteTiming[asc[2n]]
AbsoluteTiming[fnc[2n]]
gave the following results:
8000144
96000080
128382000
0
0.142235, 1000000
0.805691, 1000000
0.00001, 1000000
4.*10^-6, 1000000
Thus Function is the fastest, but how do I get its real memory requirement?
functions table associations memory data-structures
asked Aug 25 at 17:09
Leon
298111
add a comment |Â
up vote
3
down vote
favorite
I was comparing RAM and CPU efficiency for List, Rule, Association, Function. The following code
n=10^6;
tbl=Table[2i,i,n]; ByteCount[tbl]
rls=Table[2i->i,i,n]; ByteCount[rls]
asc=Association@Table[2i->i,i,n]; ByteCount[asc]
Do[fnc[2i]=i,i,n]; ByteCount[fnc]
AbsoluteTiming[Position[tbl,2n][[1,1]]]
AbsoluteTiming[2n/.rls]
AbsoluteTiming[asc[2n]]
AbsoluteTiming[fnc[2n]]
gave the following results:
8000144
96000080
128382000
0
0.142235, 1000000
0.805691, 1000000
0.00001, 1000000
4.*10^-6, 1000000
Thus Function is the fastest, but how do I get its real memory requirement?
functions table associations memory data-structures
asked Aug 25 at 17:09
Leon
298111
add a comment |Â
up vote
3
down vote
favorite
up vote
3
down vote
favorite
I was comparing RAM and CPU efficiency for List, Rule, Association, Function. The following code
n=10^6;
tbl=Table[2i,i,n]; ByteCount[tbl]
rls=Table[2i->i,i,n]; ByteCount[rls]
asc=Association@Table[2i->i,i,n]; ByteCount[asc]
Do[fnc[2i]=i,i,n]; ByteCount[fnc]
AbsoluteTiming[Position[tbl,2n][[1,1]]]
AbsoluteTiming[2n/.rls]
AbsoluteTiming[asc[2n]]
AbsoluteTiming[fnc[2n]]
gave the following results:
8000144
96000080
128382000
0
0.142235, 1000000
0.805691, 1000000
0.00001, 1000000
4.*10^-6, 1000000
Thus Function is the fastest, but how do I get its real memory requirement?
functions table associations memory data-structures
asked Aug 25 at 17:09
Leon
298111
I was comparing RAM and CPU efficiency for List, Rule, Association, Function. The following code
n=10^6;
tbl=Table[2i,i,n]; ByteCount[tbl]
rls=Table[2i->i,i,n]; ByteCount[rls]
asc=Association@Table[2i->i,i,n]; ByteCount[asc]
Do[fnc[2i]=i,i,n]; ByteCount[fnc]
AbsoluteTiming[Position[tbl,2n][[1,1]]]
AbsoluteTiming[2n/.rls]
AbsoluteTiming[asc[2n]]
AbsoluteTiming[fnc[2n]]
gave the following results:
8000144
96000080
128382000
0
0.142235, 1000000
0.805691, 1000000
0.00001, 1000000
4.*10^-6, 1000000
Thus Function is the fastest, but how do I get its real memory requirement?
functions table associations memory data-structures
asked Aug 25 at 17:09
Leon
298111
asked Aug 25 at 17:09
Leon
298111
asked Aug 25 at 17:09
Leon
298111
asked Aug 25 at 17:09
Leon
298111
298111
add a comment |Â
add a comment |Â
2 Answers
2
active
oldest
votes
up vote
3
down vote
DownValues should get close to the actual value I believe.
ByteCount[DownValues[fnc]]
192000080
You could also use MaxMemoryUsed during the construction:
ClearAll[fnc]
MaxMemoryUsed[Do[fnc[2 i] = i, i, 10^6];]
168327840
answered Aug 25 at 18:10
Mr.Wizard♦
227k284631014
DownValuesreturns way more, than only the right hand side. Is it a fair comparison? I guess all memory structures should be compared byDownValuesthen.
– Johu
Aug 25 at 18:17
@Johu I think it's a good first order approximation at least. I did provide a second method that I presume is more accurate.
– Mr.Wizard♦
Aug 25 at 18:18
add a comment |Â
up vote
3
down vote
In your example the value of the argument is a part of the definition. Value of fnc[2 i], where i is a symbol, is not defined in your MWE.
Total@Table[ByteCount[fnc[2 i]], i, n]
16 000 000
(Edit: note, that my solution only gives only the size of the right hand side and probably underestimates the real memory cost. See the other solution.)
Note also, that your timing measurement lead to misleading results as you apply it to so simple operations. Compare to
n2 = 10;
AbsoluteTiming[Position[tbl, #] & /@ (2 RandomInteger[n, n2])] // First
AbsoluteTiming[Replace[rls] /@ (2 RandomInteger[n, n2]);] // First
2.19423
1.8028
Both of these approaches are very slow, as they require going through the whole list to fine the element.
These are much much faster:
n2 = 100000;
AbsoluteTiming[fnc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[asc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[Lookup[asc, 2 RandomInteger[n, n2]];] // First
> 0.267781
> 0.226777
> 0.171752
I think it is misleading to call fnc[i] a function, as a function usually is evaluated runtime. In your MWE you save a precomputed value. This technique is referred to as memoization.
When you wonder which one you should use, I would always use what makes sense semantically, because the engineers behind the kernel and native commands have but a lot of effort into finding a balance between all the features one usually needs from a List, Rules, Associations and Symbols. Associations and Symbols are the ones, which require fast random access.
answered Aug 25 at 17:15
Johu
2,451826
Hmm, this is very low RAM usage, comparable to Table, yet faster than all of them. Why wouldn't one use Function instead of Table or Association?
– Leon
Aug 25 at 17:21
Please see the updated answer.
– Johu
Aug 25 at 17:43
1
Your problem is thatfoo /@ 2 RandomInteger[n, n2]is parsed as(foo /@ 2) RandomInteger[n, n2].
– Carl Woll
Aug 25 at 20:42
@CarlWoll oh lord. how stupid.
– Johu
Aug 25 at 22:36
I fixed it now and adjusted my conclusions. I did think it had to be impossible for the lookup fromListto be that fast. Now it all makes sense.
– Johu
Aug 25 at 22:46
add a comment |Â
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
3
down vote
DownValues should get close to the actual value I believe.
ByteCount[DownValues[fnc]]
192000080
You could also use MaxMemoryUsed during the construction:
ClearAll[fnc]
MaxMemoryUsed[Do[fnc[2 i] = i, i, 10^6];]
168327840
answered Aug 25 at 18:10
Mr.Wizard♦
227k284631014
DownValuesreturns way more, than only the right hand side. Is it a fair comparison? I guess all memory structures should be compared byDownValuesthen.
– Johu
Aug 25 at 18:17
@Johu I think it's a good first order approximation at least. I did provide a second method that I presume is more accurate.
– Mr.Wizard♦
Aug 25 at 18:18
add a comment |Â
up vote
3
down vote
DownValues should get close to the actual value I believe.
ByteCount[DownValues[fnc]]
192000080
You could also use MaxMemoryUsed during the construction:
ClearAll[fnc]
MaxMemoryUsed[Do[fnc[2 i] = i, i, 10^6];]
168327840
answered Aug 25 at 18:10
Mr.Wizard♦
227k284631014
DownValuesreturns way more, than only the right hand side. Is it a fair comparison? I guess all memory structures should be compared byDownValuesthen.
– Johu
Aug 25 at 18:17
@Johu I think it's a good first order approximation at least. I did provide a second method that I presume is more accurate.
– Mr.Wizard♦
Aug 25 at 18:18
add a comment |Â
up vote
3
down vote
up vote
3
down vote
DownValues should get close to the actual value I believe.
ByteCount[DownValues[fnc]]
192000080
You could also use MaxMemoryUsed during the construction:
ClearAll[fnc]
MaxMemoryUsed[Do[fnc[2 i] = i, i, 10^6];]
168327840
answered Aug 25 at 18:10
Mr.Wizard♦
227k284631014
DownValues should get close to the actual value I believe.
ByteCount[DownValues[fnc]]
192000080
You could also use MaxMemoryUsed during the construction:
ClearAll[fnc]
MaxMemoryUsed[Do[fnc[2 i] = i, i, 10^6];]
168327840
answered Aug 25 at 18:10
Mr.Wizard♦
227k284631014
answered Aug 25 at 18:10
Mr.Wizard♦
227k284631014
answered Aug 25 at 18:10
Mr.Wizard♦
227k284631014
answered Aug 25 at 18:10
Mr.Wizard♦
227k284631014
227k284631014
DownValuesreturns way more, than only the right hand side. Is it a fair comparison? I guess all memory structures should be compared byDownValuesthen.
– Johu
Aug 25 at 18:17
@Johu I think it's a good first order approximation at least. I did provide a second method that I presume is more accurate.
– Mr.Wizard♦
Aug 25 at 18:18
add a comment |Â
DownValuesreturns way more, than only the right hand side. Is it a fair comparison? I guess all memory structures should be compared byDownValuesthen.
– Johu
Aug 25 at 18:17
@Johu I think it's a good first order approximation at least. I did provide a second method that I presume is more accurate.
– Mr.Wizard♦
Aug 25 at 18:18
DownValues returns way more, than only the right hand side. Is it a fair comparison? I guess all memory structures should be compared by DownValues then.– Johu
Aug 25 at 18:17
DownValues returns way more, than only the right hand side. Is it a fair comparison? I guess all memory structures should be compared by DownValues then.– Johu
Aug 25 at 18:17
@Johu I think it's a good first order approximation at least. I did provide a second method that I presume is more accurate.
– Mr.Wizard♦
Aug 25 at 18:18
@Johu I think it's a good first order approximation at least. I did provide a second method that I presume is more accurate.
– Mr.Wizard♦
Aug 25 at 18:18
add a comment |Â
up vote
3
down vote
In your example the value of the argument is a part of the definition. Value of fnc[2 i], where i is a symbol, is not defined in your MWE.
Total@Table[ByteCount[fnc[2 i]], i, n]
16 000 000
(Edit: note, that my solution only gives only the size of the right hand side and probably underestimates the real memory cost. See the other solution.)
Note also, that your timing measurement lead to misleading results as you apply it to so simple operations. Compare to
n2 = 10;
AbsoluteTiming[Position[tbl, #] & /@ (2 RandomInteger[n, n2])] // First
AbsoluteTiming[Replace[rls] /@ (2 RandomInteger[n, n2]);] // First
2.19423
1.8028
Both of these approaches are very slow, as they require going through the whole list to fine the element.
These are much much faster:
n2 = 100000;
AbsoluteTiming[fnc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[asc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[Lookup[asc, 2 RandomInteger[n, n2]];] // First
> 0.267781
> 0.226777
> 0.171752
I think it is misleading to call fnc[i] a function, as a function usually is evaluated runtime. In your MWE you save a precomputed value. This technique is referred to as memoization.
When you wonder which one you should use, I would always use what makes sense semantically, because the engineers behind the kernel and native commands have but a lot of effort into finding a balance between all the features one usually needs from a List, Rules, Associations and Symbols. Associations and Symbols are the ones, which require fast random access.
answered Aug 25 at 17:15
Johu
2,451826
Hmm, this is very low RAM usage, comparable to Table, yet faster than all of them. Why wouldn't one use Function instead of Table or Association?
– Leon
Aug 25 at 17:21
Please see the updated answer.
– Johu
Aug 25 at 17:43
1
Your problem is thatfoo /@ 2 RandomInteger[n, n2]is parsed as(foo /@ 2) RandomInteger[n, n2].
– Carl Woll
Aug 25 at 20:42
@CarlWoll oh lord. how stupid.
– Johu
Aug 25 at 22:36
I fixed it now and adjusted my conclusions. I did think it had to be impossible for the lookup fromListto be that fast. Now it all makes sense.
– Johu
Aug 25 at 22:46
add a comment |Â
up vote
3
down vote
In your example the value of the argument is a part of the definition. Value of fnc[2 i], where i is a symbol, is not defined in your MWE.
Total@Table[ByteCount[fnc[2 i]], i, n]
16 000 000
(Edit: note, that my solution only gives only the size of the right hand side and probably underestimates the real memory cost. See the other solution.)
Note also, that your timing measurement lead to misleading results as you apply it to so simple operations. Compare to
n2 = 10;
AbsoluteTiming[Position[tbl, #] & /@ (2 RandomInteger[n, n2])] // First
AbsoluteTiming[Replace[rls] /@ (2 RandomInteger[n, n2]);] // First
2.19423
1.8028
Both of these approaches are very slow, as they require going through the whole list to fine the element.
These are much much faster:
n2 = 100000;
AbsoluteTiming[fnc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[asc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[Lookup[asc, 2 RandomInteger[n, n2]];] // First
> 0.267781
> 0.226777
> 0.171752
I think it is misleading to call fnc[i] a function, as a function usually is evaluated runtime. In your MWE you save a precomputed value. This technique is referred to as memoization.
When you wonder which one you should use, I would always use what makes sense semantically, because the engineers behind the kernel and native commands have but a lot of effort into finding a balance between all the features one usually needs from a List, Rules, Associations and Symbols. Associations and Symbols are the ones, which require fast random access.
answered Aug 25 at 17:15
Johu
2,451826
Hmm, this is very low RAM usage, comparable to Table, yet faster than all of them. Why wouldn't one use Function instead of Table or Association?
– Leon
Aug 25 at 17:21
Please see the updated answer.
– Johu
Aug 25 at 17:43
1
Your problem is thatfoo /@ 2 RandomInteger[n, n2]is parsed as(foo /@ 2) RandomInteger[n, n2].
– Carl Woll
Aug 25 at 20:42
@CarlWoll oh lord. how stupid.
– Johu
Aug 25 at 22:36
I fixed it now and adjusted my conclusions. I did think it had to be impossible for the lookup fromListto be that fast. Now it all makes sense.
– Johu
Aug 25 at 22:46
add a comment |Â
up vote
3
down vote
up vote
3
down vote
In your example the value of the argument is a part of the definition. Value of fnc[2 i], where i is a symbol, is not defined in your MWE.
Total@Table[ByteCount[fnc[2 i]], i, n]
16 000 000
(Edit: note, that my solution only gives only the size of the right hand side and probably underestimates the real memory cost. See the other solution.)
Note also, that your timing measurement lead to misleading results as you apply it to so simple operations. Compare to
n2 = 10;
AbsoluteTiming[Position[tbl, #] & /@ (2 RandomInteger[n, n2])] // First
AbsoluteTiming[Replace[rls] /@ (2 RandomInteger[n, n2]);] // First
2.19423
1.8028
Both of these approaches are very slow, as they require going through the whole list to fine the element.
These are much much faster:
n2 = 100000;
AbsoluteTiming[fnc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[asc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[Lookup[asc, 2 RandomInteger[n, n2]];] // First
> 0.267781
> 0.226777
> 0.171752
I think it is misleading to call fnc[i] a function, as a function usually is evaluated runtime. In your MWE you save a precomputed value. This technique is referred to as memoization.
When you wonder which one you should use, I would always use what makes sense semantically, because the engineers behind the kernel and native commands have but a lot of effort into finding a balance between all the features one usually needs from a List, Rules, Associations and Symbols. Associations and Symbols are the ones, which require fast random access.
answered Aug 25 at 17:15
Johu
2,451826
In your example the value of the argument is a part of the definition. Value of fnc[2 i], where i is a symbol, is not defined in your MWE.
Total@Table[ByteCount[fnc[2 i]], i, n]
16 000 000
(Edit: note, that my solution only gives only the size of the right hand side and probably underestimates the real memory cost. See the other solution.)
Note also, that your timing measurement lead to misleading results as you apply it to so simple operations. Compare to
n2 = 10;
AbsoluteTiming[Position[tbl, #] & /@ (2 RandomInteger[n, n2])] // First
AbsoluteTiming[Replace[rls] /@ (2 RandomInteger[n, n2]);] // First
2.19423
1.8028
Both of these approaches are very slow, as they require going through the whole list to fine the element.
These are much much faster:
n2 = 100000;
AbsoluteTiming[fnc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[asc /@ (2 RandomInteger[n, n2]);] // First
AbsoluteTiming[Lookup[asc, 2 RandomInteger[n, n2]];] // First
> 0.267781
> 0.226777
> 0.171752
I think it is misleading to call fnc[i] a function, as a function usually is evaluated runtime. In your MWE you save a precomputed value. This technique is referred to as memoization.
When you wonder which one you should use, I would always use what makes sense semantically, because the engineers behind the kernel and native commands have but a lot of effort into finding a balance between all the features one usually needs from a List, Rules, Associations and Symbols. Associations and Symbols are the ones, which require fast random access.
answered Aug 25 at 17:15
Johu
2,451826
edited Aug 25 at 22:45
answered Aug 25 at 17:15
Johu
2,451826
answered Aug 25 at 17:15
Johu
2,451826
answered Aug 25 at 17:15
Johu
2,451826
2,451826
Hmm, this is very low RAM usage, comparable to Table, yet faster than all of them. Why wouldn't one use Function instead of Table or Association?
– Leon
Aug 25 at 17:21
Please see the updated answer.
– Johu
Aug 25 at 17:43
1
Your problem is thatfoo /@ 2 RandomInteger[n, n2]is parsed as(foo /@ 2) RandomInteger[n, n2].
– Carl Woll
Aug 25 at 20:42
@CarlWoll oh lord. how stupid.
– Johu
Aug 25 at 22:36
I fixed it now and adjusted my conclusions. I did think it had to be impossible for the lookup fromListto be that fast. Now it all makes sense.
– Johu
Aug 25 at 22:46
add a comment |Â
Hmm, this is very low RAM usage, comparable to Table, yet faster than all of them. Why wouldn't one use Function instead of Table or Association?
– Leon
Aug 25 at 17:21
Please see the updated answer.
– Johu
Aug 25 at 17:43
1
Your problem is thatfoo /@ 2 RandomInteger[n, n2]is parsed as(foo /@ 2) RandomInteger[n, n2].
– Carl Woll
Aug 25 at 20:42
@CarlWoll oh lord. how stupid.
– Johu
Aug 25 at 22:36
I fixed it now and adjusted my conclusions. I did think it had to be impossible for the lookup fromListto be that fast. Now it all makes sense.
– Johu
Aug 25 at 22:46
Hmm, this is very low RAM usage, comparable to Table, yet faster than all of them. Why wouldn't one use Function instead of Table or Association?
– Leon
Aug 25 at 17:21
Hmm, this is very low RAM usage, comparable to Table, yet faster than all of them. Why wouldn't one use Function instead of Table or Association?
– Leon
Aug 25 at 17:21
Please see the updated answer.
– Johu
Aug 25 at 17:43
Please see the updated answer.
– Johu
Aug 25 at 17:43
1
1
Your problem is that
foo /@ 2 RandomInteger[n, n2] is parsed as (foo /@ 2) RandomInteger[n, n2].– Carl Woll
Aug 25 at 20:42
Your problem is that
foo /@ 2 RandomInteger[n, n2] is parsed as (foo /@ 2) RandomInteger[n, n2].– Carl Woll
Aug 25 at 20:42
@CarlWoll oh lord. how stupid.
– Johu
Aug 25 at 22:36
@CarlWoll oh lord. how stupid.
– Johu
Aug 25 at 22:36
I fixed it now and adjusted my conclusions. I did think it had to be impossible for the lookup from
List to be that fast. Now it all makes sense.– Johu
Aug 25 at 22:46
I fixed it now and adjusted my conclusions. I did think it had to be impossible for the lookup from
List to be that fast. Now it all makes sense.– Johu
Aug 25 at 22:46
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f180642%2fbytecount-on-function%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password