Ways to speed up Pick

up vote
1
down vote

favorite

I have a few large arrays (100 times larger than in the example) and I need to re-arrange them. Basically using tests on b and c, I need to pick elements of a.

a = RandomReal[1, 10^7];
b = RandomReal[1, 10^7];
c = RandomReal[1, 10^7];

sel = MapThread[(Abs[#1] <= 0.1) Ã¢ÂˆÂ§ (Abs[#2] > 0.9)&, b, c]; // AbsoluteTiming

10.9262,Null

d = Pick[a, sel]; // AbsoluteTiming

0.584831,Null

MapThread and Pick are the only ways to do this, as far as I found, I can't re-arrange the code any more.
The selection generation takes awhile, many times longer than on other platforms, is there a faster way to speed this up?

edited 28 mins ago

kglr

168k8190393

asked 1 hour ago

Anatoly

807

add a commentÂ |Â

up vote
1
down vote

favorite

I have a few large arrays (100 times larger than in the example) and I need to re-arrange them. Basically using tests on b and c, I need to pick elements of a.

a = RandomReal[1, 10^7];
b = RandomReal[1, 10^7];
c = RandomReal[1, 10^7];

sel = MapThread[(Abs[#1] <= 0.1) Ã¢ÂˆÂ§ (Abs[#2] > 0.9)&, b, c]; // AbsoluteTiming

10.9262,Null

d = Pick[a, sel]; // AbsoluteTiming

0.584831,Null

edited 28 mins ago

kglr

168k8190393

asked 1 hour ago

Anatoly

807

add a commentÂ |Â

up vote
1
down vote

favorite

I have a few large arrays (100 times larger than in the example) and I need to re-arrange them. Basically using tests on b and c, I need to pick elements of a.

a = RandomReal[1, 10^7];
b = RandomReal[1, 10^7];
c = RandomReal[1, 10^7];

sel = MapThread[(Abs[#1] <= 0.1) Ã¢ÂˆÂ§ (Abs[#2] > 0.9)&, b, c]; // AbsoluteTiming

10.9262,Null

d = Pick[a, sel]; // AbsoluteTiming

0.584831,Null

edited 28 mins ago

kglr

168k8190393

asked 1 hour ago

Anatoly

807

I have a few large arrays (100 times larger than in the example) and I need to re-arrange them. Basically using tests on b and c, I need to pick elements of a.

a = RandomReal[1, 10^7];
b = RandomReal[1, 10^7];
c = RandomReal[1, 10^7];

sel = MapThread[(Abs[#1] <= 0.1) Ã¢ÂˆÂ§ (Abs[#2] > 0.9)&, b, c]; // AbsoluteTiming

10.9262,Null

d = Pick[a, sel]; // AbsoluteTiming

0.584831,Null

list-manipulation

edited 28 mins ago

kglr

168k8190393

asked 1 hour ago

Anatoly

807

edited 28 mins ago

kglr

168k8190393

asked 1 hour ago

Anatoly

807

edited 28 mins ago

kglr

168k8190393

edited 28 mins ago

kglr

168k8190393

edited 28 mins ago

kglr

168k8190393

asked 1 hour ago

Anatoly

807

asked 1 hour ago

Anatoly

807

asked 1 hour ago

Anatoly

807

add a commentÂ |Â

1 Answer
1

active

oldest

votes

up vote
4
down vote

You can use UnitStep to construct your selector array:

SeedRandom[1]
a = RandomReal[1, 10^7];
b = RandomReal[1, 10^7];
c = RandomReal[1, 10^7];

sel = MapThread[(Abs[#1] <= 0.1) Ã¢ÂˆÂ§ (Abs[#2] > 0.9) &, b, c]; //
 AbsoluteTiming // First

15.332532

d = Pick[a, sel]; // AbsoluteTiming // First

0.605384

sel2 = UnitStep[.1 - Abs[b]] UnitStep[Abs[c] - .9]; // AbsoluteTiming // First

0.439621

d2 = Pick[a, sel2, 1]; // AbsoluteTiming // First

0.031240

d == d2

True

If sel is already created, you can convert it a packed array. Additional processing time to do this is more than compensated for by speed gains in Pick:

sel3 = Developer`ToPackedArray[With[True = 1, False = 0, Evaluate[sel]]]; // 
 AbsoluteTiming // First

0.37475

d3 = Pick[a, sel3, 1]; // AbsoluteTiming // First

0.031245

d == d3

True

See this answer by Mr.Wizard for the origin of the With[True = 1, False = 0, ...] trick.

edited 29 mins ago

answered 37 mins ago

kglr

168k8190393

That's exactly what I was going to post. Always use vectorized functions if you can and always stay with machine integers/reals if you can. Furthermore, sel is not a packed array because MapThread doesn't pack automatically for some reason. That makes MapThread even less appealing.
â€“Â Sjoerd Smit
35 mins ago

add a commentÂ |Â

Your Answer

StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\$","\$"]]);
);
);
, "mathjax-editing");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "387"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f184719%2fways-to-speed-up-pick%23new-answer', 'question_page');

);

Post as a guest

Name

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

up vote
4
down vote

You can use UnitStep to construct your selector array:

SeedRandom[1]
a = RandomReal[1, 10^7];
b = RandomReal[1, 10^7];
c = RandomReal[1, 10^7];

sel = MapThread[(Abs[#1] <= 0.1) Ã¢ÂˆÂ§ (Abs[#2] > 0.9) &, b, c]; //
 AbsoluteTiming // First

15.332532

d = Pick[a, sel]; // AbsoluteTiming // First

0.605384

sel2 = UnitStep[.1 - Abs[b]] UnitStep[Abs[c] - .9]; // AbsoluteTiming // First

0.439621

d2 = Pick[a, sel2, 1]; // AbsoluteTiming // First

0.031240

d == d2

True

If sel is already created, you can convert it a packed array. Additional processing time to do this is more than compensated for by speed gains in Pick:

sel3 = Developer`ToPackedArray[With[True = 1, False = 0, Evaluate[sel]]]; // 
 AbsoluteTiming // First

0.37475

d3 = Pick[a, sel3, 1]; // AbsoluteTiming // First

0.031245

d == d3

True

See this answer by Mr.Wizard for the origin of the With[True = 1, False = 0, ...] trick.

edited 29 mins ago

answered 37 mins ago

kglr

168k8190393

That's exactly what I was going to post. Always use vectorized functions if you can and always stay with machine integers/reals if you can. Furthermore, sel is not a packed array because MapThread doesn't pack automatically for some reason. That makes MapThread even less appealing.
â€“Â Sjoerd Smit
35 mins ago

add a commentÂ |Â

up vote
4
down vote

You can use UnitStep to construct your selector array:

SeedRandom[1]
a = RandomReal[1, 10^7];
b = RandomReal[1, 10^7];
c = RandomReal[1, 10^7];

sel = MapThread[(Abs[#1] <= 0.1) Ã¢ÂˆÂ§ (Abs[#2] > 0.9) &, b, c]; //
 AbsoluteTiming // First

15.332532

d = Pick[a, sel]; // AbsoluteTiming // First

0.605384

sel2 = UnitStep[.1 - Abs[b]] UnitStep[Abs[c] - .9]; // AbsoluteTiming // First

0.439621

d2 = Pick[a, sel2, 1]; // AbsoluteTiming // First

0.031240

d == d2

True

If sel is already created, you can convert it a packed array. Additional processing time to do this is more than compensated for by speed gains in Pick:

sel3 = Developer`ToPackedArray[With[True = 1, False = 0, Evaluate[sel]]]; // 
 AbsoluteTiming // First

0.37475

d3 = Pick[a, sel3, 1]; // AbsoluteTiming // First

0.031245

d == d3

True

See this answer by Mr.Wizard for the origin of the With[True = 1, False = 0, ...] trick.

edited 29 mins ago

answered 37 mins ago

kglr

168k8190393

That's exactly what I was going to post. Always use vectorized functions if you can and always stay with machine integers/reals if you can. Furthermore, sel is not a packed array because MapThread doesn't pack automatically for some reason. That makes MapThread even less appealing.
â€“Â Sjoerd Smit
35 mins ago

add a commentÂ |Â

up vote
4
down vote

You can use UnitStep to construct your selector array:

SeedRandom[1]
a = RandomReal[1, 10^7];
b = RandomReal[1, 10^7];
c = RandomReal[1, 10^7];

sel = MapThread[(Abs[#1] <= 0.1) Ã¢ÂˆÂ§ (Abs[#2] > 0.9) &, b, c]; //
 AbsoluteTiming // First

15.332532

d = Pick[a, sel]; // AbsoluteTiming // First

0.605384

sel2 = UnitStep[.1 - Abs[b]] UnitStep[Abs[c] - .9]; // AbsoluteTiming // First

0.439621

d2 = Pick[a, sel2, 1]; // AbsoluteTiming // First

0.031240

d == d2

True

If sel is already created, you can convert it a packed array. Additional processing time to do this is more than compensated for by speed gains in Pick:

sel3 = Developer`ToPackedArray[With[True = 1, False = 0, Evaluate[sel]]]; // 
 AbsoluteTiming // First

0.37475

d3 = Pick[a, sel3, 1]; // AbsoluteTiming // First

0.031245

d == d3

True

See this answer by Mr.Wizard for the origin of the With[True = 1, False = 0, ...] trick.

edited 29 mins ago

answered 37 mins ago

kglr

168k8190393

You can use UnitStep to construct your selector array:

SeedRandom[1]
a = RandomReal[1, 10^7];
b = RandomReal[1, 10^7];
c = RandomReal[1, 10^7];

sel = MapThread[(Abs[#1] <= 0.1) Ã¢ÂˆÂ§ (Abs[#2] > 0.9) &, b, c]; //
 AbsoluteTiming // First

15.332532

d = Pick[a, sel]; // AbsoluteTiming // First

0.605384

sel2 = UnitStep[.1 - Abs[b]] UnitStep[Abs[c] - .9]; // AbsoluteTiming // First

0.439621

d2 = Pick[a, sel2, 1]; // AbsoluteTiming // First

0.031240

d == d2

True

If sel is already created, you can convert it a packed array. Additional processing time to do this is more than compensated for by speed gains in Pick:

sel3 = Developer`ToPackedArray[With[True = 1, False = 0, Evaluate[sel]]]; // 
 AbsoluteTiming // First

0.37475

d3 = Pick[a, sel3, 1]; // AbsoluteTiming // First

0.031245

d == d3

True

See this answer by Mr.Wizard for the origin of the With[True = 1, False = 0, ...] trick.

edited 29 mins ago

answered 37 mins ago

kglr

168k8190393

edited 29 mins ago

answered 37 mins ago

kglr

168k8190393

answered 37 mins ago

kglr

168k8190393

answered 37 mins ago

kglr

168k8190393

That's exactly what I was going to post. Always use vectorized functions if you can and always stay with machine integers/reals if you can. Furthermore, sel is not a packed array because MapThread doesn't pack automatically for some reason. That makes MapThread even less appealing.
â€“Â Sjoerd Smit
35 mins ago

add a commentÂ |Â

That's exactly what I was going to post. Always use vectorized functions if you can and always stay with machine integers/reals if you can. Furthermore, sel is not a packed array because MapThread doesn't pack automatically for some reason. That makes MapThread even less appealing.
â€“Â Sjoerd Smit
35 mins ago

That's exactly what I was going to post. Always use vectorized functions if you can and always stay with machine integers/reals if you can. Furthermore, sel is not a packed array because MapThread doesn't pack automatically for some reason. That makes MapThread even less appealing.
â€“Â Sjoerd Smit
35 mins ago

add a commentÂ |Â

draft saved

draft discarded

draft saved

draft discarded

Post as a guest

Name

Search This Blog

Iyfjky