Topic Options
#52483 - 10/05/15 01:33 AM Imaging's multi-threading bottleneck
Farid Offline
OL User

Registered: 05/15/03
Posts: 100
Loc: Montreal
Hi,

I run 4 processes that capture text files and generate pdf output using Imaging.
The 4 processes run in parallel while generating the ps files, each in its tmp folder under PP Watch spool, but these ps file are then moved to PP Imaging's QueuedJobs folder where they are processed sequentially one after the other, creating a bottleneck.
Having several thousands of large files, this is taking a lot of time.
How to improve the performance?

The server actually has 24 cores.

Thanks.

Top
#52486 - 10/05/15 07:50 AM Re: Imaging's multi-threading bottleneck [Re: Farid]
Evelyne Lachance
Unregistered


Hello Farid,

You can bump up the number of PlanetPress Image engines that do work in parallel, by going to the PlanetPress Button, then User Options, and then the Messenger section.

In this dialog (http://help.objectiflune.com/en/planetpress-workflow-user-guide/7.6/Default_CSH.html#/2669.html), check the "Let me set up how many instances to run" box, and change the number of instances.

To be very basic: One instance per core, minus 2 for the OS and Workflow. So, in your case, 22 cores (assuming this server doesn't run anything else) should run. Test and adjust, of course.

Top
#52490 - 10/05/15 07:19 PM Re: Imaging's multi-threading bottleneck [Re: Farid]
Farid Offline
OL User

Registered: 05/15/03
Posts: 100
Loc: Montreal
Thank you Evelyne. In the default configuration (when the option "Let me set setup the instances" is not checked), does it use multi-threading or it runs on one core only?

Regards.

Top
#52494 - 10/06/15 07:49 AM Re: Imaging's multi-threading bottleneck [Re: Farid]
Evelyne Lachance
Unregistered


I do believe it makes a calculation and adjusts the number of instances using the "cores-2", but if my memory serves me well, it has a hardcoded maximum of... I want to say 4 but I'm not sure.

Top
#52496 - 10/06/15 08:00 AM Re: Imaging's multi-threading bottleneck [Re: Farid]
Philippe F. Offline
OL Expert

Registered: 09/06/00
Posts: 1970
Loc: Objectif Lune, Montreal, Qc
Each instance runs on its own thread. The default number of instances is set to 1.5 times the number of cores the PC has, in order to maximize core/thread usage. Those values can be adjusted (for instance, if you are running mostly small jobs and you have a lot of RAM available, you could have 2 or 2.5 instances per core).
It's pretty much empirical how you can tweak those values, but note that at some point, overtweaking will result in degradation of performance, so monitor your logs closely to determine when you've hit that threshold.
_________________________
Technical Product Manager
I don't want to achieve immortality through my work; I want to achieve immortality through not dying - Woody Allen

Top