Coding dialog: Python => Cross platform way of finding number of processors on a machine?

Members

Help

Home

Home » Coding » Python » Cross platform way of finding number of processors on a machine?

Show: Today's Messages :: Show Polls :: Message Navigator

Cross platform way of finding number of processors on a machine? [message #15178]

Thu, 04 October 2007 23:28

John
Messages: 10
Registered: August 2007

Junior Member

Is there a way to find the number of processors on a machine (on linux/
windows/macos/cygwin) using python code (using the same code/cross
platform code)?

Report message to a moderator

Re: Cross platform way of finding number of processors on a machine? [message #15183 is a reply to message #15178 ]

Fri, 05 October 2007 00:02

Nicholas Bastin
Messages: 7
Registered: October 2007

Junior Member

On 10/4/07, John <weekender_ny@yahoo.com> wrote:
>
> Is there a way to find the number of processors on a machine (on linux/
> windows/macos/cygwin) using python code (using the same code/cross
> platform code)?

There's no single call that will give you the same info on every
platform, but you can obviously write this yourself and switch based
on os.uname()[0] in most cases.

For Darwin, you can just use the subprocess module to call 'sysctl
hw.logicalcpu'. I'm not sure if there's a more direct way in python
to use sysctl. (hw.logicalcpu_max is what the hardware maximally
supports, but someone may have started their machine with OF blocking
some of the processors and you should probably respect that decision)

For Linux you can read /proc/cpuinfo and parse that information. Be
somewhat careful with this, however, if your processors support HT,
they will show as 2, and that may or may not be what you want. You
can deterministically parse this information out if you know which
processor families are truly multi-core, and which are HT.

For Win32, the cheap and dirty way is to read the NUMBER_OF_PROCESSORS
environment variable. 99% of the time, this will be correct, so it
might be sufficient for your purposes. There is a system call
available on windows that will net you the true number of virtual
cpus, but I don't know what it is.

I don't have a cygwin install laying around, but my guess is there's a
sysctl option there too.

One note: *all* of these methods will tell you the virtual number of
CPUs in a machine, not the physical number. There's almost no reason
why you care about the distinction between these two numbers, but if
you do, you'll have to go to great lengths to probe the actual
hardware on each platform. (And pre-WinXP, Windows doesn't actually
know the difference - all processors are presumed to be physical).

Also, Darwin and Linux will easily allow you to get the speed of the
processors, but on x86 these numbers are not the maximums due to C1E
and EIST (x86 processors from all vendors are capable of changing
speeds depending on load).

--
Nick

Report message to a moderator

Re: Cross platform way of finding number of processors on a machine? [message #15237 is a reply to message #15178 ]

Fri, 05 October 2007 03:58

Tim Golden
Messages: 51
Registered: August 2007

Member

Nicholas Bastin wrote:
> On 10/4/07, John <weekender_ny@yahoo.com> wrote:
>> Is there a way to find the number of processors on a machine (on linux/
>> windows/macos/cygwin) using python code (using the same code/cross
>> platform code)?
>
> There's no single call that will give you the same info on every
> platform, but you can obviously write this yourself and switch based
> on os.uname()[0] in most cases.

Second that point about not getting one cross-platform
answer. Under Windows, WMI is often the way to go for
these things:

http://msdn2.microsoft.com/en-us/library/aa394373.aspx

but, as Nicholas noted:

"""
Windows Server 2003, Windows XP, and Windows 2000:
This property is not available.
"""

since it's presumably not available in anything earlier either,
that leaves you with Vista or the early-adopter editions of the
next Windows Server product.

TJG

Report message to a moderator

Re: Cross platform way of finding number of processors on a machine? [message #15242 is a reply to message #15237 ]

Fri, 05 October 2007 04:09

Kay Schluehr
Messages: 35
Registered: August 2007

Member

On 5 Okt., 09:58, Tim Golden <m...@timgolden.me.uk> wrote:
> Nicholas Bastin wrote:
> > On 10/4/07, John <weekender...@yahoo.com> wrote:
> >> Is there a way to find the number of processors on a machine (on linux/
> >> windows/macos/cygwin) using python code (using the same code/cross
> >> platform code)?
>
> > There's no single call that will give you the same info on every
> > platform, but you can obviously write this yourself and switch based
> > on os.uname()[0] in most cases.
>
> Second that point about not getting one cross-platform
> answer. Under Windows, WMI is often the way to go for
> these things:
>
> http://msdn2.microsoft.com/en-us/library/aa394373.aspx
>
> but, as Nicholas noted:
>
> """
> Windows Server 2003, Windows XP, and Windows 2000:
> This property is not available.
> """
>
> since it's presumably not available in anything earlier either,
> that leaves you with Vista or the early-adopter editions of the
> next Windows Server product.
>
> TJG

Remarkable. I've a two years old Windows XP dual core notebook and
when I'm asking Python I get the correct answer:

>>> import os
>>> os.environ['NUMBER_OF_PROCESSORS']
2

However this feature might not be guaranteed by Microsoft for
arbitrary computers using their OS?

Kay

Report message to a moderator

Re: Cross platform way of finding number of processors on a machine? [message #15250 is a reply to message #15237 ]

Fri, 05 October 2007 04:42

Tim Golden
Messages: 51
Registered: August 2007

Member

[Tim Golden]
>> """
>> Windows Server 2003, Windows XP, and Windows 2000:
>> This property is not available.
>> """
>>
>> since it's presumably not available in anything earlier either,
>> that leaves you with Vista or the early-adopter editions of the
>> next Windows Server product.

Kay Schluehr wrote:

> Remarkable. I've a two years old Windows XP dual core notebook and
> when I'm asking Python I get the correct answer:
>
>>>> import os
>>>> os.environ['NUMBER_OF_PROCESSORS']
> 2
>
> However this feature might not be guaranteed by Microsoft for
> arbitrary computers using their OS?

Ahem. I was referring (and not very clearly, now I look
back at my post) to a particular property within the
WMI class, not to the general concept of counting
processors.

TJG

Report message to a moderator

Re: Cross platform way of finding number of processors on a machine? [message #15279 is a reply to message #15183 ]

Fri, 05 October 2007 06:30

Nick Craig-Wood
Messages: 25
Registered: August 2007

Junior Member

Nicholas Bastin <nick.bastin@gmail.com> wrote:
> On 10/4/07, John <weekender_ny@yahoo.com> wrote:
> >
> > Is there a way to find the number of processors on a machine (on linux/
> > windows/macos/cygwin) using python code (using the same code/cross
> > platform code)?
>
> There's no single call that will give you the same info on every
> platform, but you can obviously write this yourself and switch based
> on os.uname()[0] in most cases.
>
> For Darwin, you can just use the subprocess module to call 'sysctl
> hw.logicalcpu'. I'm not sure if there's a more direct way in python
> to use sysctl. (hw.logicalcpu_max is what the hardware maximally
> supports, but someone may have started their machine with OF blocking
> some of the processors and you should probably respect that decision)
>
> For Linux you can read /proc/cpuinfo and parse that information. Be
> somewhat careful with this, however, if your processors support HT,
> they will show as 2, and that may or may not be what you want. You
> can deterministically parse this information out if you know which
> processor families are truly multi-core, and which are HT.

On any unix/posix system (OSX and linux should both qualify) you can use

>>> import os
>>> os.sysconf('SC_NPROCESSORS_ONLN')
2
>>>

(From my Core 2 Duo laptop running linux)

--
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

Report message to a moderator

Re: Cross platform way of finding number of processors on a machine? [message #15509 is a reply to message #15178 ]

Sat, 06 October 2007 06:45

raims
Messages: 30
Registered: July 2007

Member

John <weekender_ny@yahoo.com> wrote:
> Is there a way to find the number of processors on a machine (on linux/
> windows/macos/cygwin) using python code (using the same code/cross
> platform code)?

From processing <http://cheeseshop.python.org/pypi/processing/0.34> :

def cpuCount():
'''
Returns the number of CPUs in the system
'''
if sys.platform == 'win32':
try:
num = int(os.environ['NUMBER_OF_PROCESSORS'])
except (ValueError, KeyError):
pass
elif sys.platform == 'darwin':
try:
num = int(os.popen('sysctl -n hw.ncpu').read())
except ValueError:
pass
else:
try:
num = os.sysconf('SC_NPROCESSORS_ONLN')
except (ValueError, OSError, AttributeError):
pass

if num >= 1:
return num
else:
raise NotImplementedError

--
Lawrence, oluyede.org - neropercaso.it
"It is difficult to get a man to understand
something when his salary depends on not
understanding it" - Upton Sinclair

Report message to a moderator

Re: Cross platform way of finding number of processors on a machine? [message #15536 is a reply to message #15509 ]

Sat, 06 October 2007 09:26

Paul Boddie
Messages: 36
Registered: August 2007

Member

On 6 Okt, 12:45, ra...@dot.com (Lawrence Oluyede) wrote:
>
> From processing <http://cheeseshop.python.org/pypi/processing/0.34> :

[...]

> num = os.sysconf('SC_NPROCESSORS_ONLN')

It's interesting what new (or obscure) standard library functions (and
system functions) can be discovered through these kinds of
discussions. However, this one seems to report the number of "virtual
CPUs" in situations like mine where the CPU supports hyperthreading,
as Nicholas Bastin says. One way to get the real number of CPUs (on
Linux-based systems) is to parse /proc/cpuinfo, and to count the
number of distinct "physical id" values.

>From the experiments I've done with pprocess [1], pretending that a
hyperthreading "virtual CPU" is as good as a real CPU (or CPU core)
tends to be counter-productive: the performance benefits in moving
from the utilisation of one to two "virtual CPUs" are virtually non-
existent.

Paul

[1] http://www.python.org/pypi/pprocess

Report message to a moderator

Re: Cross platform way of finding number of processors on a machine? [message #15543 is a reply to message #15536 ]

Sat, 06 October 2007 10:04

raims
Messages: 30
Registered: July 2007

Member

Paul Boddie <paul@boddie.org.uk> wrote:
> From the experiments I've done with pprocess [1], pretending that a
> hyperthreading "virtual CPU" is as good as a real CPU (or CPU core)
> tends to be counter-productive: the performance benefits in moving
> from the utilisation of one to two "virtual CPUs" are virtually non-
> existent.

We came the same conclusion at work. One of my colleagues has a machine
with HyperThreading and he noticed no real performance increase with the
processing library altough on my machine (which is a dual Core 2 Duo)
the performance boost was definitely there :-)

The good side of this is hyperthreading with parallel processes is like
using a mono-CPU machine so we don't have to take into account it like
if it was an entirely different kind of architecture.

--
Lawrence, oluyede.org - neropercaso.it
"It is difficult to get a man to understand
something when his salary depends on not
understanding it" - Upton Sinclair

Report message to a moderator

Re: Cross platform way of finding number of processors on a machine? [message #15573 is a reply to message #15509 ]

Sat, 06 October 2007 14:30

Nick Craig-Wood
Messages: 25
Registered: August 2007

Junior Member

Lawrence Oluyede <raims@dot.com> wrote:
> John <weekender_ny@yahoo.com> wrote:
> > Is there a way to find the number of processors on a machine (on linux/
> > windows/macos/cygwin) using python code (using the same code/cross
> > platform code)?
>
> From processing <http://cheeseshop.python.org/pypi/processing/0.34> :
>
> def cpuCount():
> '''
> Returns the number of CPUs in the system
> '''
> if sys.platform == 'win32':
> try:
> num = int(os.environ['NUMBER_OF_PROCESSORS'])
> except (ValueError, KeyError):
> pass
> elif sys.platform == 'darwin':
> try:
> num = int(os.popen('sysctl -n hw.ncpu').read())
> except ValueError:
> pass
> else:
> try:
> num = os.sysconf('SC_NPROCESSORS_ONLN')
> except (ValueError, OSError, AttributeError):
> pass
>
> if num >= 1:
> return num
> else:
> raise NotImplementedError

There is a bug in that code...

NotImplementedError will never be raised because num won't have been
set. It will raise "UnboundLocalError: local variable 'num'
referenced before assignment" instead

--
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

Report message to a moderator

Re: Cross platform way of finding number of processors on a machine? [message #17449 is a reply to message #15178 ]

Sun, 07 October 2007 20:26

Nicholas Bastin
Messages: 7
Registered: October 2007

Junior Member

On 10/6/07, Lawrence Oluyede <raims@dot.com> wrote:
> John <weekender_ny@yahoo.com> wrote:
> > Is there a way to find the number of processors on a machine (on linux/
> > windows/macos/cygwin) using python code (using the same code/cross
> > platform code)?
>
> >From processing <http://cheeseshop.python.org/pypi/processing/0.34> :

A few reiterated notes inline, from my previous message.

> def cpuCount():
> '''
> Returns the number of CPUs in the system
> '''
> if sys.platform == 'win32':
> try:
> num = int(os.environ['NUMBER_OF_PROCESSORS'])

The user can do bad things to this environment variable, but it's
probably ok most of the time. (Hey, they change it, they pay the
consequences).

> else:
> try:
> num = os.sysconf('SC_NPROCESSORS_ONLN')

This is really bad on linux. You really want to parse /proc/cpuinfo
because HT 'cpus' are almost useless, and can be bad in situations
where you try to treat them as general purpose cpus.

--
Nick

Report message to a moderator

Previous Topic:	Newbie packages Q
Next Topic:	supplying password to subprocess.call('rsync ...'), os.system('rsync ...')

Goto Forum:

-=] Back to Top [=-

[ Generate printable PDF ] [ Syndicate this forum (XML) ] [

]

Current Time: Fri May 16 02:30:43 EDT 2008

Total time taken to generate the page: 0.20418 seconds

.:: Contact :: Home ::.