huang zuxing blog @ home the quieter you become, the more you are able to hear.

25Jun/09Off

platform lsf “Total”

Use lsf so many years, every time a word "total" in lsf  always confuses us.

Case study:

A machine has 256G physical memory.

User A  submitted 2 jobs to the machine

one asks for 130G memory, the other asks for 110G memory.

>bsub -R "rusage[mem=130000]"  joba

>bsub -R "rusage[mem=110000]"  jobb

after jobs launch, below commands output:

>lshost  machine

HOST_NAME  type    model    cpuf   ncpus  maxmem     maxswp     server RESOURCES
machine    X86_64  x86_64   63.1   16     258081M    559921M    Yes ()

>lsload machine

HOST_NAME  status  r15s   r1m  r15m    ut    pg    ls    it  tmp      swp    mem
machine    ok      0.0    0.4   6.5    1%    0.0   1     2   120G     547G   64G

When user B typed bhosts and lsload, found there were 64G free memory and free cpu in machine, submitted a job and asked for 30G memory.

>bsub -R "rusage[mem=30000]" jobc

But the jobc  just pending , saying don't have enough resources,  user looked for us.
>bhosts -l machine

.....

CURRENT LOAD USED FOR SCHEDULING:
          r15s  r1m     r15m    ut    pg    io     ls    it    tmp   swp   mem
Total     0.0   0.4     6.0     0%    0.0   772    1     1     120G  546G  0G
Reserved  0.0   0.0     0.0     0%    0.0   0      0     0     0M    0M    64G
......

We told users there was no more memory for your  expected resource base on bhosts -l ; but the "total" in bhosts -l output confused all users.

Why lsload saying have 64G free memory, bhosts -l saying memory "Total" 0 ?  what is the meaning of the 0G memory in above output ? what is the meaning of "total" here ?

Let us do some math:

joba (130G) + jobb(110G) = 240G memory

We found jobb actually used around 125G memory.joba actually used around 66G memory.

joba(reserve 130G) + jobb( actual 125G) = 255G ~=256G memory

That is why bhosts -l report memory = 0 ,  "Total " free/availble memory =0

This word "Total" happens in somewhere when users type lsf command,  like bhosts -s output to report some dynamic resource info, be carefull, those "Total" mean "Free/Available" !!

Tagged as: , Comments Off
Comments (0) Trackbacks (0)

Sorry, the comment form is closed at this time.

Trackbacks are disabled.