#287 oprofile in kvm mode not working

[Jegathesan]

I built oprofile 0.9.9 and it is working fine with my kernel (Linux sw0 2.6.34.6) when I loaded into my box directly.

But I did the same steps in virtual machine mode where oprofile starts, but no report is generated in sample directory. Do I need to do anything specific for kvm mode? Is it an issue with kernel module or oprofile user daemon?

opcontrol --deinit
opcontrol --init
opcontrol --vmlinux /root/vmlinux
opcontrol --separate=kernel
opcontrol --cpu-buffer-size=9000000
opcontrol --event=CPU_CLK:100000:0:1:1
opcontrol --callgraph=0
opcontrol --status
opcontrol --start

opcontrol --stop
opcontrol --dump

opreport -l -p /mymod/modules
Error

sw0:FID128:root> opreport -l -p /fabos/modules
Using /var/lib/oprofile/samples/ for samples directory.
error: no sample files found: profile specification too strict ?
sw0:FID128:root>
Sample directory structure (no {kern} and {root} directory)

sw0:FID128:root> pwd
/var/lib/oprofile/samples/current
sw0:FID128:root> ls
stats/
sw0:FID128:root>
Sample directory structure (working case)

sw0:FID128:root> pwd
/var/lib/oprofile/samples/current
sw0:FID128:root> ls
stats/ {kern}/ {root}/
sw0:FID128:root>

[William Cohen]

Whether oprofile works on a guest vm depends on the hosts support of performance monitoring hardware. With older kernels running on the host such as RHEL6 2.6.32 kernel the guest vms do not have access to the machine. Also the guest vm needs to be configured to have the same processor configuration as host machine. The performance monitoring hardware is very processor implementation specific and the default cpu types used for guest vm are going to present the wrong hardware.

Since you are running a 2.6.34 kernel you should be able to check to see if the performance hardware is working properly. For example on a Intel Ivy Bridge host machine running fedora 24 and rhel6 guest the "perf stat true" can be run to see if the performance monitoring hardware is working properly:

$ perf stat true

Performance counter stats for 'true':

      0.653303      task-clock (msec)         #    0.117 CPUs utilized          
             1      context-switches          #    0.002 M/sec                  
             0      cpu-migrations            #    0.000 K/sec                  
            81      page-faults               #    0.124 M/sec                  
       620,732      cycles                    #    0.950 GHz                    
       443,430      stalled-cycles-frontend   #   71.44% frontend cycles idle   
       380,514      stalled-cycles-backend    #   61.30% backend  cycles idle   
       319,575      instructions              #    0.51  insns per cycle        
                                              #    1.39  stalled cycles per insn
        60,994      branches                  #   93.362 M/sec                  
         4,375      branch-misses             #    7.17% of all branches

   0.005569705 seconds time elapsed

If that doesn't work, then oprofile isn't going to work either.

Assuming that perf shows access is working then see whether oprofile is having the issue. On the same configuration as above:

[root@rhel664 tmp]# rpm -q oprofile
oprofile-0.9.9-13.el6.x86_64
[root@rhel664 tmp]# opcontrol --deinit
Unloading oprofile module
[root@rhel664 tmp]# opcontrol --init
[root@rhel664 tmp]# opcontrol --setup --vmlinux /usr/lib/debug/lib/modules/uname -r/vmlinux \

--separate=kernel --cpu-buffer-size=9000000
[root@rhel664 tmp]# --callgraph=0
bash: --callgraph=0: command not found
[root@rhel664 tmp]# opcontrol --status
Daemon not running
Session-dir: /var/lib/oprofile
Separate options: library kernel
vmlinux file: /usr/lib/debug/lib/modules/2.6.32-642.4.2.el6.x86_64/vmlinux
Image filter: none
Call-graph depth: 0
CPU buffer size: 9000000
[root@rhel664 tmp]# opcontrol --start
ATTENTION: Use of opcontrol is discouraged. Please see the man page for operf.
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Using 2.6+ OProfile kernel interface.
Reading module info.
Using log file /var/lib/oprofile/samples/oprofiled.log
Daemon started.
Profiler running.
[root@rhel664 tmp]# opcontrol --shutdown
Stopping profiling.
Killing daemon.
[root@rhel664 tmp]#
[root@rhel664 tmp]# opreport -l
Using /var/lib/oprofile/samples/ for samples directory.
CPU: Intel Sandy Bridge microarchitecture, speed 2294.8 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000
samples % image name symbol name
1 100.000 libcrypto.so.1.0.1e /usr/lib64/libcrypto.so.1.0.1e

You should look to see if there is something /var/log/messages about oprofile setup. If using performance monitoring hardware should see:

Aug 24 11:54:30 rhel664 kernel: oprofile: using NMI interrupt.

[Jegathesan]

I do not have perf in my system. Same setup is working fine with normal setup whereas it does not work in VM mode. Is there anything I can do with VM to eliminate this difference? Do I need to confiure anything to make it working?

[William Cohen]

You haven't stated what distribution the host system is running. That would decide whether the performance monitoring hardware is usable within a guest machine.

If the host is rhel6, the performance monitoring won't work in the guest because of the older kernels.

If the host is rhel7 or newer versions of Fedora you can use the performance monitoring hardware inside the guest. However, you will need to make the guest machine use the same cpuid information as the hostl. This can be set up using the information in section 3.3.2 of https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html

Installing perf is pretty trivial with root permission with:

yum install perf

[Jegathesan]

HOSTOS where VM runs in ubuntu

HOST OS

PRETTY_NAME="Ubuntu 14.04 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
root@HOST:~# uname -a
Linux HOST 3.14.17 #0 SMP Tue Sep 20 19:56:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root@HOST:~#

VM

It is customized distribution (kind of redhat model).

sw0:FID128:root> uname -a
Linux sw0 2.6.34.6 #1 SMP Tue Sep 20 16:50:05 UTC 2016 i686 GNU/Linux
sw0:FID128:root>