Perf KVM

Perf KVM in RHEL6.1:
===================

Thanks to supriya kannery for documenting perf kvm

RPM
=====
[root @]# rpm -qa | grep perf
perf-2.6.32-131.0.15.el6.x86_64

Preparation:
==========
Before running perf kvm in host, need to save files /proc/modules and /proc/kallsyms from guest into host.
One of the easiest methods is as follows:

> start VM with host portforwarding
start guest

> Login to the guest and save modules and kallsyms
cat /proc/modules > /tmp/modules
cat /proc/modules > /tmp/kallsyms
>> From host get the above two files.
scp -P 5555 root@localhost:/tmp/kallsyms guest-kallsyms
scp -P 5555 root@localhost:/tmp/modules guest-modules

Note: if directly scp from /proc/modules, zero sized files are copied. Hence first save /proc/xx into locate files in guest and then copy to host.

Start recording events in guest and host
===========================
perf kvm --host --guest --guestkallsyms=guest-kallsyms --guestmodules=guest-modules record -a -o perf.data

Output will be stored in perf.data.kvm if both --host and --guest are specified. If only --host is specified the file will be named perf.data.host and if only --guest is specified file will be named perf.data.guest

On pressing Ctl+C or SIGINT perf stops recording.

Report the events:
===============
perf kvm --host --guest --guestkallsyms=guest-kallsyms --guestmodules=guest-modules report -i perf.data --force

Note: without using --force I couldn't collect report even though the recorded file and report was generated using same userid or root. This might be a bug to be looked at. But for now, reporting works with --force option.

Sample results:
[root]# perf kvm --host --guest --guestmodules=guest-modules report -i perf.data.kvm --force > analyse
[root]# cat analyse
# Events: 7K cycles
#
# Overhead Command Shared Object Symbol
# ........ ............ ................. .....................................
#
95.06% vi vi [.] 0x48287
0.61% init [kernel.kallsyms] [k] intel_idle
0.36% vi libc-2.12.so [.] _wordcopy_fwd_aligned
0.32% vi libc-2.12.so [.] __strlen_sse42
0.14% swapper [kernel.kallsyms] [k] intel_idle
0.13% init [kernel.kallsyms] [k] uhci_irq
0.11% perf [kernel.kallsyms] [k] generic_exec_single
0.11% init [kernel.kallsyms] [k] tg_shares_up
0.10% qemu-kvm [kernel.kallsyms] [k] tg_shares_up


[root]# perf report --sort comm,dso -i perf.data.kvm --force > highlevel
[root]# cat highlevel
# Events: 7K cycles
#
# Overhead Command Shared Object
# ........ ............ .................
#
95.06% vi vi
1.55% init [kernel.kallsyms]
0.75% vi [kernel.kallsyms]
0.69% vi libc-2.12.so
0.50% perf [kernel.kallsyms]
0.30% swapper [kernel.kallsyms]
0.19% ps [kernel.kallsyms]
0.17% qemu-kvm [kernel.kallsyms]
0.09% events/5 [kernel.kallsyms]
0.09% events/1 [kernel.kallsyms]
0.09% events/24 [kernel.kallsyms]
0.08% ps libc-2.12.so
0.08% events/25 [kernel.kallsyms]
0.07% kondemand/8 [kernel.kallsyms]
0.07% kondemand/23 [kernel.kallsyms]
0.07% perf libc-2.12.so
0.07% watchdog/26 [kernel.kallsyms]
0.01% kondemand/15 [kernel.kallsyms]
0.01% ksoftirqd/18 [kernel.kallsyms]
0.01% perf [nfs]
0.01% ps libproc-3.2.8.so
0.01% qemu-kvm [unknown]
0.01% qemu-kvm qemu-kvm
0.00% sleep [kernel.kallsyms]
0.00% awk [kernel.kallsyms]