perf trace will show the events associated with the
target, initially syscalls, but other system events like
pagefaults, task lifetime events, scheduling events, etc.
Based on a session I saw at the Linux Collaboration Summit 2011 in San Francisco there are actually a few competing profiling toolkits. While most of these are aimed at kernel developers there is also the potential to use these more generally either (1) for optimization, or (2) in conjunction with various security toolkits to automate the systematic reduction of kernel-level attack space.
While both are a can of worms, (1) is more so. With regards to (2), AFAIK it is generally accepted that grsec provides the easiest profiling tools here; whilst 'roll your own' is never as secure as something locked down by multiple third parties 'in the know' and to a finite extent (see: NSA SELinux), it is far better than nothing, and very frequently custom code or the latest or patched version of a certain daemon will not have publicly accredited rulesets. Therefore grsec's solution is a reasonable basis for beginning here. Docs @ http://en.wikibooks.org/wiki/Grsecurity/The_Administration_U...
Linux is suffering from a lack of a good debugging API. There is a decent progress in the kernel debugging and profiling tools, but for userspace no changes were made for a long time.
"ptrace" is the main userspace debugging API, used behind tools like "strace" or "gdb", but it's old and clumsy. Quite new "perf" tool, on the other hand, allows user to get various CPU/Kernel stats mostly useful for profiling. Before "perf" you could only try to emulate your program using "valgrind" to get, say cache misses.
The command "perf trace", seems to move "perf" more into the "ptrace" domain. This is indeed exciting.
There is, strace does just that. perf trace is still evolving though and appears promising. See, for example, scripting support:
http://lwn.net/Articles/371448/
