It's doing al sorts of interesting things. They subscribe to event feeds supplied by the operating system to keep track of security related events: modules that are loaded, processes that start, connections that open and close, users that log in and log off.

Then they also inject themselves into processes and 'hook' into operating system routines to check things the feeds to not provide: which files are being accessed, how often, what memory is allocated, what type of memory is allocated, which threads are running, where they are running and what they are doing, if there's mismatches between what the operating system told are the modules loaded, and what it can find in memory.

Most of this software can be configured so that the resource usage is relatively tame, but then on the next pentest the security people will notice all sorts of ways the products did not catch them (this is the usual case). And then things are tuned to max in short order :) And then you have security software running multiple rules and scans on each file any process opens. And processes open a lot of files, all the time.

And they're rarely tested with other competing software, and once they start both hooking on the same thing (and each other) you can get a death spiral.

Yea, certain routines in windows actually check if the OS is hooked. The PE loader implemented in ntdll verifies that NtOpenSection() (or NtOpenFile, not sure right now) has not been hooked. When I was looking into that it looked like it disabled concurrent module loading if detected a hook(so it became slower), probably as a bug fix for whatever software inserts itself in that place (Stuxnet did too!).