Self-modifying the underlying machine code isn't what it used to be. Besides the difficulty in writing it, there's lot's of caveats about how it interacts with the cache and the instruction pipeline. It also requires setup, because with modern memory protection all the machine code is read-only. Changing the memory protection for some machine code to be executable and writable at once will set off some alarms (And isn't even possible on systems with W^X). So you need to change it to just writable, make your modifications, then change it back to just executable, which is less suspicious, it just looks like what JIT compilers do. But all in all self-modifying code doesn't really give you anything.
The exception to that is packers and other obfuscation techniques, which are related to self-modifying code. The general idea with these is that you take your real program and compress/encrypt/mangle/etc it and store that data in an executable. The code in that executable de-compresses/decrypts/demangles that data, sets it as executable, and then runs it. Unlike traditional self-modifying code, packing is orders of magnitude easier to write for the malware developer. The advantage here is that an antivirus tool can't determine what your real program does statically unless it understands how you mangled it, which is hard to do in general. To "unpack" an executable you've got three general techniques:
1. Packers tend to get reused a lot, so just have a person write an unpacker for popular packers by hand, and do some pattern matching to figure out which packer an executable is using. This doesn't work for everything, but it's fairly simple.
2. Dynamic Analysis. Run the executable and watch the contents of memory as the program unpacks itself, the real program should pop right out. Of course you have to run the executable in some sort of sandbox environment, and there's ways for the malware to detect that and alter it's behavior. This also isn't the most efficient process, so you can't really do this to executables during, say, an antivirus scan.
3. Symbolic Analysis. Basically static analysis on steroids to figure out what the executable will do without actually running it. The malware can't stop this with sandbox detection. But it's super slow and is still an active area of research.
You can always run it on a real, unimportant machine not connected to anything. (And never connect that machine to anything ever again.) That feature just makes it slightly more difficult and costly to compromise program security.
The exception to that is packers and other obfuscation techniques, which are related to self-modifying code. The general idea with these is that you take your real program and compress/encrypt/mangle/etc it and store that data in an executable. The code in that executable de-compresses/decrypts/demangles that data, sets it as executable, and then runs it. Unlike traditional self-modifying code, packing is orders of magnitude easier to write for the malware developer. The advantage here is that an antivirus tool can't determine what your real program does statically unless it understands how you mangled it, which is hard to do in general. To "unpack" an executable you've got three general techniques:
1. Packers tend to get reused a lot, so just have a person write an unpacker for popular packers by hand, and do some pattern matching to figure out which packer an executable is using. This doesn't work for everything, but it's fairly simple.
2. Dynamic Analysis. Run the executable and watch the contents of memory as the program unpacks itself, the real program should pop right out. Of course you have to run the executable in some sort of sandbox environment, and there's ways for the malware to detect that and alter it's behavior. This also isn't the most efficient process, so you can't really do this to executables during, say, an antivirus scan.
3. Symbolic Analysis. Basically static analysis on steroids to figure out what the executable will do without actually running it. The malware can't stop this with sandbox detection. But it's super slow and is still an active area of research.