I don’t understand why miners wouldn’t include all pending transactions in their block. Seeing as how pow is a guessing game, there are plenty of permutations of the Merkle tree to generate plenty of roots. I guess to dissencentivize free loaders so miners can continue to make money once Coinbase txns are no longer a thing? It’s certainly not free to mine!
As the other commenter said, there is about 1 MB of room in each block, and almost always >1 MB worth of transactions in the "mempool" (actually it's often >12MB for the most part, because people will make tons and tons of transactions at 1sat/byte and not really care if they take forever to go through).
And you are correct that bitcoin's ultimate goal is to have mining income come solely from transaction fees eventually. But by that point the hope is that "second layer" systems will take the brunt of the transactions and the actual core blockchain will only be a settlement layer where a $500 fee is no big deal for 2 banks settling hundreds of millions of dollars directly on the blockchain.
Fees are set as a "flat rate" per transaction, but normally expressed as a fraction of "price over size"
So if a fee is 150 "sat/byte" or satoshi's per byte. A satoshi is 0.00000001 of a bitcoin.
So if your transaction is 250 bytes (which is average for a single transaction to a single person), then you will pay (250 * 150) satoshis (37500 sat), which is about $5.
If your transaction is larger, you'll pay more, if it's smaller, you'll pay less.
But they are "ranked" in terms of who is picked first by "sat/byte" since the amount of room in the block is only 1MB.
So a 500 byte transaction with 37500 sats in fees is worth less than a 250 byte transaction with 37500 sats in fees, because the 500 byte transaction takes up more room that other higher paying transactions could fit in.
A transaction can have many inputs and many outputs. These are all Bitcoin addresses.
Sally and Bob each have 1 Bitcoin. Sally received her bitcoin to a single address. If Sally wants to send that 1 Bitcoin to Charlie, that transaction would have 1 input (the address she received it on) and one output (the address Charlie wants to receive it to). Bob has 1 Bitcoin total, but his 1 Bitcoin came from 4 x 0.25 transactions. Bob didn't want to reuse addresses, so for him to send 1 Bitcoin to Charlie he has a transaction with 4 inputs (each address which contains 0.25 BTC) and 1 output. Thus, Bob's transaction is much bigger than Alice's, despite the same amount of Bitcoin being sent.
If they could, miners probably would put all the pending transactions into a block. That's why the block size limit exists; every other node in the network has to process all those transactions and they're not paid for it in any way, so without the size limit miners would be likely to act in a way that hurts the rest of the network.
The number of transactions that can be included in a block is limited. A block is generated on average 10 minutes (a bit quicker due to increasing hash rate) so the number of transactions in an hour is also limited. Naturally the transactions that pay most to the miners are getting selected by them and included in these blocks.
