Uber Idea

I was in the middle of my evening walk yesterday when I thought of an idea. 

Why don't people who use middle men to get jobs, circumvent the medium once they have a relationship with the buyer?

For example - I work with the same people on Fiverr. Why not just hire the people directly - as long as the person is ok with it? 

Another example - I use Amazon marketplace for a few tasks around the house. Now that we are well acquainted, we don't go through Amazon any more. What is the problem with that?

Especially when it comes to driving Uber, I have heard of so many drivers complain that they get a small fraction of what we pay Uber. So why not work out a deal with someone that needs a ride on a consistent basis to pick them up and drop them off for a fare that is good for both parties?

I think that the middleman economy is good to bring the demand and supply to one place. But after that introduction, the buyer and seller will benefit from not having to pay excess fees.