Obscured nailed it there, the only reason you have a soft spot there is drainage.
Looking at the pics, your going to have water in that area, period. Scrape it out with the loader, and use some biger rock to build it up even with, or above the grade next to it. Then add your gravel of choice on top.
A little grading work on the left side of the drive would be a good idea. I say left side because it seems the right side, by the trees, is another foot or so higher than the left side.
If you could, scrape that area between the trees and the drive down to form a 'v' ditch, it would help the driveway out. Depending on what you mow with, of course. I would make sure to leave enough room to get a mower width on each side of the ditch.
BTW, that's going to really look nice, with a tree-lined driveway. In the pic, is that the roadway we're looking at, and does that roadway have a ditch?