I've done this job any times while trying to getting the handles and door locks working on my car. The little ball joint and I have had our moments, with many crude words spoken in the heat of passion.
Soon, when the car finally gets back from the paint shop, I'll get to do it again.
The worst part of the whole job is getting the handle actuator attached to the ball joint on the release mechanism. For that I advise taking out the door lock actuator to give you a bit more room, then take out the door vent guard.
I use the handle of a fairly large screwdriver that fits into the ball socket on the actuator, with a hand through the hole for the vent supporting the thin metal piece with the ball on it. It's the best way I've found so far to do it. Maybe other people have better methods?