i didn't mean to incinuate that the world would come to an end, but merely that some compensation needed to be done if one were to go that route - it is a frequent thing to replace sheet metal with fiberglass or carbon fiber in a race car, but then you have also added a bunch of tubes to reinforce the chassis
the concept of a unibody is an old one - rather than provide a frame for rigidity in the chassis, instead a structure of load transferring interconnected panels is used - the key to this working though is that all of the panels have to be there, intact, and reinforced in key areas
unibody construction is used to reduce mass, while still providing the necessary rigidity and crash protection - it also tends to provide a less "truckish" feel and ride - it is a very cost effective way to build a car
the downside of unibody cars is increased cost to repair, and the lack of ability to adequately repair them in a severe crash - the results are frequently beyond the ability to restore the car at a cost below the value of that car
as for fenders and stiffness, to some degree, every component of the body is a structural member - you will find in a unibody car, a lot more nuts and bolts holding things together along torsional axes than you would on a framed car - for example, a fender on a framed car might have 4 or 5 bolts along the longitudinal line, but a unibody car will often have as many as 10 and often larger in size - this is to hold that member in place, and allow the sheet metal to transfer and absorb load
as for replacing with other materials, while carbon fiber is actually stiffer than sheet metal, the problem is going to be the connection - if you reinforced the mounting edge with embedded metal, then a carbon fiber fender could probably add as much rigidity as a sheet metal fender - the cost of course would increase - without it though, it would rip under load at the connection points (been there done that)
over the years, i've had a chance to play around with some composite body panels on a number of cars, both framed and unibody - in fact, the entire front end of my last toy was fiberglass - the change is noticeable, but not unmanageable - reinforcing members are usually pretty easy to add to most cars
another thing to consider is the effect of mass - if you reduce a car's mass, you reduce the load on the chassis, and in turn the amount of work any given component is doing - so, in theory, you could mitigate a lot of the structural loss by eliminating a lot of weight from the car in non-structural areas (wheels, seats, etc)
the hard part though is getting fiberglass to look good - that takes a LOT of work - vacuum formed carbon fiber is easier in some respects, as the panels are generally closer to the oem size and shape, but it still has fitment issues - these kinds of panels are generally fine for race cars, but fail to come up to the aesthetic standards that most people want in a street car
did that help?