Let's look at the data. Again, I will post the computer forecasts (ECMWF, GFS, GEM ensembles) for Friday evening and then Sunday morning. This will give you perspective on where the models are trending.
Now Sunday morning:
The data continues to accelerate Irene. The intensity is about the same and the track is only slightly eastward. Historically speaking, tropical systems that have greatly affected Richmond in the past (Isabel, Gaston) did not take this path. Irene is moving fairly quickly and along the coast, so the brunt of the storm will miss central VA.
I don't want to give Richmond the "all clear" (data can sometimes change), but if I still lived there, I wouldn't be worried at all. Impacts along the Virginia coast will be greater with 70 mph wind gusts and rain totals as high as 9 inches (I'd be moderately worried). Farther north into New England, rain totals could easily exceed 15 inches in some isolated areas (I'd be worried).