This time last year, I was at 6,7% body fat and had no abs, up from 4,8% the year before (no abs at then either). Haven't gone and had it checked again (and only got it checked because our employer was doing a voluntary health awareness thing, and encouraged everyone to have a thorough check-up, on their dime. A couple of guys were found to be in such a poor state that they were advised to find a less demanding job for the sake of their health.)
Which brings me to my point. I worked out (still do) and had a physically demanding job (still do), and never felt like shit or felt fatigued or lethargic. Even when talking to colleagues about it, a couple of them told me I had no business functioning with that little body fat. But the doctors, nurses and nutritionists said nothing of it. I asked them about it, as I'd heard that having as little as I had could be bad for you, and they told me that as long as I felt good, and based on my test results, I had nothing to worry about. They did say that I was on the low end, and that I probably shouldn't make it a goal to go any lower. Which is fair, I don't really care about body fat percentage. A couple of my colleagues, on the other hand, who are all about beach bodies, were mad jealous of the fact that I, who couldn't care less, had less than half of their body fat.
So without being an expert, it seems to be highly individual.