That sounds like your ISB. If it were your throw out or clutch release bearing it would make noise with the clutch pushed in. I just replaced one on my spare B000 S40 and it wasn't too bad. My A000 S40 currently in the car is noisy,it'll make a slight squeal when taking off from a stop and rattles/rumbles with the clutch out. Parts were less than $50 for the ISB and seal. Not sure how long you can drive like that,maybe a day or maybe a year. When the bearing finally goes the plastic cage around the balls (not sure if 2001+ uses this design) will get everywhere in your trans and booger up the shaft. Earlier D series trans are notorious for this.