Add remaining 2 Tbsp. oil to skillet and heat over medium-high. Arrange mushrooms in pan in a single layer and cook, undisturbed, until golden brown underneath, about 3 minutes. Season with salt and pepper, toss, and continue to cook, tossing often, until golden brown all over, about 5 minutes. Transfer to bowl with dressing, but don’t toss. Strip rosemary leaves off stems into bowl and add radicchio.