I guess I'm not the best person to comment on the reason for Oblivion's popularity. Even with the game modded to the gills, I wasn't able to maintain an interest in it for more than a few hours. However, judging by what Elder Scrolls fans talk about, I think the main reason for the popularity of TES is the sandbox aspect. No other RPG series does the "go anywhere and be anybody" thing better than Elder Scrolls.
When I browse the official Bethesda forums, I see that free-form open-world exploration is really stressed at the thing players love most, with "immersion" being another big selling point. They want to be able to go anywhere, pick up any object, and kill everything that moves. In some ways, Bethesda's games are more comparable to Rockstar's than other RPG developers. They've got open worlds, lots of minigames, and the constant temptation to cause complete mayhem.
The one Elder Scrolls game I liked was Morrowind, which was actually my first PC RPG. Ironically, it was the story of Morrowind which most drew me in (it is ironic because the series isn't known for having strong stories). I saw in Morrowind the potential for an entirely new kind of narrative structure, with in-game books describing multiple versions of events that could be discovered at the viewer's leisure. Bethesda took the "unreliable narrator" device and applied it to an open-world sandbox game. I thought, and still think, it was brilliant. And I found that brilliance to be completely lacking in Oblivion (mind you, I've never played Shivering Isles).