I feel like a dope--all these years of looking at the White House, and I never realized: The Oval Office isn't even in the main building (a/k/a The Residence). I always assumed that the Oval Office's shape corresponded with the curved facade of the South Portico:
But while there are indeed oval-shaped rooms behind that curved façade...
...none of those rooms are the Oval Office. That latter room is off in a separate-but-connected structure, the West Wing, which I'd probably know if I ever bothered to watch the show.
In any case, 3D renderer Jared Owen has put together this cool video with exploded views of the White House, showing you how all of the spaces of the three-building-compound are utilized: