The Peter van de Kamp Observatory is on the roof of the Science Center on the Swarthmore campus.
Note the public parking lot (#17 on the map) next to the Science Center, off of Whittier Place. It is easily accessed via the north entrance to campus, off of Elm Ave. You can supplement this map with the annotated map on the department site.
Go up the stairs or elevator to the third floor. Turn left and you will be facing an unmarked door at the end of the hall. Go through that doorway and out onto the roof. Walk to your right, and you will see the door to the observatory dome around the corner.