Who swears in a new chief justice?

I believe it is the Associate Justice with the most seniority (in this case, it's JP Stevens), but I'm having trouble finding a confirmation of that.

I believe that is correct, although that wouldn't be a requirement. A government official who is required to take an oath of office can take it from anyone who is authorized to administer oaths. For instance, in 1963, Lyndon Johnson's oath was administered by Judge Sarah T. Hughes. Calvin Coolidge, after Warren Harding's death, took the presidential oath from his own father, who was a notary public. For a complete list of presidential oaths of office, click here