Here's a twist for you.
This site gives a breakdown of the officials and their specific play-by-play responsibilities. It turns out that, despite the common usage of "referee" applying to all seven officials, "referee" is a specific position. The other six are line judge, back judge, side judge, field judge, down judge, and umpire.
Aren't sports terms fun?
A companion site listing the history of NFL officials' uniforms calls it a "shirt" and not a "jersey."
I don't know if all this throws a monkey wrench in your story or not. I can tell you, though, that only the most diehard NFL fans (and, of course, the officials themselves) care about the names of the officials' positions. They're all "zebras" to the players.