I’m sure I can think of vertebrates that have things that make them “them” that don’t also apply to fish. For example do any fish have hips of any type? If not i can use “hip or not” to group vertebrates distinctly.

It’s similar to the process that lets me differentiate vertebrates from invertebrates in the first place. squishy worm, worm with teeth, worm with jaws, worm with backbone etc.