Then Wikipedia misled me. (Not surprising: it is Wikipedia.) I took my info from the article on Brighton proper, not the Brighton-Hove city, but it appears that the population figure there given is actually the one for the city, not the specifically Brighton part. The articles on consolidated Toronto/Toronto proper are more reliable.

"city" vs. "town" terminology makes no difference here. That's an indication of the form of local government, not the size or importance of the municipality.