If someone stole or by some other means gained access to the phone and knows password to unlock it, [s]he can gain access to all servers even not knowing them beforehand, because [s]he can view list of known hosts with their IPs, pair with any new device, approve auth.
Password used to unlock a phone is usually shorter than password used to encrypt an ssh key, also it is
used more frequently.

Suggestions:

1. require user to generate a separate password (not device password) for the mobile app. Add settings to ask password each time/once in a period of time when app is opened or auth is requested
2. always ask this password when pairing with a new device or opening "known hosts"
3. maybe send email alert when pairing with a new device