> what irritates me is that granting access to my location allows the app to continually track me while it's running in the background

There's an app called Bouncer [1] that can auto-remove permissions as soon as an app is closed.

[1] https://play.google.com/store/apps/details?id=com.samruston....