One of the things that confused me on the Apple TV was when the music I was listening to kept playing even after I closed the app and began navigating to other apps. I would then navigate back to the Music app and stop it from playing. As you might expect, there is a simpler way to do this.
To pause music that's playing in the background on your Apple TV even though you aren't currently in the Music app, simply press the Play/Pause button on the Siri remote and the music will stop playing.
If you want your music to resume playing, you'll need to return to the Music app and then press the Play/Pause button again. Note, however, that you don't need to navigate back to the app to do so. Instead, you can simply press and hold the Play/Pause button for at least three seconds and you'll jump back to the Music app. This only works, though, if you're in the Home screen. You can use this long press to get back to the Music app regardless of whether the music is currently playing or paused.
Of course, another way to quickly stop music from playing in the background is to use Multitasking mode to quit the Music app. Similar to iOS, you can invoke the Apple TV multitasking carousel by pressing the Home button twice. Then simply swipe up on the Music app to quit it.