As @sfdcfox says in his comment, I needed to first turn on the replay debugger and then run either Debug All Tests or Debug Test. (To turn on the replay debugger in vscode, command+shif+p/control+shift+p and type "turn on replay" and you'll see that option appearing).
Now if you click on this feature, (which shows up right next to the Run Test at the top of each unit test in your test class), it'll run the test(s) and starts a debugging session (with the Apex Replay Debugger). If you already have set any breakpoints it will jump-start into the first one.
This is much more convenient than having to pull down the log file and execute the debug command on the file, which is the method I've been using up to this point.
I hope this helps! Thanks to @sfdcfox for pretty much giving me this answer in his helpful comment.