It's a little more complicated than just saying you have a limit of, say, 100 queries. You have two governor limits. The moment a unit test starts, you have your testing governor limits. They are always the same as synchronous governor limits (e.g. 10,000 ms CPU, 6,000,000 bytes heap, 100 queries, etc).
When you call Test.startTest, those limits are suspended, and the very next statement you execute after that will set new governor limits. This may be synchronous limits (e.g. you call a controller), or it might be asynchronous limits (e.g. because you called System.enqueueJob). This is further compounded by the fact that you may have installed packages, which each also have their own limits (except for some shared limits, like CPU).
Once you call Test.stopTest, the limits that were used after Test.startTest are discarded, and the suspended limits become active once again. If you used 20 queries before Test.startTest, then you now have 80 queries remaining.
There's one wrinkle to this, though. If you use @testSetup, then those limits are used as part of the testing limits. For example, if you use 5 queries in a @testSetup method, then each unit test will be limited to only 95 additional queries. In order to avoid this behavior, it is imperative that you use Test.startTest at the beginning of your @testSetup method.
Each unit test is a separate transaction, and does get its own set of governor limits. This lets you have, for example, 100 unit tests in a class that each perform 100 queries (10,000 total queries!).
You'll want to do some profiling so you can optimize your organization's transactions, as failures in unit tests are often canaries in a coalmine. You should not assume that it's just the test is poorly written.