If you want to make use of metadata components (including custom fields, for example) added by third party AppExchange packages, the only way to achieve this is to install those packages because they will have a namespace in which all their metadata exists. You cannot reproduce these in your own source base.
Dependencies on third-party packages are captured in the sfdx-project.json. Unfortunately, scratch org creation does not automatically ensure dependencies get installed (a significant oversight in SFDX IMHO) so you will have to write your own scripting to deal with this. Once done, using SFDX for your development works quite nicely, though (depending on the size of the packages on which you depend) scratch org creation can be pretty slow.
Personally I would say you should always view your version control system as the source of truth, never an org. I would also say that, due to undesirable behaviour in sfdx around pulling metadata from a scratch org (it will see some profiles as changed by Salesforce and insist on pulling spurious updates to these - at least, that is what I have found), you should consider explicitly retrieving rather than using pull if you have to create or modify metadata on the org rather than locally in your IDE.