Do I have to create a scratch org to be able to use 2GP?
Yes, 2GP packages are completely developed in scratch orgs.
Or can I create my changes in the SF developer org, test, and push to production org, pull down prod changes to VSCode and run the sfdx cli commands to push to a beta package?
2GPs do not use a "production org". A developer edition org reserves the namespace for the 2GP but is never used otherwise and contains no metadata. If you are developing a package in a developer edition org with a namespace, you may be in fact working with a 1GP, which uses a packaging org.
If I can do it this way, how do I connect to the beta package if I don't have a partner community login? I assume I would have to connect the same way I would have to connect to a GitHub repo and I would be able to view it in some platform like GitHub has.
The partner community hosts the Publishing Console for the AppExchange. That is a step you would take after you create a new released version of your managed package and wish to add it to your AppExchange listing or submit it for Security Review. You don't need to connect to the Partner Community or the Publishing Console during the development process.
Does force:package:version:promote take your package from beta to production? I have to assume that "released status" means released to the public but I don't like to assume.
Yes, but moving to released status is about the package version itself, and does not change your AppExchange listing or perform an upgrade of any customers. You move to released status when your development reaches a conclusion on a new version; this allows the package version to be installed in a production org and makes it possible for you to submit for Security Review and/or update your AppExchange listing.
If I can't just work out of the developer org and VSCode - how do I get my code changes from a GitHub repo to the beta app and then beta to production?
You're thinking about this as if "beta" and "production" are persistent environments you deploy to. That's not how packaging works. If you're working in source control (GitHub) with a second-generation package, you use the sfdx force:package:version:create command to create a new beta package from your repository, then promote that version when you're ready to release it with sfdx force:package:version:promote.
When you're working in a scratch org, you use the sfdx force:source:push and sfdx force:source:pull commands to move metadata between source control and the scratch org.
Clarification on terminology:
- A Developer Edition is a type of org, which is often used as the packaging org for 1GP managed packages.
- The Dev Hub is an org that manages your scratch orgs. The Dev Hub and the packaging org (for 1GP) or namespace org (for 2GP) are not the same org.
- A package is not an org. A package is an artifact that can be installed in an org.
- 1GP managed packages are created from the metadata in a packaging org.
- 2GP managed packages are created from metadata stored in source control (not an org).
- Packages can be in "beta" or "released" state. When a package is in released state, it can be installed in production orgs and published to the AppExchange. Beta packages are for testing.
- "Promoting" a 2GP package version changes the state of that version from beta to released. It doesn't publish it to the AppExchange or upgrade and customer orgs.
- Creating a release package version is a prerequisite to listing on the AppExchange and/or submitting for Security Review, but that is a separate operation.
