There needs to be a relationship between the two objects if you want to create a formula field to grab data from a "parent" record. You don't necessarily need to create that relationship (i.e. A relationship between Opportunity and Account already exists as the standard AccountId field on Opportunity), but there does need to be a relationship.
Consider using Apex instead of formula.
Here's why as of January 2021 (SFDC API Version 50.0) . (@ for edits):
Be skeptical of using formula fields for datetime difference calculations
because of a limitation with formula fields, quoted below:
Important: Be aware of timezone conversion issues when using formula
fields to calculate datetime differences: