Custom fields that don't have 'required: True' are not appearing in describe call

Question

I create a Custom Object named 'SomeTemplate__c' with the Metadata API, and I have fields:

'created_at__c'
'name__c'
'template_id__c'

which are created with required: True set in the parameters. Those fields appear in the describe response's 'fields' field. These fields:

'updated_at__c'
'is_active__c'
'is_weighted__c'

do not have required set in the parameters at all. Though these fields appear in the Object Manager UI for 'SomeTemplate__c''s "Fields & Relationships" (so I know the Metadata API call succeeded for them), they do NOT appear in the desrcibe response's 'fields' field, like the required fields do.

Not only do they not show in that describe call, I also can't set those fields when inserting objects via the Bulk API. The insert call errors and tells me

Field name not found : is_active__c

So it feels like those fields weren't created (except that I can see them in the Salesforce website's Object Manager).

What is this weird behavior with required fields vs. non-required fields?

QUICK UPDATE:
I can update the visibility using the Salesforce website (Object Manager -> Field Accessibility) by checking the box for "Visible", under both "Field-Level Security" and "Page Layout". After doing that I can now see 'is_active__c' in the describe call. But using the website is not a valid solution, since I need to do this via the API for many accounts, which I don't have dashboard access to since my API access for them is only via oAuth.

NOTE:
I can't simply make those fields required because 'updated_at__c' is not always going to exist, and the others are of type 'Checkbox' which, according to a different insert error message, cannot be required.

Answer 1

Your profile doesn't have at least "read" access to the field(s) in question. When you deploy via the Metadata API, the default access is "hidden" for non-required fields, and "editable" for required fields. You would need to either bundle in the appropriate permissions for the fields in the various Profile objects, or modify them after deployment in the UI (e.g. Setup > Create > Objects > Object Name > Field Name > Set Field-Level Security). You'll want to look at the Profile object for more details.

In your deployment, you'll want to have a "profiles" folder, with files for each profile, such as Admin.profile, with the appropriate permission set:

<fieldLevelSecurities>
  <editable>true</editable>
  <readable>true</readable>
  <field>Object__c.Field__c</field>
</fieldLevelSecurities>

You'll probably want to retrieve a profile with a field to see how it's designed. Try a package.xml that looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<Package xmlns="http://soap.sforce.com/2006/04/metadata">
    <types>
        <members>Object__c.Field__c</members>
        <name>CustomField</name>
    </types>
    <types>
        <members>Admin</members>
        <name>Profile</name>
    </types>
    <version>40.0</version>
</Package>

Answer 2

If you are creating CustomObject and CustomField metadata objects via some API call in whatever language or tool, when you go to insert objects of that custom type you made, you won't be able to set any fields that were not required: true, because non-required fields are created with visibility as false.

(You can see this in your Salesforce account's Object Manager => your object => Fields and Relationships => your field => Set Field Level Security, and see the unchecked boxes.)

An easy fix is to modify your existing Profile metadata object. For me, that meant setting the Profile whose fullName was Admin, to have these fieldPermissions:

'fieldPermissions': {
    'editable': True,
    'field': 'SomeCustomObject__c.is_active__c'
}

I'm not sure if all API calls are done as Admin, but for me that was the relevent Profile to update.

As for HOW to do this update, however you're making your createMetadata call for CustomObjects and CustomFields, instead do an updateMetadata call, passing in 'fullName': 'Admin', or whichever Profile you're updating.

So for me, in Python, I did:

sapi.request('updateMetadata', {
    'metadata': sapi._get_api_obj('Profile', {
        'fullName': 'Admin',
        'fieldPermissions': {
            'editable': True,
            'field': 'SomeCustomObject__c.is_active__c'
        }
    })
})

to allow me to then insert SomeCustomObject__c objects including the field is_active__c.

(I'm using zeep, and sapi.request, and sapi._get_api_obj are part of a little wrapper I made for Salesforce's SOAP api.)