Take a look at the Salesforce Bulk API described here.
When to Use REST API
REST API provides a powerful, convenient, and simple REST-based Web services interface for interacting with Salesforce.
Its advantages include ease of integration and development, and it’s an excellent choice of technology for use with mobile
applications and Web projects. However, if you have a large number of records to process, you may wish to useBulk APIBulk
API, which is based on REST principles and optimized for large sets of data.
When to Use Bulk API
Bulk API is based on REST principles and is optimized for loading or deleting large sets of data. You can use it to query, insert,
update, upsert, or delete a large number of records asynchronously by submitting batches which are processed in the background
How Bulk API Works
You process a set of records by creating a job that contains one or more batches. The job specifies which object is being processed
and what type of action is being used (query, insert, upsert, update, or delete). A batch is a set of records sent to the server in
an HTTP POST request. Each batch is processed independently by the server, not necessarily in the order it is received. Batches
may be processed in parallel. It's up to the client to decide how to divide the entire data set into a suitable number of batches.
A job is represented by the JobInfo resource. This resource is used to create a new job, get status for an existing job, and change
status for a job. A batch is created by submitting a CSV or XML representation of a set of records and any references to binary
attachments in an HTTP POST request. Once created, the status of a batch is represented by a BatchInfo resource. When a
batch is complete, the result for each record is available in a result set resource.
Processing data typically consists of the following steps:
- Create a new job that specifies the object and action.
- Send data to the server in a number of batches.
- Once all data has been submitted, close the job. Once closed, no more batches can be sent as part of the job.
- Check status of all batches at a reasonable interval. Each status check returns the state of each batch.
- When all batches have either completed or failed, retrieve the result for each batch.
- Match the result sets with the original data set to determine which records failed and succeeded, and take appropriate action.