Bulk data load make the global search unusable

Question

We are migrating data from saleforce org A to org B. While migrating we are processing data out of salesforce and its out of scope of this question. When we load data into target org it breaks the indexes and make global search unusable. I understand that the reason behind this. The problem is CSR are significantly dependent on global search. When we load millions of rows the re-indexing can takes week to catch up with current state. I know that there are workaround like create view or VF page that let the CSR search with specific criteria but still it will change their flow and will need training. The worst could be they take longer time and break the SLA.

So my questions are:

1. Is there any way to prioritize the global search re-indexes and avoid the delay of weeks? SFDC support said No.
2. Does any one have beach marking on X number of records would take Y number of hours for indexing? Any documentation with this information will help.
3. If we load the data in batches (for ex. 100k records every night) would it less or more time for indexing? What i mean here is,does time for indexing increase in relational with existing records in table? For ex:

Batch 1 takes 10 minutes for re-indexing and total records in objects are 10000
Batch 2 takes 12 minutes for re-indexing and total records in objects are 20000
Batch 3 takes 15 minutes for re-indexing and total records in objects are 30000
Batch 4 takes 18 minutes for re-indexing and total records in objects are 40000
Batch 5 takes 21 minutes for re-indexing and total records in objects are 50000

In other words, Will the total records count increase will the indexing time increase?

Update 1: In below answer CoryCowgill is saying load below 9k rows should be processed in less than 2-3 minutes. But did not find authentic source of this info in documentation.

Bulk data load make the global search unusable

Update 2 One possible way to solve this problem is trickle data load. Here is what I am planning to do:

1. Using ETL tool (like Talend or Cast Iron) create data load job
2. In this job insert data in small burst for ex. 1k records in batch
3. Then check for some random samples using SOQL to know if its indexed
4. After certain time interval (for ex. 3 minutes ) re start job with next small burst / batch

I am still not sure whether it will work or not.

Answer 1

There is the article Salesforce Bulk Data Loads and Full-Text Search Indexes by Salesforce engineering.

Strategies for Addressing Lags in Search Indexing

So how can you architect acceptable solutions that address inevitable lags in search indexing after data loads? Here are a couple of things to consider.

• Disable full-text search indexing for custom objects (especially large ones) that don’t need to be searchable. This best practice helps to avoid unnecessary load on search indexing. Disabling this feature only affects full-text searches, SOSL queries, and enhanced lookups–it does not affect SOQL queries.
• Instead of relying on the full-text search engine and SOSL, implement your application’s search feature using SOQL. Because SOQL queries target the transactional database, they’ll always return results that correspond to the latest set of committed records.

Turning off the indexing where you don't need it would be one way to speed up the process.

---

Also from the Best Practices for Deployments with Large Data Volumes:

For data to be searched, it must first be indexed. The indexes are created using the search indexing servers, which also generate and asynchronously process queue entries of newly created or modified data. After a searchable object’s record is created or updated, it could take about 15 minutes or more for the updated text to become searchable

There are likely factors outside your control that are going to affect performance here which is why no concrete metrics are published. It would largely depend on what other orgs are doing on the instance at that time and what sort of throttling Salesforce are using to prevent one org monopolizing the search indexing.

---

From Multitenant Search Processing

As applications update data in text fields (CLOBs, Name, etc.), a pool of background processes called indexing servers are responsible for asynchronously updating corresponding indexes, which the search engine maintains outside the core transaction engine. To optimize the indexing process, Force.com synchronously copies modified chunks of text data to an internal “to-be-indexed” table as transactions commit, thus providing a relatively small data source that minimizes the amount of data that indexing servers must read from disk. The search engine automatically maintains separate indexes for each organization (tenant).

Depending on the current load and utilization of indexing servers, text index updates may lag behind actual transactions.

---

A number of these resources are authored by Steve Bobrowski from Salesforce. Maybe reaching out to him on twitter will yield a more official response.

Answer 2

Found Solution: The approach is load data as slowly as possible. If you load relatively less records (say 5k or less) the global search is available in the matter of seconds (for me, it was less than 30 seconds). So, instead of loading data in bulk we choose to load data in “Trickle Load” fashion. In a way it’s exactly opposite of bulk data loading. In Bulk data loading we upload all data and periodically keep checking the status. In Trickle load we load small chunk and wait for salesforce to finish updating global indexes. After some time, we submit again another chunk and this can go on and on till we finish. If you think of this in a way, it’s just tread of between limits.

How to Trickle Load huge data?

1. Split large CSV file of data into batches of 5k (php code is provided at the bottom)
2. Trigger job of data loading at every 5 minutes to insert one batch file
3. The easiest way for me was using data loader CLI to run from command prompt

Things to keep in mind:

• With this approach we can load max 1,440,000 records per 24 hours (5k * (24* 12(batches per hour)))

• As the wait time increase you will notice less problem of jamming the global search indexes. As the wait time reduced you will notice more the problem of jamming the global search indexes

• Higher API consumption can be result with this approach.
• We consider to wait for 5 minutes but you can choose another number as per your ability to wait after jamming problem.

More details on blog: http://atulrsfdc.blogspot.co.nz/

<?php

$size = 5000;
$cnt = 0;
$num = 1;
$base = 'C:\Data\Load\\';
$file = 'Contact';
$output = $base . 'batches\\';

$handle = fopen( $base . $file .'.csv', 'r');
if ($handle) {
    $header = fgetcsv($handle, 0,",",'"') ;
    $fp = fopen( $output. $file. '_batch_'. $num .'.csv', 'w'); 
    fputcsv ($fp, $header, ",", '"');
    while (($line = fgetcsv($handle, 0,",",'"')) !== false) {
        $cnt++;
        fputcsv ($fp, $line, ",", '"');
        if(($cnt % $size) === 0){
            fclose ($fp);
            $num++;
            $fp = fopen( $output. $file. '_batch_'. $num .'.csv', 'w'); 
            fputcsv ($fp, $header,",", '"');
        }       
        //if($num > 5) break;   
    }
    fclose ($fp);
    fclose($handle);
} else {
    echo "\n\nThere was error \n\n";
} 
?>