Skip to main content
New
Stack Overflow Jobs powered by Indeed: A job site that puts thousands of tech jobs at your fingertips (U.S. only). Search jobs

Questions tagged [largedatavolumes]

Questions dealing with handling very large data volumes (greater than 2 Million records in an object) present in large enterprise organizations.

Filter by
Sorted by
Tagged with
0 votes
0 answers
88 views

Salesforce API to retrieve files with size 2GB

So, we upload files approximately with size of 2GB in salesforce cases in lightning. What would be the best way to receive it? (Download through api). Currently, I use /ContentVersion endpoint to get ...
anyman's user avatar
  • 1
1 vote
0 answers
36 views

Why does loading LDV in this sequence reduce computation?

I'm reading the documentation on loading LDV and to reduce computation the docs recommend this sequence of tasks below. I don't understand why #3 is called out and why it should be done after step 2. ...
PartOfTheOhana's user avatar
1 vote
0 answers
66 views

Can you index the isDeleted field? If so, how do you calculate the selectivity threshold

Can you index the standard isDeleted field? If so, in terms of calculating the selectivity threshold, would it be considered a unary standard or unary custom index? I'm working with large data volumes ...
PartOfTheOhana's user avatar
1 vote
0 answers
39 views

Are RecordTypes Indexed on Custom Objects?

Is RecordtypeId indexed on custom objects? According to two places in the documentation it says just standard objects, but that seems like a mistake. https://developer.salesforce.com/docs/atlas.en-us....
PartOfTheOhana's user avatar
2 votes
0 answers
256 views

How does ownership skew impact queues?

I have queues that are assigned more than 10k records, and their records are included in reports and list views. Most of the documentation about ownership skew is around a User owning records. How ...
PartOfTheOhana's user avatar
1 vote
1 answer
419 views

How indexes work when the WHERE clause contains AND

When we use AND condition in our SOQL query and lets assume both the fields are indexed and both are standard index. eg) Select id from child__c where RecordType.Name = 'a' and parent__c ='...
R Ramki's user avatar
  • 11
0 votes
1 answer
747 views

How to list all record ids if "SELECT Id FROM ObjectName" returns fails with QUERY_TIMEOUT?

I need literally all the records. Documentation says that if I hit 120 sec. limit then I should refactor my query to retrieve less amount of data. The only option not to skip data but collect it part ...
Alexander's user avatar
0 votes
1 answer
560 views

Custom Indexes of Null Records

I'm reading up on Custom Indexes and the documentation says: By default, the index tables do not include records that are null (records with empty values). You can work with Salesforce Customer ...
Jesse Twum-Boafo's user avatar
1 vote
1 answer
348 views

Owner Data Skew Prevention for smaller orgs

We have a small org with ~50 users in the B2C space. Their business model includes importing large lists of records (~500 per day) and working on them. We can minimize lookup skew by not using lookup ...
JimG's user avatar
  • 743
0 votes
1 answer
175 views

How to store information on Product stocks with variable number of warehouses

In my recent Salesforce project, we came across tough problem storing product stocks coming from integration with ERP API. Initially there were only 10 warehouses and thus there was 10 fields on ...
Michał Piątek's user avatar
3 votes
2 answers
1k views

Difference between Salesforce Connect and Heroku Connect

Can someone explain the difference between Salesforce Connect and Heroku Connect. In the below image why they have to use both Salesforce connect and Heroku Connect. Cant they just use Heroku connect ...
Shivananda Yalpi's user avatar
10 votes
0 answers
538 views

Move Large Data Volume objects to Big Objects

I have built an Open-Source tool that collects App Errors in Managed Packages and creates Custom Object records for each of them in a Salesforce org. This app can produce thousands of records per day. ...
Robert Sösemann's user avatar
2 votes
0 answers
76 views

Options to store Large Configuration Data in an AppExchange Managed Package

We have an AppExchange App which needs to connect with an external application via API Calls. API calls need to happen in individual user context and not in a single Integration User context. This ...
Ranjit P's user avatar
1 vote
1 answer
811 views

Dynamic Queries on Dynamic Objects at Volume

We are trying to create a mechanism that would process user dynamically created queries and create "Contributors" for each matching record. The object model is as follows: Account -< PO ...
James Ridge's user avatar
4 votes
1 answer
155 views

Alternate of Protected Custom Settings In an AppExchange Managed Package for Large Configuration Data

We have an AppExchange App which needs to connect with an external application via API Calls. API calls need to happen in individual user context and not in a single Integration User context. This ...
Ashish's user avatar
  • 2,370
1 vote
1 answer
570 views

Working with large data volume

I have a design question and I want an expert opinion from the group. Here is the use case: Whenever there is any update on the Booking record (let's say change the status to "Cancelled"), ...
Noob_NoVoice's user avatar
  • 2,709
1 vote
1 answer
430 views

Case Object with 2 Self Lookups to Cases - Considerations

We have functionality around Moves(Shipping request from one location to another) , claims and service requests for one of our clients. we are planning to put all 3 of them in Salesforce Case object ...
RedDevil's user avatar
  • 7,821
0 votes
1 answer
859 views

Aggregate Query on 900K records - how to do this?

Hello fellow developers, I need to run an aggregate query on 900K telephone records to add up a field (grouping by phone line id) and record results in the "summaries logbook". I have tried ...
wmin's user avatar
  • 69
3 votes
1 answer
1k views

System.LimitException: Too many query rows: 50001 - How to solve it with a batch?

I need to create a summaries record for each result generated by the following query, which returns around 12K. The trouble is the aggregate result has to query 600K records, so I get 'System....
wmin's user avatar
  • 69
1 vote
1 answer
309 views

Treatment of large data sets (>1M) in standard objects

Question: At what point Salesforce tags a standard object as big object? We are planning to load about 10+ million (one time) records in Asset and Order object and there would be incremental records ...
Bugude's user avatar
  • 390
0 votes
1 answer
281 views

Exporting of and capturing changes on a large set of records

We have to load the entire Salesforce Lead data(this data might be millions records)to external system using Bulk API. Once all the data loaded and we thought to initiated to catch the added/modified/...
Madhu Nayaki's user avatar
1 vote
1 answer
277 views

Testing bulk clone operation

After doing enough Google search and scratching my brain, I have decided to post here and seek for suggestions. Here is the background: An opportunity has multiple order & order line item ...
goabhigo's user avatar
  • 344
0 votes
2 answers
1k views

Data skew exists whereby the number of matching rows is very large

I am getting the following Error. works fine in DEV org but throwing error in UAT. Please suggest me as I was unable to move forward. The Actual Error is The first validation error encountered was ...
Muzeeb's user avatar
  • 3
0 votes
1 answer
643 views

Question on Negative operator SOQL Index or Table scan

I was reading SFDC documentation and they state that using != or Contains will almost run the query in table scan. With that said, we have 5 Million case records in production. When i check the ...
Darkdog's user avatar
  • 95
4 votes
0 answers
54 views

Should I be concerned about thousands of write operations per minute?

I am setting up a POC which would create a junction between User and Case (using Lookup relationships only). We would have 10k+ users each touching several records per hour, so we might expect this ...
Adrian Larson's user avatar
  • 150k
7 votes
1 answer
8k views

How to get MAX value using SOQL when there are more than 50,000 rows

I would like to run this query: SELECT MAX(Auto_Number__c) MaxNumber FROM Opportunity But I am getting this error: System.LimitException: Too many query rows: 50001 Because there are more than a ...
Robs's user avatar
  • 9,356
1 vote
1 answer
104 views

Real time synch with external application on insert and update of frequently changing records

In order to synch very large amount of frequently changing data to an external system, what will be best way to do it ? We can't use trigger or schedule batches as these will fire so many times that ...
goose's user avatar
  • 81
0 votes
1 answer
178 views

LDV : What is max records that aggregate query can work with?

I am having difficulty understanding limits on aggregate queries. Salesforce Documentation for ReadOnly Attribute of Visualforce Page says Normally, queries for a single Visualforce page request ...
ClassCastException's user avatar
0 votes
0 answers
2k views

Retrieve large data sets from salesforce using Rest API or Bulk API

We have external system which retrieves the data from Salesforce by using Rest Api. It was working fine till the return resultSet is limited. However, when we are working on large data set it works as ...
Abhishek Daware's user avatar
1 vote
1 answer
185 views

Query not selective error is happening only in Trigger context but not in VF,Batch. Is this expected?

I have a query with query plan as below: The lowest cost is greater than 1, It means query is not selective. Trigger context: I used this query in a before insert trigger on a empty object and it ...
javanoob's user avatar
  • 8,887
4 votes
2 answers
909 views

Bulk deletion of Case Team Member records, data skew, and performance issues

We're proposing to delete no-longer-relevant Case Team Member records, numbering about 4.3 million (2.8 million total Cases). I'm concerned about the performance implications that might be caused by ...
David Reed's user avatar
  • 93.1k
1 vote
0 answers
572 views

How do you avoid a timeout on custom objects with a lot of data?

We have a batch performing a query on a custom object that returns a timeout. The table contains 2.4 million records and the query should return 125 000 records. That's 5% of total records. Our ...
stvnhg's user avatar
  • 337
3 votes
1 answer
245 views

500K record transaction: Trigger vs Scheduled Apex?

I'm working on an instance where records are sent from Eloqua over to Salesforce in a single transaction of 300,000 to 500,000 records. I didn't find out until NOW. Based on requirements, I've ...
RC82's user avatar
  • 595
2 votes
0 answers
311 views

How to migrate large data volume from On-premise to Heroku Postgres [closed]

We are planning to migrate large volume of data from on-premise Oracle DB to Heroku Postgres in Private Space. We are using Heroku to build our API Services for various channels e.g. portal / mobile ...
Kaon Z's user avatar
  • 326
5 votes
0 answers
525 views

Working with Very Large SOQL Queries

Background This is a question about your experiences managing huge amount of data for UIs. With an ERD like this, where the arrow directions reflect the fields lookup directions. (e.g. entity "B" ...
Lucas Buyo's user avatar
2 votes
1 answer
60 views

How to Filter New Accounts - Large Data Volumes?

My business partners are adding in 1000000 new records into Salesforce and they want to document at a location level as well as keep the current data documentation at an org level untouched with no ...
KLM's user avatar
  • 33
1 vote
1 answer
197 views

Reading large number of records in REST service

This is a apex class for a REST service fro inserting multiple records into a custom salesforce object- I am getting a System.LimitException: Too many query rows: 50001 error because my org data is ...
goSF's user avatar
  • 385
6 votes
1 answer
2k views

Bulk data API: How to Avoid Upserts

I'm working on an daily integration of about 5 million records (file delta's is not an option). Best practices for large data volumes recommends avoiding upserts. The only alternative solution I can ...
NSjonas's user avatar
  • 10.2k
1 vote
1 answer
69 views

What is the best practice method to summarize contact information to leads?

Here's the business case. I want to have a reportable field on a lead that displays the number of contacts for, say, a city field match. This value could be used in reports, dashboards, snapshots, ...
Chris's user avatar
  • 63
0 votes
1 answer
19 views

High volume imports

I would like to know best practices when dealing with nightly loads of high volume of information, let's say, importing around 2.500.000 regs. per day into CRM. thanks, Daniel
Daniel's user avatar
  • 1
1 vote
1 answer
403 views

Rest API PATCH method with Large Data Volumes

Are there any best practices relating to the use of the Rest API within an environment with large data volumes? I am aware of the SOQL best practices relating to LDV. Does this relate at all to ...
dacology's user avatar
  • 2,992
0 votes
1 answer
75 views

Multiple File Data Set Import Cleaning

It is pre-Salesforce import, I have designed the Named Account structure inside Salesforce, and I have 1500 Hoover's D&B Family Trees with the same columns and headers in .xls form. What do I use ...
MeighanRocksSF's user avatar
0 votes
0 answers
129 views

Is there a way to tell if custom indexes exist on an object?

I want to be able to design to use indexes and custom indexes where they exist. Is there a way to tell if custom indexes have previously been created on a field?
dacology's user avatar
  • 2,992
2 votes
1 answer
644 views

Best approach for creating Junction object with millions of records

I have two objects Object1 and Object2 in Salesforce. These two objects doesn't have Master-Detail relationship. But they are related through a Text column X. I want to create many to many junction ...
karthikselva's user avatar
  • 2,476
1 vote
1 answer
2k views

Querying more than 15000 records into a VF page using RemoteAction

I have a page that uses AngularJS for front-end rendering and filtering. To make use of the dynamic filtering capability of AngularJS filters, I am loading all the records onto the HTML page, so the ...
prashanthkr's user avatar
13 votes
2 answers
2k views

Can SOSL perform full text search on chatter files?

The Document sobject has an IsBodySearchable flag which controls if you can full text search it via SOSL. ContentVersion has no such flag but I know that it is searchable in certain cases. However, ...
Greg Grinberg's user avatar
3 votes
1 answer
4k views

query against large object type

I have my own custom code(before insert trigger) for check on Dead Leads. for storing dead Leads there is a custom object. Currently the custom object has more than 200000 records. Due to large ...
Bharadwaj Urma's user avatar
6 votes
1 answer
2k views

Querying Large volumes of data in Apex

I am looking for a solution where i have a scenario to query large volume data and do an update on them. Is there a way i can do, query once and then querymore on that object instead of getting all ...
nvr's user avatar
  • 423
3 votes
2 answers
3k views

Batch Apex throwing error when executing query ( SQLException [common.exception.SfdcSqlException: ORA-01013:)

I am trying to execute a Batch job that is processing around 2 million contacts, I batch Apex whenever I try to execute is throwing an error First error: SQLException [common.exception....
logontokartik's user avatar
4 votes
1 answer
4k views

Reports and formula fields

I read this very helpful article, Designing Dashboards and Reports for Force.com Implementations with Large Data Volumes I have one question regarding formula fields - if I use formula fields instead ...
Dedo's user avatar
  • 5,007