Questions tagged [largedatavolumes]
Questions dealing with handling very large data volumes (greater than 2 Million records in an object) present in large enterprise organizations.
60
questions
21
votes
2
answers
23k
views
Breaking out of a nested loop
Just for a bit of background I am trying to write a Depth First Sort in Apex. I have been successful writing the method using recursion but when testing with some large data volumes I have hit the ...
- 7,942
13
votes
4
answers
20k
views
How to query more than 50000 records in apex and bind it in vf page in jqgrid and paging?
How to query more than 50000 records (78000 records) in apex and bind it in vf page in jqgrid and implement paging in it?
I tried querying through SOQL and assigning it to list.. Am unable to query ...
- 3,232
13
votes
2
answers
2k
views
Can SOSL perform full text search on chatter files?
The Document sobject has an IsBodySearchable flag which controls if you can full text search it via SOSL.
ContentVersion has no such flag but I know that it is searchable in certain cases. However, ...
- 7,481
11
votes
3
answers
4k
views
Governor Limit question: Large data set. How do I avoid hitting governor limits?
We have a pretty complex trigger suite on our org. We've built a lot of custom functionality. I'm starting to worry about governor limits and not really understanding how to manage them. I've read ...
- 1,697
10
votes
0
answers
584
views
Move Large Data Volume objects to Big Objects
I have built an Open-Source tool that collects App Errors in Managed Packages and creates Custom Object records for each of them in a Salesforce org. This app can produce thousands of records per day. ...
- 38k
7
votes
1
answer
9k
views
How to get MAX value using SOQL when there are more than 50,000 rows
I would like to run this query:
SELECT MAX(Auto_Number__c) MaxNumber FROM Opportunity
But I am getting this error:
System.LimitException: Too many query rows: 50001
Because there are more than a ...
- 9,416
7
votes
3
answers
4k
views
Unable to perform a COUNT() query on object with many million records
I have a custom object with around 25.000.000 records and need to perform a single count query to find those records where a field has a certain value.
I could get none of the following queries to ...
- 38k
6
votes
1
answer
2k
views
Querying Large volumes of data in Apex
I am looking for a solution where i have a scenario to query large volume data and do an update on them. Is there a way i can do, query once and then querymore on that object instead of getting all ...
- 423
6
votes
1
answer
2k
views
Bulk data API: How to Avoid Upserts
I'm working on an daily integration of about 5 million records (file delta's is not an option).
Best practices for large data volumes recommends avoiding upserts.
The only alternative solution I can ...
- 10.2k
5
votes
0
answers
528
views
Working with Very Large SOQL Queries
Background
This is a question about your experiences managing huge amount of data for UIs.
With an ERD like this, where the arrow directions reflect the fields lookup directions. (e.g. entity "B" ...
- 51
4
votes
3
answers
467
views
Moving Salesforce data into Heroku Java app for LDV processing?
I have read a few blog posts about moving parts of an app into a Ruby Heroku app, but no example for moving data between a Salesforce org and a custom Java app on Heroku.
Do you know of any in posts ...
- 38k
4
votes
1
answer
4k
views
Reports and formula fields
I read this very helpful article, Designing Dashboards and Reports for Force.com Implementations with Large Data Volumes
I have one question regarding formula fields - if I use formula fields instead ...
- 5,017
4
votes
1
answer
162
views
Alternate of Protected Custom Settings In an AppExchange Managed Package for Large Configuration Data
We have an AppExchange App which needs to connect with an external application via API Calls. API calls need to happen in individual user context and not in a single Integration User context.
This ...
- 2,372
4
votes
2
answers
960
views
Bulk deletion of Case Team Member records, data skew, and performance issues
We're proposing to delete no-longer-relevant Case Team Member records, numbering about 4.3 million (2.8 million total Cases). I'm concerned about the performance implications that might be caused by ...
- 93.4k
4
votes
0
answers
54
views
Should I be concerned about thousands of write operations per minute?
I am setting up a POC which would create a junction between User and Case (using Lookup relationships only). We would have 10k+ users each touching several records per hour, so we might expect this ...
- 151k
3
votes
1
answer
10k
views
More than 50000 items inside a list in apex
Can we store more than 500000 items into a list in apex? I tried this and is working fine. So the limit of 50000 has been lift up? When this update was made to salesforce? any idea?
- 4,595
3
votes
1
answer
247
views
500K record transaction: Trigger vs Scheduled Apex?
I'm working on an instance where records are sent from Eloqua over to Salesforce in a single transaction of 300,000 to 500,000 records. I didn't find out until NOW.
Based on requirements, I've ...
- 595
3
votes
2
answers
6k
views
"Non-selective query against large object type" only occurs in trigger context despite filtering on lookups
I have a very simple after delete trigger that searches for orphan records and deletes them:
List<my_ns__Orphanable_Object__c> orphans = Database.query(
' SELECT ' +
' Id ' +
' ...
- 7,942
3
votes
1
answer
1k
views
System.LimitException: Too many query rows: 50001 - How to solve it with a batch?
I need to create a summaries record for each result generated by the following query, which returns around 12K.
The trouble is the aggregate result has to query 600K records, so I get 'System....
- 69
3
votes
1
answer
4k
views
query against large object type
I have my own custom code(before insert trigger) for check on Dead Leads. for storing dead Leads there is a custom object. Currently the custom object has more than 200000 records. Due to large ...
- 337
3
votes
2
answers
175
views
Prevent LDV-problems in own ISV package by auto-indexing packaged fields
In our managed ISV package we have a custom object that in many customer orgs might run into LDV issues as the nature of this object lead to million of records.
I know that Salesforce support can
...
- 38k
3
votes
2
answers
1k
views
Difference between Salesforce Connect and Heroku Connect
Can someone explain the difference between Salesforce Connect and Heroku Connect.
In the below image why they have to use both Salesforce connect and Heroku Connect. Cant they just use Heroku connect ...
- 113
3
votes
2
answers
3k
views
Batch Apex throwing error when executing query ( SQLException [common.exception.SfdcSqlException: ORA-01013:)
I am trying to execute a Batch job that is processing around 2 million contacts, I batch Apex whenever I try to execute is throwing an error
First error: SQLException [common.exception....
- 811
2
votes
0
answers
294
views
How does ownership skew impact queues?
I have queues that are assigned more than 10k records, and their records are included in reports and list views.
Most of the documentation about ownership skew is around a User owning records. How ...
- 10.6k
2
votes
0
answers
76
views
Options to store Large Configuration Data in an AppExchange Managed Package
We have an AppExchange App which needs to connect with an external application via API Calls. API calls need to happen in individual user context and not in a single Integration User context.
This ...
- 49
2
votes
0
answers
313
views
How to migrate large data volume from On-premise to Heroku Postgres [closed]
We are planning to migrate large volume of data from on-premise Oracle DB to Heroku Postgres in Private Space. We are using Heroku to build our API Services for various channels e.g. portal / mobile
...
- 326
2
votes
1
answer
62
views
How to Filter New Accounts - Large Data Volumes?
My business partners are adding in 1000000 new records into Salesforce and they want to document at a location level as well
as keep the current data documentation at an org level untouched with no ...
- 33
2
votes
1
answer
679
views
Best approach for creating Junction object with millions of records
I have two objects Object1 and Object2 in Salesforce. These two objects doesn't have Master-Detail relationship. But they are related through a Text column X. I want to create many to many junction ...
- 2,476
1
vote
1
answer
413
views
Owner Data Skew Prevention for smaller orgs
We have a small org with ~50 users in the B2C space. Their business model includes importing large lists of records (~500 per day) and working on them.
We can minimize lookup skew by not using lookup ...
- 763
1
vote
1
answer
324
views
Treatment of large data sets (>1M) in standard objects
Question: At what point Salesforce tags a standard object as big object?
We are planning to load about 10+ million (one time) records in Asset and Order object and there would be incremental records ...
- 390
1
vote
1
answer
186
views
Query not selective error is happening only in Trigger context but not in VF,Batch. Is this expected?
I have a query with query plan as below:
The lowest cost is greater than 1, It means query is not selective.
Trigger context:
I used this query in a before insert trigger on a empty object and it ...
- 8,917
1
vote
1
answer
199
views
Reading large number of records in REST service
This is a apex class for a REST service fro inserting multiple records into a custom salesforce object- I am getting a System.LimitException: Too many query rows: 50001 error because my org data is ...
- 395
1
vote
1
answer
2k
views
Querying more than 15000 records into a VF page using RemoteAction
I have a page that uses AngularJS for front-end rendering and filtering. To make use of the dynamic filtering capability of AngularJS filters, I am loading all the records onto the HTML page, so the ...
- 906
1
vote
1
answer
830
views
Dynamic Queries on Dynamic Objects at Volume
We are trying to create a mechanism that would process user dynamically created queries and create "Contributors" for each matching record.
The object model is as follows:
Account -< PO ...
- 85
1
vote
1
answer
689
views
Working with large data volume
I have a design question and I want an expert opinion from the group.
Here is the use case:
Whenever there is any update on the Booking record (let's say change the status to "Cancelled"), ...
- 2,709
1
vote
1
answer
475
views
Case Object with 2 Self Lookups to Cases - Considerations
We have functionality around Moves(Shipping request from one location to another) , claims and service requests for one of our clients.
we are planning to put all 3 of them in Salesforce Case object ...
- 7,851
1
vote
1
answer
116
views
Real time synch with external application on insert and update of frequently changing records
In order to synch very large amount of frequently changing data to an external system, what will be best way to do it ?
We can't use trigger or schedule batches as these will fire so many times that ...
- 81
1
vote
1
answer
478
views
How indexes work when the WHERE clause contains AND
When we use AND condition in our SOQL query and lets assume both the fields are indexed and both are standard index.
eg) Select id from child__c where RecordType.Name = 'a' and parent__c ='...
- 11
1
vote
1
answer
281
views
Testing bulk clone operation
After doing enough Google search and scratching my brain, I have decided to post here and seek for suggestions.
Here is the background:
An opportunity has multiple order & order line item ...
- 344
1
vote
1
answer
413
views
Rest API PATCH method with Large Data Volumes
Are there any best practices relating to the use of the Rest API within an environment with large data volumes?
I am aware of the SOQL best practices relating to LDV. Does this relate at all to ...
- 2,992
1
vote
1
answer
512
views
Heapsize problem on VF page
I am VF page which display records for a custom object. When there are a large number of custom records - I get: HEAPSIZE 6014176 error on page.
Any tips?
- 4,283
1
vote
0
answers
48
views
Calculating temporal metrics with a large dataset
I'm having an issue with a requirement to calculate different metrics for groups of users within a specific date range. We expect that by pressing a button, we can generate a PDF displaying these ...
1
vote
0
answers
36
views
Why does loading LDV in this sequence reduce computation?
I'm reading the documentation on loading LDV and to reduce computation the docs recommend this sequence of tasks below. I don't understand why #3 is called out and why it should be done after step 2.
...
- 10.6k
1
vote
0
answers
81
views
Can you index the isDeleted field? If so, how do you calculate the selectivity threshold
Can you index the standard isDeleted field? If so, in terms of calculating the selectivity threshold, would it be considered a unary standard or unary custom index?
I'm working with large data volumes ...
- 10.6k
1
vote
0
answers
41
views
Are RecordTypes Indexed on Custom Objects?
Is RecordtypeId indexed on custom objects? According to two places in the documentation it says just standard objects, but that seems like a mistake.
https://developer.salesforce.com/docs/atlas.en-us....
- 10.6k
1
vote
0
answers
578
views
How do you avoid a timeout on custom objects with a lot of data?
We have a batch performing a query on a custom object that returns a timeout. The table contains 2.4 million records and the query should return 125 000 records. That's 5% of total records.
Our ...
- 337
1
vote
1
answer
70
views
What is the best practice method to summarize contact information to leads?
Here's the business case. I want to have a reportable field on a lead that displays the number of contacts for, say, a city field match. This value could be used in reports, dashboards, snapshots, ...
- 63
0
votes
2
answers
1k
views
Data skew exists whereby the number of matching rows is very large
I am getting the following Error.
works fine in DEV org but throwing error in UAT. Please suggest me as I was unable to move forward.
The Actual Error is
The first validation error encountered was ...
- 3
0
votes
1
answer
936
views
Aggregate Query on 900K records - how to do this?
Hello fellow developers,
I need to run an aggregate query on 900K telephone records to add up a field (grouping by phone line id) and record results in the "summaries logbook".
I have tried ...
- 69
0
votes
1
answer
201
views
How to address large data?
AtaHi guys：
i have a things about salesforce account tables. In my scenario, I have more than 600000 record in my account table. And currently, we want to admin all account territory. So, I want to ...
- 319