I have been debugging some slow code and it seems that the culprit is the EF code posted below. It takes 4-5 seconds when the query is evaluated at a later stage. I'm trying to get it to run in under 1 second.
I have tested this using the SQL Server Profiler, and it seems that a bunch of SQL scripts are executed. It also confirms that it takes 3-4 seconds before SQL server is done with the executions.
I have read other similar questions about the use of Include() and it does seem that there is a performance penalty when using it. I've tried to split the below code into several different queries but it's not making much of difference.
Any idea how I can get the below to execute faster?
Currently the web app I'm working on is just showing an empty iframe while waiting for the below to complete. If I cannot get faster execution time I have to split it up and partially load the iframe with data or go with another asynchronous solution. Any ideas here would also be appreciated!
using (var scope = new TransactionScope(TransactionScopeOption.Required, new TransactionOptions { IsolationLevel = System.Transactions.IsolationLevel.ReadUncommitted }))
{
formInstance = context.FormInstanceSet
.Includes(x => x.Include(fi => fi.FormDefinition).Include(fd => fd.FormSectionDefinitions).Include(fs => fs.FormStateDefinitionEditableSections))
.Includes(x => x.Include(fi => fi.FormDefinition).Include(fd => fd.FormStateDefinitions))
.Includes(x => x.Include(fi => fi.FormSectionInstances).Include(fs => fs.FormFieldInstances).Include(ff => ff.FormFieldDefinition).Include(ffd => ffd.FormFieldMetaDataDefinition).Include(ffmdd => ffmdd.ComplexTypePropertyNames))
.Include(x => x.CurrentFormStateInstance)
.Include(x => x.Files)
.FirstOrDefault(x => x.FormInstanceIdentifier == formInstanceIdentifier);
scope.Complete();
}
5条答案
按热度按时间o4hqfura1#
tl;dr Multiple
Include
s blow up the SQL result set. Soon it becomes cheaper to load data by multiple database calls instead of running one mega statement. Try to find the best mixture ofInclude
andLoad
statements.it does seem that there is a performance penalty when using Include
That's an understatement! Multiple
Include
s quickly blow up the SQL query result both in width and in length. Why is that?Growth factor of
Include
s(This part applies Entity Framework classic, v6 and earlier)
Let's say we have
Root
Root.Parent
Root.Children1
andRoot.Children2
Root.Include("Parent").Include("Children1").Include("Children2")
This builds a SQL statement that has the following structure:
These
<PseudoColumns>
consist of expressions likeCAST(NULL AS int) AS [C2],
and they serve to have the same amount of columns in allUNION
-ed queries. The first part adds pseudo columns forChild2
, the second part adds pseudo columns forChild1
.This is what it means for the size of the SQL result set:
SELECT
clause is the sum of all columns in the four tablesSince the total number of data points is
columns * rows
, each additionalInclude
exponentially increases the total number of data points in the result set. Let me demonstrate that by takingRoot
again, now with an additionalChildren3
collection. If all tables have 5 columns and 100 rows, we get:One
Include
(Root
+ 1 child collection): 10 columns * 100 rows = 1000 data points.Two
Include
s (Root
+ 2 child collections): 15 columns * 200 rows = 3000 data points.Three
Include
s (Root
+ 3 child collections): 20 columns * 300 rows = 6000 data points.With 12
Includes
this would amount to 78000 data points!Conversely, if you get all records for each table separately instead of 12
Includes
, you have13 * 5 * 100
data points: 6500, less than 10%!Now these numbers are somewhat exaggerated in that many of these data points will be
null
, so they don't contribute much to the actual size of the result set that is sent to the client. But the query size and the task for the query optimizer certainly get affected negatively by increasing numbers ofInclude
s.Balance
So using
Includes
is a delicate balance between the cost of database calls and data volume. It's hard to give a rule of the thumb, but by now you can imagine that the data volume generally quickly outgrows the cost of extra calls if there are more than ~3Includes
for child collections (but quite a bit more for parentIncludes
, that only widen the result set).Alternative
The alternative to
Include
is to load data in separate queries:This loads all required data into the context's cache. During this process, EF executes relationship fixup by which it auto-populates navigation properties (
Root.Children
etc.) by loaded entities. The end result is identical to the statement withInclude
s, except for one important difference: the child collections are not marked as loaded in the entity state manager, so EF will try to trigger lazy loading if you access them. That's why it's important to turn off lazy loading.In reality, you will have to figure out which combination of
Include
andLoad
statements work best for you.Other aspects to consider
Each
Include
also increases query complexity, so the database's query optimizer will have to make increasingly more effort to find the best query plan. At some point this may no longer succeed. Also, when some vital indexes are missing (esp. on foreign keys) performance may suffer by addingInclude
s, even with the best query plan.Entity Framework core
Cartesian explosion
For some reason, the behavior described above, UNIONed queries, was abandoned as of EF core 3. It now builds one query with joins. When the query is "star" shaped1 this leads to Cartesian explosion (in the SQL result set). I can only find a note announcing this breaking change , but it doesn't say why.
Split queries
To counter this Cartesian explosion, Entity Framework core 5 introduced the concept of split queries that enables loading related data in multiple queries. It prevents building one massive, multiplied SQL result set. Also, because of lower query complexity, it may reduce the time it takes to fetch data even with multiple roundtrips. However, it may lead to inconsistent data when concurrent updates occur.
1Multiple 1:n relationships off of the query root.
jljoyd4f2#
with dot net core 5 I have use this solution
3hvapo4f3#
I've got a similar issue with a query that had 15+ "Include" statements and generated a 2M+ rows result in 7 minutes.
The solution that worked for me was:
A sample can be found below:
IQueryable is needed in order to do all the filtering at the server side. IEnumerable would perform the filtering in memory and this is a very time consuming process. Entity Framework will fix up any associations in memory.
b09cbbtk4#
Do you have correctly configured relationships between all the entities that you try to 'include'? If at least one entity does not have a relationship to some of other entities, then EF will not able to construct one complex query using SQL join syntax - instead it will execute as many queries as many 'includes' you have. And of course, that will lead to performance issues. Could you please post the exact query(-es) that EF generates in order to get the data?
0sgqnhkj5#
SET STATISTICS TIME ON; SET STATISTICS IO ON;
and paste the Output window text into statisticsparser.com