Software Engineer

13 minute read

19 min read 3982 words

LINQ (Language Integrated Query) is one of the most beloved features in C#. It allows us to write declarative code to manipulate collections, making our code more readable and expressive. However, to truly master .NET, we need to understand that LINQ is not just “magic” syntax—it’s a sophisticated set of library methods built upon core C# language features like generics, delegates, and iterators.

In this post, we’ll explore the internals of how LINQ works, from the fundamental interfaces to the performance optimizations that make it efficient.

1. The Foundation: IEnumerable and IEnumerator

At the heart of LINQ lies the IEnumerable<T> interface. Every LINQ method is an extension method on this interface. But what is it exactly?

  • IEnumerable<T>: A provider that can give you an IEnumerator<T>. It’s like a Book.
  • IEnumerator<T>: The actual “cursor” or “worker” that moves through the collection. It’s like your Finger pointing at a specific word in the book.

The Enumerator Interface

public interface IEnumerator<T> : IDisposable, IEnumerator
{
    T Current { get; } // The item your finger is pointing at
    bool MoveNext();   // Move your finger to the next word. Returns false if you reached the end.
    void Reset();      // Move back to the very beginning (rarely used).
}

How foreach Actually Works

When you write a foreach loop, the C# compiler transforms it into a while loop using an IEnumerator.

Practical Example: Imagine you have a List<string> colors = new() { "Red", "Green", "Blue" };.

Your Code:

foreach (var color in colors)
{
    Console.WriteLine(color);
}

What the Compiler Sees:

using (var enumerator = colors.GetEnumerator())
{
    while (enumerator.MoveNext())
    {
        var color = enumerator.Current;
        Console.WriteLine(color);
    }
}

Why two interfaces? (The Separation of Concerns)

Imagine if the List itself tracked your position (the IEnumerator logic). You could only have one person reading the list at a time!

By separating them:

  1. IEnumerable is the data (The Book).
  2. IEnumerator is the state (The Reader).

This allows multiple foreach loops to run on the same list at the same time, each with its own “finger” pointing at a different spot.

A Simple Custom Iterator

You don’t always need to implement these interfaces manually. C# provides the yield keyword to do it for you.

public IEnumerable<int> GetNumbers()
{
    yield return 1;
    yield return 2;
    yield return 3;
}

When you call GetNumbers(), C# creates a “state machine” (a hidden class) that implements both IEnumerable and IEnumerator for you!

2. Deferred Execution: The “Yield” Magic

One of the most important concepts in LINQ is Deferred Execution. LINQ queries are not executed when they are defined; they are executed only when you start iterating over the results (e.g., using foreach, .ToList(), or .Count()).

This is implemented using the yield keyword, which allows the compiler to generate a state machine behind the scenes.

Deep Dive: What is the yield keyword?

In C#, yield is a special keyword used to create an Iterator. It allows you to produce a sequence of values one at a time, without having to create a whole list in memory first.

Think of it as the “I’ll be right back” keyword.

1. yield return: Producing a value

When you use yield return, you’re telling the method: “Here is one item for now. I’m going to pause right here and wait for you to ask for the next one.”

The Vending Machine Analogy: Imagine a Vending Machine.

  • In a normal method (using a List), you pay for a snack, and the machine dumps the entire inventory of the machine into your lap at once. You have to carry all of it.
  • In a yield method, you pay for a snack, the machine drops one item (yield return), and then it stays put. It remembers exactly where it is. It doesn’t give you the next snack until you put in another coin.

2. yield break: Stopping the sequence

yield break is like the machine saying: “I’m out of snacks! No more for you.” It immediately ends the iteration, even if there is more code after it.

public IEnumerable<int> GetSmallNumbers()
{
    yield return 1;
    yield return 2;
    yield break; // The iteration stops HERE.
    yield return 3; // This code will NEVER run!
}

How it Works: The Hidden “State Machine”

When the C# compiler sees the yield keyword, it does something incredible. It rewrites your method into a hidden class called a State Machine.

  1. It Saves the Spot: It creates a variable to remember which line of code it was on (the “state”).
  2. It Saves the Variables: It “lifts” your local variables into fields so they don’t disappear when the method pauses.
  3. It Pauses and Resumes: When MoveNext() is called, the machine jumps to the saved “state,” runs until the next yield return, saves the new state, and pauses again.

This is why yield is so memory-efficient. You could have a method that “produces” a billion numbers, but it only ever uses the memory for one number at a time!

How Where is Implemented (Simplified)

public static IEnumerable<TSource> MyWhere<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate)
{
    foreach (var item in source)
    {
        if (predicate(item))
        {
            yield return item;
        }
    }
}

What is .Where? (The Filter) Think of .Where as a Security Guard or a Sieve.

  • It looks at every item in your collection.
  • It asks a True/False question (the predicate).
  • If the answer is True, it lets the item pass through.
  • If the answer is False, it discards it.
  • Key Point: The items that come out are the exact same objects that went in. If you have a list of User objects, .Where will give you a list of those same User objects.

How Select is Implemented (Simplified)

    public static IEnumerable<TResult> MySelect<TSource, TResult>(
        this IEnumerable<TSource> source, 
        Func<TSource, TResult> selector)
    {
        // Use ArgumentNullException.ThrowIfNull for cleaner validation
        ArgumentNullException.ThrowIfNull(source);
        ArgumentNullException.ThrowIfNull(selector);

        foreach (var item in source)
        {
            yield return selector(item);
        }
    }

What is ArgumentNullException.ThrowIfNull?

Introduced in .NET 6, ArgumentNullException.ThrowIfNull is a “guard” method. It’s a modern and concise way to ensure that your method’s inputs are not null.

Before .NET 6, you had to write:

if (source == null)
{
    throw new ArgumentNullException(nameof(source));
}

Now, you can simply write:

ArgumentNullException.ThrowIfNull(source);

Why use it?

  1. Readability: It makes your code cleaner and easier to read.
  2. Less Boilerplate: You don’t have to repeat the nameof(...) manually.
  3. Consistency: It’s the standard way to validate arguments in modern .NET code.

What is .Select? (The Projection) Think of .Select as a Transformer or a Projector.

  • It doesn’t throw anything away (it processes every item).
  • It takes an item and “projects” it into something new.
  • Key Point: The items that come out are different from what went in. You are “projecting” the data into a new shape.

What is “Projection”?

In math and graphics, “Projection” means taking an object in one space and representing it in another. In C#, it’s the same:

  • You might take a User object and project only their Name (a string).
  • You might take an int and project its squared value (another int).
  • You might take a Product and project it into a ProductViewModel for your website.

Projection = Changing the shape of the data.

Understanding the Generics: TSource and TResult

If you’re a beginner, the letters inside the angle brackets < > might look like a foreign language. Think of them as placeholders or labels for types that haven’t been decided yet.

  1. TSource (The Input): This stands for “Type of the Source.” It’s like a box that can hold anything—an int, a string, or a User object. LINQ doesn’t care what’s inside; it just knows it’s getting a collection of “Sources.”
  2. TResult (The Output): This stands for “Type of the Result.” This is what comes out of the transformation.
  3. Func<TSource, TResult> (The Machine): This is a function that takes one TSource and turns it into one TResult.

The Factory Analogy: Imagine a factory machine.

  • You feed it Raw Wood (TSource).
  • The machine follows a set of instructions (selector).
  • It spits out a Chair (TResult).

The machine itself is “Generic”—it doesn’t care if you feed it wood to make chairs or metal to make spoons. It just follows the pattern: Input -> Process -> Output.

Because of yield return, these methods don’t create a new list immediately. They return an object that “remembers” where it is in the iteration. It only pulls and transforms the next item from the source when it’s needed (Deferred Execution).

Example of Deferred Execution in Action:

var numbers = new List<int> { 1, 2, 3 };

// This line does NOTHING yet. No processing has happened.
var query = numbers.MySelect(n => {
    Console.WriteLine($"Processing {n}");
    return n * 2;
});

Console.WriteLine("Query defined.");

// The processing only happens HERE, during the foreach loop.
foreach (var n in query) 
{
    Console.WriteLine($"Result: {n}");
}

/* 
Output:
Query defined.
Processing 1
Result: 2
Processing 2
Result: 4
Processing 3
Result: 6
*/

3. Common Operators: Sorting, Grouping, and Materialization

While .Where and .Select are the most common, understanding how to sort, group, and store your data is critical for real-world development.

Ordering: OrderBy and OrderByDescending

LINQ makes sorting collections incredibly easy. Instead of writing complex sorting algorithms, you just specify the property you want to sort by.

  • OrderBy: Sorts items in Ascending order (A-Z, 0-9).
  • OrderByDescending: Sorts items in Descending order (Z-A, 9-0).
var users = new List<User> { ... };

// Sort by Age (Smallest to Largest)
var youngestToOldest = users.OrderBy(u => u.Age);

// Sort by Name (Z to A)
var reverseNames = users.OrderByDescending(u => u.Name);

Analogy: Imagine a deck of cards. OrderBy is like laying them out from Ace to King. OrderByDescending is like laying them out from King to Ace.

Grouping: GroupBy (The “Buckets” Analogy)

GroupBy is used when you want to categorize your data into “buckets” based on a common key. This is a One-to-Many relationship: one key leads to many items.

Analogy: Imagine a big box of Lego bricks. You want to group them by color.

  1. You pick up a brick (the TSource).
  2. You check its color (the key).
  3. You put it into the “Red” bucket, the “Blue” bucket, etc.
var products = new List<Product> { ... };

// Group products by their Category
var groupedByCategory = products.GroupBy(p => p.Category);

foreach (var group in groupedByCategory)
{
    Console.WriteLine($"Category: {group.Key}"); // The "Bucket" label
    foreach (var product in group)
    {
        Console.WriteLine($" - {product.Name}"); // The items in the bucket
    }
}

Key Point: The result of a GroupBy is a collection of IGrouping<TKey, TElement> objects. Each group has a Key property and itself is an IEnumerable containing all the items that matched that key.

Materialization: ToList and ToDictionary

Up until now, we’ve talked about Deferred Execution (the code doesn’t run until you ask for it). Materialization is when you say: “I want the results NOW, and I want to save them.”

1. .ToList() (The “Snapshot”)

When you call .ToList(), LINQ iterates through the entire source, runs all your filters and transformations, and stores the final results in a brand-new List<T>.

  • Use it when: You need to access the results multiple times without re-running the query.
  • Analogy: It’s like taking a Snapshot of a moving stream. The stream keeps flowing, but your photo (the List) stays the same.

2. .ToDictionary() (The “Phonebook”)

This turns your collection into a Dictionary<TKey, TValue>, which allows for extremely fast (O(1)) lookups using a Key. This is typically a One-to-One relationship: one key leads to exactly one item.

The Basic Usage:

// Key: User ID, Value: The User object itself
var userLookup = users.ToDictionary(u => u.Id);

// Find a user instantly
var user = userLookup[123];

Using a Value Selector: You don’t have to store the whole object as the value. You can pick just what you need.

// Key: User ID, Value: User's Email address
var emailLookup = users.ToDictionary(u => u.Id, u => u.Email);

string email = emailLookup[123];

⚠️ The Golden Rule of ToDictionary: The key you choose must be unique for every item in the list. If LINQ finds two items with the same key (e.g., two users with the same ID), it will throw an ArgumentException. If you have duplicate keys, you should use GroupBy instead!

Analogy: It’s like a Phonebook. Instead of reading the whole book from start to finish to find “Alice”, you just jump straight to the “A” section using her name.

3.4. The Power Move: Combining GroupBy and ToDictionary

Sometimes you want to group your data and then immediately turn those groups into a lookup table.

Scenario: You have a list of Orders and you want to create a dictionary where the key is the CustomerId and the value is the total amount they have spent.

var customerSpending = orders
    .GroupBy(o => o.CustomerId)
    .ToDictionary(
        group => group.Key,                // The Key for the Dictionary
        group => group.Sum(o => o.Amount)  // The Value (calculated from the group)
    );

// Usage:
decimal totalSpentByCustomer42 = customerSpending[42];

This pattern is incredibly common in data processing and reporting!

4. The LINQ Provider Pattern: IEnumerable vs. IQueryable

LINQ is split into two main domains:

  1. LINQ to Objects (IEnumerable<T>): Operates on in-memory collections using delegates (compiled code).
  2. LINQ to Entities/SQL (IQueryable<T>): Operates on external data sources (like databases). Instead of delegates, it uses Expression Trees.

Example of IEnumerable<T> (In-Memory):

var names = new List<string> { "Alice", "Bob" };
// The filtering happens in your C# code on the computer's RAM.
var filtered = names.Where(n => n.StartsWith("A"));

Example of IQueryable<T> (Database/EF Core):

// 'db' is an Entity Framework DbContext
// This DOES NOT pull all users from the database.
// It translates the C# logic into a SQL "SELECT ... WHERE ..." statement.
var query = db.Users.Where(u => u.IsActive); 

// The SQL is only sent to the database when you iterate over 'query'.
var activeUsers = query.ToList();

5. Performance Optimizations in .NET

While LINQ is powerful, it has historically been criticized for being “slower” than manual loops due to allocations and virtual calls. However, modern .NET has introduced many optimizations:

Special-Casing for Collections

Many LINQ methods check if the source implements ICollection<T> or IList<T> to provide a faster path. For example, Count() will check if the source has a .Count property before iterating through the whole collection.

Avoid Boxing with ValueTypes

Historically, LINQ would box structs. Newer optimizations in the .NET runtime and specialized LINQ implementations (like those found in System.Linq in .NET 6+) try to minimize these allocations.

Struct-based Iterators

Some modern .NET types use custom struct-based enumerators to avoid heap allocations during foreach loops.

6. Standard Query Operators to Master

To use LINQ effectively, you should be familiar with these categories of operators:

  • Filtering: Where, OfType 
    var adults = users.Where(u => u.Age >= 18);
  • Projection: Select, SelectMany 
    var names = users.Select(u => u.Name);
  • Aggregation: Count, Sum, Min, Max, Aggregate 
    var totalAge = users.Sum(u => u.Age);
  • Quantifiers: Any, All, Contains 
    bool hasVip = users.Any(u => u.IsVip);
  • Set Operations: Distinct, Union, Intersect, Except 
    var uniqueTags = tags.Distinct();
  • Ordering: OrderBy, OrderByDescending, ThenBy (See Section 3 for details) 
    var sorted = users.OrderBy(u => u.LastName).ThenBy(u => u.FirstName);
  • Grouping: GroupBy (See Section 3 for details) 
    var ordersByCustomer = orders.GroupBy(o => o.CustomerId);
  • Partitioning: Take, Skip 
    var firstTen = users.Take(10);
  • Materialization: ToList, ToArray, ToDictionary (See Section 3 for details) 
    var userList = users.ToList();

7. Best Practices

  1. Use Any() instead of Count() > 0: 
    // Good (stops at first item found)
if (users.Any()) { ... } 
    
// Bad (might count thousands of items unnecessarily)
if (users.Count() > 0) { ... }
  2. Avoid Multiple Enumerations: 
    var query = users.Where(u => u.Active);
    
// BAD: This runs the filter twice!
var list = query.ToList();
var count = query.Count(); 
    
// GOOD: Materialize once, then use the result.
var activeUsers = query.ToList();
var count = activeUsers.Count;
  3. Use ConfigureAwait(false): (See my previous post on Async/Await) When mixing LINQ with async operations (like in EF Core).
  4. Consider ValueTask: When writing your own async LINQ-like streams (IAsyncEnumerable<T>).

Conclusion

Understanding LINQ internals transforms it from a “black box” into a powerful tool you can use with confidence. By mastering IEnumerable, deferred execution, and the differences between IEnumerable and IQueryable, you can write code that is both elegant and performant.

References & Further Reading

Leave a comment

You may also enjoy