FSharp vs CSharp - a performance comparison with BenchmarkDotNet
Forenotes from 2023
This post was originally published on my blog on April 25th, 2017 as part of my my Daj Się Poznać 2017 blog post series 🖋️. My aim, as explained at the beginning of the post, was to debunk a few myths, like "F# is slow". However, although I specifically put a disclaimer for it, I got quite a few comments and messages telling me "this code is not optimised" or "it could go much faster if you did this or that". I know. There are always ways to make code faster. In practice though, it's about balance. Performant code is sometimes less readable. Or more difficult to change. Sometimes it may take much, much longer to write. Performance is a complex topic that can quickly evolve in heated discussions, and that's why I was simply attempting to do a rough comparison of the two languages using boring, day-to-day code snippets that you will find in most applications.
F# vs C#: a performance comparison with BenchmarkDotNet
[EDIT - 02/05/2017] I received quite a lot of comments regarding this post, about how it does not fairly reflect the differences between F# and C#, and also suggesting possible performance optimisations. I want to highlight from the get-go that this was done on purpose (this is explained in the post below but was maybe not explicit enough). My goal was to compare the traditional OOP approach of C# with the more FP'ish approach of F# on a real use-case. I left out any performance optimisation from both versions because the aim was not to optimise code - but rather to show how both versions compare if we write them in a simple and naive way.
Ever since I became interested in functional programming, I've read and heard multiple times something along the lines of "F# (or functional programming in general) is slow and that's why you should stay with OOP languages that guarantee good performance, like Java, C# or C++".
Because I am curious by nature, I decided to see for myself if indeed I should sacrifice F# on the altar of performance. Specifically, there are two parts of the sentence above that deserve our attention:
"F# is slow"
What does it mean for a programming language to be slow? How can we measure the slowness of a given language? In what context? On what machine, OS, or architecture? Under what conditions? This is definitely not an easy question. Let's try to put it in a more reasonable way:
Is F# so much slower than other OOP languages like C# that we should automatically refrain from using it?
This is easier to answer already. We can definitely measure if a language is slower than another and we can also judge if the difference in performance is something we can live with. This is precisely what we'll try to do in this post.
"OOP languages guarantee good performance"
Thankfully this is an easy one to debunk. I've seen enough systems written in C# that were riddled with performance issues to know this is not the exclusivity of FP languages. I trust in the C# programmers' ability to create slow and unusable code, too. The same applies to Java and C++, of course.
Actually, there is one thing OOP languages are ultra fast at: introducing bugs in the codebase! I still remember a discussion from a few years ago with some of my former colleagues: "This module sure doesn't work, but look at how fast it runs!".
All jokes aside, I hope you get my point. Some languages may be faster than others, but they do not automatically guarantee you good performance. This is something we, programmers, must take care of. With that said, let's benchmark!
BenchmarkDotNet to the rescue!
We are lucky! The fantastic BenchmarkDotNet tool supports both C# and F# (and also VB if it's something you're into). So we will be using it today. But first, we need to know what we will be measuring. I didn't want to do yet another benchmark comparing the speed of strings concatenation, so I came up with a more realistic business case. We will be comparing the performance between F# and C# for the following tasks:
- Generate a list of customers.
- Iterate through the list of customers and set as VIP those whose seniority is greater than 2 years.
To have a little more to discuss, we will conduct the experience twice, first with 100 000 customers, and finally with 1 000 000 customers.
Obviously, I wrote the C# and F# samples using different approaches. Therefore:
- The C# sample uses good ol' for loops, mutable classes and standard lists.
- The F# sample uses recursive functions, immutable records and immutable lists.
I wrote both versions in a naive way, with no attempt at performance optimization! Also, before I started benchmarking, I knew that F# would most probably be slower than C# in all cases due to the additional allocations required to achieve immutability, but I had no idea how much slower it would be.
As usual, the full code is available on my GitHub repo here (C# version) and here (F# version).
The C# solution
Let's have a look at the general skeleton for the C# solution first:
namespace BenchmarkDotNetCsharp
{
public class Program
{
public static void Main(string[] args)
{
var summary = BenchmarkRunner.Run<PerformanceTests>();
Console.WriteLine(summary);
}
}
[MemoryDiagnoser]
public class PerformanceTests
{
private List<Customer> _customers;
[Benchmark]
public void CSharp_GenerateCustomers()
{
// TODO implement
}
[Benchmark]
public void CSharp_ModifyCustomers()
{
// TODO implement
}
}
public class Customer
{
public int Id { get; set; }
public string Name { get; set; }
public int Age { get; set; }
public bool IsVip { get; set; }
public DateTime CustomerSince { get; set; }
}
}
The structure is really simple. The Program class runs our benchmark using
BenchmarkRunner.Run<PerformanceTests>();
The PerformanceTests class contains the two methods that we want to benchmark, CSharp_GenerateCustomers and CSharp_ModifyCustomers. They are decorated by the Benchmark attribute from BenchmarkDotNet. And finally we have the straightforward Customer class that contains 5 properties.
The customers (respectively 100 000 and 1 000 000) will be held in a standard .NET list:
private List<Customer> _customers;
The F# solution
Now to the F# equivalent:
type Customer = {
Id: int
Name: string
Age: int
IsVip: bool
CustomerSince: DateTime
}
[<MemoryDiagnoser>]
type PerformanceTests_ImmutableList() =
let mutable _customers : Customer list = [] // Immutable F# linked-list
[<Benchmark>]
member this.FSharp_GenerateCustomers_Immutable() =
// TODO implement
[<Benchmark>]
member this.FSharp_ModifyCustomers_Immutable() =
// TODO implement
[<EntryPoint>]
let main argv =
BenchmarkRunner.Run<PerformanceTests_ImmutableList>() |> printfn "%A"
Console.ReadKey() |> ignore
0
Damn, I love the conciseness of the F# version! Here we have an immutable Customer record type containing the same fields as its C# counterpart, followed by PerformanceTests_ImmutableList, the class we want to benchmark. It contains both FSharp_GenerateCustomers_Immutable and FSharp_ModifyCustomers_Immutable, also decorated by the Benchmark attribute. The customers will be stored in a F# immutable list:
let mutable _customers : Customer list = [] // Immutable F# linked-list
The reference to the list itself is mutable, purely because we want to assign it from the first function and reuse it in the second one.
In case you were wondering, the MemoryDiagnoser attribute from BenchmarkDotNet allows to retrieves additional information about memory (Gen0, Gen1 and Gen2 allocations). This is the setup I used for the benchmarks:
But enough talking, let's the fight begin!
Generating customers
This is the implementation of the C# method that generates customers:
[Benchmark]
public void CSharp_GenerateCustomers()
{
var numberToGenerate = 1000000;
var random = new Random(10);
var customers = new List<Customer>();
for (int index = 1; index <= numberToGenerate; index++)
{
var customer = new Customer()
{
Id = index,
Age = random.Next(18, 100),
Name = _names[random.Next(0, _names.Length)],
IsVip = _booleans[random.Next(0, _booleans.Length)],
CustomerSince = DateTime.Today.AddMonths(-random.Next(0, 121))
};
customers.Add(customer);
}
_customers = customers;
}
And here's the F# equivalent:
[<Benchmark>]
member this.FSharp_GenerateCustomers_Immutable() =
let numberToGenerate = 1000000
let random = Random(10)
let rec loop customers max current =
if current > max then
customers
else
let customer = {
Id = current
Age = random.Next(18, 100)
Name = names.[random.Next(0, names.Length)]
IsVip = booleans.[random.Next(0, booleans.Length)]
CustomerSince = DateTime.Today.AddMonths(-random.Next(0, 121))
}
loop customers) max (current+1
// Assign the results to our F# immutable list of customers
_customers <- loop [] numberToGenerate 1
In both cases, the code produces the exact same data (the Random instance uses the same seed).
Results for 100 000 customers
Below are the results for the generation of 100 000 customers:
And this is the legend for all columns:
As you can see the performance is almost the same: 98.4ms for C# versus 101.6ms for F#. We can also notice that there's a lot more GC collections on F#'s side, but fortunately no Gen2 collection in sight!
I call it a tie! Score: C# 1 / 1 F#
Results for 1 000 000 customers
Below are the results for the generation of 1 000 000 customers:
Here we obtain the following timings: 1.01s for C# versus 1.12s for F#. F# is almost 11% slower than C# in that specific case. Nothing too bad but a difference still.
Let's give the point to C# here. Score: C# 2 / 1 F#
_Modifying customers
Below is the C# code that iterates through the list and updates customers if they have a seniority greater than 2 years:
[Benchmark]
public void CSharp_ModifyCustomers()
{
var today = DateTime.Today;
foreach (var customer in _customers)
{
// If customer for more than 2 years
if ((today - customer.CustomerSince).TotalDays > 365*2)
{
customer.IsVip = true;
}
}
}
Straightforward. Here's the F# version:
let setVip today customer =
match (today - customer.CustomerSince).TotalDays > 365. * 2. with
| true -> { customer with IsVip = true }
| false -> customer
[<Benchmark>]
member this.FSharp_ModifyCustomers_Immutable() =
let today = DateTime.Today
_customers
|> List.map (setVip today)
|> ignore
As F# Customers are immutable, we need to return a new instance every time we need to update IsVip. This is done here:
// ...
| true -> { customer with IsVip = true }
// ...
Results for 100 000 customers
Let's check the results for 100 000 customers first:
As I suspected, the difference starts to show here. At 4,61ms vs 1,40ms, F# is almost 3.3 times slower than C#. The most significant difference is on the memory side though: zero collections for C# (the modifications are made in-place) while F# has quite a lot of Gen0 and Gen1 collections! This is due to the immutability of both the F# list and the Customer type.
Another point for C#! Score: C# 3 / 1 F#
_Results for 1 000 000 customers
This time we go iterate through the list of 1 000 000 customers!
Now, that's a huge difference! 14.21ms for C# versus 212.22ms for F#. This time F# is almost 15 times slower than C#, with again a significant amount of Gen0 and Gen1 collections. This is a clear win for C#!
Final Score: C# 4 / 1 F#
Analysing the results
Now that we have a slightly better view on the topic, let's go back to our original concern then:
Is F# so much slower than other OOP languages like C# that we should automatically refrain from using it?
The first part of the answer is Yes, F# is slower than C# (at least in the scenarios that we saw above).
Now, would the figures above discourage me from using F#? My answer is No. Except for the last test where the difference was pretty significant, I wouldn't sweat too much over the slight performance hit on the F# side.This is something I am willing to sacrifice in return for the better correctness and readability that F# brings to the table. Not to mention that immutability makes it much easier to introduce concurrency in your application, if you ever need to scale out.
Also, we are only talking about a few milliseconds difference for a simple and naive implementation. In a real business application, this would be completely drown in the total time dedicated to IO operations. You should spend your time on optimising those instead!
Final words
Out of curiosity, I decided to run once again the last test with 1 000 000 customers in F#, but this time by swapping the immutable list for the .NET standard list (just like in the C# version):
let mutable _customers = List<Customer>() // Mutable standard .NET list
[<Benchmark>]
member this.FSharp_ModifyCustomers_Mutable() =
let today = DateTime.Today
let rec loop index (customers:List<Customer>) =
if index >= customers.Count
then ignore
else
customers.[index] <- setVip today customers.[index]
loop (index+1) customers
loop 0 _customers |> ignore
And here are the results:
As you can see, the results are much better, almost twice as fast than before! This shows that there is some room for improvement, but it comes with a trade-off.
That's it for today. If you have anecdotes or data to share about performance in a functional code base, I'd be more than happy to hear about it!
Until next time!