Cloud Economics: Computational Load Testing and Stress Testing for Azure App Service Web API (Performance Benchmark)

App Service, Service Fabric and Serverless Azure Function are becoming flagship computational service for Microsoft Azure. I am trying to find an economical benchmark for Computing in Azure. It is not a straight-forward task, but I am employing a simple methodology to find cost/computing rational between all three computational platforms.

Summary

Following is the highest throughput and zero error load test’s summary out 5 various load patterns for Azure App Service Server Farm of 3 instances.

Hourly Cost (3 x Instances)0.131183505 GBP x 3 =
0.393550515 GBP per Hour
Azure Pricing
Hourly Test Count579,600 Api CallsLoad Test Stats & Profile
Zero-Fault Average CPU Consumption91.490875%Performance Stats

Load Test – Detail Breakdown

Test Methodology

I ran user load of 600, 1300, 500, 500, 500, 50, 50, 50 respectively. My objective was to find Zero-Fault maximum throughput on the Azure App Service.

The user load was well distributed with planned load spikes for a successful run (I am using the same scenario pattern for all Azure Compute comparisons).

App Service Profile

Sites1
Instances3
Api (SDK)Web Api 2.0

Machine Profile

"sku": {
  "name": "B3",
  "tier": "Basic",
  "size": "B3",
  "family": "B",
  "capacity": 3
}

Machine Configuration

SizeB3 Basic
CPU cores4
Memory GiB7
Local Storage GiB10

Azure Pricing [as of 25th April 2017]

  • 0.131183505 per Hour per Instance – Based on Detail Usage Report from Azure Portal
  • 0.131183505 x 3 = 0.393550515 per Hour per App Service Farm (3xNode)

Web API Harness – Sample Code

Generate random Guid and apply list and hash operations.

# API Endpoint - 1

var guidList = new List<dynamic>();
for (var i = 0; i < 300000; i++)
{
    var randomGuid = Guid.NewGuid().ToString();
    var guid = randomGuid;
    var hash = randomGuid.GetHashCode();
    var reverseGuid = Reverse(randomGuid);
}

for (var i = 0; i < 1000; i++)
{
    var randomGuid = Guid.NewGuid().ToString();
    guidList.Add(new { Guid = randomGuid, Hash = randomGuid.GetHashCode(), ReverseGuid = Reverse(randomGuid) });
}

// Return GuidList using JsonConvert
# API Endpoint - 2

ISupportedImageFormat format = new JpegFormat { Quality = 70 };
Size size = new Size(150, 0)
using (MemoryStream inStream = new MemoryStream(photoBytes))
{
    using (MemoryStream outStream = new MemoryStream())
    {
        using (ImageFactory imageFactory = new ImageFactory(preserveExifData:true))
        {
            imageFactory.Load(inStream)
                        .Hue(180, true)
                        .Brightness(40)
                        .Filter(matrixFilter)
                        .Resize(size)
                        .Pixelate(20)
                        .Format(format)
                        .Save(outStream);
        }

        // Converting Outstream to Base64 and add into HttpResponse.
    }
}

Accept Base64 2048×2048+300dpi High-Resolution image and applying some rigorous image manipulation.

Load Test Stats & Profile

I find TestId/3015 was the best match for comparison. Following is the summary of the test run.

Max User Load500 (No warm up or think time)
Tests/Sec161 (579,600 Tests per Hour)
Tests Failed11
Avg. Test Time (sec)0.37 (370 ms)
Avg. Http Response Size (KiB)137.21
Throughput/Sec (KiB)22,090.81 (21.5731 MiB)

Azure Metrics Query


https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/{{your resource group}}/providers/Microsoft.Web/sites/{{your site farm}}/providers/microsoft.insights/metrics?api-version=2016-09-01&$filter=(name.value eq 'CpuPercentage') and aggregationType eq 'Average' and startTime eq 2017-04-25T21:20:00Z and endTime eq 2017-04-25T21:40:00Z and timeGrain eq duration'PT1M'

Performance Stats

You can also find raw result set at Gist.

TimestampAverage CPU
2017-04-25T21:20:00Z92.78
2017-04-25T21:21:00Z97.43
2017-04-25T21:22:00Z93.185
2017-04-25T21:23:00Z92.375
2017-04-25T21:24:00Z96.125
2017-04-25T21:25:00Z90.585
2017-04-25T21:26:00Z92.425
2017-04-25T21:27:00Z91.45
2017-04-25T21:28:00Z90.35
2017-04-25T21:29:00Z91.88
2017-04-25T21:30:00Z93.87
2017-04-25T21:31:00Z91.29
2017-04-25T21:32:00Z92.5275
2017-04-25T21:33:00Z87.2025
2017-04-25T21:34:00Z90.84
2017-04-25T21:35:00Z90.32
2017-04-25T21:36:00Z90.3725
2017-04-25T21:37:00Z90.5775
2017-04-25T21:38:00Z83.9275
2017-04-25T21:39:00Z90.305

Disclaimer

The views expressed on this site are personal opinions only and have no affiliation. See full disclaimerterms & conditions, and privacy policy. No obligations assumed.