How to Start With Your Application's Performance Optimization

If you want to compete with the best, your system needs to be fast and scalable. I have recently seen a large number of clients facing problems with their application performance. Many systems run into performance and scalability problems as they grow. How to start and what should the first step be? Optimization can be a deadly trap and a time-waster. There is always something you can do better if you have the time, money and motivation.

The real trick is knowing where to look for performance and what steps to take first. What steps to consider when doing performance improvements and how to avoid common mistakes?

Modern applications are getting increasingly complex - they process huge amounts of data, communicate with external systems, and support for different time zones became an engineering nightmare. Some systems need to be re-engineered from scratch to handle their growth. If you are building a new application, performance work should never be left at the end! The larger and more complex your application is, the earlier you need to start considering performance as a major feature.

Before we dive in, let’s highlight the difference between performance and scalability, due to frequent misunderstanding between the two concepts.

The Difference Between Performance and Scalability

Everybody knows how important it is to keep a system fast. Very often our clients require their system to be fast. The first question I ask is What do you mean by fast? The usual answer is that It needs to support thousands of users. A system can be slow and still support thousands of users.

A site’s performance decreases the time it takes for a particular page to load or for a particular user-visible action to complete. What a single user sees while sitting at their computer is “performance”.

Scalability is connected to the number of users a site can support. If additional users can be supported by adding more hardware and network bandwidth, the site can be considered as “scalable”. If more hardware and network bandwidth do not solve the problem, additional users cause the site to slow down, which implies that the site has reached its scalability threshold.

Planning a Performance Investigation

Application involves complicated business processes that do not always focus on performance. When a feature works slowly, is it not always possible to accelerate the code. Before you dive deep into the code and try to accelerate part of the code related to your feature, make a plan for your performance.

1. Identify problems and goals and determine what to measure
2. Select methods and tools  
3. Analysis and Conclusion

Identify Problems and Goals and Determine What to Measure

First, ask yourself a few important questions:

• What is wrong with the current performance level?
• What do I want to achieve?

If you start to randomly optimize your application, it will be a waste of time. Define problems and goals. Determine a set of performance requirements. Those requirements will determine which metrics you need to collect. You need concise, well-defined metrics that correspond to the goals. After obtaining the results, you should be flexible to change your goals.

A lot of things are involved and make sure you consider the right path. Observe aspects such as memory - measurement of memory usage and reduction. You want to measure memory usage and minimize it. But which kind of memory? Commit size, large object heap size, .NET heap size or some other variant? For tracking memory usage over time, the average per hour or peak? Be as specific as possible when describing what you want to measure and define your goals well.
Not well-defined: Memory should be less than 1 GB.
Well-defined: Working set memory usage should never exceed 1 GB during peak load of 100 queries per second.

Let’s check out some of the most popular methods and tools.

Select Methods and Tools

There are many tools and methods that provide performance metrics. After performance analysis, make sure that the tool you select has the required characteristics such as precision, portability, etc.

Coder review

Particular code performance issues can sometimes be identified without measurements. Check the asymptotic complexity, big O notation, database query, or ineffective piece of code. You can solve some performance issues without analysis. Keep in mind that personal feelings can easily deceive you as well as technology change.

Profiling

Where to start when optimizing your application? Optimizing a random method may not affect the performance of the whole application. Trying to write fast code can increase code complexity and introduce new bugs.
The most important step is to find where the application spends a significant part of its time. The best way to do it is profiling. That means you take a profile, attach it to the application, take a snapshot and look at the profile. Profiling shows methods that are called frequently, and bottlenecks of your application. That helps you locate the most critical methods that require optimization first.

Monitoring

Certain problems can not sometimes be reproduced locally since they occur rarely or only in the production environment. In this case, profiling can not help and monitoring needs to be conducted to find a method with performance problems. There are a lot of built-in monitoring tools such as event log. Take data from logs and try to find the source of this problem.

Performance tests

It’s common practice to write unit tests that ensure that the business logic works fine after any changes to your code. Unit tests are not enough to measure performances. To make sure your performance stays the same before and after changes, you need performance tests. To maintain the same level of performances in your tests, you need to have exactly the same environment (hardware + software) for all the benchmarking configurations.

If your tests are executed in a different environment each time, you will most likely have different results. Executing the same tests in an environment that has more hardware will affect your tests to be executed faster. Therefore, it is crucial to run tests on the same server environment. The performance tests can be executed on a build server, as part of a continuous integration (CI) pipeline.

Benchmarking

A benchmark measures performance properties of a piece of code especially to compare it to an alternative implementation. For benchmark tests, it’s important to maintain the same version of a .NET runtime, an operating system, hardware, and so on. Run your benchmark tests in more iterations to achieve sufficient accuracy. Benchmarks are amazing when they test a single resource such as CPU time. You can test other things like reading files off an SSD, but you will need to isolate those resources from outside influence.

Analysis and Conclusion

“If you have spent less than a week studying a benchmark result, it is probably wrong.”
- Brendan Gregg, author of Systems Performance: Enterprise

An analysis is the most important part of any performance investigation. Do not take results fast and adapt the solution on the whole application, for example: “Profiler shows that method A is faster than method B. Let’s use A everywhere instead of B”. We should understand why we got such results in profiler sessions. If you use one set of data,  method A may be faster, but you should take in consideration the whole application and different set of data. Method A can probably be substantially slower than method B in some other cases.

Most of the performance mistakes are caused in the measurement methodology. Check other data patterns before you decide on the method in the production code. Be careful with analysis before you make a conclusion.

Conclusion

Planning a performance investigation requires more steps. Those steps can be customized to your needs. I covered the three most important steps that should be involved in any investigation. Identifying problems and goals, and determining what to measure should be your first step. Next step is monitoring, which helps you find a slow user scenario. The profiling will help you identify the bottleneck of your methods. Benchmarks will find the fastest way to implement the feature. Performance tests will help keep the performance on the same level in the future.

All approaches have a purpose and use. Last step of any investigation is data analysis and conclusion. The more time you spend in data analysis and measurement, it’s more likely that your performance investigation is done properly.