Application processing at lightning performance – The hourglass view of access times

HourglassEven in these modern times, when lots of things are changing in the ICT world, some lessons from the past still hold true.

Previously, I discussed the I/O stack in a typical database environment. As virtualization has complicated things a bit, the fundamental principles of performance tuning stay the same.

Recently I was browsing through old presentations of colleagues and found another interesting view on response times in an application stack. Again, I polished it up a bit and modified it to reflect a few innovations and personal insights.

The idea is as follows. We as humans have problems getting a feel of how fast modern microprocessors work. We talk in milliseconds, microseconds, nanoseconds. So – in the comparison we assume a 1 Gigahertz processor and then scale up one nanosecond to match one second – because this fits better in human’s view of the world. Then we compare various sorts of storage on the “indexed” timescale and see how they relate to each other.

Read more of this post

Performance – The I/O stack

Concorde Mach Indicator

In my last post, I gave a highly simplified representation of “The I/O stack” of a database. In reality, it’s much more complex.

I found an old picture where the whole I/O stack of a database was described and I decided to brush it off and include some additional layers (application and middleware) and show how the I/O flows if you run on a virtual server with a hypervisor.

Also, the storage network can provide virtualization which in turn adds a layer of complexity.

Read more of this post

The missing link in application performance tuning

When dealing with application performance problems, the quick and easy solution is often to throw hardware at the problem. Typically this is one out of processing power (CPU), memory, or more I/O resources. With a bit of luck, the bottleneck is removed and shifts somewhere else (but now the total stack is – hopefully – capable of handling more workload).

Missing Link

Read more of this post

POC: Piece Of Cake or Point Of Contradiction?

Every now and then I get involved in Customer Proof of Concepts. A Proof of Concept (POC) is, according to Wikipedia, something like a demonstration of feasibility of a certain idea, concept or theory.

Concept Performance Aircraft

Concept Aircraft

Read more of this post

Monkey Business

Monkey eating bananaMaybe you have heard the story of the Monkey Experiment. It is about an experiment with a bunch of monkeys in a cage, a ladder, and a banana. At a certain point one of the monkeys sees the banana hanging up high, starts climbing the ladder, and then the researcher sprays all monkeys with cold water. The climbing monkey tumbles down before even getting the banana, looks puzzled, wait until he’s dry again and his ego back on its feet. He tries again, same result, all monkeys are sprayed wet. Some of the others try it a few times until they learn: don’t climb for the banana or you will get wet and cold.

The second part of the experiment becomes more interesting. The researcher removes one of the monkeys and replaces him with a fresh, dry monkey with an unharmed ego. After a while he spots the banana, wonders to himself why the other monkeys are so stupid not to go for the banana, and gives it a try. But when reaching the ladder, the other monkeys kick his ass and make it very clear he is not supposed to do so. After the new monkey is conditioned not to go for the banana, the researcher replaces the “old” monkeys, one by one, with new ones. Every new monkey goes for the banana until he learns not to do so.

Eventually the cage is full of monkeys who know that they are not allowed to climb the ladder to get the banana. None of them knows why – it’s just the way it is and always has been…
Read more of this post

Managing database performance SLA’s with quality of service

A guy walks into the showroom of a Porsche dealer. He wants to buy a new set of hot wheels. The sales guy tells him about the latest technology in sports car design. This year’s model has active 4-wheel drive traction control, a very powerful engine (over 500 horsepower) with direct fuel injection, semi-automatic dual-clutch with seven speeds, and the whole car is weight-balanced to offer the best handling and cornering speeds. At the same time, carbon emissions per kilometer are the lowest in years and the car actually has green labels, so at least you can make yourself believe that it does not ruin the environment too much ;-)


“Great,” says the customer. “What’s the acceleration and top speed?”

Read more of this post

Pinpointing I/O bottlenecks on Linux

Does this story sound familiar?

The end users of a database application start complaining about poor system response and long running batch jobs. The DBA team starts investigating the problem. DBA’s look at their database tools such as Enterprise Manager, Automatic Workload Repository (AWR) reports, etc. They find that storage I/O response times are too high (such as an average of 50 milliseconds or more) and involve the storage team to resolve it.

The storage guys, in turn, look at their tooling – in case of EMC this could be Navisphere Analyzer, Symmetrix Performance Analyzer (SPA) or similar tools. They find completely normal response times – less than 10 milliseconds average.

The users still complain but the storage and database administrators point to each other to resolve the problem. There is no real progress in solving the problem though.

Two Way Communications

Read more of this post