Richard BF

Before *nix systems made their way out of academia, back when mainframes ruled the corporate I.T. world, performance tuning was a job in itself. When the cost of a few extra megabytes of memory was in the six figure range, small tuning improvements here and there meant saving hundreds of thousands if not millions of dollars.

For this reason, IBM machines, upon which I spent many of my early years, provided a vast range of performance tuning functionality.

At Synop, we were recently faced the performance tuning of one of our production Linux boxes, and I was completely dismayed to find out there’s virtually nothing for the seasoned tuning engineer to play with.

In the IBM mainframe world, particularly under VM, metrics are immediately available on what memory is being used by what, how often code is dispatched or time sliced, and a vast range of other useful information. And once you have these metrics, you can pretty much tune what you like. For example, you can reserve any number of pages for a particular process, to use *nix speak, which will never be swapped out, or you can assign specific pages that will be locked into memory within the processes memory space, so not only won’t they be swapped out, but they won’t move around in memory either.

To minimise dispatching overheads, you can lock processes into one of several dispatching queues, optimised for short and quick responses, like user sessions, or long running batch jobs. The O/S has logic to transition processes between the queues appropriately, but you can also lock processes into a specific queue, as well as tune the time slicing interval and frequency for processes in each queue. Individual processes can also be allocated a percentage priority for dispatch ordering.

CPU usage limits can be set for specific processes, so they don’t exceed a certain level during peak times. Say a batch job churns in the background until a user interaction process kicks off, like a web server for example. The dispatching of the web request could be optimised by reducing the CPU limit for the batch process during business hours, and ensuring the web server was permanently in the user interaction queue for fast dispatching.

For I/O problems, devices can be located on different physical channels, or busses to balance load, and once you have the available metrics for disk seek and latency, you can manually rearrange the layout of your drive for optimum seek latency. With multiple heads per disk platter, you have ample opportunity to reduce the time it takes to seek between popular blocks on the disk, reducing head chatter/thrashing.

When you have processes which are starting up several instances at once, like an inet daemon or shell utility, you can optionally lock shared segments of the code, so that only one copy needs to be in memory. The CPU shares the code amongst all the instances of the process. Forking and threading addresses this to some extent, but not if you have multiple unique instances running at the same time. You can then apply the reservation and locking logic mentioned above, to the shared segments as well, because you can easily determine where they are in memory.

These are only some of the unique performance tuning features of the mainframe world. And while the cost of memory, CPU and disk storage is continuing to fall, it is still nice to know that an engineer can take virtually any level of control over performance and process dispatching, things just not available in standard *nix. However, even though there is no such control, at least knowing some of what is possible, will help engineers have a better understanding of what to look for when tuning *nix systems.

As the Linux market grows, technical personnel in corporate mainframe environments are still hesitant to embrace it, even as the speed of microprocessors continues to climb. This is partially due to the lack of tools they’d need to protect and tune their systems, and ultimately to save themselves from redundancy. While IBM are indeed embracing Linux, they still only run it on top of their mainframe operating systems, which effectively gives them the best of both worlds. The mainstream I.T. media tend to ignore this fact.

There’s several lessons here for the Linux world.

(Originally posted to Synop weblog)