Boris's Blog on Predictive Analytics for IT

Oracle Open World 2011 and Capacity Management Challenges

2011-10-16T02:17:00.000-04:00

Oracle OpenWorld 2011 had more than 45,000 attendees. It took me significant efforts to follow news and visit booths at Exhibit Hall showing Exadata, Exalogic, learn about new Exalitics, Big Data, mix workload management and systems management announcements.

I met with Oracle developers, vendors and customers. It looks like that several organizations in Oracle including OEM developers and RAC/Exadata developers approaching workload management from the different angles. One angle is based on Resource Management and development rules for allocation of CPU resources, concurrency or number of active threads/sessions control, level of parallelism control the second on development technology to dynamic manage re allocation of resources for Oracle instances.

Many companies are buying Exadata appliances expecting that it is turn key operation, but in reality it take significant efforts to move applications to Exadata and organize production environment. Customers experience a lot of challenges in sizing of the new applications and capacity management of the mix workload environment.

A lot of attention at the conference was on Big Data. I feel that capacity management for big data, including setting realistic SLAs based on business demand, configuring Hadoop environment and making decisions about the number of nodes, number of disks per node, configuring DBMS and analytic environment to support existing and future workloads ensuring that it will scale.

Presentation at IOUG 2011 on "Predictive Analytics for Management Oracle Exalogic and Exadata Cloud Environments"

2011-04-20T21:00:00.011-04:00

Last week, on April 13, I presented at IOUG 2011 in Orlando a paper on 'Predictive Analytics for Management Oracle Exalogic and Exadata Cloud Environments". Oracle Exalogic and Exadata x2-8 is a base for construction of private clouds providing a platform for workload consolidation.

Each workload has a unique profile and specific SLOs. Oracle Resource Manager enables creation of rules for controlling usage resources, but it is difficult for IT people to set rules to assure that SLOs of the individual workloads will be met.

In this presentation, we reviewed modeling results showing the interdependence between workloads and servers in multi-tier environments and discussed best practices to organize a continuous proactive performance management process.

We reviewed several case studies to illustrate how to apply predictive analytics, based on statistical methods of workload characterization, analytic performance prediction models and optimization technology to private clouds with multi-tier distributed Oracle Exalogic and Exadata environment supporting mix workloads. We clearly showed how this enables evaluation of options and justification of strategic capacity planning, tactical performance management and operational workload management actions required to meet SLOs.

The weather in Florida was about 85 degrees, which is a big contrast with Chicago, where everything was under snow on April 18th.

Presentation on Workload Management Optimization in Barcelona

2011-01-30T22:17:00.003-05:00

On January 26, I presented jointly with Doug Brown a paper on Workload Management Optimization for Teradata Architects at the Architecture Summit at Barcelona. We reviewed a competition in mixed workload management between Data Warehouse Appliance vendors, including Teradata, Oracle Exadata, IBM DB2 and Netezza, and EMC Greenplum.

We discussed a case study illustrating how predictive analytics can be used to answer typical what if questions, evaluate options and justify operational workload management, tactical performance tuning and strategic capacity planning decisions, including:

1. What will be the impact of the expected growth and planned changes?

2. How to set realistic SLG?

3. How to change workload’s priority (relative weight) to meet SLGs?

4. How to set workloads’ concurrency level (TASM throttling) to met SLGs?

5. How to justify tuning measures to meet SLGs?

6. How to predict new application implementation impact?

7. How to justify the hardware upgrade required to meet SLGs?

8. How to compare actual performance with expected?

Doug completed this well attended presentation by providing an insightful overview of TASM Workload Management TD14.0.

One of the highlights of this trip for me was an opportunity to see the best in the world Barcelona soccer teams in action.

BEZ Systems Acquisition by Compuware Corporation

2010-12-29T22:00:00.002-05:00

In December of 2010, Compuware Corporation acquired BEZ Systems. At Compuware, I am now the CTO of Modeling and Optimization, focusing on applying predictive analytics to application performance management in cloud environments.

I founded BEZ Systems in 1983 and it was an exciting journey. We developed unique modeling and performance optimization technology used by many Fortune 100 companies to justify strategic capacity planning, tactical performance tuning and operational workload management decisions and actions.

I would like to take this opportunity to thank my colleagues at BEZ Systems who made great contributions to our success. I'd also like to thank our investors from JMI, Ascent, Velocity, the State of Massachusetts and others who supported our efforts and growth. Another word of thanks goes to my friends and advisers, including Jeff Buzen, Dan Kaberon, Claudia Imhopf, Bill Inmon, Mark Friedman and Amit Ghosh. I would like to thank many of our customers for their trust and advice. Many of the requirements generated by John Hootman, Dennis Cooper and Beth Withers from Kmart, Rick Dolzel from Wal-Mart, Mike Bankowsky, Al Amato and Russ Fisher from AT & T, Barry Hicks and Jeff Watson from JCP, Oleg Paltoratskiy from GAP, Ellen Reys from VISA, John Vernon from Lowe's were incorporated into products. Thanks to our partners, including Doug Brown and Anita Richards from Teradata, Lee Goddard and Igor Urisman from IBM, Charlie Garry from Oracle and Mark Friedman from Microsoft.

At Compuware, I will be focusing on development of new technology for modeling and application performance optimization in the cloud. I will also form a new company which will provide capacity management services to our Teradata, Oracle and DB2 customers.

MOUG - Capacity Management for Oracle Exalogic and Exadata Private Cloud

2010-11-06T20:03:00.011-04:00

Response time for many eCommerce applications is critical for a business's bottom line. Recently, at OOW 2010, Oracle announced Exalogic and Exadatax2 appliances for the private cloud. On Friday, November 5, I presented a paper at MOUG on "Capacity Management for Oracle Exalogic and Exadata Private Cloud".

Oracle Exalogic and Exadata enable server consolidation and concurrent support of both OLTP and Data Warehouse mixed workloads. Understanding individual workload performance, resource utilization and data access profiles, along with the ability to set realistic SLOs and compare different options within and outside of the private cloud are critical considerations for making effective capacity management decisions. End user response time includes service time, queueing time and different types of delays caused by software parameters limiting the concurrency levels within and outside of the private cloud. We reviewed several examples illustrating the need for workload management optimization, proactive performance management and capacity planning for a private cloud behind the firewall as well as for an Internet environment and content caching outside of the firewall.

In a complex multi-tier distributed virtualized environment, any change can affect workloads differently. For example, changing physical or virtual configuration, workload volume and data growth, implementating new applications, DB and application tuning, changing software parameters, selecting an ISP provider with geographically distributed servers and even selecting a browser can affect performance and the business's bottom line. It's important to determine how to set realistic SLOs and organize Proactive Service Level Management, set resource management rules and justify tuning and capacity planning measures to continuously satisfy SLOs of the individual lines of business.

The graph below shows end-to-end response time components:

The goal of capacity management is to apply available performance measurement data to build profiles of each workload, build models to evaluate different options and justify operational, tactical and strategic decisions to recommend specific actions.

All required measurement data are not always available and sometimes data extracted from different sources contradict each other. Oracle and other DBMS vendors have sufficient measurement data characterizing DBMS performance, but they typically do not synchronize this data with performance measurements collected on OS level.

Many vendors collect measurement data characterizing the components of end user response time, but they do not collect data about usage of resources. We showed how during model calibration, missing parameters can be derived to successfully address a variety of capacity management scenarios.

We reviewed the application of the hybrid models based on queueing network and non-linear regression models. Queueing network models can describe the private cloud, where measurement data characterizing response time, throughput, usage of systems resources and usage of data are available. Non-linear regression models of the Internet and content caching servers outside of the firewall can be used when limited information about response time of each of the network components is available.

We discussed how modeling results can be used to evaluate the relationship between response time requirements and IT configuration requirements and corresponding cost vs possible impact on the business's bottom line in order to set realistic SLOs.

Several examples were used to illustrate how modeling results and optimization techniques can be used to evaluate different strategic capacity planning, tactical performance management and operational workload management decisions and justify the set of steps necessary to support continuously meeting SLO for each workload with minimum cost. A a result of the modeling and optimization, a plan of operational, tactical and strategic actions is generated and performance expectations are set to ensure that SLOs for each workload will be met during next fiscal year.

This plan of action is based on the application of modeling and optimization technology which generate near real time operational recommendations, proactive DBMS and application tuning recommendations and resource procurement capacity planning recommendations.

Teradata Workload Management Optimization

2010-10-29T12:13:00.004-04:00

From October 24 to 28, I attended the Teradata Partners Conference in San Diego. Over 3000 people were in attendance. The conference was very well organized, but I expected better weather in San Diego.

Some major news for me was the focus of Teradata on cloud computing and how Teradata supports Map Reduce. With increased volume of non-structured data processed by companies, Map Reduce is becoming popular not only for scientific applications, but for many business applications as well. I participated in reviewing problems with high volumes of data load (petabytes, exabytes and zetabytes) and access to scientific applications at a High Performance Computing conference in May, but I did not expect to see that Oracle, IBM and Teradata had invested so much in supporting Map Reduce for business applications so fast.

With accelerating rate of server, data and application consolidation, workload management optimization has become a critical factor for all vendors. Teradata, in particular, is playing a leading role in Data Warehouse mixed workload management.

At the Teradata conference, I presented a paper in collaboration with Software Architects Doug Brown and Anita Richards on "TASM Workload Management Optimization". We have been collaborating with Doug and Anita for about 10 years in this area and this presentation summarized the results of this collaboration by integrating BEZVision with Teradata TASM.

The main idea is to select TASM rules and Priorites, Concurrency and Resource allocation for each workload in order to continuously satisfy Service Level Goals (SLG) for individual workloads. We reviewed several BEZVision case studies illustrating how predictive analytics can be used to justify changing rules and continuously supporting SLGs. Below are several case studies which were covered in this presentation.

The major focus was on optimizing selection priorities and concurrency level for each workload:

In conclusion, we presented an optimized plan for priorities, concurrency, tuning and hardware upgrades to meet SLGs of individual workloads:

In summary, major recommendations include:

•Know your workloads’ profiles

>(performance, resource utilization and data usage)

•Set Realistic SLG

>Evaluate if SLG will satisfy business needs and how it will affect cost to be sure that you have realistic SLGs for each workload

•Keep in Mind That Everything Is Interdependent

>Workloads are interdependent and any change of TASM parameters can improve performance of one of the workloads, but negatively affect others

•Set Expectations

>Before you change TASM parameters, implement a new application, tune an application or database, or upgrade hardware ask yourself “what should I expect?”

•Always Compare Actual Results with Expected

>If actual results are significantly different from expected, find out why

•Develop Corrective Actions and Organize Continuous Proactive Performance Management

Oracle Open World 2010

2010-09-28T11:24:00.009-04:00

On September 19 at Oracle Open World I presented a paper on Capacity Management for Oracle Exadata Machine v2. During the presentation, we reviewed several case studies and covered some interesting points, including:

According to the prediction results, workload growth and volume of data will increase contention for resources and RAC nodes will become the performance bottlenecks. We evaluated the impact of increasing the number of nodes or potentially increasing the number of processors per node (assuming that Oracle will make this option available). What is interesting is that the same day at 6pm Larry Ellison, in his keynote presentation announced new Exadata x2 - 8, which will include more powerful nodes.

We also discussed the importance of the holistic approach to Exadata capacity management. All workloads and elements of the multi-tier environment are interdependent. We demonstrated that any change in hardware configuration, software parameters, performance tuning or workload management of Exadata systems in DBMS tiers affects performance of the application servers and vice versa.

During the keynote presentation, Larry Ellison also announced Exalogic - a new application tier appliance. Exalogic and Exadata will become Oracle's building blocks for the construction of private clouds.

Exadata and Exalogic systems will be used for server consolidation. Mixed workload management will become critical for effective use of this new technology. We reviewed how performance prediction results can be used for strategic capacity planning, tactical performance management and operational workload management of Oracle Exadata and Exalogic environment.

Over 41,000 people attended the OOW conference. Oracle made many announcements, including Exadata Database Machine x2-8, Exalogic Elastic Cloud (cloud in the box), Fusion Applications, Unbreakable Enterprise Kernel for Oracle Linux and My SQL Release Candidates 5.5.

It was quite a challenge to find my way among 41,000 attendees and hundreds exhibitors in order to attend the right sessions. Proactive scheduling and planning was required to use time effectively. But, I have to admit that OOW was well organized. Can you imagine the level of service required to serve lunch for all those people with minimal waiting time; or bringing thousands of attendees to the concert of mega stars like Berlin, The Black Eyed Peas, Steve Miller, The English Beat, Don Henley and Montgomery Gentry?

ACM HPDC 2010

2010-07-01T12:38:00.003-04:00

Last week I attended the 19^th ACM HPDC 2010 conference. Researches and PhD students from 26 countries discussed challenges of supporting large scale academic applications. Some of the topics included application of modeling for evaluation the impact of virtualization in distributed and parallel processing systems, cloud computing, and storage subsystem optimization.

Many presenters were concerned with large scale computational applications accessing very large volumes of data. Computational applications which used only occasionally are good candidates for cloud computing especially if they retrieve limited amount of data from large files.

One of the challenges is to be able to load large volume of data on time and concurrently be able to perform calculations.

For applications which are scientists run occasionally and service level objectives are not critical the cloud provides great benefits. Several papers were focusing on taking advantage of the parallel processing. It is difficult to train scientist to design effective applications, on the other hand it is even more difficult to explain computer professional the specific problems related to astronomy, chemistry or nuclear physics. Participants discussed several frameworks supporting needs of the academic research, including MapReduce.

Capacity Management for Oracle Database Machine Exadata v2

2010-05-28T00:45:00.003-04:00

On May 20th I presented a paper on "Capacity Management for Oracle Database Machine Exadata v2" at NOCOUG. We reviewed new features of Oracle Database Machine V2, including Smart Scan, Hybrid Columnar Compression, Storage Indexes and Flash Storage. We discussed how to apply regression analysis and queueing network models to predict the impact of the workload growth, increase in volume of data on performance of the individual database machine workloads. We used a case study to illustrate how predictive analytics can be used to to justify strategic capacity planning decisions, tactical performance management and operational workload management actions in Oracle Database Machine V2 environment.

https://docs.google.com/viewer?url=http://www.nocoug.org/download/2010-05/DB_Machine_5_17_2010.pdf&pli=1

Conference was at Oracle's HQ and I spent a couple of days in San Francisco. Surprisingly the weather in San Francisco was much cooler than in Chicago. We appreciate attention of all people who visited our booth, asked questions about "Predictive Analytics for IT" and look at the demo of the new release of BEZVision 3.5. The winner of our drawing become Kamran Rassouli from Activant Solutions, Inc.

Predictive Analytics for IT

2010-04-24T20:42:00.007-04:00

The term "predictive analytics" has become an increasingly common word in the business intelligence (BI).

Predictive analytics, according to Gartner, focus on prediction of future business probabilities and business trends. Multiple predictors are combined in one predictive model which can be used to forecast future business probabilities with an acceptable level of confidence.

“Predictive Analytics for IT" provide information to justify strategic capacity planning, tactical performance management and operational workload management decisions and generate proactive actions required to supporting business needs.

Predictive analytics incorporates statistical methods, regression analysis, and statistical process control, as well as analytical queueing network models and optimization technology.

Applied to Information Technology predictive analytics enables modeling the multi-tier distributed IT infrastructures with mix workloads and predict the impact of the expected growth and planned changes, compare different options and generate proactive alerts and recommendation enabling continuous proactive performance management process

Our product BEZVision currently support DBMS servers based on Oracle, Teradata and SQL Server and application servers based on WebLogic and WebSphere. BEZPlus supports DB2 UDB. Holistic view of the complex IT infrastructure with the dynamic workloads enables evaluation and comparison of the different options. It provides a platform for collaborative efforts in justification of decisions and proactive actions. Modeling results set realistic expectations reducing the risk of surprises.

Differentiators of the Predictive Analytics for IT:
• Automation of data collection, workload aggregation and workload characterization
• Tying Business demand (Key Business Indicators (KBI)) with IT
• Automation of building performance, resources utilization and data usage profiles for each application/workload
• Incorporate statistical process control to identify significant changes in performance, resource utilization and data usage by workloads
• Universal prediction and optimization engines enable modeling of the heterogeneous multi-tier distributed parallel processing environments with mix workloads
• Automatic creation, calibration and evaluation of closed queueing network models enabling:
o setting realistic SLOs, negotiation of SLAs
o justification of the Strategic (capacity planning), Tactical (performance management) and Operational (workload management) decisions
o automatic generation of alerts and proactive advise
o automatic comparison of the actual and expected results
o organization of the continuous proactive performance management process

Misconceptions About the Requirements to Accuracy and Value of Modeling

2010-02-06T14:49:00.007-05:00

Introduction
We often hear questions about the accuracy of models and value of modeling. Hardware is cheap so why bother with spending time and effort on modeling. Accuracy of modeling results depends on many factors. If modeling results cannot guarantee 100% accuracy, why bother with data collection, workload characterization, workload forecasting, and building and calibrating models.

My typical answer is: “How can you manage your system effectively if you do not know what to expect?”

Let’s review factors affecting model accuracy, the role of modeling, limitations of commercial modeling tools, and success stories of our customers when modeling results helped to compare options, justify decisions, reduce risk of surprises and enable organization of the continuous proactive performance management.

There is no doubt that benchmark tests can provide more accurate results than modeling, but benchmark-test preparation time and its cost prohibit use of the benchmarks to justify every decision. Modeling complements benchmarks and expands the horizon of performance evaluation.

Indeed, if you are at a fork in the road and have to make a decision about how to reach the destination sooner, you do not need 100% accuracy in measurements of the distance to the destination to make a decision. You just need the answer to the question “which path is shorter?”

Work on building model and evaluating modeling results create a collaborative environment between DBAs, capacity planners, architects, managers, and other representatives of the business and IT in describing business requirements/workloads, setting realistic SLOs and evaluating prediction results. It helps to understand the environment, and avoid mistakes in making decisions. Modeling results enable comparison of the actual results with expected.

Modeling Overhead
Commercial modeling tools perform similar functions, including data collection, workload characterization, workload forecasting, scenario planning, model calibration and model evaluation. Tools include:

• BEZVision
• BMC Perform Predict
• Hyperformix
• TeamQuest
• Metron
• OptNet
• VMWare

Typical Overhead of Modeling Tools: 1-3%

Typical Cost of Modeling Tool: $100K - $500K

Typical Manpower Required to Support Modeling Tool: 0.2 – 1 FTE

Differences between Modeling Tools Affecting Modeling Accuracy
• Accuracy of measurement data
• Workload characterization and ability to represent non-exponential distribution of the arrival rate, service time
• Ability to represent current and planned IT Infrastructure, including hardware configuration, software configuration, database design, parallel processing
• Ability to take into consideration the interdependence between workloads and servers in a multi-tier distributed environment
• Ability to correlate workload performance, resource utilization and data usage profiles in multi-tier environment
• Use of the open queueing network models assuming that the arrival rate is constant, regardless of server utilization
• Use of the closed queueing network models
• Ability to predict how change in database design and hardware upgrade will affect the level of parallelism within DBMS servers
• Ability to identify users, programs, tables and SQL requests that will cause performance in the future
• Ability to take into consideration the impact, not only of hardware configuration, but also changes/tuning of software parameters affecting the level of concurrency within application tier and DBMS tier.
• Ability to predict the impact of the database tuning
• Ability to predict the impact of the expected growth and change in hardware and software configurations, not only on usage of resources, but also on response time and throughput.
• Ability to model virtualized environments and predict the impact of planned growth and changes on Hypervisor overhead and scalability of the virtualized environment
• Ability to predict how change in pattern of usage of data and level of parallel processing will affect the performance of different workloads
• Ability to predict how change in priority and level of concurrency will affect workload performance
• Accuracy of workload forecasting
• Ability to extract performance measurement data during stress testing and predict the impact of new application implementation in production environment

How Modeling Tools Support Virtualization
VMWare capacity planning tool focuses on application tier. VMware predicts the resource consumption of virtualization and helps to determine size of the host server required to support expected workload growth. VMware does not predict how virtualization will affect the response time and throughput of the individual workloads. It does not take into consideration how expected workload growth will affect the contention for DBMS server resources and affect the response time. It does not take into consideration the interdependence between workloads, interdependence between servers of application tier and DBMS tier, impact of the changes of the software parameters, impact of the changes/tuning database design.

TeamQuest uses a simplified approach in representing workloads and modeling. It does not take into consideration the details of architecture and workloads affecting the parallel processing and concurrency in workload management of DBMS server.
BEZVision, in contrast to other modeling tools, uses closed queueing network models and focuses on detailed representation of both application tier and DBMS tier distributed environments, enabling it to predict the impact of different changes, not only on resource usage, but also on response time and throughput for individual workloads. BEZVision workload characterization automates building hourly performance, resource utilization and data usage profiles for each workload. BEZVision has been modeling complex systems based on parallel processing for years. BEZVision takes into consideration the impact of workload growth and increase in the number of VMs on Hypervisor overhead and scalability of the application servers.

Value of Modeling
Accuracy of modeling is limited but the value is high. It can help:
• Justify strategic capacity planning, tactical performance management and operational workload management decisions during the application and systems life cycle starting from the feasibility study, performance management, capacity planning, workload management, disaster recovery and finally, with server consolidation
• Organize collaborative process
• Reduce uncertainty and risk of surprises
• Answer “what is” questions, compare alternatives
• Predict the impact of the workload growth and increase in volume of data
• Justify hardware upgrade required to support SLO of multiple workloads in multi-tier distributed environment
• Predict the impact of virtualization
• Predict the impact of server consolidation
• Predict the impact of new application implementation
• Enable organization of continuous proactive performance management by comparing actual results with expected and developing corrective measures

BEZ Customers Success Stories
Companies in different industries successfully used our modeling technology to justify strategic capacity planning, tactical performance management and operational workload management decisions since 1992.
• Retail: Immediately after the announcement of our commercial modeling tools in 1992, major retailers Wal-Mart and Kmart started using the modeling tools to justify hardware configuration upgrades for complex systems supporting multiple workloads based on Massively Parallel Processing architecture.
• Wal-Mart—Rick Dolzel, VP of Operations, brought BEZPlus tools in to evaluate the impact of the expected workload and database size growth; justify hardware configurations; balance the utilization of the MPP environments and justify hardware upgrades.
• Kmart—Used BEZPlus to justify upgrades. John Hootman, Manager of Operations, wanted to incorporate database performance tuning functionality. Modeling was used to justify multiple multi-million configuration upgrades. The products not only demonstrated high accuracy in performance prediction, but also helped analysts to understand their complex environment and the impact of the expected workload and database growth. According to Dennis Cooper, Manager of DBAs, “Using BEZPlus, system analysts at Kmart, we were able to convince management that a two-phase, multi-million-dollar upgrade of its existing data warehouse was necessary. By going ahead with the upgrade to our data warehouse, we got an overall boost in performance of about 22%—exactly what the BEZ analysis had indicated we would get." According to John Hootman, who was head of operations at Kmart, the cost of the modeling tool is a drop in the bucket compared with its value.

• JC Penney—According to Barry Hicks, Manager of Corporate Capacity Planning, "Using BEZPlus, we were able to identify architectural bottlenecks that needed to be corrected. In the process, we balanced the utilization of our high capacity EMC2 storage disks and improved the throughput of the system.” He continues, “BEZ has incorporated our Statistical Process Control (SPC) approach into the CorpView product. This function gives us a strategic advantage that insures the efficient use of our corporate resources."
• Gap, Lowe’s, Sears, American Stores, Safeway, Kroger, The Limited and several other retail stores used BEZPlus and BEZVision to plan and manage their data warehouse environment supporting mixed workload environment.

• Communication: AT&T—uses BEZVision for both strategic and tactical purposes during planning and managing distributed environments, server consolidation to support growing business needs. Mike Bankowsky requested incorporation of the SQL tables use analysis and developing root cause analysis identifying SQL and tables that will cause problems in the future.
• VIVO is using BEZVision for capacity planning and performance management of the distributed environment
• Lucent Technologies, Sprint, SBC, NYNEX, Bell Canada, AT&T Canada—used BEZPlus for strategic planning, performance management and tuning, and for justification of hardware configuration upgrades.

• Companies from Finance: Visa, Bank of America, Canadian Imperial Bank of Commerce, The Royal Bank of Canada, Fidelity Investments, Hewitt Associates, Insurance: Blue Cross/Blue Shield, Nationwide Insurance, Prudential Insurance, CNA Insurance, Transportation: Delta and Canadian Airlines, Manufacturing: FedEx, Xerox, Packaging Corporation of America, 3M and others used our modeling solutions, Government: US Internal Revenue Service, Public Work Supply and Service, Canada, US Senate and US Department of Agriculture used our modeling solutions for capacity planning and performance management, Energy: South California Edison, Public Services Company of Colorado and American Electric Power used our modeling solutions, and Health Care: Intermountain Health Care, and Medco used our modeling solution for capacity management and performance tuning and evaluation of new application implementation impact

CMG Conference 2009

2009-12-14T13:57:00.002-05:00

During CMG 2009 I presented Half a day Workshop "Hands on Workshop on Modeling and Optimization in Virtualized Multi-tier Environment"
Paper on "Capacity Management Opportunities for Oracle Database Machine Exadata v2"
and participated in panel discussion on "Role of modeling"

On Sunday I presented half a day hands on workshop . Each year I include in this workshop new examples reflecting latest announcements and challenges in planning and managing IT resources.

Outline of the workshop this year included:
Introduction and Objectives
Simple Analytical Queueing Network Model
Modeling Inputs
How to Predict the Impact of Expected Workload Growth and Hardware Upgrade
How Modeling Helps to Evaluate Performance of New Oracle, IBM DB2 and Teradata Data Warehouse Appliances
How Modeling Helps to Set Realistic SLO
How to Justify Performance Management and Tuning Measures
How to Predict Impact of New Application Implementation
How to Verify Accuracy of Modeling Results and Organize Continuous Proactive Service Level Management
Summary and Next Steps

Objectives of the workshop were:
Learn how simple open and closed queueing network models and commercial modeling tools can be used for proactive performance management of multi-tier virtualized distributed environments
Learn how to predict:
The impact of the expected workload and database size growth
The impact of implementation of new applications
The impact of virtualization and server consolidation
The impact of database, application and software tuning
Learn how modeling can help to justify:
Strategic capacity planning decisions
Tactical performance management and database tuning decisions
Operational workload management decisions
Learn how to organize a proactive performance management:
How to compare actual results with expected
How to compare options and develop timely corrective actions

Workshop is based on usage of examples presented on Excell spreadsheets.

Participants were answering the following questions:
Hardware’s cheap, so why do modeling?
Can you give an example of challenges related to strategic capacity planning,
tactical performance management and operational workload management
What is Utilization Law?
What is Response Time Law?
What is Little’s Law?What is the objective of data collection and workload characterization?
What is a workload performance profile?
What is a workload resource utilization profile?
What is a workload data usage profile?
What are the typical workload aggregation rules?
How do you predict the impact of the expected workload growth?
How do you justify hardware upgrade?
How do you predict the impact of virtualization?How do you compare scalability of the different DBMS and hardware platforms?How do you set up realistic SLO?
How do you negotiate SLA?How do you justify tuning efforts?
How does a change in the number of JVM threads affect performance?
How does an increase in connection pool size affect performance?
How does reduction in level of concurrency for one workload affect performance of other workloads?
How do you collect new application performance data?
How do you predict new application performance prior to implementation on a production system?
How do you predict what impact implementing a new application will have on current production workload?How do you improve modeling accuracy?
How do you verify modeling accuracy?
How do you set up realistic SLO and negotiate SLA?
How do you organize proactive SLM?
At the end of workshop participants have enough material to present a formal capacity management report with findings and recommendations.

Tuesday morning I presented a join paper with Charlie Garry on "Capacity Management Opportunities for Oracle Database Machine Exadata v2"
On September 15, 2009 Oracle announced the world’s first database appliance designed to run both OLTP and data warehousing workloads. Oracle’s Database Machine V2 is based on Sun hardware utilizing commodity components and x86 processors.
In this presentation we will review architecture and functionality of Database Machine enabling high performance and scalability and will discuss challenges of strategic capacity planning, tactical performance management and operational workload management for this environment.

In this paper we discussed:

Intro to the Oracle Database Machine V2
How to measure performance?
Challenges of the workload characterization
How to predict performance and justify capacity planning, performance management and workload management solutions
Case study, including:
How to define the best strategy, tactics and workload management to support SLOs effectively?
What will be the impact of the expected workload growth and changes, and when will the current system be out of capacity?
What will be the impact of implementing Oracle DB Machine v2?
What will be the impact hardware upgrade?
What will be the impact of performance tuning?
What will be the impact of new application?
What will be the impact of limiting CPU utilization?
What will be the impact of changing workload concurrency?
What will be the impact of changing workload priority?
What is the best plan of action?
We empasized that:
Oracle Database Machine includes smart scan, columnar data compression and flash memory significantly improving performance of storage subsystem and enabling concurrent support of OLTP and BI/DSS workloads
It includes mechanism allowing to change priority, resource allocation and concurrency for individual workloads
We demonstrated how BEZVision modeling and performance optimization results can be used to justify strategic capacity planning, tactical performance management and operational workload management decisions
It enables organization of the continuous proactive performance management process.

Our Paper at Teradata Partners Conference

2009-10-28T19:48:00.002-04:00

At the recent Teradata Partners conference, Teradata announced the Extreme Performance Appliance 4555 - the new first solid state data warehouse and cloud computing device. In contrast to Oracle's database machine which uses up to 5 terabytes of flash storage, Teradata is using solid state drives which improve performance for both read and write operations.

According to Scott Gnau, Teradata Chief Development Officer, the new appliance based on multi-core Intel processor technology and the 64-bit SLES operating system will allow scaling from seven to 200 terabytes of user data.

Teradata announced versions of its Teradata Express software for Amazon's Elastic Compute Cloud (EC2) and VMware Player. Teradata Express provides Teradata developers and testers access to a database at no charge. This announcement directly competes with Greenplum’s "Enterprise Data Cloud" strategy.

Planning and managing a DW environment is difficult. Teradata TASM simplifies creation of rules to manage performance of mixed workload environments, but it can still be difficult to select the right TASM parameters capable of satisfying SLO for each workload. In our joint paper “Capacity Management and Optimization in TASM Environments” co-authored with Doug Brown we demonstrated that:

• It is easy to change TASM settings, but it is difficult to decide how to change values to satisfy SLGs for each workload.
• Modeling and optimization technology can be used to justify strategic capacity planning and tactical performance management decisions and set TASM rules to satisfy workloads SLGs
• Workload characterization and performance prediction results can be used to justify realistic SLGs, set throttling, priorities and resource allocation TASM rules and organize a continuous process of proactive Service Level Management.

Progress in technology provides many options to decision makers for planning, managing and controlling performance of critical applications supporting business processes. Even with the currently available tools, it is still difficult to evaluate different options and make the right decisions. The role of modeling and optimization is to automate the process of evaluation and provide information to justify capacity planning, performance management and workload management decisions and enable verification of actual results with expectations while helping define a process for continuous proactive performance management.

Our Presentation at Oracle Open World 2009

2009-10-28T19:42:00.003-04:00

Over 40,000 people from over 120 countries attended Oracle Open World 2009 earlier this month. One of the most important announcements during the showw was Oracle OLTP DB Machine V2. Oracle conducted several benchmark tests and demonstrated excellent performance for both OLTP and BI DW applications.

On Tuesday, I presented a paper that I co-authored with Alex Lupersolsky on “Modeling and Optimization for Multi-tier Virtualized Oracle Environments”. In this paper, we reviewed the challenges of planning and managing a complex environment, where easy-to-add hardware, changing software parameters controlling workload concurrency, priorities, and allocation of CPU and memory resources are all available but it is difficult to make decisions which will satisfy SLOs effectively.

We also reviewed several case studies illustrating the impact of workload growth and evaluating different options, including creation of RAC and an Oracle OLTP Database Machine V2.

We analyzed modeling results predicting the impact of migration ETL, OLTP, BI and archiving workloads to Oracle DB Machine v2. We used the following measurement data to build the model:

1) For each RAC node we used performance measurement data contained in GV$ views and Oracle OEM Grid Control:
• total physical CPU utilization
• the number of CPUs used
• total I/O rate in IOps and KBps
• the number of OS-visible disks
• read/write ratio
• average I/O operation response time

2) for each database instance per workload element (User/Program/Machine/Module):
• #executions
• total or average server response time per execution
• average number of parallel sessions (client sessions existing at the same time)
• parsing and execution CPU time consumed
• # physical IO operations with storage
• GV$ views contain information about master and slave sessions running in the same or different instances allowing us to estimate average intra-request parallelism and an average amount of data transferred between master and slave sessions trough a "node interconnect"

3) For each Exadata cell:
• arrival rate/throughput in number of SQL requests/hour,
• average response time,
• CPU utilization,
• number of logical and physical I/Os per hour per User/Program/Machine/Module

We discussed how modeling and optimization can be used to compare alternatives, justify and verify operational workload management, tactical performance tuning and strategic capacity planning decisions to ensure SLO support for the critical workloads.

We illustrated the importance of workload management. Without any constraints, low priority ETL workloads can monopolize resources. Workload management, database tuning and hardware configuration changes can all improve performance for one workload, but they also carry the risk of moving rather than eliminating bottlenecks and negatively affect other workloads. Strategic capacity planning, tactical performance management and operational workload management decisions should take into consideration the interdependence between servers and workloads and virtualization overhead.

It is impossible to manually evaluate all of the possible permutations of changes in concurrency, priority or resource allocation, database tuning or hardware upgrade options. We demonstrated how comparing the actual with expected results, based on modeling and optimization, enables organizations to practice continuous, proactive service level management.

Challenges of Teradata Workload Management in TASM Environment

2009-08-31T23:58:00.030-04:00

It is easy to change Teradata Active Systems Management (TASM) parameters affecting workloads priorities, concurrency and resource allocation settings, but it is difficult to decide how to change values to satisfy Service Level Goals for each workload.

Modeling and optimization technology can be used to justify not only strategic capacity planning, tactical performance management, but also operational workload management TASM parameters to satisfy workloads SLGs.

Let's review how workload characterization and performance prediction results can be used to justify realistic SLGs, set Concurrency/Throttling, Priorities and Resource Allocation TASM rules and organize continuous proactive Service Level Management.

Reducing the level of concurrency/throtling reduces the number of concurrently processed requests (Multi Programming Level (MPL) ), but it increase the number of requests waiting for the tread as it shown on Figure below:

One of the challenges is to find for each workload the approximation of the distribution of the probability requests in the system, number of requests waiting for service and number of requests being processed.

Below are performance prediction results illustrating how throttling for Batch workload can improve performance of other workloads

Change of priority for one of the workloads can improve it's performance, but negatively affect the performance of other workloads.

One of the approaches is to reduce priority for the not critical workloads using excesive amount of resources.

Hardware upgrade, change of the DBMS or OS release can change balance in usage of resources and it require reevaluation of the workload management TASM parameters.

Below are performance prediction results illustrating the impact of the proposed hardware upgrade and change workload management TASM parameters.

As we can see the challenge in Teradata workload management is to coordinate selection of TASM parameters to satisfy SLGs for each workload.
Modeling and optimization technology can be used to justify strategic capacity planning and tactical performance management decisions and set TASM rules to satisfy workloads SLGs
Workload characterization and performance prediction results can be used to justify realistic SLGs, set Throtteling, Priorities and Resource Allocation TASM rules and organize continuous proactive Service Level Management.

Hot Summer

2009-08-23T11:12:00.012-04:00

During last several months, we've seen a significant burst of activity. Many customers, in spite of the budget cuts, are starting to evaluate how to streamline and optimize their IT operations. In the next couple of postings, I will describe several examples illustrating how analytic modeling technology is used to justify movement of workloads and data from one system to another, how modeling technology is used to reduce the risk of performance surprises during implementation of new applications, and why planning of hardware upgrades, changes of OS and migration to a new release of the DBMS should include reevaluation of the workload management rules. Next week I will be on vacation and plan to finish several papers.

I am working on paper for Oracle World 2009: "Modeling and Optimization of Virtualized Multi-Tier Distributed Environment. We will review the challenges of planning and managing complex multi-tier virtualized distributed environments with many interdependent servers supporting multiple workloads.

Any change in workload management, database tuning, or hardware upgrades can improve performance for one workload while also moving one or more bottlenecks to another server on another tier and negatively affect other workloads for variety of reasons:

There many parameters you can change, including concurrency, priority or resource allocation by workload, you can change database design, create new indexes, materialized views or upgrade the hardware configuration

It is impossible to evaluate all possible permutations of parameters

We will discuss how modeling technology can answer specific "what if" questions

We will also review how optimization technology iteratively and intelligently generates "what if" questions for the modeling engine to find what should be changed within workload management, performance tuning and hardware upgrades to satisfy SLOs for critical workloads

We will also review how comparison of the actual results (after the change) with expected results enables organizations to implement a continuous proactive performance management process

Another paper I am preparing for the upcoming Teradata Partners Conference about the application of modeling and optimization for workload management and creation of the continuous, closed loop proactive performance management titled "Teradata Infrastructure Optimization in TASM Environment". In this paper we will discuss:

Challenges of setting workload management Teradata TASM parameters

The role of modeling and optimization in finding optimal workload management parameters to meet Service Level Goals for each workload

Workload characterization in TASM environment

Strategic capacity planning in a TASM environmen

Tactical performance management in a TASM environment

Operational workload management in a TASM environment

How to optimize the selection of TASM throttling, priority and resource allocation rules based on SLG for each workload

How to use performance prediction results to organize a continuous, closed loop proactive performance management process in a TASM environment

For CMG 2009 I am preparing a half day session titles "Hands on Workshop on Modeling and Optimization in Virtualized Multi-tier Environments". This is an intensive "hands on" workshop for performance management professionals who would like to learn how to build and apply analytic models to proactively manage the performance of applications in virtualized multi-tier environments based on VMware, WebLogic and WebSphere Application Servers as well as Oracle, DB2, Teradata and SQL Server Database Servers. During the workshop, attendees will learn how to build and apply analytic models to predict the impact of workload and database size, growth, the impact of implementing new applications, adding or moving VMs and upgrading hardware. We will not use our commercial modeling tool, BEZVision, but instead I will teach attendees how to use an Excel spreadsheet with prepared exercises to illustrate how to perform workload characterization, build simple analytic queueing network models, and apply modeling results to justify strategic capacity planning, tactical performance management and operational workload management recommendations. At the end of the workshop, participants will summarize results and will be ready to present a report with capacity management recommendations.

In addition, Tim R. Norton invited me to participate in a Panel Discussion at CMG titled "Hardware’s Cheap so Why Do Modeling?". I will be preparing some materials for this panel as well. The cost of hardware is rapidly trending down while other costs are rising even more rapidly. The result of this interplay is that cost saving opportunities are shrinking while the analysis takes increasingly more time, effort and money. This panel of world-renowned experts in application and systems modeling will candidly discuss this and other questions related to the future of modeling as a tool to achieve business objectives. Panel discussion areas:

Hardware’s cheap so why do modeling at all?

Does analysis cost more than just buying the hardware?

How close is good enough?

Hardware’s evermore powerful so why try for prediction precision?

Business vs. Math: What’s the trade-off between political costs and technical value?

How can the modeler find the tipping-point?

How does application and infrastructure complexity affect the value of modeling?

Is there such a thing as a “simple” model anymore?

Is modeling headed to the clouds?

Is traditional modeling at odds with the utility model of cloud computing?

Where’s the value as datacenters move toward commodity pricing and “on demand” capacity?

What’s driving the costs du jour?

Can a modeling analysis effort be successful before it is superseded by the next management priority?

How can modeling optimize multiple mutually exclusive objectives?

My wife does not know yet, but if I have time left between hiking and finishing papers, I have an obligation to prepare an abstract for CMG on a Late Breaking paper with Charlie Gary on "New application infrastructure modeling and optimization"

Predicting New Application Implementation Impact

2009-07-15T20:34:00.005-04:00

Business people, application developers and IT management are all concerned that even after thorough stress testing, new applications in production environments will not perform as expected and they may negatively impact existing production applications. Indeed, HP LoadRunner, Jmeter and other stress testing software can evaluate the impact of increasing the number of users in pre-production environments, but it can not evaluate the impact of implementing new applications on systems with different architecture, different hardware, software and DBMS platforms. For example, a significant increase in the volume of data, a change of the policy and rules of distribution of resources in virtualized application server environments (priorities, concurrency and resource allocation) and changes in the workload management policies of DBMS servers can all shift bottlenecks from one tier to another tier, or server, and negatively affect performance of new and existing applications.

Oracle Real Application Testing (RAT) allows you to capture, analyze and replay production transactions on a small test system to evaluate the impact of upgrades and system changes, including implementing a new OS or DBMS patch or version, the impact of the performance tuning, the impact of Database upgrades, patches, parameters, schema changes, configuration changes, such as conversion from a single instance to RAC, ASM, etc.

DBAs can test and upgrade data center infrastructure components. In fact, the goal of RAT is to assist DBAs in testing and identifying the full impact of upgrades and system changes and include them in a certification process.

Value of new application certification

· Identify potential problems with new application and justify changes required to be sure that new application will perform well and to be sure that existing applications will be able to meet their SLOs after new application implementation
· Organize collaboration between business people, application developers and IT management in setting realistic SLO, negotiating SLA and organizing proactive SLM
· Provide a basis for comparison of the actual with expected results and organizing a continuous Proactive Performance Management (PPM) process during application life cycle

Collaborate09 IOUG Conference

2009-05-17T18:47:00.003-04:00

I April I presented paper on Modeling and Optimization for Multi-Tier Virtualized Oracle environment at Collaborate09 IOUG conference in Orlando. In spite of the bad economy and swine fly epidemic danger the Collaborate09 attracted about 4500 people and over 200 exhibitors. Conference had many tracks divided into 3 major sub conferences, including International Oracle User Group (IOUG), Oracle Application User Group (OAUG) and Quest User Group united EJ Edwards and People Soft users.

I attended Oracle’s presentations on new development in Oracle Enterprise Manager and Grid Control. It contains valuable information for modeling and performance optimization of the multitier virtualized distributed environment.
I had an opportunity to see new development in the systems management area presented by Oracle, IBM, CA, HP, BMC. All of them have repositories, containing performance measurement data with valuable information for building and calibrating analytical models.
Several speakers presented papers about challenges in planning and managing virtualization. VMware introduced performance measurement data characterizing the Hypervisor overhead. several presenters described their experience in virtualization of the DBMS servers.

Customers like the fact that virtualization can potentially reduce cost hardware and software by more than 50%.

We saw attempts to virtualize DBMS servers supporting test and development environment, where there performance level is not so critical. Most of the customers are still skeptical about virtualization of the DBMS servers in production environment.

MODELING AND OPTIMIZATION IN VIRTUALIZED MULTI-TIER DISTRIBUTED ENVIRONMENT

2009-03-21T22:22:00.030-04:00

At the end of April, Cisco will announce their Unified Computing System (UCS) which will compete with HP, IBM and Dell. This is part of Cisco's plan of partnering with EMC. EMC is planning to announce a new Symmetrix and VMware device. VMware is also planning to announce VMware vSphere 4.0 at the same time on April 21. (EMC's announcement was actually reported by the Boston Globe on April 14). Cisco is also buying Tidal software which is focusing on job scheduling, application performance management, and automation software products.

It is clear that virtualization solutions are becoming cheaper, but complexity of planning and management of the multi-tier virtualized environment increases the risk of surprises. Systems management tools will play an important role in competition between virtualization and cloud computing solution providers.

In order to avoid surprises, you will need to know what the impact of virtualization and cloud computing is on the performance of your applications. Analytical queueing network models can be used to evaluate capacity planning, performance tuning and workload management options, provide reasonable expectations and justify proactive performance management decisions.

We will review how prediction results can be used to set realistic Service Level Objectives, find the most effective solutions, set expectations, verify the results and organize proactive Service Level Management. We will discuss how performance prediction can be used to find the right candidates for virtualization, justify hardware and software upgrades for the application tier and DBMS tier, optimize VM migration, predict the impact of new VMs and new workload implementation, set optimum level of concurrency, priorities, and resource allocation for each workload to support critical workloads’ SLOs with minimum cost.

Virtualization in a Multi-Tier Distributed Environment
Virtualization can reduce cost, but hypervisor overhead can negatively affect performance. As a result, not all applications are good candidates for virtualization. For example, applications with high I/O rate can have significant performance degradation after virtualization. Hypervisor overhead that depends on the number of VMs and workload parameters can affect performance of all applications.

Workload and database size growth, implementation of the new applications, adding new VMs can increase hypervisor’s CPU, memory and I/O overhead and can negatively affect applications performance.

We will review how modeling and optimization technology can be used to evaluate options and justify strategic capacity planning, tactical performance management and operational workload management decisions, verify results and enable organization of continuous proactive performance management.

Response time in the multi-tier environment includes service time and queueing time for CPU, I/O and interprocessor communication in application servers and DBMS servers, plus different types of delays caused by limited concurrency. Workloads have different resource utilization profiles.

Complexity of requests, volume of data and processing speed all affect service time. Virtualization overhead of managing VMs affects CPU service time, elongates I/O response time due to hypervisor scheduling overhead.

Increase in number of users, implementation of new applications, and concurrency limitations increase contention for resources and affect queueing time.

Workloads are interdependent because they compete for the shared resources, and changing the priority of one workload can improve its response time, but it can increase queueing time and response time for others.

Let’s review the application of modeling and optimization of the simple configuration shown in Figure 1. Application servers have unbalanced usage of resources by Java EE applications. Java EE applications generate SQL requests accessing data from Oracle DBMS servers. We will predict the impact of virtualization, replacing the servers with VMs placed on one physical server. Each JVM running the application server software has a limited number of execution threads and a limited number of connections to the databases. JVM thread pool size and connection pool size limit the number of requests that can access DBMS concurrently.

Figure 1. An example of Server Consolidation and Virtualization. Two physical application servers are replaced by two VMs in one physical host.

In order to support workload growth and implementation of new applications, you can add a VM to an existing physical server or create a cluster of application servers and place the new VM in a new physical server. Decisions about migration of VMs between physical servers, change of a VM’s priority, tuning decisions, and change of concurrency should take into consideration that all components of the system are interdependent, and a change in one place can move a bottleneck and affect other workloads.

VIRTUALIZATION IS FOUNDATION OF CLOUD COMPUTING

Virtualization is the first step toward Cloud Computing. An individual VM can run on a desktop or can be moved to a private Cloud inside the Computer Center where the VM will run on a so called “software mainframe” – a shared, high availability, high performance computing platform based on distributed physical machines. For example, VMware VMotion can move a hosted operating environment from one physical machine to another.

Workloads in this environment can then be moved from an in-house private Cloud to an Internet Accessible Cloud provider. New capacity can be added to the VM runtime environment when needed during high peak processing within the physical server, or VMs can be moved to a bigger physical machine using VMotion, and finally, the total physical capability of the Cloud hosting environment can be increased as well.

The customer’s challenge is to decide what type of workloads should be placed on an in-house private cloud and which applications should be moved to the Internet accessible cloud provider, and which provider should be selected to support SLO with minimum cost.

The provider’s challenge is to manage concurrently different workloads, dynamically move VMs and reallocate resources depending on SLO and actual activity within VMs. New capacity can be added to the VM run time environment during high peak processing within the physical server. VMs can be moved to the bigger physical machine. The total physical capability of the Cloud hosting environment can be increased as well.

Figure 2. Virtualization is a first step toward Cloud Computing. Within the data center there are traditional computing resources and private clouds. Applications can be moved back and forth to external an Cloud Host Environment. One of the challenges is to find candidates for virtualization. Another challenge is workload management and migration of VMs between physical hosts within private cloud of the data center. And finally, a decision has to be made about which applications and when they should be moved to an external cloud host environment.

Challenges of modeling virtualized multi-tier environments

Closed queueing network models based on MVA algorithms can be applied to modeling Java EE applications in multi-tier distributed environments with Oracle DBMS.

The queueing network model of clusters of virtual servers with hundreds of VMs containing JVMs with limited number of JVM threads and limited connection pool sizes affecting the flow of requests between application and DBMS server can be very complex. On the other hand, the structure of each individual server model is the same. We will review use of the hierarchical modeling approach where the lower level tier is treated as an additional I/O device (like disk) for the current tier.

We apply a central server, closed queuing network model to model each server. An iterative multi-class Mean Value Analysis algorithm has been modified to reflect intra-request parallelism and take into consideration the software limitations on the number of requests that can be concurrently processed by the server (MPL) and limitation of the CPU utilization by workload.
Modeling parameters are dynamically recalculated during the modeling iterations to reflect contention for memory between requests:

• Response time of DBMS server workloads is calculated during workload characterization based on measurement data extracted from SQLArea for SQL requests that belong to the corresponding workloads.

• Response time of the workload’s request to application server includes the application server’s own response time and total time the request will spend in the DBMS server, calculated as multiplication of the average DBMS response time for corresponding workload by the number of calls to the DBMS server per one request to the application server.

• Measured response time of the Web server includes average response time of the Web server plus corresponding response time of the application server.

• Calibration results synchronize and coordinate all model parameters, correct inaccuracies of measurement data and workload characterization assumptions.

• To be able to calibrate each server independently, calibration should fix external parameters: response time, throughput, think time and the average number of processes generating requests. Only CPU utilization and I/O rate are redistributed between workloads while preserving total node metrics.

Figure 3. Iterative Modeling Algorithm Applied to Interdependent Queueing Network Models of Servers in a Multi-Tier Distributed Environment.

If one physical server hosts both VM/AS and VM/DBMS servers, the interconnection between VMs should be taken into consideration. Bottom-up effect incorporates next tier delay change for calling server workloads. Top-down effect reflects change of concurrency level (equivalent number of parallel sessions) and equivalent think time of workloads on the called server. Queueing network MVA algorithm of each server should take into consideration Regression Analysis coefficients to predict how different planned changes will affect response time (service, queuing and delays), throughput and resource utilization for each workload.

PREDICTING IMPACT OF THE WORKLOAD AND DATABASE SIZE GROWTH

The graph below shows the result of performance prediction showing the impact of the workload and database size increase in a multi-tier environment with the two application servers and two DBMS servers shown in Figure 1. As you can see in Figure 5, the response time of one of the workloads will not be affected, but the response time of the Admin Cluster workload will be significantly increased.

Predicted Impact of Workload and Database Growth on Response Time and Throughput without Virtualization

Figure 4. As a result of the expected workload growth the response time for the Physical Cluster and Admin Cluster workloads will grow significantly, but other workloads’ response times will not be significantly affected. Throughput for all workloads, except DBora1-Catch All will be reduced by 10-30% in a year.

One of the areas where analytical queueing network models can provide value is selection of candidates for virtualization. Planning virtualization includes analysis of many alternatives, including selection of candidates for virtualization, justification for the size and number of the physical servers required to support selected VMs, making decisions about which VMs should be placed on which physical servers, etc.

Let’s review performance prediction results based on measurement data collected on the system shown in Figure 1. The CPU of Application server #1 is underutilized, while the utilization of Application server #2 CPU is almost 100% (Figure 6). We have unbalanced application servers, and one of the solutions is to evaluate the impact of virtualization: what if we place Application server #1 workloads into VM1 and workloads of Application server #2 into VM2 onto the physical application tier server #2?

Predicted Impact of Workload and Database Growth on CPU Utilization of Application Server #1 and Application Server #2 Without Virtualization

Figure 5. Application server #1 is underutilized, but Application server #2 is saturated. CPU utilization of Application server #1 will be growing from 7% to almost 12%, and Application server #2 CPU utilization will be saturated at the level of 100%.

Predicted Impact of Virtualization
Replacement of the physical servers with VMs and placement of the VMs on one host with double CPU capacity is shown in Figure 7 where the hypervisor controls VMs to balance the utilization of resources, but the impact of the workload growth on performance of applications is a concern. Analytical models take into consideration workload profiles [5], and also the hypervisor overhead, which will be increasing with workload growth. Performance prediction results shown in Figure 6 illustrate that response time and throughput of different workloads will be affected differently depending on the workloads’ profiles. Prediction results set realistic expectations, reduce risk of surprises and provide the information to review different proactive performance management measures, including change of workloads’ priorities, level of concurrency, resource allocation, etc.

Figure 6. Physical Cluster and Admin Cluster workloads response time and OE workload throughput will be very sensitive to workload growth, and CPU utilization of the host server will be increased from 60% to 85% in a virtual environment with two VMs, representing workloads from Application servers 1 and 2.

Predicted Impact change of Workload Priority

Performance prediction results can be used to identify when the SLO for critical workload is not satisfied, where there is a potential problem (is it the application server or the DBMS server, CPU or I/O, service time or queueing time, or are delays caused by concurrency limitations?, etc.). One of the possible tuning options is to increase dispatching priority for the critical workload. Figure 8 illustrates the performance prediction results reflecting the impact of the proposed change of the priority for one of the workloads. As a result, this workload will have improvement in response time and throughput, but it will negatively affect performance of other workloads. The model takes into consideration that all workloads compete for resources, and improvement in one place can create a bottleneck in another. Sometimes improved performance on an application server can create a bottleneck in a DBMS server and vice versa.

Figure 7. Increase of the priority for VM hosting Physician Cluster workload will improve response time for this workload, but other workloads, especially Admin Cluster workload, will be negatively affected. As a result of the change, throughput and CPU utilization by each workload will be affected as well.

Predicted Impact of Concurrency limitation

Enforcement of concurrency limitation for one of the workloads limits the number of concurrent requests for this workload and can have a very different impact on all other workloads. Reduction in the number of JVM threads in the application server can limit resource consumption for one workload, but increase consumption of resources by other workloads using a different JVM. It can move a bottleneck from the application server to the DBMS server. Performance prediction results in Figure 8 illustrate the change in response time, throughput and CPU utilization as a result of change in level of concurrency for the Physician Cluster workload.

Figure 8 . Implementation of throttling and limitation of the level of concurrency for Physician Cluster will increase response time for the Physician Cluster workload, but will significantly improve the response time for all other workloads.

Predicted Impact of Change in CPU Consumption Limit

Limitation of the CPU utilization by one of the workloads can have a similar impact. As is shown in Figure 9, setting a limit on CPU consumption on the DBMS server for the OE workload will increase the response time and reduce throughput for the OE workload, but significantly improve performance for other workloads, especially DBora1. Modeling results take into consideration not only contention for resources of the application server, but also the impact on performance of the DBMS servers.

Figure 9. Setting limit on CPU consumption on the DBMS server for the OE workload will increase the response time and reduce throughput for the OE workload, but significantly improve performance for other workloads, especially DBora1

Figure 10. As a result of adding a new node to an Oracle RAC system, response time improved and throughput increased. CPU time consumed increased because the throughput increased.

Performance prediction results allow evaluation of the impact of the proposed Oracle RAC hardware upgrade on each workload. Shared disk subsystems, variable degree of parallelism, contention for the interconnect, and memory limitations can affect Oracle RAC scalability. Potentially, it also can limit the ability of RAC to provide consistent service for dynamic environments with mixed workloads.

Additional nodes will allow redistribution of requests between nodes, and the arrival rate to each node will diminish, which will reduce average CPU utilization, and each node will process requests faster. Users will wait less time for the response and will be able to generate more requests. It is positive, because the system will be able to process more business transactions – systems’ throughout will increase.

According to the performance prediction results, after adding a new index, performance of one of the workloads will be improved, but several workloads that do not include SQL accessing tables with the new index will experience performance degradation.

Figure 11. Prediction results show different expected improvements of the new index creation on response time for different workloads. OE and DBora1 workloads will benefit the most, while Admin Cluster and Physician Cluster will not. Modeling expectations provides the basis for comparing the actual results with expected, and to verify that the goal of the change is reached.

Predicted impact of adding new vm containing new application dentist on new virtual server

Tuning will reduce contention for storage subsystems and DBMS throughput will increase. Suddenly, the maximum number of JVM treads will become a bottleneck. Increasing the number of JVM threads will increase the number of concurrent requests within the application server and the amount of heap memory used by all concurrent requests. Heap size is limited to 2 GB, and creation of an additional JVM will be required to support the increased number of concurrent requests. Creation of a JVM within the same application server will increase contention for the CPU, and adding a new physical application server will be required. Creation of a new JVM on a new application server will balance the application servers’ utilization and reduce time requests spent within the application tier, but it will increase the arrival rate of requests to the DBMS tier, and increase contention for the DBMS server again. The DBA can decide to increase the degree of parallelism or change priority and resource allocation for one of the workloads, but it will affect different workloads differently. Some of them may benefit by that, but some of them may not. Modeling results show that adding a new VM containing a new application accessing data from the same DBMS server will increase the contention for the DBMS and affect performance of all workloads.

Figure 12. Predicted impact of adding a new VM containing a new application DENTIST on a new virtual server accessing data from the same DBMS server

OPTIMIZATION AND AUTOMATION OF THE PERFORMANCE MANAGEMENT DECISIONS

Each workload has unique performance, resource utilization and data usage profiles. When changing the hardware and software configurations, applications and database tuning can affect performance of workloads differently. Finding the best configuration and rules defining concurrency, priority, resource allocation and migration of VMs and JVMs between virtualization servers to support individual SLOs is a very difficult task.

Analytical models can be used to evaluate different options to justify workload management, performance tuning and capacity planning decisions [7,8,9]. To make the model independent from the number of servers in the system, we can build models hierarchically. Each server is modeled by the separate queuing model where called servers are included as additional data sources, and calling servers determine the equivalent number of users (sessions) and the equivalent think time. The whole model is solved iteratively server by server with several iterations until convergence.

Dynamically adjusted evaluation scenarios can be used to evaluate different software parameters and find the level of concurrency, priority and resource allocation for each workload that will satisfy SLO for each workload, and minimize Total Cost of Ownership (TCO).

The algorithm starts with evaluating the impact of the software parameters change. If changing the software parameters is not sufficient to satisfy SLO, then migration of VMs and JVMs to balance usage of resources is evaluated. If that is not enough, then hardware a capacity increase on overutilized servers is considered.

We can describe such optimization as a multi-criteria (each workload has own SLO), multi-dimensional (system and workloads’ software parameters and hardware parameters) optimization. It’s not possible to optimize for any workload separately because they all use the same physical resources, thus, affect each other. There’s not even approximate analytical expression that link variables and goal functions. We have to run the multi-tier model for each and every combination of software and hardware parameters of all workloads and all servers to get the corresponding performance metrics and compare them with the SLO.

The following steps make the search for the optimum solution more effective:

1. Select a workload where the SLO will not be met first
2. Select the server corresponding to the greatest component of the workload’s system response time.
3. Look at the components of the workload’s request elapsed time on this server and make corresponding actions: if the request spends the most time on (or waiting for) CPU, increase the workload’s CPU limit or priority; if the request spends the most time waiting for execution thread, increase the number of threads available, if the available memory allows (application tier, there are similar concurrency limitation in DBMS as well); etc.

Each such change affects all workloads, so after the model runs another workload can violate its SLO earlier or another server can become a bottleneck.

If after workload control adjustments all SLOs are not satisfied, we turn to the server-level software parameters mostly available for virtual servers: CPU share, memory share (affects swapping IO rate), etc. Finally, if all attempts to change software parameters, move VMs and JVMs cannot satisfy SLO, a hardware upgrade is evaluated.

SUMMARY
Complexity of virtualized multi-tier distributed Oracle environment makes it difficult to plan and manage dynamic environments effectively. We presented a methodology and approach to modeling of the complex, multi-tier distributed environment with virtualization. We demonstrated how modeling and optimization improves effectiveness and reduces risk of performance surprises during planning and management of the virtualized, multi-tier distributed Oracle environments. We reviewed how to model the impact of the workload growth and other changes on hypervisor overhead. We demonstrated how performance prediction and optimization technology allows evaluation of different options, setting realistic SLO, finding virtualization candidates, predicting the impact of workload growth and adding new VMs, justification of migration of VMs, predicting impact of new applications implementation, justification of the application tier servers and Oracle DBMS servers hardware upgrades and provides a framework to organize a continuous proactive performance management process.

REFERENCES

B. Zibitsker, IOUG 2008. Reducing Risk of Surprises in Changing Multi-tier Distributed Oracle RAC Environment
B. Zibitsker, DAMA 2007, Enterprise Data Management and Optimization
B. Zibitsker, CMG 2008, Hands on Workshop on Performance Prediction for Multi-tier Distributed Environments
J. Buzen, B. Zibitsker, CMG 2006, Challenges of Performance Prediction in Multi-tier Parallel Processing Environments
B, Zibitsker, G. Sigalov, A. Lupersolsky Modeling and Proactive Performance Management of Multi-tier Distributed Environments, International conference "mathematical methods for analysis and optimization of information and telecommunication networks" (Byelorussian Winter Workshop in Queueing Theory – 2007)
Mark Friedman and Stephen Marksamer, Measure IT, March 2007 A Realistic Assessment of the Performance of Windows Guest Virtual Machines
Nocedal, Jorge. Stephen J. Wright, Numerical optimization, ISBN 0-387-98793-2
Michael W. Trosset, Numerical Optimization Using Computer Experiments, Adjunct Associate Professor, Department of Computational & Applied Mathematics, Rice University, Houston, TX, Virginia Torczon, Assistant Professor, Department of Computer Science, College of William & Mary, Williamsburg, VA
Charbonneau, High altitude observatory, national center for atmospheric research, Boulder, Colorado

Optimization of Teradata Data Warehouse Performance and Reducing the Cost

2009-02-24T09:30:00.003-05:00

Today we'll review how modeling and optimization can be used to automate data warehouse workload management, improve efficiency and reduce cost. In a Teradata environment, a DBA can set Teradata Active Systems Management (TASM) rules to change workload priorities, concurrency and resource allocation depending on the time of day and resource usage. TASM simplifies setting rules, but does not help to do it right. As a result, there is uncertainty and high risk of surprises during execution of these rules.

One of the many challenges affecting management decisions is setting realistic goals. We will review how workload characterization and performance prediction results can be used to set realistic Service Level Objectives (SLO) for each workload. We will also discuss how modeling results can be used to justify strategic capacity planning, tactical performance management and operational workload management decisions. And finally, we will review how modeling technology and iterative optimization can be used to automate operational workload management and organize a continuous proactive performance management process, reducing cost and risk of surprises.

Challenges of Managing Mixed-Workload Environments:

Difficult to justify strategic capacity planning decisions, set realistic SLO and negotiate SLA in constantly changing environment

Difficult to justify tactical performance management decisions, preventing performance degradation for most critical workloads, identify what should be tuned in applications and databases, identify opportunities for new indexes, JI, PPI, data compression

Difficult to define operational workload management changes, finding the optimal concurrency/throttle for each workload and set optimal priorities to satisfy individual workloads SLO

Difficult to verify if changes were successful - If there is no expectation prior to changes, then it is impossible to determine if changes were successful

Teradata Active Systems Management is a goal-oriented automatic technology supporting performance tuning, workload management, capacity planning and configuration management. It is easy to set up and change TASM rules, but it is difficult to find the right parameters capable of satisfying SLO for major workloads.

This example shows how throttling rules limit the number of concurrent requests to 20 by processing different workloads with different priorities and by changing the priority based on usage of resources.

If an administrator selects a small value for throttling, the limited number of requests are processed concurrently which reduces utilization of resources and requests are going thru the system faster. However, many requests are waiting longer for an available thread.
Increasing allocation group weight can improve the performance of all requests for one workload, but it can slow down the time of processing requests of other workloads.
The number of possible combinations of setting threads and allocation of the weight for multiple workloads is very large and selection of the parameters capable to satisfy SLO for different workloads is a challenging problem.

Measurement data collected by Teradata can be extracted from ResUsage, DBQL and AmpUsage and after aggregation they can be used as input to a performance prediction engine. An expected growth and change plan will dictate what kind of "What if" questions should be evaluated.
Critical SQL, which will use an excessive amount of resources in the future, can be processed by a DBMS wizard to determine database tuning candidates.
Options provide a list of hardware, software and DBMS options which can be used to change the system.
Performance prediction results show how expected changes will affect performance RT and THR for each workload.
A performance optimization engine allows automating evaluation of the many options to find the most effective solution.

Let’s review how to organize a continuous proactive performance management process based on optimization of operational, tactical and strategic capacity planning decisions

Optimization of the Operational Decisions

In Data Warehouse (DW) environment, DBAs can dynamically change priorities and the level of concurrency for individual workloads. Modeling results can provide value in evaluation of the different options for changing priorities, resource allocation and level of concurrency for individual workloads and help to select the most effective option, satisfying SLO of the different workloads.

Continuous data collection and workload characterization can detect the velocity of change of response time, throughput and usage of resources and usage of data by individual workloads and predict when the current configuration and set of software parameters will not be able to satisfy SLO. Performance prediction results can be used to evaluate different options and proactively select the steps required to avoid the problem.

Throttling for the Sales workload will improve its response time, but will have negative impact on others. When # of workloads is greater than 10, the number of permutations of assigning throttling is so high that setting Throttling rules becomes very difficult.

ETL processing continuously loading and backing up data use significant amount of resources. If the number of concurrent ETL utilities is relatively low, it can take too long to load data. If the number of concurrent ETL utilities is high, it can negatively affect the performance of OLTP and BI/DSS applications retrieving data. The number of utility slots assigned to Data Load can be increased during night time or varied depending on utilization of the system resources. The number of AMP Worker Tasks and the dispatching priority of MultiLoad and FastLoad utilities can be changed dynamically. TASM simplifies the mechanism of making the change, but it does not offer advice for setting up the parameters effectively to satisfy SLO of major workloads.

Performance prediction results shown on the Graph show that planned change of ETL Sales data load priority will improve data loading performance, but it will negatively affect performance of other workloads.

Optimization of Tactical Decisions
Analyze workloads profile and identify significant changes in performance, usage of resources and data

Use modeling results to find critical workloads, users and SQL which will cause problems in a future

Use DB wizards to identify tuning options, including new Indexes, Join Indexes, candidates for Data Partitioning, Data Compression, etc.

Use modeling to predict the impact of the different options and define tuning efforts to continuously satisfy SLO of the most critical workloads

How to Optimize Strategic Capacity Planning Decisions

Setting realistic SLO and negotiating SLA
Optimizing physical configuration satisfying SLO for major workloads with minimum TCO
How to predict the impact of a New Application
How to plan for Disaster Recovery
How to plan for Server Consolidation

Set Realistic SLO and Negotiate SLA

Service Level Objectives – Desired level of service for each individual workload typically – response time and throughput

SLO should satisfy business needs
–Throughput should be sufficient to process business transactions in time
–Response time should be good enough to satisfy users’ needs
Selection of the workloads SLO affects capacity planning decisions and TCO

What is an acceptable SLO level capable of supporting business demand with min TCO?

Organization of a Continuous Proactive Performance Management Process

Summary and Review

We reviewed major challenges of managing mixed-workload environments

We highlighted the role of predictive modeling and iterative optimization in managing performance and reducing cost

We provided several examples illustrating how modeling helps in justification of the strategic capacity planning, defining tactical performance management and operational workload management decisions

We showed how to set realistic Service Level Objectives and negotiate reasonable Service Level Agreements

We provided verification of the results and organization of a continuous proactive service level management process

Next Steps in Applying Modeling to Improving Performance and Reducing Cost

Identify major lines of business and their workloads

Build performance, resource utilization and data usage profiles for each workload

Analyze patterns and trends of usage resources

Document expectations about workload growth

Use modeling technology to evaluate how workload growth will affect performance of the different workloads to set up realistic expectations, SLO and predict TCO

In your plan, document expectations prior to major changes and compare actual with expected results

Always Remember: "A bad plan is better than none at all" - Garry Kasparov - World Chess Champion

How to Use Modeling Results to Reduce Cost

2009-02-23T10:42:00.003-05:00

I got many questions recently on how to use modeling results to reduce the cost of IT. In summary modeling results help to make more effective strategic capacity planning, tactical performance management and operational workload management decisions and control their implementation. You can manage more effectively if you know what to expect as a result of the different changes, if you can compare different options and find better decision and finally if you can verify how close are the actual results to expected.

Modeling results can be used to evaluate different options, justify more effective strategic capacity planning decisions and control the resurlts of their implementation:

Better tracking the changes in workload and data volume growth, performance, use of resources and data and predicting their impact on performance of the individual workloads can help to set realistic expectations, reduce risk of surprises and justify proactive measures
Better analysis and predicting how new applications will perform in production environment and how they will affect performance of existing applications can be used to justify proactive measures
Predicting how multiple concurrent changes of application functionality, server, data and application consolidation, movement of applications to other platforms and how it will affect SLO and SLA helps to make more informed decisions
Analysis of the performance prediction results and cost of the software licenses help to justify better decisions. Sometimes investment in new more powerful hardware technology can reduce software license cost and reduce Total Cost of Ownership
Justification of the most effective architecture, parallel processing, virtualization and hardware upgrades requires analysis of many alternatives and modeling can help to make better decisions
Analysis of the modeling results can be used to reevaluate workloads' SLO and SLA and find a compromise between business demands and budgeting constraints

Modeling results can be used to evaluate different options and find better tactical performance management decisions affecting performance of the most critical workloads:

Finding new tuning opportunities for Applications, DBMS, and software to better balance use of the exiting infrastructure, avoid problems and delay the hardware upgrades
Focusing tuning on critical workloads which are identified by modeling results as violators of SLO and SLA
Finding the critical components of the response time related to service time, queueing time and delays caused by software limitations
Finding the most resource consuming applications, users, SQL and tables accessed as candidates for tuning
Justification of indexing, data compression, materialized views, data partitioning strategies
Finding candidates and archiving dormant data
Finding and eliminating background batch processes loading, aggregating and building regular reports which never accessed
Reevaluating scheduling and finding more effective distribution of the workloads between systems

Modeling results can be used to find better operational workload management decision:

Improving scheduling, assigning workload priority and optimizing the application servers level of concurrency decisions

Modeling results enable organization of the continuous proactive performance management process

Automation of routine data collection, workload characterization and performance prediction functions reduce time which experienced analysts and DBA spend on preparing information and spending more time on making, implementing changes and verifying the results
Modeling results enable implementation of the cooperative proactive performance management by cooperative analysis of the automatically updated dashboards, and analysis of alerts and notification reports.

Hands On Modeling Workshop

2009-02-05T10:03:00.031-05:00

Yesterday, we hosted a "hands-on" Modeling Workshop for a number of attendees of the recent annual CMG meeting in Las Vegas.

The main objectives of the workshop were to learn how:"

Modeling helps improve planning and proactively managing multi-tier distributed environments
Even simple models can answer complex “What If” questions
Modeling helps justify strategic capacity planning, tactical performance management and operational workload management decisions both large and small
Comparing actual with expected results helps organize a proactive performance management plan

During the first part of the workshop, participants used Excel spreadsheets to model a simple multi-tier environment and address several typical scenarios using open queueuing network models. We were assuming that OS, application server and DBMS server measurement data were readily available, along with the hardware configuration of the system with one application server and one DBMS server, SLO and expected workload growth plan for implementing a new application.

During second part of the workshop we reviewed how the same problem can be addressed by closed queueing network models.
Below are several examples illustrating how to build very simple analytical models for Application and DBMS servers during Capacity Planning Study:

Business Perspective
Business plan - increase Sales with the rate of 20% per Quarter
SLO - Response Time

Predict the impact of workload growth
Predict when the existing hardware configuration will not be able to support SLO for Sales workload
Identify potential bottlenecks
Predict the expected impact of a hardware upgrade by doubling DBMS and Application Server CPU capacity
Predict how long the upgraded system will be able to support SLO
Predict the impact of balancing disk utilization
# JVMs, # Treads/JVM, Connection Pool Size, Memory redistribution impact
Predict the impact of a new Marketing application implementation, assuming that 10% of Sales users will use Marketing
Model a migration to Oracle 10g RAC

Prediction results show that as a result of the expected workload growth, Response Time SLO will not be satisfied. We reviewed the impact of the strategic capacity planning, tactical performance management and operational workload management decisions.

We started with evaluation of the hardware upgrade. Doubling Application server and DBMS server CPU capacity will not be sufficient, because the performance bottleneck is in the storage subsystem.

Analysis of the performance management options show that balancing Disk utilization, database tuning and creation of a new index will help improve performance, but SLO will not be met by Q4.

Implementation of parallel processing by migrating from Oracle to Oracle RAC will help achieve the goal and satisfy SLO, delaying a hardware upgrade for one year.

We reviewed how to apply models to evaluating the operational - workload management options. According to prediction results, reducing the level of concurrency - "throttling" will help to improve the response time.

During second part of the workshop, Paul Lowder reviewed modeling results generated by a commercial product, BEZVision, which incorporates the closed queueing network models.

Paul illustrated how BEZVision automates data collection, workload characterization and performance prediction functions.

BEZVision generated capacity planning and performance management advice and automatically compared the actual results with expected results. The ability to set realistic expectations for each workload and compare actual with expeted results after implementation changes enables organization within the proactive performance management plan.

During this workshop, we reviewed how to build and apply simple open and closed queueing network models to address complex "what if" questions. We also considered many factors, including accuracy of data collection, workload characterization, level of details reflecting computer system architecture, software and hardware configurtaion that can affect modeling accuracy.

Challenges of Modeling Virtual Environment

2009-01-31T12:12:00.023-05:00

There are three major challenges of modeling virtualization:

Variable Hypervisor overhead
Performance interdependence between applications within different VMs
Interdependence between servers in multi-tier distributed environment.

Workload and database size growth, implementation of the new applications, increase in number of VM within physical server increases the Hypervisor overhead in usage of CPU resources, increase memory usage, paging and swapping, elongate I/O response time due to increase delays in scheduling I/O by Hypervisor, which can negatively affect applications performance.

In a multi-tier distributed environment change of the hardware, software parameters, tuning one of the application or database, dynamic reallocation of resources for VMs and movement VMs between physical servers can affect performance of all other workloads.
Goal of strategic capacity planning, tacticl performance management and operational workload management decisions is to ensure satisfaction of SLO and SLA of major workloads with minimum cost. Multiple criteria should be taking into consideration, including performance requirements for each workload, growing and changing business demand, total cost of ownership, consistency of service, flexibility, scalability, etc. There many factors should taken into consideration and there is a high risk that wrong decisions will affect performance of the business applications.

Role of modeling is to evaluate how specific decisions will affect performance, scalability and ability to support SLA of the individual workloads. Major performance requirements include response time and throughput. Workload's Response Time , include Service Time, Queueing Time and different types of Delays, including delays caused by limited level of concurrency. In Virtual environment we should take into consideration:

How CPU service time for each workload will be affected by CPU overhead by Hypervisor
How I/O Response time for each workload will is elongated due to delays caused by increase of the scheduling I/O by Hypervisor
How Response time will be elongated due to increase of Demand Paging and Swapping within each VM and physical server
How resource reallocation between VMs can affect performance

In order to model virtualization we need measurement data characterizing Hypervisor overhead for each of the virtualization environments, including VMware, Hardware Virtualization IBM, OS Virtualization - Sun/Solaris, Microsoft Hyper-V, Oracle VM environments.

Example below show measurement data collected in VMware environment using
VMware ESX VMware utility: esxtop -b -a -d 300 -n 12 > my_file.csv
( sampling interval is 300 and number of samples is 12 )
Or resxtop -b -a -d 300 -n 12 -server > my_file.csv
from remote Linux client or in the RCLI appliance console
VMware measurement data include CPU utilization by each VM and Hypervisor, Detail information about I/O, storage and memory utilization

CPU utilization by Hypervisor and each VM is important source of data for workload characterization

VMware provide very detail information about I/O operations and utilization of storage subsystem

Memory is one of the frequent bottlenecks in virtual environment. Hypervisor statistics provide valuable information about memory usage on physical server and within each VM.

OS, Hypervisor, Application Server and DBMS Server measurement data are source of the information for workload characterization and synchronization of the workload characterization across different servers in a multi-tier distributed environment.

Workload characterization generates performance, resource and data utilization profile for each of the major workloads, which is used as input for performance prediction.

Model of the virtual environment should take into consideration how workload and volume of data growth, increase in # of VMs, tuning measures, change of the software and hardware configurations will affect Hypervisor CPU and I/O overhead, I/O response time elongation due to increasing delay of Hypervisors scheduling and Increase of the Paging and Swapping rate within each VM and physical server.
Model of the multi-tier distributed environment can include queueing network models of the individual servers of application and DBMS tiers. Workload growth and changes within one VMs can affect performance of application residing within other VMs can affect performance of applications within other VMs and this type of interconnections between VMs should be taken into consideration. In order to model this type of interconnections iterative optimization algorithm can be used incorporating bottom up and top down approach, where Bottom – up algorithm shows how change of DBMS server performance will affect performance of applications of application server and Top – down algorithm reflect how changes in application tier will affect performance of the DBMS tier.

Ability to predict the impact of the expected changes and evaluate different scenarios is an important part of building autonomic computing environment, dynamically changing software parameters and resource allocation to satisfy SLO and SLA for critical line of business, business processes and workloads.

Happy New Year 2009!

2008-12-31T19:43:00.021-05:00

As the year comes to a close, I want to take this opportunity to review modeling and capacity management challenges in supporting latest IT trends. Gartner analysts identified the top technologies and trends ( http://www.gartner.com/it/page.jsp?id=777212 ) which will affect strategic long term and operational short term IT decisions for most organizations in 2009:

1) Virtualization. Including Server, Storage and Client devices virtualization.
2) Cloud Computing. Cloud computing has built in elasticity and scalability, which will reduce risk for business that IT will not be able to meet changing business demand and provide benefits for both small and large companies.
3) Servers — Beyond Blades. Simplification of provisioning separately memory, storage and processing resources.
4) Web-Oriented Architectures.
5) EnterpriseMashups.
6) Specialized Systems. Appliances and heterogeneous systems focusing on supporting needs of the most demanding workloads.
7) Social Software and Social Networking.
8) Unified Communications.
9) Business Intelligence. BI affects companies strategic and tactical decisions. Focus on enabling faster, better and more informed decisions.
10) Green IT.

During fourth quarter I attended several conferences, where I presenting several papers and had many meetings with customers, analysts, vendors and old friends discussing these trends and reviewing the changing role of modeling and capacity management in supporting critical changes of IT.

Virtualization and cloud computing was one of the major topic at all recent conferences. Many presenters expect that effective implementation of virtualization and cloud computing can reduce TCO. Virtualization and cloud computing will change approach of planning and manage IT.

During CMG 2008 conference in December many discussions were about virtualization and cloud computing. I conducted a half day "Hands on Workshop on Performance Prediction for Multi-tier Distributed Environments ." we discussed how even simple analytical queueing network models can be used to justify strategic, tactical and operational decisions for DBMS server, multitier environment with many application and DBMS servers, virtualization and even cloud computing. I promised participants to organize an additional webex session in January to review the value of analytical models in addressing typical capacity management problems.

Debbie Sheetz from BMC Software presented a paper on Modeling/Sizing Techniques for Different Virtualization Strategies (a.k.a. One Size Doesn’t Fit All). Debbie
discussed modeling and sizing techniques specific to the architecture of the virtualization’s measurement data as well as to the architecture of the virtualization of IBM AIX partitions, HP nPar and vPars, Sun Solaris LDoms and containers, and VMware virtual machines. I did not find in her presentation how to take into consideration te impact of workload growth and increase in number of VMs on Hypervizor overhead.

Ellen Friedman presented a paper on "VMware Resource Balancing and High Availability Capacity Planning and Implementation Considerations" , which prides an overview of available metrics and performance management and capacity planning rules of thumb for VMware High Availability and DRS for ESX 3.5.

Andrew Hillier from CiRBA Inc. presented "CAPACITY MODELING AND PLANNING IN VIRTUAL ENVIRONMENTS". Andrew discussed capacity supply in virtual environments, and how to model the demands that workloads place on this supply. Andrew reviewed objectives of Dynamic Capacity Management in matching supply and demand by putting the right
workloads on the right servers at the right times.

Jie Lu from BMC Software presenred paper on "Modeling the Performance of Virtual I/O Server". Jie Lu discussed method of modeling performance of applications sharing physical disks and network adapters in virtual environment

Tad Kellogg from The Boeing Company presented a paper on
E S X G U E S T C A PA C I T Y D E T E RMI N AT I O N U S I N G G U E S T
R E A D Y- T IME ME T R I C A S A N I N D I C ATO R

Ted reviewed the relationship of various host
attributes and performance metrics to aggregated %Ready-Time (RT) performance. Using
guest RT as a host capacity indicator, other host capacity indicators, such as CPU utilization,
and distribution of guest types, i.e. single, dual, or quad virtual CPU, are quantified.

Several presenters discussed limited acuracy of measurement data collected in VMware environment. Many of them indicated that memory is a most serious performance bottleneck in their virtual environment.

During Teradata Partners conference in November I presented a paper on "Best Practice of Proactive Performance Management of Data Warehouses with Mix Workloads."
Teradata is a leader in a mix data warehouse workload management. Teradata Active Systems Management (TASM) provides a mechanism allowing systems administrator dynamically change rules including change of workload priority, level of concurrency and resource allocation. One of the challenges is to find a way to change these parameters to satisfy SLA of major workloads most effectively. In my presentation I described how modeling results can be used to evaluate different options and justify most effective strategic capacity planning, tactical performance management and operational workload management decisions. I illustrated BEZVision’s mechanism of setting realistic expectations and opportunity to compare actual results with expected. Mechanism which allows to evaluate different options and justify decision, verify actual results with expected and generate corrective actions is a foundation of autonomic computing or dynamic capacity management.

In October I attended Oracle Open World, where Larry Elison announced Oracle Database Machine.

A lot of customers interested in building data warehouses are comparing Oracle Database Machine with IBM, Teradata, Netizza, Greenplum and other data warehouse appliances. This is the area where analytical models can be very useful.

I made a presentations, at MCMG in Chicago on "How New Oracle/HP, IBM DB2, Teradata, Netezza, Greenplum, Data Warehouse Appliances AffectCapacity Management", where I illustrated how to compare different data warehouse appliances not in isolation, but as a part of multi-tier distributed architecture - private cloud. Inspite of economical problems companies continue investments in BI and DW Appliances

Several analysts believe that progress with virtualization and SOA will be the building blocks toward organizing a cloud computing. In order to play constructive role, capacity management should change focus from justification of the hardware configurations toward selection of the architecture, defining rules and policies of the workload management and dynamic allocation of the physical resources through the virtualization, design new applications and cloud computing to satisfy SLA of the individual workloads with minimum total cost of ownership.
We believe that modeling and optimization technology has an opportunity to play a significant role in implementing dynamic capacity management or autonomic computing.

Moving in this direction BEZ Systems achieved a very significant milestone in 2008 by successful launch of BEZVision proactive performance management technology for multi-tier distributed environment. Holistic modeling of the interdependent application and DBMS servers supporting mixed workloads takes into consideration how workload and database size growth, change of the hardware and software configurations, change of parameters affecting concurrency and priority of the individual workloads, database tuning and other proposed changes will affect response time, throughput, different types of delays and usage of resources.

Happy New Year!

Role of Modeling in Virtualization Optimization

2008-08-03T23:13:00.009-04:00

On September 7th 2008 it will be 25 years as I founded BEZ Systems. All these years we were focusing on applying modeling technology to justification of the most critical IT strategic, tactical and operational decisions. In 80th it was mainframes. In 90th mainframes became too small to build very large data warehouses and focus moved toward large massively parallel processing systems. When distributed systems became very complex we enhanced modeling technology focusing on multi-tier distributed systems. During last couple of years virtualization started addressing the proliferation of under-utilized distributed servers. Consolidation of the distributed servers through virtualization simplify management, can reduce administrative and other cost, can benefit some applications, but CPU overhead of handling I/O and I/O latency can cause performance degradation for others.
Gartner reports that more than half of the mission critical systems companies built in 2007 were based on SOA principles, and it predicts that the figure will exceed 80% by 2010 and SOA and Virtualization will be the building blocks for future IT and cloud computing
We will review several examples where modeling technology can be used to identify candidates for virtualization, justify and optimize strategic, tactical and operational decisions controlling virtualized environment, reduce risk of surprises and set up realistic expectations

Which applications are good candidates for virtualization?
–I/O intensive applications are not good candidates for virtualization. Modeling results can provide a information helping to select candidates

How to optimize the resource allocations between Virtual Servers?
–Different workloads have different profiles, different growth and different SLO. Modeling and optimization can help to evaluate different options and find the most effective static polices and rules affective distribution of the physical resources between different virtual servers. For example change of the resource allocation during day time and night time.

How to optimize policies of dynamic resource reallocation, concurrency level and priorities between Virtual Servers to satisfy SLOs most effectively?
–In dynamic environment workload characterization and modeling results can be used to modify resource allocation dynamically. Actual measurement data can be compared right away with performance prediction results or expectations.

How to predict the impact of new Virtual Server supporting implementation of a new application/workload?
–Measurement data collected in test environment can be used to build a model of the new virtual server. Before implementing a new workload performance prediction results can show what will be an impact of their implementation on other workloads, set realistic expectations and reduce risk of surprises.

How proposed Application tuning in one of the Virtual Servers will affect performance of other workloads in other Virtual Servers in multi-tier distributed environment?
–Change application or software parameters within one virtual server can potentially affect performance of other virtual servers and workloads in a distributed multi-tier environment. Modeling results can identify potential problems and implement proactive performance management actions to avoid risk of surprises.

How to apply modeling and optimization technology for creation of a closed loop proactive performance management process for virtualized environment?
-Automation of the data collection, workload characterization, performance prediction and optimization can help to organize a continuous process of the service level management of multiple workloads.

What is the physical server configuration required to support a given set of Virtual Servers and workloads? -Modeling results can be used to find a minimum configuration required to support SLO of the individual workloads supported by different virtualization servers.

What will be more effective: very large SMP server or a cluster of servers to implement virtualization?
-Modeling results can be used to evaluate different platforms and architectures

What is the best virtualization technology for a group of specific workloads?
-Modeling results can be used to take into consideration differences in overhead of supporting of the different workloads by different virtualization technologies to justify the most appropriate technology for specific applications

Which Virtual Servers should be placed on each of the physical servers?
-Modeling and optimization can take into consideration difference in workload profile, difference in SLO, seasonal peaks and help to redistribute Virtualization Servers between different Physical Servers in a most effective manner.

How to predict the impact of replacement of spinning disks with Flash memory?
-Modeling results can help to evaluate cost/performance of the replacement of the disks with more expensive, but faster Flash memory.

When additional resources will be needed to the pool of the physical servers and how much resource will be required to support expected workload growth?
-Prediction of the workload growth within each of the virtualization server can be used to justify physical server pool upgrade requirements

Role of Workload Characterization and Performance Prediction in Autonomic Computing and Self Healing

2008-06-08T23:20:00.004-04:00

Last week I attended “The 5th IEEE International Conference on Autonomic Computing”. http://www.acis.ufl.edu/~icac2008/. This Conference (ICAC-08) was sponsored by Microsoft, Hewlett‐Packard, Intel and IBM and brought together researchers and practitioners to address all aspects of self-management in computing systems. I attended several sessions on Virtualization and Data Centers.

One of the organizers Jeff Kephart from IBM invited me to demonstrate our product BEZVision, which provides organizations with continuous proactive performance management. I had many questions related to optimization of data collection, automation of the workload characterization, and application modeling to justify strategic capacity planning, tactical performance management, provisioning and organizational workload management decisions. We discussed how automating comparison of the actual and expected/predicted results enables closed loop continuous proactive performance management process.

Several papers at the conference described how to develop rules for autonomic computing, self healing process and learning algorithms for periodic adjustments of rules.

Modeling results can be used to optimize selection of rules and dynamic adjustments of rules to satisfy SLO for mixed and dynamic workloads. The new generation of autonomic computing tools should incorporate a holistic approach. Modeling and optimization technology can increase the effectiveness of planning, management and control by dynamic reallocation of the physical and virtual resources, change of the software parameters affecting concurrency, priorities, and parallelization. Applications of the multi-criterion approach will allow finding a compromised solution satisfying the requirements of the critical workloads to response time, throughput, availability, power consumption, cooling, and cost.

2008-05-12T19:41:00.017-04:00

Workload Characterization
If justification of strategic, tactical and operational changes is made based on gut feelings, or vendors’ recommendations without a clear understanding of how they will impact Service Level Objectives (SLOs) for different workloads, performance surprises may arise. In order to manage effectively, it is important to be able to compare actual with expected results.

Workload characterization provides an integrated view of business demand and level of service, usage of resources by each workload on each tier. Workload characterization uses measurement data collected from OS, DBMS and JVMs. Each workload’s profile reflects interconnection between JVMs, application servers and DBMS servers supporting the workload. The performance profile includes the average response time and throughput. The resource utilization profile includes CPU utilization, I/O rate and disk utilization, average level of concurrency, memory usage, level of parallelism, and interconnect utilization. The data usage profile includes the frequency of each table access and the type of access.
Results of the workload characterization are used by the analytical model to evaluate the impact of workload growth and other expected changes. Results of the workload characterization are used to identify trends, unusual changes in workload profiles and to justify performance management actions. They are also one of the inputs to analytical models.

Performance Prediction and Optimization
Modeling answers what-if questions. Optimization techniques applied to modeling allow evaluation of many options and tradeoffs to determine the most effective changes. Modeling results can be used during different phases of application and information life cycles to find solutions that satisfy the SLO of individual workloads.

Let’s consider several examples of the typical surprises and how modeling can be used to identify future problems and generate recommendations to avoid these surprises:

Surprise #1. Unexpected Workload Growth Impact

A multi-tier environment’s workload was growing at a consistent rate. CPU utilization was low, but suddenly users started complaining about performance. Some of them could not load all their data on time. Some of them were experiencing problems with response time. It was an unexpected surprise. An urgent performance-tuning project was initiated and questions about an unexpected hardware upgrade were raised.

What could have been done to avoid surprise?
Workload characterization and performance prediction could identify that as a result of workload growth, CPU utilization will gradually increase:

Expected workload growth will nearly double CPU utilization from 38% to 70% within a year.

The workloads’ response time will increase almost exponentially and one of the workloads will be the most sensitive to workload growth increase.

Each workload has a different profile of CPU usage, memory, I/O rate and a different profile for usage of data and will be affected by expected workload growth differently. SLO for one of the workloads will not be met in a half year.

As the system ages, each transaction begins to process more and more historical data which makes every transaction bigger (using more CPU and doing more I/O). This leads to lower
throughput for most of the workloads , but one of the workloads will suffer the most.

Throughput for most of the workloads will be reduced. Current configuration will be able to process only 50% of one of the workload’s requests in a year.

Several proactive tuning measures could have been done to avoid the surprise. Changing the number of concurrent database loads, changing the maximum number of JVM threads, and a hardware upgrade could have been helpful. Most importantly, realistic expectations could have been presented to management to justify a change and to avoid an unpleasant performance surprise.

2008-05-08T21:50:00.007-04:00

Simplified Analytical Model of a Multi-tier System

A production environment can include hundreds of application and DBMS servers supporting multiple workloads.

Here is a simplified model of a multi-tier system with one web server, one application server and one DBMS server supporting a single workload. The figure below illustrates how the hardware and software configuration of the application server and DBMS server can affect performance of the multi-tier system.
Simplified analytical model illustrate interdependence between hardware and software configurations of application and DBMS servers affecting balance, scalability and performance of the multi-tier environment.

User requests are sent through the network and processed by a web server be able to process up to 75 concurrent requests with up to 125 requests waiting for web server service. After 200 requests are made of the web server (75 active plus 125 queued), new requests are rejected. After processing in the web server , the transactions use CPU, memory and disk requests at the application server. In this example, the maximum number of app server JVM threads is equal to 60 and up to 15 requests can wait for a JVM thread. After being serviced by the application server,transactions arrive at the DBMS server. The DBMS server can process up to 25 concurrent sessions and 25 requests can wait for connection.

Each element of the configuration has a defined capacity. When demand exceeds the capacity, requests are thrown away. When one part of the configuration is enhanced with additional capacity, it is now capable of passing more requests to the next elements in line for processing. In short, relieving some bottlenecks will inevitably expose others as soon as the arrival rate gets high enough.

If business is growing and an increased number of users generate more requests, the number of concurrent requests within an application server is increased. The maximum number of concurrent requests is limited by the number of JVM threads. If the maximum number of JVM threads is small, contention for resources is small, but time waiting for a JVM thread is high. If the number of JVM threads is large, then requests do not wait for the JVM thread, but contention for the next transaction resources is high.

Connection pool size similarly affects contention for DBMS server resources and the time transaction wait for connection.

Each transaction requests a certain amount of memory. Heap size within the JVM is limited and when the demand is growing, free memory to process new requests disappears. The number of JVM threads is limited by the number of JVM threads and by available memory (JVM heap size). To support growing numbers of users, new JVMs are created. If application server capacity is sufficient, the JVM can be created within the same application server. If not, a new application server is required to support the new JVM.

Java applications generate SQL, which is sent to the DBMS server through the connection between the application server and the DBMS server. Connection pool size limits the number of requests that can be sent for concurrent processing by the DBMS server.

In an Oracle RAC environment, each request can be processed in parallel by one or several nodes accessing data from the shared storage subsystem. Workload growth increases contention for shared resources and affects response time and throughout.

2008-04-09T15:12:00.004-04:00

Exciting Times!

BEZ Systems is very happy to be selected as one of the Cool Vendors of 2008 by a leading analyst firm. Vendors selected for this report are considered innovative, impactful and intriguing. BEZVision provides visibility into future application and database performance problems providing IT professionals the time and data necessary to make 'trusted changes'. BEZVision captures performance data, often from existing database and J2EE application performance monitors, and stores it in a Performance Management Database (PMDB) that BEZ refers to as the Consolidated Performance Warehouse. Cross-tier data is aggregated into business workloads to show past and present service delivery by business function. Forward looking automatic predictions alert users to future service breaches. Real-time changes to prediction models provide "what-if" analysis to understand how future change(s) will affect service delivery. To learn more check out http://www.bez.com/

2008-03-25T11:00:00.001-04:00

Welcome!
Welcome to my first blog entry on Practical Analytic Modeling and Predictive Performance Management.

I hope this blog will provide a useful forum for sharing ideas and successes in applying models and predictions for solving IT challenges. When properly applied you increase the effectiveness of capacity planning, performance management and operational workload management decisions while reducing the risk of surprises.

Over the past few months I have visited many companies and taught a modeling workshop at the CMG conference in San Diego and spoke at the St. Louis CMG regional meeting. During this time I received a lot of questions on modeling applications based on Oracle, DB2 and Teradata in a multi-tier environment. This type of environment challenges people who are making strategic capacity planning, performance management and operational workload management decisions since data collection, workload characterization, and correlation across tiers and modeling concurrent workloads proves to be difficult.
The presentation given at the St. Louis reginonal meeting can be found here: http://regions.cmg.org/regions/stlcmg/Meetings_02-08.html

In January I visited a company in Brazil with very large and rapidly growing multi-tier distributed environment with many Oracle RAC systems. Many questions related to justifying a hardware upgrade based on the usage trends of CPU resources versus predicted response time and throughput.

In a dynamic multi-tier distributed environment, a planned change can affect the performance of all workloads. The following example illustrates how results of workload characterization and performance predictions allow IT to; evaluate impact of the expected workload and database growth, justify the most effective capacity planning, performance management and operational workload management decisions set realistic expectations and verify results.

Applications execute in an environment composed of hardware, network and application components, including all the controls and configuration decisions that govern these elements. Insufficient understanding of how these pieces interact leads to disappointing results. Correcting these complex configuration problems is usually difficult and spans multiple organizational elements. Big surprises after planned changes to any part of an application environment may force unexpected resource upgrades while testing the tolerance of its community of users.

When decisions on architecture, hardware and software platforms, server consolidation, application and database tuning implementations are made based on just gut feelings or vendors’ recommendations without concrete expectations, you have a high risk of surprise.

Many factors affect performance in a dynamic multi-tier environment with multiple workloads. Workload growth can affect contention for application and DBMS servers. Even minor changes with software parameters can affect balance, contention for resources and cause significant performance degradation.

Business users demand consistent service with predictable cost. The level of service required by different lines of business and corresponding Service Level Objectives (SLO) to timeliness of data, data access, and throughput are different.

Today’s applications queue to consume resources as if they were shopping and checking out at several stores in a mall on the busiest shopping days. Shoppers, or transactions, wait for and acquire JVM threads then queue for and are dispatched on a CPU, then wait for JDBC connection and repeatedly queue for and acquire their data from their favorite storage devices. As applications and their hosting configurations evolve, the number of unique shops our transactions visit and their contention for service in their mall leads to many more contention opportunities than ever before. How long it takes to complete a shopping adventure is the sum of all the stops for resources, including all the lines.

User response time includes time waiting for JVM thread and time of being serviced and waiting for service by CPU, storage subsystem in application tier, time waiting for connection to DBMS server and similar service time and queuing time for CPU, disk and interconnect within DBMS server. Changes in request rates do not cause linear changes in response time. When utilization gets higher, even small changes in volume can lead to major performance swings.
Balancing the usage of resources within one of the servers in one of the tiers will unbalance the usage of resources between tiers causing unexpected results. For example, the implementation of new applications, server consolidation, performance tuning and hardware upgrades can eliminate performance bottlenecks in one place, leading to congestion downstream causing unexpected performance degradation for some of the workloads.

In my next entry we will review how prediction results based on analytic models help to evaluate the impact of the expected changes, identify future bottlenecks and justify the most effective proactive measures to ensure continuous support of the business needs.