Macleod
Performance Impact Analysis with KPP using Application Response Measurement in E-Government Systems Namho Voo, Ph. D. DoD/HA Falls Church, Virgínia, USA Namho. yoo@us. army. mil Abstract In this paper, the performance impact analysis of egovernment systems with key performance parameters is being considered. Meaningfu impact analysis in sustained government systems is required for considering non-functional requirements and functional requirements. Performance requirements are a critical component of non-functional areas.
For example, if a new system change is set to the s time must be imple n XML-based frame impacts on sub-syste impact analysis by pe 2 rk Swipe nentp ms of the response m. In this paper, lyze performance heme to enhance sing application response measurement. Through a health system example as a case study, a performance requirement model to describe extended trees and adapting analysis result of performance monitoring using application response measurement and XML tree representation are addressed.
Th•s paper also proposes a scheme for prioritized processing and an algorithm for effectively enhancing impact analysis in a timely fashion. 1. Introduction In a sustained system, impact analysis is required to make decision Sv. ‘ipe to View next page decisions prior to making system changes associated with new requirements [11. System engineers are faced with the problem of analyzing nonfunctional impacts as well as functional impacts.
There is a growing recognition of the role of the system engineer in determining the quality of a system. This analysis is driven by non-functional requirements such as performance, security, reliability, modifiability, reusability, and economic constraints. Among these requirements, the performance requirement is a ritical component in the non-functional area. Most performance failures occur due to a lack of consideration of performance issues in the early stage of analysis.
As impact analysis is covered by several systems connected via interface in a sustained system, several supporting engineers would be involved in the engineering process suggested by the technical committee to meet the users requirement [2]. One of the most typical requirements from the user with respect to performance is application response time. prior to deployment, performance testing to measure the response time must be conducted for a horough engineering impact analysis. As system environments change rapidly and engineering efforts for impact analysis increase significantly, performance specifications require careful examination.
One of the key documents for such performance engineering efforts is the Operational Requirement Document (ORD), including the 20F 12 performance engineering efforts is the Operational Requirement Document (ORD), including the Key Performance Parameter (KPP) that contains a description about performance requests/ requirements and how they should be implemented and evaluated. In this paper, the experience gained dunng engineering projects for impact analysis is discussed. Performance requirement are commonly applicable to all sub-systems.
In this paper, a typical government health system as a case study. Thus, this paper suggests a customized process to monitor performance requirement during the impact analysis. As shown in Figure 1, the performance requirement is described as one of the non-functional requirements. To focus on the response time, application response measurement (ARM) for performance monitoring is discussed as a lightweight tool The proposed cheme and implementation are discussed in order to connect the users requirement described in the document and system enwronment for performance monitoring. this paper, Extensible Markup Language (XML) technology is proposed for supporting performance engineering efforts for impact analysis in sustained systems as a common vehicle for information exchange. 503 978-1 -4244-4828-9/09/$25. 00 2009 IEEE Proc. ICSM 2009, Edmonton, Canada NFR Types Performance Security Change Case 1 2 34 5 6 System A Insert Delete Up 3 2 nvironment Combined System A Insert Delete Update Volume Environment Combined
System B Yes/No Yes/No Yes/No NIA NIA NIA Time Space Avallability Confidentiality Integrity Response Through Main Secondary Time -put Memory Storage Figure 1- Non-FunctionaI Requirement (NFR) Types Tree XML is widely used as a standard for efficient information exchange on the World Wide Web. An efficient data management scheme associated with performance requirement is discussed and algorithms are proposed for decision support 2. Background and Problem Statements System impact analysis is essential for good decision support for given change requirements.
If change requirements are given as xternal inputs which should be applied to the entire system, it is obvious that change impacts between sub-systems must be carefully examned. Moreover, in case a given requirement has performance parameters for application, it should be efficiently examined with the overall view as well as the technical view. n table l, it takes some time to find the result of whether or not to analyze the impact in change cases (1-3). The analysis result is dependent on the knowledge and experience of the System Engineer and there are man cases in which the result is uncertain.
If change cases idered, the situation for 4 12 he previous case. System Engineers need to conduct system impact analysis and evaluate testing results against the ORD with KPP. In sustained systems, performing testing and evaluating the results take up a lot of time and the project schedule is delayed accordingly. In the current situation, less consideration is given for reducing the time for testing and mitigating the inaccuracy of performance testing result. The customized process to support the System Engineers’ efforts for performance impact analysis is required.
Table l- Change Case Table for The System System performance requirements are categorized as threshold r objective, and some threshold requirements are further defined as key performance parameters. The threshold value is the minimum acceptable value that, in the user’s judgment, is necessary to satisfy the need. This paper suggests the usage of a performance monitoring tool to improve the process for performance testing. ARM is defined as Application Response Measurement. It comprises of an open standard application program interface (API) for performance monitoring in applications.
The ARM standard is administrated and overseen by the Computer Measurement Group (CMG) and is an open tandard, derived with input and agreement, from multiple third party vendors[2]. Consequently, the ARM API is supported by a number of third party commercial offthe-shelf (COTS) products targetng application performance monito 2 of third party commercial offthe-shelf (COTS) products targeting application performance monitoring, including Tivoli, Candle, and others. • arm_init : initializes ARM environment for the application. • arm_getid : Names each transaction that Will be monitored. ?? arm_start : Signifies the start of a unique transaction instance. • arm_update : Updates statistics for a long running transaction optional) • arm_stop : Signifies the end of a unique transaction instance. • arm_end : Cleans up the ARM environment prior to application shutdown. Although the ARM environment is recommended for performance impact analysis, implementation difficulties still exist for evaluation against KPP, based on the operational requirement document. This brings up several questions: 1) How does one map the correlation between the ARM result and threshold? 2) How does one communicate each of the other system for efficient performance impact analysis? ; 3) How does one track the measured result and update the pecification on performance requirements in accordance With the monitored result? ; and 4) Are there any simple and povverful ways to execute performance impact analysis? 504 3. XML-based performance monitoring with ARM To monitor performance with ARM, XML is a good vehicle for supporting engineering environments and enhancing the process of performance impact analysis.
To achieve this goal, in this paper, an XML-based representation of gather 6 2 performance impact analysis. To achieve this goal, in this paper, an XML-based representation of gathered specification for performance impact analysis is discussed. By customizing and implementation scripted by Perl programming, as well as applying ARM Instrumentation, the metrics are produced in accordance with the useds requirement for performance evaluation.
Active Impact Analysis Report Database Profile XML is related to the non-functional requirement but the security issue is taken care of the full version of the paper. Log XML and Message XML are also omitted intentionally. 4. Enhancing performance impact analysis A manually performed impact analysis is time consuming and prone to defect. The System Engineer needs information about all elated documentation entities of a selected requirement. Using a performance monitoring tool, the time and efforts for analysis, as well as potential errar, is mitigated.
But in arder to increase practical usability for performance impact analysis, the end user’s performance requirement based on engineering document must be considered with the tool. Based on an example in Table II, the data for ten transactions are demonstrated in terms of response time. The data on threshold is defined in accordance with KPP in the ORD. The items are identified as “use cases” to be measured nd are comprised of the relevant performance requirement. Specifically, they are like follows: • KPP 2 of the relevant performance requirement.
Specifically, they are like follows: • KPP I-l(response time): user Actions within a screen(l sec) • KPP 1-2(response time): Person Data Retrieval – Multiple Resultsmultiple TestS(25 sec) • KPP 1-3(response time): User actions to change screens(6 sec) • KPP 1-4(response time): Person Data Retrieval – Single Result/Single Test(6 sec) For the purpose of initial verification, the stopwatch testing at engineering lab can be used also as seen duration column of the table. Table II also includes Run 4, which means the number of runs tested with stop-watch.
Based on the experience, stop-watch testing Increases the reliability of ARMed result and is useful to mitigate the users concern. This information that has metric can be maintained using XML as well. Table II- Sample Response Time (Manual and ARMed) Number 12345678910 ARM XML Application NonFunctional Analysis Parsed Data Document XML Documents Profile XML Functional Analysis Security Input Mapped Data Log XM L Decoded Data HL7 Msg HL7 Ms 80F 12 To propose PIA architecture, Figure 2 (left hand Side) shows the
XML-based information architecture mapping to each offline or online components. Among system component, the relevant entity associated with performance impact analysis is considered. The focus is on the application and ensurng that the ORD has performance information. using related artifacts, the Document XML based on key performance parameter metrics of the ORD, is generated. By embedding ARM API functions to relevant component of application, the measured duration of response time is obtained.
Transaction Security-User logon Change User Open-Rewew Open- Appointment Cancel-Search Dlg Run-Search Select Person Close- Module Open-ltem Run-Calculate Item Run # Duration* ARMed Threshold Pass/FaiI 31 32232131 7. 53 17. 09 1. 44 1. 72 0. 47 2. 28 1. 56 1. 07 11. 62 0. 72 6. 45 14. 99 1. 81 1. 32 0. 38 2. 25 1. 73 1. 36 10. 95 0. 91 25 25 66 1 6666 1 pass pass pass pass pass pass pass pass Fail pass * Using Stop-Watch For reporting purposes on the implementation, the response time falling withln a certan range near each 505 threshold “meet” the threshold.
More significantly, in order to answer about the relationship between the systems, it is sufficient to consider the path information from system to system Algo rith m PIA In put:ARMedme tric s In form a tio n, P erformance -b ase d document&HL7 Lo g F ile O u tp n form a tio n, Performance -b ase d document&HL 7 Lo g F ile O u tp u t: U pd a tin g ance m pa c t A na ly s is Doc 1 ) Access Engin ee rin g la b Server fo r te s tin g 2) M ove to re Ie van td ire c to ryw ith respect to th e s y s te m change in fo rm a tio n3 )Checkth epos s ib ility o f H c te d by change. ) Ifyes, th end o fo Ilo w in g b e lo w proced u re . O th e r w is e e x it. 5) Ex tra c t th eARMeddataofre Ie vantsys te m . 6) Parse th e H L 7 lo gaccord In g to th eru le base donsystem cha g e d 7) Pars e th e re le va ntKPPm e tric s in form a tio n 8) ConvertARMme tr ic , K P P m e tric s, andparsedH L 7 lo g to x ML 9) ChangexMLformat M e ere presen ta tio n 1 0) Sea rchandCompa re th ere lev a n tin fo rm a tio n. 1 ) C a Ic u la te th e pa th in form a tio n us in g X P a th w ith cons id e rin g p rio rityl 2) Compare itw ith th e prev io us in forma tio n. 13) W ith re gu la r ba s is,upda te th eh IS to D’ data. 1 4) Sys te meng in eerp rod u c e th e m e tr ic in fo r m a tio n ba s e do n X M L DO M d a ta . 1 5) Ap p ly th eres u Itfo r th e Perfo rm ancelm pac t Analys is. 16)Upd 0 DF 12