How to make PeopleSoft run faster: Episode 2


How do you know if you have a performance problem with PeopleSoft?

With Episode 1 of this new series on How to make PeopleSoft run faster we got straight into some specifics with some key configuration settings for Application Server domains.

This episode will focus on identifying if you have a speed problem in the first place.   There's no point in fixing a problem that does not exist!  How do you know if you have a performance problem and therefore need to make PeopleSoft run faster?  There are 5 tools and techniques that I want to highlight that can help us determine this.


PeopleSoft Ping  

    • This is a benchmark indicator for general infrastructure health.  The test payload has been the same ever since PSPing was introduced in PeopleTools well over 20 years ago (probably around the time PIA was introduced).
    • Because PSPing is a constant payload, regardless of what PeopleTools version or application you run, then we can make direct comparisons of one PeopleSoft site against another. If there's a PSPing difference between one PeopleSoft installation and another then it will be because of the underlying stack and not the PSPing utility itself.
      PSPING chart
    • Creating a benchmark for your site can be very useful as, when things start to look like you're having performance problem, PSPing can sometimes help highlight where in the stack the problem is.
    • Back in 2015 I started capturing anonymous PSPing data from various PeopleSoft installations.  You enter your own results too. The chart below shows some of the data over time. 
    • Infrastructure will have an effect on PSPing times.  CPU speed, available memory, speed of the network between WebLogic, App domains and database server, is the data encrypted in transit between the layers, speed/latency of the network, etc. 
    • These two PSPing measurements were taken on the same PeopleSoft environment running Web, App, Batch and Database all on the same server.  All I changed was the compute shape in OCI from Flex E3 to Flex E5 and the effect is startling !  The E5 has a much faster CPU.  All measurements in PSPing are in seconds (NOTE: I tend to discard the Client Time as this can be too variable as it's dependent on user's local broadband, fiber, satellite, or even modem dialup connectivity! )  
      PS Ping. Same environment, different CPU speed

    • Again, benchmarks are really useful here as if you introduce a change in the infrastructure you can see if the impact affects PSPing.   PS Ping is a "real" PeopleSoft transaction so if the change impacts PSPing then it's likely to impact end-users.

 PeopleSoft Performance Monitor

    • This is an outstanding piece of application instrumentation.  Every click, by every user is captured and broken down into minute detail as it travels through each layer of the PeopleTools stack and written to a database.
    • Each PeopleSoft Performance Unit (PMU) is logged and can be analysed. In the example below the total time taken for the PIA request was 16967ms (PMU 101) of which 16933ms was spent in executing PeopleCode and SQL in the ICPanel (PMU 401). This example below does not necessarily show there's a performance problem unless we compare similar transactions over time.  Maybe, this 16 second transaction normally runs in 2 seconds!  

    • Simple SQL to examine if duration of the 401 is > 6 seconds would easy enough
    • Or to profile the 401 durations by payload .
    • Another example below shows 4 second transaction (101) but the ICPanel (App Server) part of the transaction (401) was only 400ms (less than 1/2 second).  Clearly, there's a problem with either the Web Server or the connectivity between Web and App domains. 

    • Having a long 101 in proportion to the heavy lifting SQL and PeopleCode part of the transaction (401) would indicate there's a performance problem in the Web layer or connectivity between web and app domains.  <SQL>



WebLogic Access Logs
If you don't have access to PPM then a more crude method of detecting preformance problems in transactions is to examine the WebLogic Access logs.

The example below shows a 43.27 response time for a POST.  Other than the URI , we don't have much more detail.  but, if you want  a quick and easy way to profile your transactions then look at all your POST transactions.   In PeopleSoft, a GET request will be a component retrieving things like images, CSS and Javascript . These all load either from the client browser cache or if not present in browser cache from the Web Server cache.  We're not particularly interested in these unless they are taking seconds to retriev. They will generally be quite quick - less than 0.5 second.   The GET requests were most interested in are those where a user is opening a component.  There are 2 that we wan to track
1) /psc/<site>/C/efdewdewd.   This has no parameters and will therefore load the Component Search page (if it has one). 
2) /psc/<site>/c/ededed?&BU=XYZ&POID=12345 These have parameters in the URL and therefore will open the component and execute the search.

PeopleSoft POST requests could be any of dozens of different operations like Search, Next in List, Save, Cancel, Page tab, etc. They responsiveness of these will vey much contribute to the user's perception of how fast is PeopleSoft.




 DDDAUDIT process

DDDAUDIT is a great litmus test for a number of potential performance problems.   It's good because it's a constant  payload.   Every time you run it the same SQL is executed against a largely static set of PeopleTools and database meta data.  Because it examines PeopleTools meta data, it will run in the same time whether you run a Finance system that is 8 Terabytes or your a small HR implementation with less than 100GB of database. Just like the PS Ping utility, the DDDAUDIT SQR process is a constant SQL payload regardless of PeopleTools version. Has the process changed recently?  I don't think so.  The PeopleSoft CFO from 8.47 to 8.62 contains 15 references to modifications to SYSAUDIT but none to DDDAUDIT.  

So what can affect the execution time of DDDAUDIT?   

Latency. The characteristics of DDDAUDIT process are that it executes thousands of SQL statements.  Each SQL SELECT statement needs to be sent from the client process to the database server.  This transmission of the request and the transmission of the response from the database server takes time.  We call this time latency.  Latency is the time delay between the initiation of a request for data and moment of receipt of that data.   A latency delay in sending a single request would generally go unnoticed. But, send thousands of requests and even an apparant small increase in the latency delay will have a negative impact on performance.
Process scheduler and database server CPU.  The faster the CPU thread then the faster the process will execute and, up to a limit, the faster the database server will fetch the data.    Because SQR is a single threaded application throwing more CPU threads at a process like DDDAUDIT will not make it go faster.  But, if you increase the speed of the individual thread then you can make quite a dramatic positive impact on performance. This is because PeopleSoft, or should I say SQR, is single threaded.  More threads, therefore does not make a single process go faster.    The CPU utilisation chart below shows a 2 x CPU thread machine during the execution of DDDAUDIT.  Only 50% of the available CPU (ie 1 single thread) is consumed even though there is a further 50% of CPU it won't get used by this process.


NOTE:  One of the really cool features of OCI over say Axure, GCP and EC2 cloud infrastructure , that of Flexible shapes.   OCI Flex shapes allow you to configure the precise number of OCPU (an OCPU is 2 threads) and memory of a server. This means you're not trying to select the best T-shirt sized shape from a menu of fixed shape sizes. 
Over the years, Oracle have released new generations of Flexible shapes with each successive generation having a thread speed faster than the previous one .   A real example of this can be seen in the table below showing the DDDAUDIT execution times for  



 End User Feedback


The final indicator as to whether you have a performance problem is human end-user feedback.   If the users are saying it's slow, well... it probably is.   This series is about how to make PeopleSoft run faster and obviously there are limits here as we cannot achieve ANY performance.  We will eventually be constrained by configuration, hardware and tuning.  Having said that, very few of us ever reach the limits of what is possible as usually money and/or time get in the way of improvements.
 If the users are saying it's slow, well... it probably is.   
A widely recognised benchmark for end-user response time says that if the response times are :...

  • < 0.1 second response then the user is freely navigating.  It's like the user is controlling the computer. 
  • 1 second response then this is the limit for user perception of freely navigating.
  • > 1 second response then the user will believe their clicks are giving the instruction to a computer to do something and must wait until done.
NOTE: End users may be more speed tolerant of some operations than others.  For example
  • Save button
  • Run PS Query online
  • Open Component (might depend what the component is)
We expect a 200 line purchase order to take several seconds but we expect a lookup of a country code to be lightning fast.   Understanding this dynamic for your users and transactions will make sure you exert tuning energy on the right things..


As always, comments, thoughts ideas and suggestions are always welcome.

Other blogs from this series:

Comments

Post a Comment