This theme of making PeopleSoft faster, has centered around the following 3 activities. Any of these activities could make PeopleSoft run faster.
1) CONFIG: Ensure you have PeopleSoft configured correctly for your workload
2) HARDWARE: Can we throw iron at the problem? More/faster CPU, more memory, etc
3) TUNING: Can we tune SQL and PeopleCode?
As we put effort into each activity, speed should increase. The obvious one to start with is configuration. I believe, once a PeopleSoft environment is optimally configured then it won't be possible to make PeopleSoft run any faster unless you increase hardware and/or tune SQL/PeopleCode.
I'm starting a blog series on this broad topic in the hope that our PeopleSoft community might find some of my ramblings useful. As always, these are just my thoughts and I welcome comments and alternative views.
How to make PeopleSoft run faster - Config
The PSAPPSRV is one of the key Application Server Domain (app server) processes which, if not configured right, can result in sub-optimal performance. A typical production domain can have many of these processes configured - anywhere from 8 to 30+. They are designed to execute PeopleCode and SQL statements for both PIA transactions and Integration Broker synchronous messages.
An app server domain should be configured with an optimum number of PSAPPSRV processes. The number of PSAPPSRVS will depend on
- expected user concurrency
- available server memory and CPU
- number of other domains
Note #1
Set the number of PSAPPSRVS in the Application Server domain to be the same as the number of CPU threads or virtual CPU count.
Why? If a PSAPPSRV goes busy and consumes CPU then it will only use upto 100% of a single CPU thread. This is because at the PSAPPSRV level PeopleSoft is single threaded. It doesn't matter how many CPU threads you throw at an app server, a busy PSAPPSRV will only ever use 1 thread. It might use a portion of the thread if that thread is busy but it will never spread it's workload across multiple threads.
 |
| Fig 1. A server with unused CPU and transactions waiting |
This therefore means that if a server has more CPU threads than PSAPPSRVS then your server is never going to get busy as you will always have more CPU than the app server domain can use. (This assumes the server is dedicated as an app server and is not running any other applications). Fig. 1. shows that when all PSAPPSRVs are busy then even though there's spare CPU then new transactions will get queued. A queued transaction means users are waiting.
NOTE: When monitoring CPU utilisation on an app server it's important to realise that taking averages doesn't really work here. When a user clicks a link in PIA they want a fast response. When transaction is assigned to a PSAPPSRV process it will be assigned to a single CPU thread. If that thread is already busy with other work then this transaction may not get all the CPU it needs. Ensuring the PSAPPSRV is not starved of CPU or sharing a CPU is an optimal configuration.
To speed up a transaction we need a motorcycle not a bus.
Note #2
Faster CPU threads might be better than having more threads.
Why? If a transaction is heavy on CPU then having it finish as quickly as possible is the best result. We already know that adding additional CPUs won't speed this up so, with the knowledge that the PSAPPSRV is single threaded, we can see that adding more threads won't yield faster performance.
If you have a user concurrency of 40 (meaning at any one point in time there are 40 busy PSAPPSRVS) then the optimum number of CPU threads is 40. That might be 2 domains each with 20 PSAPPSRVs or 4 domains each with 10 PSAPPSRVs. Either way, when all 40 PSAPPSRVS go busy we idealy want a dedicated CPU thread for each one.
Yes... I know that many PeopleSoft transactions will be waiting on SQL and so won't use App Server CPU while it's waiting, but... we're talking about optimum performance here, not optimum cost ($) ! Real-world may well need to balance cost with optimum performance.
Extreme concurrency is a real requirement in some PeopleSoft sites. Examples of this include universities where thousands of users will be hitting the system concurrently to get exam results or to enroll or payroll where all staff login on pay day to check their payslip. PeopleSoft scales horizontally very well but regardless of how many server/domains you have when a PSAPPSRV goes busy then you always want it to finish as quickly as possible. High speed CPU can help. If you're running in OCI then switching compute shapes is relatively straight forward and can yield immediate response time improvements. More on this when we come to talk about hardware as an activity.
Note #3
Set the Recycle Count directive to manage memory.
A PSAPPSRV will allocate memory on startup and then use memory as it brings disk based cache into memory and, as it brings in the component buffer from the web tier when executing PeopleCode. It's difficult to predict how much memory a PSAPPSRV will use as there are many different influencing factors including what components are used, how many different components users are likely to use, volumes of data being processed, the JVM memory setting under App Domain PSTOOLS, etc.
As a general guide, allow at least 4GB of memory per PSAPPSRV.
The Recycle Count directive is a powerful tool to help manage memory.
When a PSAPPSRV executes a transaction a Requests Done counter is incremented. When this counter reaches the Recycle Count value the current transaction will complete and Tuxedo will attempt to stop and start the PSAPPSRV process.
Additional considerations:
- Recycling a PSAPPSRV will free up allocated memory including any memory based cache, therefore recycle as often as you need to to ensure you don't run out of memory.
- Using memory based cache on a PSAPPSRV is faster than disk based cache, therefore let the memory cache build up. Recycling will destroy memory based cache.
- Recycling the PSAPPSRV can take several seconds, therefore avoid recycling too frequently.
- You ONLY need to to Recycle if you need to manage available memory on the server. If you're not running out memory then don't recycle the PSAPPSRV frequently.
- Recycling of a PSAPPSRV is controlled by either the number of transactions since the last recycle event (the default is 10,000) or by the amount of memory currently being used by the PSAPPSRV. I've experimented with ProcessRestartMemoryLimit directive but have had mixed and unpredictable results on Windows and Linux. I find the just using Recycle Count works very efficiently.
- The pattern of PSAPPSRV utilisation differs significantly between Windows and Linux.
- Windows will allows send the next transaction to the first idle PSAPPSRV. Consequently, the first PSAPPSRV will always get work if it's idle.
- In contrast, Linux will send the next transaction to the next PSAPPSRV and so cycle through the available PSAPPSRVS. the effect of this is that on Linux all PSAPPSRVS will get the same number of transactions and consequently will all hit the Recycle Count threshold at the same time.
- To avoid all PSAPPSRVS recycling at the same time (and resulting in complete loss of service) the PSAPPSRV will not recycle while another one is recycling. Be aware of this when setting the Recycle Count. Values as low as 1000 are perfectly OK if you are struggling to manage memory. Better to recycle often than to run out of memory.
- On Linux, it's ok to have this value high but on Windows you should recycle more frequently as Windows port of tuxipc does not manage the memory as well as on Linux. The inter process communication (ipc) software was originally written for a Linux operating system and ported to Windows much later. The port was ok, but behaves slightly differently as noted above and in my experience suffers a little more memory-leak problems.
- There's no general rule that I know of which relates the number of transactions completed by a PSAPPSRV and the amount of memory that it's using. Using the number of transactions threshold for triggering a Recycle is therefore a crude method. Set too low and you'll be recycling unnecessarily. Set to high and you're domain may run out of memory before the Recycling will free that up.
Comments