Real Time Indexing - An Introduction

I feel like a kid in a sweetshop with so many new PeopleTools features and enhancements to explore. Over the last 18 months I've been doing quite a bit of learning and investigation into "Real Time Indexing" in the PeopleSoft Search Framework.

Elasticsearch has been the keyword searching platform in PeopleSoft since around PeopleTools 8.55/6 back in 2016.   Since then, we've seen a significant growth in it's value add to the PeopleSoft application and it's become an essential part of the PeopleSoft stack.

Due to some changes in how Elasticsearch licensed their product Oracle has switched to using OpenSearch which is just a fork of the Elasticsearch 7.10 code line.  To keep it simple there is essentially no difference between Elasticsearch and OpenSearch.   PeopleTools 8.61 already requires OpenSearch and from July 2024 there will be no further patching or support for Elasticsearch (See Doc ID 3005972.1).     If you're still on Elasticsearch then don't worry when you read this article just replace the word OpenSearch with Elasticsearch.    

Why use OpenSearch?

• find the data you want to work with first then decide on the action to perform.  This interaction model has been a focus of PeopleSoft for some time now and embedding Related Actions and the Related Content Framework in the Search Framework has been a key part of enabling this.
• keyword searching of business data.  
• Not just text information in the database but file attachments too.
• Unstructured data becomes very accessible using the PeopleSoft Search Framework.
• Keywords can be associated with facets to make the search faster and more powerful and to implement a common interaction model found in many other web applications.
• Offload the reporting and analytics from the PeopleSoft database. Finding a transaction using keys will not really impact the performance of the database but searches against unstructured data stored in a relational database will impact the performance of the database.  Extracting relevant data from PeopleSoft database and indexing it in a specialised database like OpenSearch can help with this situation.

So, what's the PROBLEM?

One of the issues with holding search index data outside the PeopleSoft database is that of keeping it up to date.  As soon as the data leaves PeopleSoft and is loaded into OpenSearch the data is potentially out of date.   A routine schedule build of indexes becomes necessary.  Every hour, every night, once a week?  Any of these options leaves the possibility that Elasticsearch data will be out of synch with PeopleSoft.   We have learned to adapt and compromise around this need for a batch based synchronisation between PeopleSoft and Elasticsearch.

Solution?

Out of date OpenSearch indexes?  PeopleTools 8.59 introduced a solution that should solve this problem.  Real Time Indexing (RTI). 

Once RTI is configured for your Search Definition then as data is added, update or deleted the corresponding OpenSearch indexes are updated in (near) real time.

The last few PUM Images have released the Search Definition code changes to support the RTI features and the latest 8.61 release has seen yet more improvements to this great feature.

If you don't have the latest application release but you do have at least PeopleTools 8.59 then you can still see RTI in action with the Portal Most Recently Used index (PTSF_RECENTMENU) and Keyword Indexing (PTSF_RECENT_KW).  These are both part of PeopleTools and so will work on HCM, FSCM, Campus, IH or ELM.

I've recorded a short video which explains how Real Time Indexing works and over the coming weeks I'll be digging deeper into 

• how to configure RTI infrastructure
• how to enable RTI for delivered Search Definitions
• how to manage bulk updates to tables