I've been working recently on my first Sitecore search project, finally i got a chance to work on this amazing product by Sitecore, I dealt with different sources types, and once of the challenges I got is to deal with paginated APIs that we need to index its data, in this blog post I will walk you through how you can index data from all pages, interested, keep reading!!
While working on a Sitecore Search implementation, one request was indexing data coming from an API in a Json format, the problem was that the index must return up to 50 items at a time, while we have thousands of items, how would we achieve this utilizing existing Sitecore Search Sources?
I decided to go with an API Crawler source type, but when checking the type of trigger it has, I only noticed that there is the following types, but the request type was not the one I'm following, as this will only index the page that you're specifying in the url, as example the first 50 items or the second 50 items...
So.. I decided to go with a JS trigger type, in this type you can add a JS code, so why not using it and add like a loop to go through the pages you have and add them as array of trigger(s) ? this is what I did in my trigger JS code below:
as you can see from the code above, you can loop through your data pages, create a URL for each page, add them into an array and then just return the array of urls. hopefully this help someone with their Sitecore Search implementation.
The only downside I see with the above approach is the dynamic approach to calculate number of pages, you should know before hand how many pages you will have from that API in order to resolve the trigger(s) correctly; if you have better way please comment on this post and lets resolve this in a better way!!
No comments:
Post a Comment