RapidIdentity Cloud Product Guide

JMESPath Next Page URL Selector

The goal of the Next Page URL selector is to build a URL that can be used to get to the next page of results from an API that can break up large result sets across multiple web service requests/responses. It is only needed for APIs that break up the data into multiple "pages" using some method other than those Studio supports natively. Currently native paging support includes the following:

  • Link HTTP response header with rel="next", e.g.:

    • Link: <https://provider.example.com/users?offset=25&maxResults=25>; rel="next"

  • top level links property in response data, with sub-property next, e.g.:

    {
  locations: [
    {"name": "Seattle", "state": "WA"},
    {"name": "New York", "state": "NY"},
    {"name": "Bellevue", "state": "WA"},
    {"name": "Olympia", "state": "WA"}
  ]
  links: {
    "self": "https://provider.example.com/users?offset=0&maxResults=25",
    "next": "https://provider.example.com/users?offset=1&maxResults=25",
    "last": "https://provider.example.com/users?offset=517&maxResults=25",
  }
}

If the API requires paging but uses a scheme other than the above, you will need to define the Next Page URL selector. Unlike the other two JMESPath selectors, input to the Next Page URL Selector is not just the response data, but rather a JSON object that contains other contextual information as well:

  • initialUrl: The URL used to get the previous page (before any redirection)

  • finalUrl: the URL used to get the previous page (after all redirections)

  • statusCode: the HTTP status code returned from the request for the previous page

  • recordOffset: the total number of records returned by previous requests

  • pageOffset: the total number of pages returned by previous requests

  • data: the previous response JSON data

  • headers: the previous responses HTTP headers, e.g.:

    Link: <https://api.github.com/user/repos?page=3&per_page=100>; rel="next",<https://api.github.com/user/repos?page=50&per_page=100>; rel="last"

  • links: the digested form of link headers from previous response, e.g.:

    {
  "next": "https://api.github.com/user/repos?page=3&per_page=100",
  "last": "https://api.github.com/user/repos?page=50&per_page=100"
}

Example: OneRoster 1.1 uses query parameters offset and limit to control paging behavior. The specification recommends that servers should return link headers as described above, but some implementations do not. So in order to implement paging, configure the initial URL path for a user's records to be something close to https://oneroster.example.com/ims/oneroster/v1p1/users?offset=0&limit=500 and then, after handling the first page to generate the URL for the next page, you may be presented with something similar to this:

{
    "initialUrl": "https//oneroster.example.com/ims/oneroster/v1p1/users?offset=0&limit=500",
    "finalUrl": "https://oneroster-1.example.com/ims/oneroster/v1p1/users?offset=0&limit=500",
    "statusCode": 200,
    "recordOffset": 500,
    "pageOffset": 1,
    "data": {"users": [...]},
    "headers": {"Content-type": ["application/json"], ...},
    "links": {}
}

And now you can use a Next Page URL selector such as:

replace(finalUrl,'offset=\d*',concat('offset=',recordOffset))

to generate the next page URL: https://oneroster-1.example.com/ims/oneroster/v1p1/users?offset=500&limit=500

In this section: