Invoke-WebRequest Canvas LMS API pagination

Question

I cannot to seem to find any examples on this topic and I would like to know how to do it. Can anyone show me an example or point me to a link on how to do pagination in powershell with Invoke web-request? The challenge I am facing is that I am making API calls to a server that only returns 100 rows at a time. In order to get any more rows, I would have to make a second call to the server. I have no clue how to do it. If it helps, here is the link provided by the Canvas LMS and my code that I have so far.

Pagination:

Pagination

Requests that return multiple items will be paginated to 10 items by default. You can set a custom per-page amount with the ?per_page parameter. There is an unspecified limit to how big you can set per_page to, so be sure to always check for the Link header.

To retrieve additional pages, the returned Link headers should be used. These links should be treated as opaque. They will be absolute urls that include all parameters necessary to retrieve the desired current, next, previous, first, or last page. The one exception is that if an access_token parameter is sent for authentication, it will not be included in the returned links, and must be re-appended.

Pagination information is provided in the Link header:

Link:
<https://<canvas>/api/v1/courses/:id/discussion_topics.json?opaqueA>; rel="current",
<https://<canvas>/api/v1/courses/:id/discussion_topics.json?opaqueB>;> rel="next",
<https://<canvas>/api/v1/courses/:id/discussion_topics.json?opaqueC>;> rel="first",
<https://<canvas>/api/v1/courses/:id/discussion_topics.json?opaqueD>;> rel="last" 

The possible rel values are:

current - link to the current page of results. next - link to the next page of results. prev - link to the previous page of results. first - link to the first page of results. last - link to the last page of results. These will only be included if they are relevant. For example, the first page of results will not contain a rel="prev" link. rel="last" may also be excluded if the total count is too expensive to compute on each request.

The beginning product

$curlly=""
$url_main="https://[instance].instructure.com/api/v1/accounts/1/courses?per_page=1"
$security_token="imhungry"
$header = @{"Authorization"="Bearer "+ $security_token; "rel"="last"}
$curlly=Invoke-WebRequest -Headers $header   -Method Get   -Uri $url_main   
$curlly = ConvertFrom-Json $curlly.Content
foreach($course in $curlly)
{
    $course.name
}
$curlly.Count

The final product

 ##This is an example on how to use pagination in powershell
$url_main="https://[instance].instructure.com/api/v1/accounts/1/courses?per_page=100"
$security_token="boyimhungry"
$header = @{"Authorization"="Bearer "+ $security_token}
$purlly=Invoke-WebRequest -Headers $header   -Method Get   -Uri $url_main   
$curlly = ConvertFrom-Json $purlly.Content
$url_main = $purlly.Headers.Link.Split(",")[1].Replace("<","").Replace(">","") ## you can get away with just doing one replace("<","") but it looks neater this way
while( !$url_main.Contains("prev"))
{
$purlly=Invoke-WebRequest -Headers $header   -Method Get   -Uri $url_main   
$curlly += ConvertFrom-Json $purlly.Content
$url_main = $purlly.Headers.Link.Split(",")[1].Replace("<","").Replace(">","")
cls
$curlly.Count
$url_main
}
foreach($course in $curlly)
{
    $course.name
}
$curlly.Count

Answer 1

I know you've accepted an answer, but I thought I'd give my code example just in case anybody needs one. This example is getting a list of all of our Canvas users. Not a terrible process -- most of the work is done with just a 4-line do..while loop.

$token = "YOUR_ACCESS_TOKEN"
$headers = @{"Authorization"="Bearer "+$token}
$allCanvasUsers = New-Object System.Collections.ArrayList @()
$pageNumber = 1

Function RequestPageOfUsers($page) {
    $script:resultsPage = Invoke-WebRequest -Method GET -Headers $headers -Uri "https://$domain/api/v1/accounts/self/users?per_page=100&search_term=P00&page=$page"
    $usersPage = ConvertFrom-Json $resultsPage.Content

    foreach ($user in $usersPage) {
        $script:allCanvasUsers.Add($user)
    }
}

do {
    RequestPageOfUsers $pageNumber
    $pageNumber++
} while ($resultsPage.Headers.Link.Contains('rel="next"'))