Click on accept cookie buton with R selenium

Question

I am trying to accept cookies on a specific site using R selenium. Normally, this is no issue but with this specific site I am not able to do it. I have tried multiple things, but none of them work. What I tried:

the website: https://myterminal.ect.nl/app/object-schedule

Initialization

library(RSelenium)

driver <- rsDriver(browser = "firefox")
remDr <- driver[["client"]]
remDr$open()

remDr$navigate("https://myterminal.ect.nl/app/object-schedule")

xpath

Normally, I use the xpath finder extension of firefox to get the xpath and use the findElement method from the Rselenium package to fnd the button. I was not able to retrieve the xpath taht way, but after inspecting I got an xpath which seemed to be the right one. But after I use the clickElement method, nothing happens (no errors). I even tried to first switch to the frame of the dialog pop up, but this also wont work;

remDr$switchToFrame("ect-cookie-dialog")
btn1 <- remDr$findElement(using = "xpath", '(//ect-cookie-dialog[@class="hydrated"])[1]')
btn1$clickElement()

css selector

As an alternative I tried to use the css selector, but this gave the same output (none, with no errors)

remDr$switchToFrame("ect-cookie-dialog")
btn1 <- remDr$findElement(using = 'css selector', 'ect-button')
btn1$clickElement()

note: I tried above methods using switch frame and without using switch frame

Mouse click

I tried to move the mouse to the button, and click it to accept the cookie. When I move the mouse, I see the button getting 'highlighted' for a very short time (<1s) like its hovering over it, but after using the click method still nothing happens, no errors. I Had to use an offset, for some reason the Privacy & cookie policy button is highlighted if I don't, maybe they have the same xpath (?)

btn1 <- remDr$findElement(using = 'css selector', 'ect-button')
btn1 <- remDr$findElement(using = "xpath", '(//ect-cookie-dialog[@class="hydrated"])[1]')

# I tried for both definations of 'btn1'

remDr$mouseMoveToLocation(x = 150, y = 0, webElement = btn1)
remDr$doubleclick()

GetAllCookies

Finally, I tried to use the getAllCookies function. I tried this in 2 ways; navigate to the page, get the cookies and apply them. And I tried getting the cookies, saving them locally, restarting R and load the cookies prior to navigating to the site, both did not work;

# Method 1; Directly use cookies;
remDr$navigate("https://myterminal.ect.nl/app/object-schedule")
cookies <- remDr$getAllCookies()
saveRDS(cookies, "cookies.rds")
for (i in 1:length(cookies)) {
  remDr$addCookie(name = cookies[[i]][["name"]], value = cookies[[i]][["value"]])
}

# Method 2; load saved cookies prior to navigating to page
cookies <- readRDS("cookies.rds")
for (i in 1:length(cookies)) {
  remDr$addCookie(name = cookies[[i]][["name"]], value = cookies[[i]][["value"]])
}
remDr$navigate("https://myterminal.ect.nl/app/object-schedule")

After trying all this, still nothing works. I know it should be possible, I've seen that the site can be scraped using Octoparse (webscraping tool) which also automaticly accepts/saves the cookies. But After trying all of the above I have ran out of methods. Probably I am not applying them well, or maybe I am overlooking a simpler solution. Either way, any help will be greatly appreciated!

Kind regards.

Answer 1

I have found the solution, thanks to @DebanjanB

remDr$executeScript('return document.querySelector("body > ect-cookie-dialog").shadowRoot.querySelector("#cookie-dialog > div._ect-cookie-dialog__links > div > ect-button").shadowRoot.querySelector("button").click()')

Executing this line clicks the button. I coppied the JS path of the element and then I added 'return' at the head, and '.click()' at the tail of the string. I believe this is a JAVA command. I enter this string in the remDr$executeScript() function and that did the trick.

Answer 2

Above solution didn't work in my case.

library(shadowr) does wonders! It's also quicker and very simple.

myurl <- remDr$navigate("https://myterminal.ect.nl/app/object-schedule")
remDr$navigate(myurl)
shadow_rd <- shadow(remDr) 
element <- find_elements(shadow_rd, 'button[class="_ect-button"]')[[5]]
element$getElementText()[[1]] #to preview the element we found
element$clickElement()[[1]]

In your case there are nested shadow-root's. That's why we used find_elements and [[5]] .

And how did I find that [[5]] ?

tofind <- find_elements(shadow_rd, 'button[class="_ect-button"]')
hey <- unlist(sapply(tofind, function(x) { x$getElementText() }))
hey

Your button is the 5th item according to the result of above code. So it's [[5]].

There's more info about shadowr library here.

I hope it becomes helpful for people whom cannot solve with above solution.