Advanced usage of selenider

Customizing the session creation

library(selenider)

[selenider_session()] is really just a wrapper around either chromote::ChromoteSession$new(), or selenium::selenium_server() and selenium::SeleniumSession$new(). selenider exposes arguments to these functions (plus some additional options) via the options argument.

The most common argument that you are going to want to use is headless in chromote_options(): it allows you to run chromote in non-headless mode, meaning that the browser you are controlling will be displayed:

session <- selenider_session(
  "chromote",
  options = chromote_options(headless = TRUE)
)

Managing selenium options is a bit more complex, since you are can provide options to the client selenium_client_options() and server selenium_server_options(). One cool thing you can do is pass NULL into the server_options parameter of selenium_options() to stop selenider from creating its own server. This is useful if you have created a server manually (using docker, for example):

session <- selenider_session(
  "selenium",
  options = selenium_options(
    server_options = NULL, # Stop selenider from creating a server
    client_options = selenium_client_options(
      host = "localhost", # Use the host and port of your manually created server
      port = 4444L
    )
  )
)

Accessing the underlying session

While selenider provides a high level interface, sometimes you need to access the underlying chromote::ChromoteSession or selenium::SeleniumSession to perform more advanced tasks. The driver field of a selenider_session() can be used to do this.

This is especially useful for chromote, since much of the configuration is done after the session is created:

session <- selenider_session()

chromote_session <- session$driver

chromote_session$Browser$setDownloadBehavior(
  behavior = "allow",
  downloadPath = "<path_to_folder>"
)

Accessing underlying elements

Much like you can access the underlying chromote/selenium session behind a selenider session, you can access the chromote/selenium element represented by a selenider_element/selenider_elements object using get_actual_element() and get_actual_elements(), respectively.

If you are using chromote, the backendNodeId of the element is returned, while in selenium’s case, the element is returned as a selenium::WebElement. It’s important to note that the element in this form is no longer lazy, so should be used as soon as possible to avoid errors as the page changes.

Element collections

Let’s use selenider to get every link element in the R Project’s website.

open_url("https://www.r-project.org/")
#> Can't find an existing selenider session.
#> ℹ Creating a new session.

links <- ss("a")

links
#> { selenider_elements (61) }
#> [1] <a href="/"><img src="/Rlogo.png" width="100" height="78" alt="R"></a>
#> [2] <a href="/">[Home]</a>
#> [3] <a href="https://cran.r-project.org/mirrors.html">CRAN</a>
#> [4] <a href="/about.html">About R</a>
#> [5] <a href="/logo/">Logo</a>
#> [6] <a href="/contributors.html">Contributors</a>
#> [7] <a href="/news.html">What’s New?</a>
#> [8] <a href="/bugs.html">Reporting Bugs</a>
#> [9] <a href="/conferences/">Conferences</a>
#> [10] <a href="/search.html">Search</a>
#> [11] <a href="/mail.html">Get Involved: Mailing Lists</a>
#> [12] <a href="https://contributor.r-project.org/">Get Involved: Contributing</a>
#> [13] <a href="https://developer.R-project.org/">Developer Pages</a>
#> [14] <a href="https://blog.r-project.org/">R Blog</a>
#> [15] <a href="/foundation/">Foundation</a>
#> [16] <a href="/foundation/board.html">Board</a>
#> [17] <a href="/foundation/members.html">Members</a>
#> [18] <a href="/foundation/donors.html">Donors</a>
#> [19] <a href="/foundation/donations.html">Donate</a>
#> [20] <a href="/help.html">Getting Help</a>
#> ...

But what actually is links? In some ways, it acts like a list:

links[[1]]
#> { selenider_element }
#> <a href="/">
#>   <img src="/Rlogo.png" width="100" height="78" alt="R">
#> </a>

links[1:2]
#> { selenider_elements (2) }
#> [1] <a href="/"><img src="/Rlogo.png" width="100" height="78" alt="R"></a>
#> [2] <a href="/">[Home]</a>

length(links)
#> [1] 61

But assuming it is a list in all scenarios can result in surprising behavior:

names(links)
#> [1] "session"     "driver"      "driver_id"   "element"     "timeout"    
#> [6] "selectors"   "to_be_found"

To reveal why this is, let’s emulate adding a new link to the page using JavaScript.

execute_js_expr("
  const link = document.createElement('a');
  link.href = 'https://ashbythorpe.github.io/selenider/';
  link.innerText = 'Selenider';
  document.body.appendChild(link);
")
#> NULL

Now let’s look at links again:

links
#> { selenider_elements (62) }
#> [1] <a href="/"><img src="/Rlogo.png" width="100" height="78" alt="R"></a>
#> [2] <a href="/">[Home]</a>
#> [3] <a href="https://cran.r-project.org/mirrors.html">CRAN</a>
#> [4] <a href="/about.html">About R</a>
#> [5] <a href="/logo/">Logo</a>
#> [6] <a href="/contributors.html">Contributors</a>
#> [7] <a href="/news.html">What’s New?</a>
#> [8] <a href="/bugs.html">Reporting Bugs</a>
#> [9] <a href="/conferences/">Conferences</a>
#> [10] <a href="/search.html">Search</a>
#> [11] <a href="/mail.html">Get Involved: Mailing Lists</a>
#> [12] <a href="https://contributor.r-project.org/">Get Involved: Contributing</a>
#> [13] <a href="https://developer.R-project.org/">Developer Pages</a>
#> [14] <a href="https://blog.r-project.org/">R Blog</a>
#> [15] <a href="/foundation/">Foundation</a>
#> [16] <a href="/foundation/board.html">Board</a>
#> [17] <a href="/foundation/members.html">Members</a>
#> [18] <a href="/foundation/donors.html">Donors</a>
#> [19] <a href="/foundation/donations.html">Donate</a>
#> [20] <a href="/help.html">Getting Help</a>
#> ...

links[[length(links)]]
#> { selenider_element }
#> <a href="https://ashbythorpe.github.io/selenider/">
#>   Selenider
#> </a>

links has been updated to include the new link!

A lazy list

The core reason behind this strange behavior is selenider’s promise of laziness. This means that elements are only ever collected from the page right before they are used by an eager function (print(), elem_text(), elem_click(), etc.). The only thing a selenider element actually stores is the path to an element (i.e. the set of steps you specified to reach the element), rather than the element itself.

This property offers an array of benefits when compared with the eager approach. It offers a far more suitable representation of a constantly-changing webpage, and as such side-steps many common errors encountered during web automation. It also powers the automatic waiting feature that is also offered by selenider.

The element collection, then, is a generalisation of this concept to sets of elements. A selenider_elements object stores the path to its elements, but not the elements itself. It therefore cannot be represented by a list; for one thing, as seen above, it is necessarily unaware of its length.

For all of the advantages of lazy elements, this choice of structure does come with some caveats. The major one is that many list operations will not work on an element collection; in fact, you should assume that any operation that works on a list will not work on a selenider_elements object. This is in part due to the fact that R does not natively support custom iterators.

So, what can I do?

selenider provides an API for working with element collections. All of the methods below preserve the laziness of the element collection, meaning that none of them will actually fetch any elements from the page until the resulting element is used.

elems[[x]] and elems[x] work with numeric indices, including negative numbers, allowing you to filter elements by position.
elem_filter() and elem_find() allow you to filter an element collection or find a single element based on a condition.
elem_flatten() allow you to combine multiple elements or element collections into a single collection.
find_each_element() and find_all_elements() allow you to easily find children of all the elements in a collection.

As seen before, length() can be used on element collections to get the number of elements. This is not lazy, meaning you shouldn’t rely on this value to always be accurate after it is called.

However, sometimes you want to perform more complex operations on a set of elements. One common example is iteration, either in a for loop or using lapply()/purrr::map(). Iteration is an operation that goes against the idea of a lazy collection: how do you iterate over a set that is constantly changing?

In this situation, if you are willing to sacrifice some of the lazy properties of an element collection, use as.list(). This function, when called on an element collection elems, converts it to the following:

list(elems[[1]], elems[[2]], ..., elems[[n]])

Where n is length(elems).

Notably, the elements of the list are still lazy, since [[ preserves laziness on element collections. However, the length of the list is not, since the call to length() is not lazy.

Since this is an actual list, it supports a much wider range of operations. For example, in selenider’s README, as.list() is used to iterate over a collection of links to find their hyperlinks. Take a look at as.list.selenider_elements() for more examples.

Forcing eager behaviour

Sometimes it may be desirable to avoid the lazy behaviour of selenider’s elements. This is usually for performance reasons: you may have an element represented by a long, complex set of steps, which needs to be used many times. By default, selenider will follow the path every time the element is used, which can end up being very slow, and may be redundant if you know the element’s position is unlikely to change.

elem_cache() can be used to force an element or set of elements to be retrieved from the DOM and stored, creating an “eager” element. Note the caveat in the docs: further elements created using this element will not also be eager, but will use this eager element as a starting point.