How can I append values from more than one page using Selenium?

I'm posting improved version. However. I cannot say that I am completely satisfied with it.I tried at least three other options, but I could not click Next button without executing Javascript.I am leaving the options I tried commented because I want you to see them.

import timeimport requestsimport pandasimport pandas as pdfrom bs4 import BeautifulSoupfrom selenium import webdriverfrom selenium.webdriver import ActionChainsfrom selenium.webdriver.common.keys import Keysfrom selenium.webdriver.firefox.options import Optionsfrom selenium.webdriver.support.ui import WebDriverWaitfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support import expected_conditions as ECimport jsondriver = webdriver.Chrome(executable_path='/snap/bin/chromium.chromedriver')driver.get(    'https://www.remax.pt/comprar?searchQueryState={%22regionName%22:%22%22,%22businessType%22:1,%22listingClass%22:1,%22page%22:1,%22sort%22:{%22fieldToSort%22:%22ContractDate%22,%22order%22:1},%22mapIsOpen%22:false}')driver.maximize_window()driver.implicitly_wait(5)wait = WebDriverWait(driver, 15)cookies = driver.find_element_by_id('rcc-decline-button')cookies.click()element_list = []for j in range(1, 2569):    try:        for i in range(1, 40, 2):            wait.until(EC.element_to_be_clickable((By.XPATH, "(//div[@class='listing-search-searchdetails-component'])[{0}]".format(i))))            link = driver.find_element_by_xpath(                "(//div[@class='listing-search-searchdetails-component'])[{0}]".format(i))            link.click()            try:                detalhes = driver.find_element_by_id('details')                preco = driver.find_element_by_id('listing-price')                tipo = driver.find_element_by_id('listing-title')                freguesia = driver.find_element_by_xpath('//h5[@class="listing-address"]')                imoveis = [detalhes.text, preco.text, tipo.text, freguesia.text]                element_list.append(imoveis)            finally:                driver.find_element_by_css_selector(".modal-close-icon").click()    finally:        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")        next_btn = driver.find_element_by_xpath("//a[@class='page-link'][.//span[.='Next']]")        # next_btn.send_keys(Keys.PAGE_DOWN)        # driver.execute_script("arguments[0].scrollIntoView();", next_btn)        wait.until(EC.element_to_be_clickable((By.XPATH, "//a[@class='page-link'][.//span[.='Next']]/span")))        # actions = ActionChains(driver)        # actions.move_to_element(next_btn)        # actions.click().perform()        driver.execute_script("arguments[0].click();", next_btn)

Also, note, that I have modified some of your code from the inside to make it more stable (added few locators).Currently, it clicks the next button.

You need to implement it further. I mean you need to grab all the listing again and loop through them. For this you need to wait until the next page completely loads.I don't have an answer for it yet. It will require more time.

Here is the great question about the difference between Selenium's click and JS click WebDriver click() vs JavaScript click()

The question is not completely answered yet, just made more stable. Your click on the next page almost never worked.

Update

for i in range(1,40,2) was the biggest problem.

You tried to click on a listing with id 21, but there are only 21 of them. So, I've changed it to for i in range(1, 20, 2), added one wait on a new page it everything works well now.I'm living the debugging code, so everything would be clear for you.Sorry, I have no more time to check how the list looks like, but it should be easy now.

from selenium import webdriverfrom selenium.webdriver.support.ui import WebDriverWaitfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support import expected_conditions as ECdriver = webdriver.Chrome(executable_path='/snap/bin/chromium.chromedriver')driver.get(    'https://www.remax.pt/comprar?searchQueryState={%22regionName%22:%22%22,%22businessType%22:1,%22listingClass%22:1,%22page%22:1,%22sort%22:{%22fieldToSort%22:%22ContractDate%22,%22order%22:1},%22mapIsOpen%22:false}')driver.maximize_window()driver.implicitly_wait(15)wait = WebDriverWait(driver, 15)cookies = driver.find_element_by_id('rcc-decline-button')cookies.click()element_list = []for j in range(1, 2569):    try:        print("Searching Page " + str(j))        wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='listing-search-searchdetails-component']")))        for i in range(1, 20, 2):            wait.until(EC.element_to_be_clickable((By.XPATH, "(//div[@class='listing-search-searchdetails-component'])[{0}]".format(i))))            el = driver.find_element_by_xpath("(//div[@class='listing-search-searchdetails-component'])[{0}]".format(i))            print("Listing number " + str(i))            link = driver.find_element_by_xpath(                "(//div[@class='listing-search-searchdetails-component'])[{0}]".format(i))            link.click()            try:                detalhes = driver.find_element_by_id('details')                preco = driver.find_element_by_id('listing-price')                tipo = driver.find_element_by_id('listing-title')                freguesia = driver.find_element_by_xpath('//h5[@class="listing-address"]')                imoveis = [detalhes.text, preco.text, tipo.text, freguesia.text]                element_list.append(imoveis)            finally:                driver.find_element_by_css_selector(".modal-close-icon").click()    finally:        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")        next_btn = driver.find_element_by_xpath("//a[@class='page-link'][.//span[.='Next']]")        wait.until(EC.element_to_be_clickable((By.XPATH, "//a[@class='page-link'][.//span[.='Next']]/span")))        driver.execute_script("arguments[0].click();", next_btn)

P.S. What you could accomplish by this time is already good.