Excel VBA / HTML Clicking next page from dropdown

[Edit: I now have a solution and the code has been replaced. -RDH]

First I want to mention that if the data retrieved in this manner is used for commercial purposes or anything other than personal use then it violates 2 sections of the Kelley Blue Book (kbb.com) Terms of Service.

FYI: Sites that collect, update, and maintain data like BlueBook or the MLS take their data very seriously, and they don't like people scraping it. I was speaking to an old classmate of mine who has her degree in Computer Science and is now a real estate agent, and I mentioned to her about how cool it is to be able scrape housing data off of MLS and she nearly flipped out on me. Just saying: people were paid to create that data and people make their lives using that data. 'Nuff said.I was able to get the problem code running by creating a web page on my own server that had the same format you were looking for since I get a different version of the bluebook.com site since I am in Canada. I get redirected to kbb.com.

+++ The real problem +++

The problem is that hrefs with an # symbol are actually the full URL with the # attached to the end, and when you check the onClick event it actually contains the full function declariation, so you have to only search for partial strings.

' A good idea to declare the proper datatypes' because IHTMLElement has the click event but IHTMLAnchorElements don'tDim l As IHTMLElementDim htmlanchors As IHTMLElementCollection' ...Set htmlanchors = ie.Document.getElementsByTagName("a")' Look through all the anchor tags on the page    For Each l In htmlanchors       ' Check to see the Href contains a # and the onclick event has specific code        If InStr(l.href, "#") And InStr(l.onclick, "changePage(3); return false;") Then            ' Click the current anchor link            l.Click            Exit For        End IfNext l

Have you tried

.FireEvent ("onclick")Or.FireEvent ("onmouseover").FireEvent ("onmousedown").FireEvent("onmouseup")

in place of .click? Sometimes the JavaScript actions don't respond to .click.

Rick – below is my entire code. I’m basically trying to scrape www.the bluebook.com.

Sub ScrapeData()Dim ie As InternetExplorerDim ele As ObjectDim RowCount As LongDim myWebsite As String, mySearch1 As String, mySearch2 As String, mySearch3 As StringDim Document As HTMLDocumentmyWebsite = Range("Website").ValuemySearch1 = Range("search1").ValuemySearch2 = Range("search2").ValuemySearch3 = Range("search3").ValueSet mySheet = Sheets("Sheet1")Range("A6").Value = "Company"Range("B6").Value = "Address"Range("C6").Value = "Contact"RowCount = 7Set ie = New InternetExplorerie.Visible = TrueWith ie.Visible = True.navigate (myWebsite)Do While .Busy Or .readyState <> 4    DoEventsLoopie.Document.getElementById("search").Value = mySearch1ie.Document.getElementById("selRegion").Value = mySearch2ie.Document.getElementsByClassName("searchBtn")(0).ClickDo While .Busy Or _    .readyState <> 4    DoEventsLoopFor Each ele In .Document.all    Select Case ele.className    Case "result_title"    RowCount = RowCount + 1    Case "cname"    mySheet.Range("A" & RowCount) = ele.innerText    Case "addy_wrapper"    mySheet.Range("B" & RowCount) = ele.innerText    End SelectNext eleEnd With'THIS IS THE CODE THAT IS NOT WORKINGFor Each l In ie.Document.getElementsByTagName("a")    If l.href = "#" And l.onclick = "changePage(3); return false;" Then        l.Item(3).Click        Exit For    End IfNext lSet ie = NothingEnd Sub