I'm using Selenium with chromedriver to scrape a list of data using HTML agility pack but when I try to navigate to this page, the url gets changed from url1 to url2. Therefore is there a way to stop this from happening?
string url = "";
Console.WriteLine("Web Crawler!");
Console.WriteLine("Enter URL :");
url = Console.ReadLine();
Console.WriteLine("Entered URL :" + url);
// selenium section
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.AddUserProfilePreference("profile.default_content_setting_values.geolocation", 2);
IWebDriver driver = new ChromeDriver(".", chromeOptions);
driver.Navigate().GoToUrl(url);
The url might not being passed correctly from the CMD. Instead, you can read it from a CSV file
using (TextFieldParser parser = new TextFieldParser("csvFile"))
{
parser.TextFieldType = FieldType.Delimited;
parser.SetDelimiters(",");
string url = string.Empty;
while (!parser.EndOfData)
{
string[] fields = parser.ReadFields();
url = fields[0];
}
}
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With