如何在使用HttpWebRequest在C＃中获取http响应时处理JavaScript？

Question

我有一个C＃代码（它是一个Web应用程序，它托管在IIS上），我使用HttpWebRequest来获取HttpWebResponse。我向任何网站提出请求&以字符串的形式得到响应，然后我分析响应字符串。但是最近我在页面加载到浏览器中后，JavaScript执行数据提取的响应。

我试图在firebug中调试这个&看到在响应的底部有一个JavaScript函数，它在页面加载后更新dom元素。有什么办法可以在我的C＃代码中做同样的事情吗？我在网上搜索了这个到现在为止找不到的解决方案。

以下是我使用的代码：

HttpWebResponse response = (HttpWebResponse)request.GetResponse(); 

     foreach (Cookie cook in response.Cookies) 
     { 
      Cookie cookie = new Cookie(); 
      cookie.Name = cook.Name; 
      cookie.Value = cook.Value; 
      cookie.Domain = cook.Domain; 
      cookie.Expires = DateTime.Now.AddDays(10); 
      cookieList.Add(cookie); 
     } 

     string postData = string.Format("username=" + txtUserID.Text + "&password=" + txtPwd.Text + "&url=https://example.com/&game="); 
     byte[] postBytes = Encoding.UTF8.GetBytes(postData); 
     HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://login.example.com/Login/authenticate"); 
     req.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0"; 
     req.KeepAlive = true; 
     req.AutomaticDecompression = DecompressionMethods.GZip; 

     ////set the cookie 
     req.CookieContainer = new CookieContainer(); 
     foreach (Cookie cook in cookieList) 
     { 
      Cookie cookie = new Cookie(); 
      cookie.Name = cook.Name; 
      cookie.Value = cook.Value; 
      cookie.Domain = cook.Domain; 
      cookie.Expires = DateTime.Now.AddDays(10); 
      req.CookieContainer.Add(cookie); 
     } 

     req.Headers.Add("Accept-Encoding", "gzip, deflate"); 
     req.Headers.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6");//en-GB,en-US;q=0.8,en;q=0.6 
     req.Method = "POST"; 
     req.Host = "login.example.com"; 
     req.Referer = "https://login.example.com/Login/logout"; 
     req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"; 

     req.ContentType = "application/x-www-form-urlencoded;"; 
     req.ContentLength = postBytes.Length; 

     //getting the request stream and posting data 
     StreamWriter requestwriter = new StreamWriter(req.GetRequestStream(), System.Text.Encoding.ASCII); 
     requestwriter.Write(postData); 
     requestwriter.Close(); 
     HttpWebResponse myHttpWebResponse = (HttpWebResponse)req.GetResponse(); 
     Stream responseStream = myHttpWebResponse.GetResponseStream(); 
     StreamReader myStreamReader = new StreamReader(responseStream, Encoding.ASCII); 
     string responseString = myStreamReader.ReadToEnd(); 
     myStreamReader.Close(); 
     responseStream.Close(); 
     myHttpWebResponse.Close(); 

Answer 1

我终于得到了简单的解决方案，以我的需要。以下是我跟着链接： Link to tutorial

以下是会得到的结果代码：

首先，您需要输入以下内容：

using System.Drawing; 
using OpenQA.Selenium; 
using OpenQA.Selenium.PhantomJS; 
using System.Text.RegularExpressions; 
using System.IO; 
using HtmlAgilityPack; 

现在代码：

 var options = new PhantomJSOptions(); 
     var driver = new PhantomJSDriver(options); 
     driver.Manage().Window.Size = new Size(1360, 728); 
     var size = driver.Manage().Window.Size; 

     driver.Navigate().GoToUrl("https://example.com/"); 
     string url = driver.Url; 
     //the driver can now provide you with what you need (it will execute the script) 
     //get the source of the page 
     var source = driver.PageSource; 
     //fully navigate the dom 
     var pathElement1 = driver.FindElementByName("username"); 
     var pathElement2 = driver.FindElementByName("password"); 
     var pathElement3 = driver.FindElementByXPath("//button[@class='SubmitButton']"); 

     pathElement1.Clear(); 
     pathElement1.SendKeys("username"); 
     pathElement2.Clear(); 
     pathElement2.SendKeys("password"); 
     pathElement3.Click(); 

     //Now get the response after login button click 
     source = driver.PageSource;