The program keeps throwing exceptions #13

ziqizh · 2019-12-04T20:17:12Z

Edit: I digged deeper and realized that the webpage https://www.ebay.com/signin/s periodically forbids GET method. For example, you will get 405 error if you use wget https://www.ebay.com/signin/s
When the GET method is allowed, the program runs into the second exception, NoSuchElementException.

The program throws two kinds of exception when logging in. Any ideas?

I ran it by: docker run -d -p 3647:3647 -e 'EBAY_USERNAME=myaccount' -e 'EBAY_PASSWORD=mypassword' ruippeixotog/ebay-snipe-server:0.2.3-SNAPSHOT

2019-12-05 14:40:11,019 INFO  Slf4jLogger - Slf4jLogger started
2019-12-05 14:40:11,112 INFO  SnipeServer$ - Logging in into eBay
2019-12-05 14:40:11,112 DEBUG EbayLoginManager - Getting the sign in cookie for ebay.com
Exception in thread "main" org.jsoup.HttpStatusException: HTTP error fetching URL. Status=405, URL=https://www.ebay.com/signin/s
	at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:760)
	at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:706)
	at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:299)
	at net.ruippeixotog.scalascraper.browser.JsoupBrowser.executeRequest(JsoupBrowser.scala:73)
	at net.ruippeixotog.scalascraper.browser.JsoupBrowser.$anonfun$executePipeline$3(JsoupBrowser.scala:84)
	at scala.Function1.$anonfun$andThen$1(Function1.scala:52)
	at scala.Function1.$anonfun$andThen$1(Function1.scala:52)
	at net.ruippeixotog.scalascraper.browser.JsoupBrowser.post(JsoupBrowser.scala:39)
	at net.ruippeixotog.scalascraper.browser.JsoupBrowser.post(JsoupBrowser.scala:30)
	at net.ruippeixotog.ebaysniper.ebay.EbayLoginManager.forceLogin(EbayLoginManager.scala:30)
	at net.ruippeixotog.ebaysniper.ebay.EbayClient.login(EbayClient.scala:30)
	at net.ruippeixotog.ebaysniper.SnipeServer$.delayedEndpoint$net$ruippeixotog$ebaysniper$SnipeServer$1(SnipeServer.scala:38)
	at net.ruippeixotog.ebaysniper.SnipeServer$delayedInit$body.apply(SnipeServer.scala:20)
	at scala.Function0.apply$mcV$sp(Function0.scala:34)
	at scala.Function0.apply$mcV$sp$(Function0.scala:34)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App.$anonfun$main$1$adapted(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:389)
	at scala.App.main(App.scala:76)
	at scala.App.main$(App.scala:74)
	at net.ruippeixotog.ebaysniper.SnipeServer$.main(SnipeServer.scala:20)
	at net.ruippeixotog.ebaysniper.SnipeServer.main(SnipeServer.scala)

2019-12-04 20:43:43,160 INFO  Slf4jLogger - Slf4jLogger started
2019-12-04 20:43:43,251 INFO  SnipeServer$ - Logging in into eBay
2019-12-04 20:43:43,251 DEBUG EbayLoginManager - Getting the sign in cookie for ebay.com
Exception in thread "main" java.util.NoSuchElementException
	at java.util.ArrayList$Itr.next(ArrayList.java:862)
	at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:40)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:448)
	at scala.collection.IterableLike.head(IterableLike.scala:106)
	at scala.collection.IterableLike.head$(IterableLike.scala:105)
	at net.ruippeixotog.scalascraper.model.LazyElementQuery.head(ElementQuery.scala:40)
	at net.ruippeixotog.scalascraper.scraper.ContentExtractors$.net$ruippeixotog$scalascraper$scraper$ContentExtractors$$$anonfun$attr$1(ContentExtractors.scala:77)
	at net.ruippeixotog.scalascraper.scraper.ContentExtractors$$anonfun$attr$2.extract(ContentExtractors.scala:77)
	at net.ruippeixotog.scalascraper.scraper.ContentExtractors$$anonfun$attr$2.extract(ContentExtractors.scala:77)
	at net.ruippeixotog.scalascraper.scraper.HtmlExtractor.apply(HtmlExtractor.scala:23)
	at net.ruippeixotog.scalascraper.scraper.HtmlExtractor.apply$(HtmlExtractor.scala:23)
	at net.ruippeixotog.scalascraper.scraper.ContentExtractors$$anonfun$attr$2.apply(ContentExtractors.scala:77)
	at net.ruippeixotog.scalascraper.scraper.ContentExtractors$.net$ruippeixotog$scalascraper$scraper$ContentExtractors$$$anonfun$formDataAndAction$1(ContentExtractors.scala:103)
	at net.ruippeixotog.scalascraper.scraper.ContentExtractors$$anonfun$8.extract(ContentExtractors.scala:102)
	at net.ruippeixotog.scalascraper.scraper.ContentExtractors$$anonfun$8.extract(ContentExtractors.scala:102)
	at net.ruippeixotog.scalascraper.scraper.HtmlExtractor.$anonfun$mapQuery$1(HtmlExtractor.scala:43)
	at net.ruippeixotog.scalascraper.scraper.HtmlExtractor$$anon$2.extract(HtmlExtractor.scala:60)
	at net.ruippeixotog.scalascraper.dsl.ScrapingOps$ElementsScrapingOps.$anonfun$extract$1(ScrapingOps.scala:15)
	at scalaz.Monad.$anonfun$map$2(Monad.scala:14)
	at scalaz.IdInstances$$anon$1.point(Id.scala:20)
	at scalaz.Monad.$anonfun$map$1(Monad.scala:14)
	at scalaz.IdInstances$$anon$1.bind(Id.scala:22)
	at scalaz.Monad.map(Monad.scala:14)
	at scalaz.Monad.map$(Monad.scala:14)
	at scalaz.IdInstances$$anon$1.map(Id.scala:19)
	at scalaz.syntax.FunctorOps.map(FunctorSyntax.scala:10)
	at net.ruippeixotog.scalascraper.dsl.ScrapingOps$ElementsScrapingOps.extract(ScrapingOps.scala:15)
	at net.ruippeixotog.scalascraper.dsl.ScrapingOps$ElementsScrapingOps.$greater$greater(ScrapingOps.scala:19)
	at net.ruippeixotog.ebaysniper.ebay.EbayLoginManager.forceLogin(EbayLoginManager.scala:27)
	at net.ruippeixotog.ebaysniper.ebay.EbayClient.login(EbayClient.scala:30)
	at net.ruippeixotog.ebaysniper.SnipeServer$.delayedEndpoint$net$ruippeixotog$ebaysniper$SnipeServer$1(SnipeServer.scala:38)
	at net.ruippeixotog.ebaysniper.SnipeServer$delayedInit$body.apply(SnipeServer.scala:20)
	at scala.Function0.apply$mcV$sp(Function0.scala:34)
	at scala.Function0.apply$mcV$sp$(Function0.scala:34)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App.$anonfun$main$1$adapted(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:389)
	at scala.App.main(App.scala:76)
	at scala.App.main$(App.scala:74)
	at net.ruippeixotog.ebaysniper.SnipeServer$.main(SnipeServer.scala:20)
	at net.ruippeixotog.ebaysniper.SnipeServer.main(SnipeServer.scala)

The text was updated successfully, but these errors were encountered:

tryangle · 2019-12-14T04:46:54Z

I get this as well. I don't know if it makes a difference but I am running:
java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1ubuntu1~16.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)

ruippeixotog · 2019-12-20T17:28:39Z

It may be because eBay changed their sign in page. I'll take a look at this.

ruippeixotog · 2019-12-20T18:17:58Z

Unfortunately this is happening because they are now showing Captcha pages to prevent automated login:

<!doctype html>
<html>
 <head> 
  <meta NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> 
  <meta http-equiv="cache-control" content="max-age=0"> 
  <meta http-equiv="cache-control" content="no-cache"> 
  <meta http-equiv="expires" content="0"> 
  <meta http-equiv="expires" content="Tue, 01 Jan 1980 1:00:00 GMT"> 
  <meta http-equiv="pragma" content="no-cache"> 
  <meta http-equiv="refresh" content="10; url=/distil_r_captcha.html?requestId=98a3a58d-e29a-4360-9916-b25a9b969158&amp;httpReferrer=%2Fsignin%2F"> 
  <script type="text/javascript">
	(function(window){
		try {
			if (typeof sessionStorage !== 'undefined'){
				sessionStorage.setItem('distil_referrer', document.referrer);
			}
		} catch (e){}
	})(window);
</script> 
  <script type="text/javascript" src="/nkfytkqtoxtljvzb.js" defer></script>
  <style type="text/css">#d__fFH{position:absolute;top:-5000px;left:-5000px}#d__fF{font-family:serif;font-size:200px;visibility:hidden}#ryutavyvevzbtareszzbwcaq{display:none!important}</style>
 </head> 
 <body> 
  <div id="distilIdentificationBlock">
   &nbsp;
  </div>   
 </body>
</html>

geotheory · 2019-12-20T19:13:00Z

Ah. Does this mean R.I.P.?

ziqizh · 2019-12-20T20:16:47Z

IMO we can solve this by enabling JavaScript. Not sure how to do it tho

geotheory · 2019-12-20T20:58:12Z

Possibly https://medium.com/@jsoverson/bypassing-captchas-with-headless-chrome-93f294518337

ruippeixotog · 2019-12-24T00:30:55Z

Enabling JavaScript is easy enough, it suffices to use an HtmlUnitBrowser instead of a JsoupBrowser in EbayClient.scala. But that is not enough, the server still gets FailingHttpStatusCodeException: 405 Method Not Allowed for https://www.ebay.com/signin/s.

The sole purpose of captchas are to prevent people to automate actions throughout the website. Even if someone found a way to do bypass them today I'm sure eBay will continue to close those holes. It's a cat-and-mouse game I don't have the time or willingness to pursue, so unless someone else wants to play with the project, it's R.I.P. 🙁

geotheory · 2019-12-24T03:03:30Z

Probably exposing my ignorance of the authentication process, but I wonder if a manual workaround might be to log in manually on a regular browser, copy the cookie across to your server machine and make available to that login request?

ruippeixotog · 2019-12-26T12:39:37Z

It all depends on how they are checking for automation. We can copy cookies as a way to bypass captchas on authentication, but there's nothing preventing eBay from adding captchas on the other requests and actions too (if they haven't already). Simple engines like jsoup and HtmlUnit are easily detectable; real browsers controlled by a web driver like Selenium may have a better chance, but it's still a considerable effort to implement support for them and there's no guarantee it will work.

geotheory · 2020-04-21T09:39:45Z

Bah I miss this

anoduck · 2020-06-03T22:00:47Z

So in the great words of Dr. McCoy, Chief Surgeon of the starship Enterprise, "It's dead, Jim."

wicadmin · 2021-04-20T04:39:22Z

What is everyone else using these days?

anoduck · 2021-04-24T04:50:45Z

Just by using duckduckgo to search for "ebay snipe" I came across this project. It was written within 9 months ago as of date of posting, and hopefully still might work.

https://sourceforge.net/projects/esniper/

ruippeixotog mentioned this issue Dec 20, 2019

Sign in failed repeatedly during bid.. cant_sign_in #14

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The program keeps throwing exceptions #13

The program keeps throwing exceptions #13

ziqizh commented Dec 4, 2019 •

edited

Loading

tryangle commented Dec 14, 2019

ruippeixotog commented Dec 20, 2019

ruippeixotog commented Dec 20, 2019

geotheory commented Dec 20, 2019

ziqizh commented Dec 20, 2019

geotheory commented Dec 20, 2019

ruippeixotog commented Dec 24, 2019

geotheory commented Dec 24, 2019

ruippeixotog commented Dec 26, 2019

geotheory commented Apr 21, 2020

anoduck commented Jun 3, 2020

wicadmin commented Apr 20, 2021

anoduck commented Apr 24, 2021

The program keeps throwing exceptions #13

The program keeps throwing exceptions #13

Comments

ziqizh commented Dec 4, 2019 • edited Loading

tryangle commented Dec 14, 2019

ruippeixotog commented Dec 20, 2019

ruippeixotog commented Dec 20, 2019

geotheory commented Dec 20, 2019

ziqizh commented Dec 20, 2019

geotheory commented Dec 20, 2019

ruippeixotog commented Dec 24, 2019

geotheory commented Dec 24, 2019

ruippeixotog commented Dec 26, 2019

geotheory commented Apr 21, 2020

anoduck commented Jun 3, 2020

wicadmin commented Apr 20, 2021

anoduck commented Apr 24, 2021

ziqizh commented Dec 4, 2019 •

edited

Loading