Pandaazuma 1,295 Posted September 19, 2016 Pardon me if I say something idiotic, but could not a file be created (as Rando described) from which the usual database grabber pulls the data, just as it did until yesterday from the kyokai site, as or when the data is manually input? Share this post Link to post Share on other sites
Jakusotsu 5,193 Posted September 19, 2016 6 minutes ago, Asashosakari said: However, I suspect no matter how many of us do that, the bottleneck remains that the collected/formatted data needs to be injected into the database, and that's definitely a non-trivial task given the complexity of the data structures. (I assume that's what Jakusotsu is also getting at.) Exactly. I could definitely work out what has to be done with the obvious table, but I have no clue which other possibly hidden parameters have to be set. 1 minute ago, Pandaazuma said: Pardon me if I say something idiotic, but could not a file be created (as Rando described) from which the usual database grabber pulls the data, just as it did until yesterday from the kyokai site, as or when the data is manually input? Not idiotic at all, I had the same idea. The problem though is to point the grabber at the new source, which is also something only Doitsuyama can do. 1 Share this post Link to post Share on other sites
Pandaazuma 1,295 Posted September 19, 2016 Ok. At least we have some hope. I have faith whatever happens in Doitsuyama! :) Share this post Link to post Share on other sites
ronnie 221 Posted September 19, 2016 Would it be possible to email entries for day 10 just in case a solution is found before the deadline for the games? And is there a list of email addresses for the affected games? Share this post Link to post Share on other sites
Asashosakari 17,372 Posted September 19, 2016 33 minutes ago, Jakusotsu said: Exactly. I could definitely work out what has to be done with the obvious table, but I have no clue which other possibly hidden parameters have to be set. Not idiotic at all, I had the same idea. The problem though is to point the grabber at the new source, which is also something only Doitsuyama can do. Also, the HTML code in the Kyokai's pages is messy enough that trying to recreate it faithfully enough that the grabber can parse it successfully is probably more trouble than it's worth. Easier to just put the results in a spreadsheet or similar, which can be imported into the database in a more straight-forward manner. But even for that we'd need to know exactly what data is required, and in which format. Years ago Doitsuyama had me fill in a sheet like that with some maezumo bout data, and I recall being surprised how complex even the "template" for that was. Share this post Link to post Share on other sites
nomadwolf 8 Posted September 19, 2016 5 hours ago, Jakusotsu said: Looks like the NSK messed with their site again. The buttons at http://www.sumo.or.jp/EnHonbashoMain/torikumi don't point to permalinks anymore but some javascript (how I hate it!!). I fear we might be screwed for the rest of the basho because that could turn into a nightmare for the grabbers. Yeah, this looks to also have taken out the @sumo twitter account... and my Excel grabber, which will make my picks that much slower... Took an hour to figure out how to manually copy the results to my sheets... Share this post Link to post Share on other sites
Pandaazuma 1,295 Posted September 19, 2016 11 minutes ago, nomadwolf said: Yeah, this looks to also have taken out the @sumo twitter account... and my Excel grabber, which will make my picks that much slower... Took an hour to figure out how to manually copy the results to my sheets... Hang on...if this has taken out their Twitter account, this might actually be a one-off screw-up by the tech people. Surely they wouldn't have wanted to mess that up. At least let's hope so... Share this post Link to post Share on other sites
Asashosakari 17,372 Posted September 19, 2016 (edited) 12 minutes ago, Pandaazuma said: Hang on...if this has taken out their Twitter account, this might actually be a one-off screw-up by the tech people. Surely they wouldn't have wanted to mess that up. At least let's hope so... That's not the Kyokai's account (which is at http://twitter.com/sumokyokai). Edited September 19, 2016 by Asashosakari Share this post Link to post Share on other sites
Pandaazuma 1,295 Posted September 19, 2016 Thanks, Asasho...I don't follow them on Twitter...so assumed incorrectly. Share this post Link to post Share on other sites
Jakusotsu 5,193 Posted September 19, 2016 If I was a pranoid conspiracist, I'd get the feeling that this is a planned move aimed to prevent exactly what Doitsuyama's doing. First they took away the live stream... Share this post Link to post Share on other sites
Doitsuyama 1,157 Posted September 19, 2016 Well yeah, they changed the html zu embedded ajax, so the clicks on day or division don't need a page change, only the table is rewritten. Nice, and not really new for me to grab. Recreating the ajax requests is easy enough for me. Problem is, these requests only give empty results, so it is likely that there is a server-side check if the request is coming from the page or from external sources which is not nice. If there are people willing to enter the results into the database (and thankfully it looks like it), I can make this possible. 4 Share this post Link to post Share on other sites
Jejima 1,082 Posted September 19, 2016 (edited) This basho is tricky for me - due to moving house this week - (plus the baby and the pregnant wife....) But I would be available (at a pinch) to help out tomorrow or Wednesday (after that, it is last minute packing before the big move on Saturday). Next basho (if the problem persists), I will be happy to volunteer Edited September 19, 2016 by Jejima Share this post Link to post Share on other sites
Pandaazuma 1,295 Posted September 19, 2016 1 minute ago, Doitsuyama said: Well yeah, they changed the html zu embedded ajax, so the clicks on day or division don't need a page change, only the table is rewritten. Nice, and not really new for me to grab. Recreating the ajax requests is easy enough for me. Problem is, these requests only give empty results, so it is likely that there is a server-side check if the request is coming from the page or from external sources which is not nice. If there are people willing to enter the results into the database (and thankfully it looks like it), I can make this possible. I wonder why they did that. Ok...I will be happy to do Makuuchi and Juryo. I can also input the next day's bouts. Just send me the info I need and I will do it as soon as I get it. For the games at least that is a priority. The lower-division results can of course be updated at a more leisurely pace. Just PM me with the info I need. Share this post Link to post Share on other sites
Jejima 1,082 Posted September 19, 2016 6 minutes ago, Jakusotsu said: If I was a pranoid conspiracist, I'd get the feeling that this is a planned move aimed to prevent exactly what Doitsuyama's doing. First they took away the live stream... I was having the same paranoid thoughts - but then I thought that no one could have that big a grudge against the sumo loving foreign community.... could they? Share this post Link to post Share on other sites
Pandaazuma 1,295 Posted September 19, 2016 The Sumo Game results are now up...and they look OK. The gamesmaster says he found another Japanese site that has results and, after some modifications, he was able to use his grabber there. I've asked him for the URL and I'll post it here as soon as I get it. Hopefully that may provide at least the game-related divisions. Share this post Link to post Share on other sites
Pandaazuma 1,295 Posted September 19, 2016 33 minutes ago, Jejima said: I was having the same paranoid thoughts - but then I thought that no one could have that big a grudge against the sumo loving foreign community.... could they? Well, I do know one guy, actually...and so do you! ;) Share this post Link to post Share on other sites
Frinkanohana 13 Posted September 19, 2016 36 minutes ago, Doitsuyama said: Well yeah, they changed the html zu embedded ajax, so the clicks on day or division don't need a page change, only the table is rewritten. Nice, and not really new for me to grab. Recreating the ajax requests is easy enough for me. Problem is, these requests only give empty results, so it is likely that there is a server-side check if the request is coming from the page or from external sources which is not nice. If there are people willing to enter the results into the database (and thankfully it looks like it), I can make this possible. I also tried to recreate the ajax. It seams like the server is checking if the HTTP header field with the name "X-Requested-With" and value "XMLHttpRequest" is present. Without that header field I get empty results, with the header I get results. Hope that helps. 9 Share this post Link to post Share on other sites
Pandaazuma 1,295 Posted September 19, 2016 (edited) Frinkanohana's comments sound promising. In the meantime, the Sumo Game guy used this site: http://www.nikkansports.com/battle/sumo/score/ It only has Makuuchi but that's something, I suppose. Edited September 19, 2016 by Pandaazuma 1 Share this post Link to post Share on other sites
ronnie 221 Posted September 19, 2016 18 minutes ago, Pandaazuma said: Frinkanohana's comments sound promising. In the meantime, the Sumo Game guy used this site: http://www.nikkansports.com/battle/sumo/score/ It only has Makuuchi but that's something, I suppose. You should do a Google translate on that link. It lists one kimarite as "Duster Included" and the shikonas are pretty good too. 2 Share this post Link to post Share on other sites
Pandaazuma 1,295 Posted September 19, 2016 I'm off to bed shortly. @Doitsuyama and @Golynohana take a look at Frinkanohana's comment - it may be useful for you in finding a better solution. If manual input is the only way, I'll do it first thing in the morning. If you need it done more quickly, maybe get another volunteer to do it for today. Share this post Link to post Share on other sites
nomadwolf 8 Posted September 19, 2016 If we are going to manual entry, I can whip up a spreadsheet that allows you to manually copy the website table and it spits out what Doitsuyama needs, but that depends on what he needs... :D Share this post Link to post Share on other sites
Doitsuyama 1,157 Posted September 19, 2016 Thanks to Frinkanohana's comment I got it working again. I tried it with this field already, but had to tweak it a bit more to make it work. Looks like it is up and running again. 13 Share this post Link to post Share on other sites
Achiyama 1,149 Posted September 19, 2016 17 minutes ago, Doitsuyama said: Thanks to Frinkanohana's comment I got it working again. I tried it with this field already, but had to tweak it a bit more to make it work. Looks like it is up and running again. Vielen Dank Doitsuyama-zeki, You are a real Wizard! I tip my hat to you! Share this post Link to post Share on other sites
Jakusotsu 5,193 Posted September 19, 2016 Excellent, Sir! Once again you save our day. And cheers to Frinkanohana as well, of course! Share this post Link to post Share on other sites
Ganzohnesushi 444 Posted September 19, 2016 Brilliant work, Doitsuyama-san! And kudos to Frinkanohana for his comment. Ganzohnesushi Share this post Link to post Share on other sites
