Jump to content

  • Curse Sites
Help
* * * * * 4 votes

GuildWars2Spidy - Graphs of the Trade Market

trade market graphs spider website

  • Please log in to reply
105 replies to this topic

#31 Drakie

Drakie

    Asuran Acolyte

  • Members
  • 56 posts

Posted 17 August 2012 - 02:09 PM

there's been a reddit post and a post on these forums with all the links they could find which had the gemstore, tradingpost etc.
I think they guessed all the URLs after arenanet gave the URL to the gemstore as preview :-)


Let's hope it's getting back online at some point and it's just maintenance ...

#32 adhome

adhome

    Pale Tree Seedling

  • New Members
  • 5 posts

Posted 17 August 2012 - 05:00 PM

Found Way to login.
h**ps://tradingpost-live.ncplatform.net/authenticate?account_name=Guild Wars 2&session_key=DD6FCAED-8BE8-E111-AFA1-7AEAA1596A6F

Session key was from s cookie. New is session_key=

PS: Search is working too.

Edited by adhome, 17 August 2012 - 05:18 PM.


#33 Drakie

Drakie

    Asuran Acolyte

  • Members
  • 56 posts

Posted 17 August 2012 - 06:10 PM

View Postadhome, on 17 August 2012 - 05:00 PM, said:

Found Way to login.
h**ps://tradingpost-live.ncplatform.net/authenticate?account_name=Guild Wars 2&session_key=DD6FCAED-8BE8-E111-AFA1-7AEAA1596A6F

Session key was from s cookie. New is session_key=

PS: Search is working too.

niiiiice, that's working :D
how did you find out??

Edited by Drakie, 17 August 2012 - 06:10 PM.


#34 adhome

adhome

    Pale Tree Seedling

  • New Members
  • 5 posts

Posted 17 August 2012 - 06:16 PM

Easy. Starting with useoldlauncher.
Start Processexplorer. Open GW2 Task. Search memory strings. "ncplat". Wola.

#35 isilnor

isilnor

    Pale Tree Seedling

  • Members
  • 9 posts

Posted 18 August 2012 - 08:18 PM

is this project still in progress? will you continue and maintain it? or have you stop cause of any problem from arenanet side?

if you continue i'd like to contribute my knowledge in economics.
i have a university degree on economics and i have many ideas how you can make it better not only mathematically but also more informative.


anyway please continue your great work. this is something i will gladly use :D

#36 Drakie

Drakie

    Asuran Acolyte

  • Members
  • 56 posts

Posted 18 August 2012 - 08:33 PM

I hope to continue it, I think I fixed the problems yesterday.

I hope to continue it, I think I fixed the problems yesterday.

#37 Key

Key

    Pale Tree Seedling

  • New Members
  • 1 posts

Posted 18 August 2012 - 09:30 PM

I think EU and US have two different markets, so for which one is it? Or is it for both ?

#38 Uttar

Uttar

    Fahrar Cub

  • Community Contributors
  • 20 posts

Posted 19 August 2012 - 05:03 AM

Great job Drakie! I'd be very tempted to contribute to the project if it wasn't for the fact I haven't used PHP in many years (and I didn't particularly enjoy using it in the past, although your code looks fairly nice and clean!)

I've managed to login and get the item lists using cURL in Lua, but I have the same problem as you do for the listings - do you believe there's actually a solution to this problem? It would be rather sad if there wasn't, hopefully it's not intentional by ArenaNet.

There is an alternative approach I was considering before noticing this project: actually get the data in-game! Automate the key presses/mouse movements and use an OCR (e.g. tesseract) to read the text at the known locations. Unfortunately GW2's fonts are rather terrible in terms of spacing and it's hard to read the prices robustly (you need to remove the coin icons so the OCR doesn't get confused, and even then the font numbers aren't good for OCRs...).

The performance is also quite bad if you want good quality (which requires scaling up the screenshot by a lot and doing some post-processing, e.g. remove UI background colour and convert to grayscale by making all color channels equal to the brightest channel) so you'd need a dedicated high-end machine (and GW2 account!) for this probably and even the it wouldn't be as fast as the web interface. And that's before even considering the risk of being banned due to ArenaNet understandably (but incorrectly) assuming it's a bot...

If you think the listings cannot be fixed, then it might be worth it for me to try implementing this before launch anyway, but it'd obviously be nicer with getting the info directly from the web page :)

#39 Drakie

Drakie

    Asuran Acolyte

  • Members
  • 56 posts

Posted 20 August 2012 - 11:16 AM

View PostUttar, on 19 August 2012 - 05:03 AM, said:

Great job Drakie! I'd be very tempted to contribute to the project if it wasn't for the fact I haven't used PHP in many years (and I didn't particularly enjoy using it in the past, although your code looks fairly nice and clean!)

I've managed to login and get the item lists using cURL in Lua, but I have the same problem as you do for the listings - do you believe there's actually a solution to this problem? It would be rather sad if there wasn't, hopefully it's not intentional by ArenaNet.

There is an alternative approach I was considering before noticing this project: actually get the data in-game! Automate the key presses/mouse movements and use an OCR (e.g. tesseract) to read the text at the known locations. Unfortunately GW2's fonts are rather terrible in terms of spacing and it's hard to read the prices robustly (you need to remove the coin icons so the OCR doesn't get confused, and even then the font numbers aren't good for OCRs...).

The performance is also quite bad if you want good quality (which requires scaling up the screenshot by a lot and doing some post-processing, e.g. remove UI background colour and convert to grayscale by making all color channels equal to the brightest channel) so you'd need a dedicated high-end machine (and GW2 account!) for this probably and even the it wouldn't be as fast as the web interface. And that's before even considering the risk of being banned due to ArenaNet understandably (but incorrectly) assuming it's a bot...

If you think the listings cannot be fixed, then it might be worth it for me to try implementing this before launch anyway, but it'd obviously be nicer with getting the info directly from the web page :)

I actually started this project with the idea to use OCR, since I had been playing with that idea in my head for the diablo3 RMAH, and it's doable but requires A LOT of work and capacity to do that at a decent speed and with enough accuracy and of course requires a 2nd account and PC to run on ..
For EVE they've build something which can read the information from shared memory or something, maybe if something like that would work that would be cool ;)

When you fetch the item list with search.json there's a 'min_sale_price' (or something like that) which I'm now using, however I need to check how accurate that is, hopefully we get 1 last stresstest this week :(

#40 Drakie

Drakie

    Asuran Acolyte

  • Members
  • 56 posts

Posted 20 August 2012 - 04:15 PM

Okay, some1 else decided to post it on reddit, while I was planning to wait for release day to post it up since more people will be interested then, but now I need to upvote love :P since I dropped to the 2nd page Y_Y

http://www.reddit.co..._watching_tool/

On a side note;
- I optimized some server sided stuff (kinda big configuration changes) so that it's now a gazillion times faster (okay, maybe just 100~1000 times faster, but A LOT!).
- I added some monitoring, so if it becomes slow again I'll know!
- The lists with items are now 50 on a page and a lot more like the traditional item database websites.
- IE8 (and lower) won't have the fancy background, since it was acting up in those browsers!
- There might be 1 final stresstest this week, that would be AAAAAAAAAAAAAAWESOME (apart from satisfying my need of GW2 :P) so I can test some final things before launch! \o/



#41 Evel

Evel

    Fahrar Cub

  • Members
  • 24 posts
  • Guild Tag:[IE]
  • Server:Gate of Madness

Posted 24 August 2012 - 04:15 PM

Great project!  Will definitely keep an eye on this one :)

#42 dinguz

dinguz

    Pale Tree Seedling

  • New Members
  • 1 posts

Posted 25 August 2012 - 01:07 PM

Is it me or search is not working? ("page not found")

#43 Drakie

Drakie

    Asuran Acolyte

  • Members
  • 56 posts

Posted 26 August 2012 - 08:10 AM

View Postdinguz, on 25 August 2012 - 01:07 PM, said:

Is it me or search is not working? ("page not found")

no problems on this side of the line afaik

#44 adhome

adhome

    Pale Tree Seedling

  • New Members
  • 5 posts

Posted 28 August 2012 - 08:59 PM

View PostDrakie, on 26 August 2012 - 08:10 AM, said:

no problems on this side of the line afaik
Sorry, but get same "Sorry, the page you are looking for could not be found." error.
But getting search inputform on open "http://www.gw2spidy.com/search"
Using FF
PS: Can you make a table view, with "Item | Sell Price | buy Price" ? You can then easy see items with big delta and trading win.

Thanks (;

#45 Drakie

Drakie

    Asuran Acolyte

  • Members
  • 56 posts

Posted 29 August 2012 - 01:33 PM

View Postadhome, on 28 August 2012 - 08:59 PM, said:


Sorry, but get same "Sorry, the page you are looking for could not be found." error.
But getting search inputform on open "http://www.gw2spidy.com/search"
Using FF
PS: Can you make a table view, with "Item | Sell Price | buy Price" ? You can then easy see items with big delta and trading win.

Thanks (;

Oh I messed up the search it seems, I fixed it now ;)
I'll add the sell and buy price to the lists, gimme a few minutes :P



#46 Drakie

Drakie

    Asuran Acolyte

  • Members
  • 56 posts

Posted 29 August 2012 - 01:48 PM

View PostDrakie, on 29 August 2012 - 01:33 PM, said:



Oh I messed up the search it seems, I fixed it now ;)
I'll add the sell and buy price to the lists, gimme a few minutes :P

aaaaaaaaaaaaaaaaaaand it's in ;)

now just waiting for the tradingpost to finaly come online!

#47 yen

yen

    Pale Tree Seedling

  • New Members
  • 7 posts
  • Guild Tag:[EDGE]
  • Server:Henge of Denravi

Posted 29 August 2012 - 05:21 PM

Loving the website + the fact that it's open source. Would be nice to add user accounts and the ability to "watch" or save certain items to a list you can easily access all the time.

#48 grimmson

grimmson

    Asuran Acolyte

  • Members
  • 147 posts

Posted 29 August 2012 - 06:10 PM

hjoly cow you just won the gw2 internetz!
really nice tool, thanks alot!
(now a working trading post wouldn't be bad...)

#49 adhome

adhome

    Pale Tree Seedling

  • New Members
  • 5 posts

Posted 29 August 2012 - 09:31 PM

View PostDrakie, on 29 August 2012 - 01:48 PM, said:

aaaaaaaaaaaaaaaaaaand it's in ;)

now just waiting for the tradingpost to finaly come online!

You are the best :P

#50 Drakie

Drakie

    Asuran Acolyte

  • Members
  • 56 posts

Posted 30 August 2012 - 01:43 PM

View Postyen, on 29 August 2012 - 05:21 PM, said:

Loving the website + the fact that it's open source. Would be nice to add user accounts and the ability to "watch" or save certain items to a list you can easily access all the time.

That's definitely one of the features that I might want to implement, I think I'll write up a small list of ideas I've heard or came up with so far and maybe use a poll so I don't have to chose what to build first myself :P
Atm I'm just patiently waiting for the tradingpost to come online, see how things turn out and then weither or not people will be returning to the site.

#51 yen

yen

    Pale Tree Seedling

  • New Members
  • 7 posts
  • Guild Tag:[EDGE]
  • Server:Henge of Denravi

Posted 30 August 2012 - 05:27 PM

View PostDrakie, on 30 August 2012 - 01:43 PM, said:

Atm I'm just patiently waiting for the tradingpost to come online, see how things turn out and then weither or not people will be returning to the site.

It wen't online like 2-3 times for a few minutes last night. Hopefully it get's fixed soon cause I have a lot of crap in the bank to sell lol.

#52 Drakie

Drakie

    Asuran Acolyte

  • Members
  • 56 posts

Posted 31 August 2012 - 09:42 AM

View Postyen, on 30 August 2012 - 05:27 PM, said:



It wen't online like 2-3 times for a few minutes last night. Hopefully it get's fixed soon cause I have a lot of crap in the bank to sell lol.

YES SIR! I got about 10hours of data, they cut us off again about 20min ago ... :(

During the BWEs and stresstests I was only collecting data every hour orso, I changed it now to 15minutes!
We can get a lot more in detail fluctuations this way!

However there's a problem with it ... the graph is really cluttered with a shitload of dots ... so I created a BETA version with some smart stuff in it to avoid the clutter, would love to get some feedback:
http://beta.gw2spidy.com/

Also created a twitter account so I can keep people updated: https://www.twitter.com/gw2spidy

#53 cascaid

cascaid

    Pale Tree Seedling

  • New Members
  • 2 posts

Posted 31 August 2012 - 09:50 AM

Love the site!
My only complaint is that when you want to examine a subsection of the data, the axis do not alter.
For example, zooming into todays data for unidentified dye:
http://www.gw2spidy.com/item/20323
The outlier at 80s throws the whole scale whether it's included in your active view or not

#54 Drakie

Drakie

    Asuran Acolyte

  • Members
  • 56 posts

Posted 31 August 2012 - 10:25 AM

View Postcascaid, on 31 August 2012 - 09:50 AM, said:

Love the site!
My only complaint is that when you want to examine a subsection of the data, the axis do not alter.
For example, zooming into todays data for unidentified dye:
http://www.gw2spidy.com/item/20323
The outlier at 80s throws the whole scale whether it's included in your active view or not

Good one! I'll fix that, shouldn't be to hard to get that to scale :)

#55 Fluid_Karma

Fluid_Karma

    Pale Tree Seedling

  • New Members
  • 8 posts

Posted 31 August 2012 - 11:53 PM

Wow,very interesting...Thanks!

#56 brightemo

brightemo

    Fahrar Cub

  • Members
  • 25 posts

Posted 03 September 2012 - 01:13 AM

If you want the listings.json to work, you have to use the session guid ("s" cookie) of a session you logged into the client with (so I definitely think it is has been intentionally made this way to stop data miners/trading bots getting too much information) . Easiest way is to do a memory search as mentioned above -- although that won't help your site on a server somewhere.

If someone wants to risk their account by trying to read what the client login code does differently to logging into the website (perhaps it is as simple as adding another parameter), then maybe that could lead to a more permanent fix. -- Although I would prefer it if AN just gave us access or perhaps API keys or something.

#57 Drakie

Drakie

    Asuran Acolyte

  • Members
  • 56 posts

Posted 03 September 2012 - 06:14 AM

View Postbrightemo, on 03 September 2012 - 01:13 AM, said:

If you want the listings.json to work, you have to use the session guid ("s" cookie) of a session you logged into the client with (so I definitely think it is has been intentionally made this way to stop data miners/trading bots getting too much information) . Easiest way is to do a memory search as mentioned above -- although that won't help your site on a server somewhere.

If someone wants to risk their account by trying to read what the client login code does differently to logging into the website (perhaps it is as simple as adding another parameter), then maybe that could lead to a more permanent fix. -- Although I would prefer it if AN just gave us access or perhaps API keys or something.

http://www.reddit.co...0v2xn?context=3
Mike O'Brien responded here that they are fine with it as long as we dont touch the client, I think this has overlap and that indeed it's intentional ;)

#58 Leftnut

Leftnut

    Pale Tree Seedling

  • Curse Premium
  • Curse Premium
  • 3 posts
  • Guild Tag:[FoE]
  • Server:Northern Shiverpeaks

Posted 04 September 2012 - 04:35 AM

Is your site down ??

#59 Drakie

Drakie

    Asuran Acolyte

  • Members
  • 56 posts

Posted 04 September 2012 - 06:33 AM

View PostLeftnut, on 04 September 2012 - 04:35 AM, said:

Is your site down ??

it was, I released a bad version ... fixed now

#60 yen

yen

    Pale Tree Seedling

  • New Members
  • 7 posts
  • Guild Tag:[EDGE]
  • Server:Henge of Denravi

Posted 04 September 2012 - 09:34 AM

Not sure if it's a cross browser issue but re sizing the window only re sizes the background. I'd have to hit refresh to get the chart to re size properly.
Posted Image

Edited by yen, 04 September 2012 - 09:35 AM.





1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users