105 replies to this topic

[Drakie]

Introduction
Guild Wars 2 loads the trademarket into the game from a website, someone on reddit posted the URL to that website a while back and you can get into that with your game account.

Now what I've build does not require you to login!
I made a niftly little tool that uses my own game account to automatically login to the Guild Wars 2 trademarket website every 15~30 minutes and go through all the listings of all the items and save those in a database.
With that information I can build graphs of sell and buy orders over time.

You can find the website here: http://www.gw2spidy.com/

Progress / Feedback
FEEDBACK IS VERY MUCH WELCOME!
I'd love to hear some feedback on what people think would be good features to build or how to improve the core feature (the graphs).

The official tradingpost site
To avoid confusion: this is an official Guild Wars 2 website! the first URL is https://account.guil...2.com/login?...
There's no need for you to login, I just posted this part so you know where I get my data from!

Is this ok with ArenaNet?
I made a ticket on their support site during BWE3 asking if what I was doing was fine with them, since I use my game account to login it automatically I didn't want to risk getting banned or anything.

Ingame graphs are coming?
There's a small section in the Trademarket called 'Market data' which might just be reserved for a new feature which gives you some graph or something... Depending on what ArenaNet puts there this might be completely useless

Colabarating
First of all, FEEDBACK IS IMPORTANT! That's something I really need.
Other then that, the project is open source so you can see the code here: https://github.com/r...sayshi/gw2spidy on github and feel free to fork or just run off on your own
There'a README included and you can contact me to get more help, I don't mind helping!

Edited by Drakie, 23 August 2012 - 03:40 PM.

[Freelancer]

Good stuff. Someone who watched the latest tales of Tyria referred me to this , and I have to say, you are a man after my own heart. Lol. You asked if this helped, I think it great. It needs some polish, but certainly is functional.

Keep up the great work. This community needs more like you.

[Drakie]

Freelancer, on 01 August 2012 - 12:11 PM, said:

Good stuff. Someone who watched the latest tales of Tyria referred me to this , and I have to say, you are a man after my own heart. Lol. You asked if this helped, I think it great. It needs some polish, but certainly is functional.

Keep up the great work. This community needs more like you.

Hey Freelancer, it was me sending the mail to ToT, since I needed some feedback!

I (of course) watch ToT and I mailed you guys like halfway through watching the latest podcast
You mentioned you had graphs yourself and maybe you can tell how much you calculated the price at a certain moment, because like I described above under 'things to figure out' I have a hard time thinking of a solid way to do this!

Edited by Drakie, 01 August 2012 - 02:30 PM.

[Beta Sprite]

This is a great tool.  Are you going to be tracking the 'buy orders' separately, too?

[Brayzz]

Wow this is really good.

[Drakie]

Beta Sprite, on 01 August 2012 - 02:28 PM, said:

This is a great tool.  Are you going to be tracking the 'buy orders' separately, too?

Oh, man I love you for asking that!
Previously the marketplace wouldn't return the buy orders when I requested them so I gave up on doing that, but since you asked I decided to check again and they seemed to have fixed it and I now can \o/
So yes, defenatly will be tracking buy orders too!

[Beta Sprite]

Drakie, on 01 August 2012 - 02:34 PM, said:

Oh, man I love you for asking that!
Previously the marketplace wouldn't return the buy orders when I requested them so I gave up on doing that, but since you asked I decided to check again and they seemed to have fixed it and I now can \o/
So yes, defenatly will be tracking buy orders too!

Excellent.  Hmm... Does it also give tracking information on how many people are buying at the given prices?  That might make it easier to know what to buy in order to play the market.  Oh... er... I mean... Track popularity!  Yeah, that's what I meant.

[Drakie]

Beta Sprite, on 01 August 2012 - 04:10 PM, said:

Excellent.  Hmm... Does it also give tracking information on how many people are buying at the given prices?  That might make it easier to know what to buy in order to play the market.  Oh... er... I mean... Track popularity!  Yeah, that's what I meant.

You can't see actual buys, so calculating prices has to been done based on listings, that provides some problems

Okay I gave the underlying code a bit of a facelift and the UI too, now the charts got zoom functionality and tooltips on every dot which make them a lot more interesting!

It's funny to see that anet has been testing or something internally because some prices actually were fluctuating xD

Anyway, tonights stress test will be interesting to see again

Hmpf unfortunately a whole lot of things were messed up during last nights stress test with the game and the trade market, so data for some items might have small gaps, unfortunately I'm completely bound by the stability of the trade market on this xD

Also since nobody even bothered to sell anything during the stress test the prices were more skewed then usual, but it's the perfect example of what I need to work on to avoid!

Let's check out Jute Scrap for example
Graph: http://gw2spidy.rube...i.com/item/4496
Trade market: https://tradingpost-...m.net/item/4496
Below the listings in the last hour

+------------+----------+
| unit_price | quantity |
+------------+----------+
|		 20 |		7 |
|		 21 |	   94 |
|		 22 |	  358 |
|		 23 |	   20 |
|		 24 |	   28 |
|		 25 |	  482 |
|		 26 |		7 |
|		 28 |		9 |
|		 29 |		2 |
|		 30 |	  278 |
|		 32 |	   15 |
|		 34 |		3 |
|		 35 |	   28 |
|		 40 |	   79 |
|		 47 |	   18 |
|		 50 |	  194 |
|		 52 |		1 |
|		 59 |		5 |
|		 60 |	   17 |
|		 68 |		6 |
+------------+----------+

The result of these listings is an average price of 30c! Which is not what I want to see, I want to see something around 25!
Atm for every listing price I just multiply it by the quantity, add that all up and devide it by the total quantity, this properly gives the average listing price because if there are a lot of listings for 25 then for 26 then it will weight the 25 a lot more!
But the problem is those listings which are way to overpriced to be sold, they aren't relevant for us since we don't care about them and since they are never sold they even influence the average price for a fairly long time!!

If I take the first 10 listed prices (20 to 30) then the average price is properly set to 24.9c, which is what I want, I think that seems like a good method to go for.

update: I implemented this idea (first 10) and updated the live site, gonna randomly go through some items and asee if this is what I want

Okay, I'm pretty confident that's the way I want the prices to be calculated atm.

I redid some of the code that takes the snapshots of the trade market and got a huge performance boost so I can process items a lot faster. I should be able to do like 2~4 snapshots per hour, but the amount of requests to do that would be rather high, gonna wait with that until I can make a ticket on the guildwars website to ask if they are ok with that and for now just do 1 snapshot per hour.

Also added a searchbox for items, though really primative atm so no smart stuff happening (eg if you search for 'Ore' you'll also find 'Armored Scale').

And last but not least the zooming / dragging around inside the graphs works a lot better now and the labels on the xaxis only show the date, while the dots on the lines also show the time

Edited by Drakie, 03 August 2012 - 08:38 AM.

[teegee543]

Great tool Drakie! Could you by any chance post a sample of the JSON returned by each of the server responses? I'd like to take a crack at writing a tool in JavaScript.

[Drakie]

teegee543, on 03 August 2012 - 10:25 PM, said:

Great tool Drakie! Could you by any chance post a sample of the JSON returned by each of the server responses? I'd like to take a crack at writing a tool in JavaScript.

Just use google chrome as browser (or firefox with firebug) and login to https://tradingpost-...tform.net/login

For all the types and shit grab the source from the first page, there's some JSON in there with all the types and subtypes.

Open the debug concole (F12) and goto the network tab

When you use the filters (by category or subcatery you'll see requests to search.json coming by, those are the ones you need, just change the URL arguments a bit to get what you want.

For the listings, when you click on an item in the list you'll see a listings.json afaik


Tbh unless you're going to use node.js I think doing everything clientside with javascript ain't very smart, since you need to fake a login first and maintain session to go through the requests and you want to store the history to get nice data out of it
And if you want other ppl to use it, you don't really want to have them login.
If you just wanna make some cool ways of displaying the data with javascript you'd be best of grabbing my code from github ( i can give you a database dump if the data I have so far ) and creating JSON with PHP and consuming that with javascript :-)

[Zyreon]

In all MMOs, I've been making gold with the marketplace, and this tool will help me alot to do so in Guild wars 2 as well.
I just have to say: I LOVE YOU FOR MAKING THIS!

Keep up the good work!

[Drakie]

I'd love to hear some feedback on what people think would be good features to build or how to improve the core feature (the graphs).

I think for the core feature it would be nice if the filtering / navigating through the item (categories) would be more like the conventional MMO item databases (a simple dropdown in the menu with the categories and some filtering based on level / rarity).
And maybe the option to sticky the items in the graphs so that you can display multiple items in 1 graphs together if that's somehow interesting to you.

[All3ks]

Really good tool cant wait to test it when the game launches.What i think would be nice is to include both buy order and sell orders in the graph.  Also it would be nice to have the option to search for item by using exact match.Now for example if i search for rawhide i get 15 pages of items and have to go trough all of them to search it.Well if you implement categories that will probably solve the problem.
Anyway good job and i hope you will keep improving this tool:).

[Bulldog]

Overall I would say this is a very nice app.  The only problem I see with this, and it's minor, is the zoom feature is a bit difficult to figure out at first.  It seems somewhat finicky and it took me a couple of minutes to actually figure out how to zoom into a day or days that I wanted to see, and then how to reset the view to see everything correctly.  I'm thinking that if you have an option where you are able to drag to select what you want to zoom into it might help, then just have a reset button or zoom out to take you to a normal view.  This is my only complaint and as I say it's a small one but most stock market charts that I've seen use some kind of variation of what I describe, keep up the good work it's still a very nice app.

[Drakie]

Thanks for the feedback,

I updated it so that when there's more then 100 copper it will display Xg Xs Xc instead of 12365 copper, should be easier on the eyes

I also removed the semi-intelegent algoritme that was using an average of the bottom 10 listings as the price to display and now I'l just displaying the minimum price, there are to many things that skew the data if I try to be smart and to many people who don't get it

You can drag the zoom around, but just not past the point where there's no data (back in time), I could live with this zoom, but indeed it ain't very intuitive yet. I'll check out some real stock market websites for how they are doing it. That's a good idea, to steal from them

The search really needs a revamp at some point, like searching within the curent category etc, but I first want all the basics working properly

[Drakie]

Ok, they had me worried for a bit, but it's all fine

The tradingpost website has been down for 2 days and got a big update, it won't function very well now in the browser since they changed some stuff, probally to increase performance ingame but also to * over people accessing it from the browser.
You also can't really login properly anymore with a browser, but I managed to get in with my tool \o/

So ArenaNet got ready for launch and we're still alive!

I think next up is looking if I can make the zoom function nicely :-)

Bulldog, on 06 August 2012 - 02:24 PM, said:

Overall I would say this is a very nice app.  The only problem I see with this, and it's minor, is the zoom feature is a bit difficult to figure out at first.  It seems somewhat finicky and it took me a couple of minutes to actually figure out how to zoom into a day or days that I wanted to see, and then how to reset the view to see everything correctly.  I'm thinking that if you have an option where you are able to drag to select what you want to zoom into it might help, then just have a reset button or zoom out to take you to a normal view.  This is my only complaint and as I say it's a small one but most stock market charts that I've seen use some kind of variation of what I describe, keep up the good work it's still a very nice app.

I just added a button just below the graphs which lets you switch to 'experimental zoom mode', I think it's exactly what you described.
Tbh I don't know which one I like better personally so I need feedback from people who are reading this

Edited by Drakie, 09 August 2012 - 01:14 PM.

[dollada06]

Wow this app in nothing short of amazing.  I run a site devoted to methods of making gold in Guild Wars 2. (Guild Wars 2 Riches if anyone wants to see it.)  This app is a must have for people who want to make gold.  We had this luxury in WOW with a site that did this as well.

Thank you a million times for creating this!!!

[miracle091]

Very very interesting tool, just one question...Are you using a php framework or all the pages\scripts is handmade?

Update: About the zoom, i prefer the experimental is more easy to use and don't have found any issues when viewing some materials. Just the background of ALL pages i have tried, is slow to rendering, nothing more just slow.

Spoiler

My Spec:
Windows 7 x64 SP1
Google Chrome 21 (stable release)
Internet Explorer 9
8 GB of RAM
Intel i3 2130
Intel Graphics HD 1000

Edited by miracle091, 09 August 2012 - 03:03 PM.

[Drakie]

miracle091, on 09 August 2012 - 02:50 PM, said:

Very very interesting tool, just one question...Are you using a php framework or all the pages\scripts is handmade?

Update: About the zoom, i prefer the experimental is more easy to use and don't have found any issues when viewing some materials. Just the background of ALL pages i have tried, is slow to rendering, nothing more just slow.

Sorry, I didn't even post it in this topic, but the code is open source on github: https://github.com/r...sayshi/gw2spidy so you can dig through it yourself if you want to!
I started with some self made routing / framework together with Propel, but later replaced a lot of code so that I'm now using Silex (lightweight Symfony2 based framework), checkout the github page for the full list of stuff that I'm using :-)

The backgrounds might be slow I suppose, depending on connection I think, it's just some added pwettyness

Edited by Drakie, 09 August 2012 - 03:03 PM.

[Drakie]

I just registered www.gw2spidy.com and updated the URLs in the first post, old URLs should keep working (they'll redirect you to the new one)

ArenaNet is messing around with a lot of stuff atm, I hope we get 1 more beta so I can spend a couple of hours to fix some issues, I wouldn't want to miss the first hours of launch! (and I wouldn't want to spend my first hours of launch debugging this )

Edited by Drakie, 14 August 2012 - 09:13 AM.

[zaza_nomades]

Drakie, on 03 August 2012 - 04:08 PM, said:

You can't see actual buys, so calculating prices has to been done based on listings, that provides some problems

Okay I gave the underlying code a bit of a facelift and the UI too, now the charts got zoom functionality and tooltips on every dot which make them a lot more interesting!

It's funny to see that anet has been testing or something internally because some prices actually were fluctuating xD

Anyway, tonights stress test will be interesting to see again

Hmpf unfortunately a whole lot of things were messed up during last nights stress test with the game and the trade market, so data for some items might have small gaps, unfortunately I'm completely bound by the stability of the trade market on this xD

Also since nobody even bothered to sell anything during the stress test the prices were more skewed then usual, but it's the perfect example of what I need to work on to avoid!

Let's check out Jute Scrap for example
Graph: http://gw2spidy.rube...i.com/item/4496
Trade market: https://tradingpost-...m.net/item/4496
Below the listings in the last hour

+------------+----------+
| unit_price | quantity |
+------------+----------+
|		 20 |		7 |
|		 21 |	 94 |
|		 22 |	 358 |
|		 23 |	 20 |
|		 24 |	 28 |
|		 25 |	 482 |
|		 26 |		7 |
|		 28 |		9 |
|		 29 |		2 |
|		 30 |	 278 |
|		 32 |	 15 |
|		 34 |		3 |
|		 35 |	 28 |
|		 40 |	 79 |
|		 47 |	 18 |
|		 50 |	 194 |
|		 52 |		1 |
|		 59 |		5 |
|		 60 |	 17 |
|		 68 |		6 |
+------------+----------+

The result of these listings is an average price of 30c! Which is not what I want to see, I want to see something around 25!
Atm for every listing price I just multiply it by the quantity, add that all up and devide it by the total quantity, this properly gives the average listing price because if there are a lot of listings for 25 then for 26 then it will weight the 25 a lot more!
But the problem is those listings which are way to overpriced to be sold, they aren't relevant for us since we don't care about them and since they are never sold they even influence the average price for a fairly long time!!

If I take the first 10 listed prices (20 to 30) then the average price is properly set to 24.9c, which is what I want, I think that seems like a good method to go for.

update: I implemented this idea (first 10) and updated the live site, gonna randomly go through some items and asee if this is what I want

Okay, I'm pretty confident that's the way I want the prices to be calculated atm.

I redid some of the code that takes the snapshots of the trade market and got a huge performance boost so I can process items a lot faster. I should be able to do like 2~4 snapshots per hour, but the amount of requests to do that would be rather high, gonna wait with that until I can make a ticket on the guildwars website to ask if they are ok with that and for now just do 1 snapshot per hour.

Also added a searchbox for items, though really primative atm so no smart stuff happening (eg if you search for 'Ore' you'll also find 'Armored Scale').

And last but not least the zooming / dragging around inside the graphs works a lot better now and the labels on the xaxis only show the date, while the dots on the lines also show the time

You should not take the first 10 listed prices... if those first prices are available in a small quantity, it won't evaluate a correct value.

What you need is not the average value but the median value: http://simple.wikipe...org/wiki/Median

in your example you have a total of  1651 items. The median value is where you have 825 items before and after it.

which is 25:


Price Quantity cumulative quantity
20 7 7
21 94 101
22 358 459
23 20 479
24 28 507
25 482 989

Edited by zaza_nomades, 14 August 2012 - 05:02 PM.

[Drakie]

Before I wrote that post I actually had the median (sorry forgot the name xD) but the results were really not the desired numbers
If you look at that data you want 20 ~ 22 imo

A lot of people will look at the chart and think it's the lowest price, no matter how obvious you'd make it that it's the median
And like I wrote, the median is to much influenced by people who are listing stuff for a price which won't sell...

[teegee543]

Hi Drakie, do you know if the login URL is dead/changed? I can't login here anymore: https://tradingpost-...tform.net/login

[Drakie]

you can still get in from this URL: https://account.guil...F&game_code=gw2

the problem however is that they changed stuff in how they retrieve the listings and it no longer works outside of the game atm
I tried package sniffing it, but I'm not very good at that and afaik it's SSL requests which I can't decode

[teegee543]

@Drakie: Are you talking about the login mechanism? Once you're logged in, the JSON responses in Firebug are still unencrypted. It just looks like the item detail retrieval is broken in the browser interface.

[Drakie]

all trafic is HTTPS, but since your browser manages the SSL connection is can show you the unencrypted data without problem in firebug (or chrome console).
indeed the listings JSON request is no longer working and I think it's intentional, as far as I can tell they're still directly using that online version to load into the game, but they've probably added a header or something like that to the listings request to prevent outside of game access

[teegee543]

Based on what I see from your source, you're using curl to retrieve the pages from the portal. Maybe this will help you authenticate and get the cookie you need to regain access with your application:
http://stackoverflow...ssl-and-cookies

[Drakie]

the login is fine, I can get the item lists etc.
they just *ed me over with the listings, it returns a 500 server error, same when you try it from the browser

[Drakie]

aaaaaaaaaaaaaand they just disabled the login @!#^^%@

[teegee543]

Drakie, on 17 August 2012 - 11:24 AM, said:

aaaaaaaaaaaaaand they just disabled the login @!#^^%@

Nooooo

By the way, how did you find this trading post website in the first place?