I've been around a long time and I've got a massive number of faves on this site at this point. Just for fun and curiosity, I was wondering if there was an easy way to find out some stats about them. You know, which artists I've faved the most, and so on. I guess it's just me being a data geek, heh. It's probably a lot easier when you're newer and only have, say, a few hundred faves at most and can just kind of know. But like I said, I've been lurking around this site for ages, and I've built up 23.7k+ faves over that time. Is there any way to look at those stats?

Unfortunately not officially, you might want to take a look at the API documentation if you truly are a data geek :p

I was working on a tool for uploading stats but I ditched that idea because I hate doing UI/UX so much. It wouldn't be too hard to do the same on user favorites, I'd do it as an example but your favorites seem to be private.

You could get Hydrus Network, download all of your favorites, and then look at the most common tags in your database.

faucet said:
Unfortunately not officially, you might want to take a look at the API documentation if you truly are a data geek :p

I was working on a tool for uploading stats but I ditched that idea because I hate doing UI/UX so much. It wouldn't be too hard to do the same on user favorites, I'd do it as an example but your favorites seem to be private.

Dang. I was really hoping there was a built-in way somehow that I'd missed, lol. And yeah my faves are private, which I was worried might be an issue if I had to resort to third party tools or the API :s

You could get Hydrus Network, download all of your favorites, and then look at the most common tags in your database.

Heh, I've toyed with Hydrus Network on my other PC, but for 23.7k+ faves, i don't even want to imagine how long that'd take to process...

lloxie said:
And yeah my faves are private, which I was worried might be an issue if I had to resort to third party tools or the API :s

Well, it's luckily not a huge problem, if you authenticate using your API key the tool will be able to access the favorites just fine.

faucet said:
Well, it's luckily not a huge problem, if you authenticate using your API key the tool will be able to access the favorites just fine.

Yeah, I just worry about getting my account in trouble in case whatever I tool I use accidentally misbehaves or something :s

lloxie said:
Yeah, I just worry about getting my account in trouble in case whatever I tool I use accidentally misbehaves or something :s

You don't actually have to download all your favorites. Your favorites are available in JSON files. ;) Just add .json to get each page.
https://e621.net/favorites.json?page=2 https://e621.net/favorites.json?page=3 and so on.

In case you haven't noticed, that file contains the tags and URLs and post numbers. It basically provides all the information you need.

alphamule said:
You don't actually have to download all your favorites. Your favorites are available in JSON files. ;) Just add .json to get each page.
https://e621.net/favorites.json?page=2 https://e621.net/favorites.json?page=3 and so on.

In case you haven't noticed, that file contains the tags and URLs and post numbers. It basically provides all the information you need.

...thanks, but I honestly don't know what to do with this info. xD I assume I'd have to make some kind of automated script to do anything with it. But I've never coded anything that interacts with the internet/browsers/etc. before.

lloxie said:
...thanks, but I honestly don't know what to do with this info. xD I assume I'd have to make some kind of automated script to do anything with it. But I've never coded anything that interacts with the internet/browsers/etc. before.

JSON is well-supported by libraries. In my case, I just needed the list of URLs and post numbers so I ignored everything except between "http" and quote mark, but I could have done it like this:
It has a hierarchy. I want contents of URL underneath files. /posts/*/files/url in other words. So just request all matching results for some expression like that. Hopefully, this returns a list (because convenient for the JS in JSON name), and then go through each member using that path.
For tags, you'd do same thing but it's binned into general, copyright, artist, and so on. You could think of each post having a bundle of tags sorted into categories like that. In C++, this is similar to the concept called a struct. You could also just do like CSVs and use an existing program like Microsoft Excel/LibreOffice Calc. Nothing says that you HAVE to use a browser for this.
What's cool is that Firefox seems to just show the structure of the JSON if you directly click that link. It lets you explore it in a friendly way to find what data you want from it.
There are also tools for handling them without coding anything. https://www.convertcsv.com/json-to-csv.htm As an example. There's TONS of sites and tools that do something like that.

alphamule said:
JSON is well-supported by libraries. In my case, I just needed the list of URLs and post numbers so I ignored everything except between "http" and quote mark, but I could have done it like this:
It has a hierarchy. I want contents of URL underneath files. /posts/*/files/url in other words. So just request all matching results for some expression like that. Hopefully, this returns a list (because convenient for the JS in JSON name), and then go through each member using that path.
For tags, you'd do same thing but it's binned into general, copyright, artist, and so on. You could think of each post having a bundle of tags sorted into categories like that. In C++, this is similar to the concept called a struct. You could also just do like CSVs and use an existing program like Microsoft Excel/LibreOffice Calc. Nothing says that you HAVE to use a browser for this.
What's cool is that Firefox seems to just show the structure of the JSON if you directly click that link. It lets you explore it in a friendly way to find what data you want from it.
There are also tools for handling them without coding anything. https://www.convertcsv.com/json-to-csv.htm As an example. There's TONS of sites and tools that do something like that.

So I've coded things in C++, heck I've even just recently dabbled with.json files and a popular C++ library for handling them. But like I said, I've never written anything that interacted with the internet in any way before. I wouldn't have the slightest idea how to write a program that scrapes e621, or any other site for that matter. I can handle .json data, but getting it in the first place is where I'm lost. I assume there must be libraries for such a thing out there, but needless to say, I have zero familiarity with them. I wouldn't even know what to search for, to be honest. And I don't want to have to sit here and manually save the json files for 316 pages of faves, y'know? xD

lloxie said:
So I've coded things in C++, heck I've even just recently dabbled with.json files and a popular C++ library for handling them. But like I said, I've never written anything that interacted with the internet in any way before. I wouldn't have the slightest idea how to write a program that scrapes e621, or any other site for that matter. I can handle .json data, but getting it in the first place is where I'm lost. I assume there must be libraries for such a thing out there, but needless to say, I have zero familiarity with them. I wouldn't even know what to search for, to be honest. And I don't want to have to sit here and manually save the json files for 316 pages of faves, y'know? xD

I generally use libcurl for stuff, which has C and C++ bindings. Though, for this kind of thing, scripting languages are generally easier. Python and PowerShell would both be easy to use here.

Also, it might be worth using the posts db export, to avoid making thousands of requests to the server. nevermind, favorites.json already includes tags.

Made a quick PowerShell script to get all favorites into a single json file:
$login = Get-Credential $posts = [System.Collections.ArrayList]::new() $page = 1 Do { $response = (Invoke-WebRequest -Uri "https://e621.net/favorites.json?page=$page" -Credential $login -Authentication Basic | ConvertFrom-JSON).posts $posts.AddRange($response) Write-Output "Retrieved page $page" $page = $page + 1 } While ($response.length -ne 0) $posts | ConvertTo-JSON -Depth 10 | Out-File -FilePath .\favs.json
It prompts for username and password, password is your API key.

I made something that does this sort of analysis and uses a binomial regression to to identify tags that are statistically significantly enriched in a user's favorites relative to the entire database. I've thought about adding it to the tools section but it its fragmented between scripts so I would need to clean it up first

fickleferret said:
uses a binomial regression to to identify tags that are statistically significantly enriched in a user's favorites relative to the entire database.

That sounds interesting, but can you explain why the need for a 'binomial regression' ?
If you build a dictionary/map of all the tags in all your favorites entries where it maps the tag name with the amount of times the tag appears in your favorites collection, then later you can make a sorted list with the tag names using their scores as rank, so you can know which tags mean the most to you.
The first/last tags in that sorted list will be the most/least important tags for you.

PS that favorites.json file is pretty cool. One thing I'm wondering is if there's a similar one for upvoted posts, as there's some posts that I really like but haven't favorited, only upvoted.

scth said:
Made a quick PowerShell script to get all favorites into a single json file:
$login = Get-Credential $posts = [System.Collections.ArrayList]::new() $page = 1 Do { $response = (Invoke-WebRequest -Uri "https://e621.net/favorites.json?page=$page" -Credential $login -Authentication Basic | ConvertFrom-JSON).posts $posts.AddRange($response) Write-Output "Retrieved page $page" $page = $page + 1 } While ($response.length -ne 0) $posts | ConvertTo-JSON -Depth 10 | Out-File -FilePath .\favs.json
It prompts for username and password, password is your API key.

Oh hell, didn't see that script. That's short and sweet!

lovestar said:
That sounds interesting, but can you explain why the need for a 'binomial regression' ?
If you build a dictionary/map of all the tags in all your favorites entries where it maps the tag name with the amount of times the tag appears in your favorites collection, then later you can make a sorted list with the tag names using their scores as rank, so you can know which tags mean the most to you.
The first/last tags in that sorted list will be the most/least important tags for you.

PS that favorites.json file is pretty cool. One thing I'm wondering is if there's a similar one for upvoted posts, as there's some posts that I really like but haven't favorited, only upvoted.

Most of the time you can just add .json to the requesting page (part before the query AKA '?') in the URL, like this:
https://e621.net/posts?tags=votedup%3Ame -> https://e621.net/posts.json?tags=votedup%3Ame
https://e621.net/posts?page=2&tags=votedup%3Ame -> https://e621.net/posts.json?page=2&tags=votedup%3Ame

The me can be any user name, or indeed, anything at all, and you'll still only get your own upvotes. Same with downvoting.