Am I the only person that has this problem?

There is so much art out there I like I can't keep track of it all. I can't save it all. I can't archive it. I certainly can't sort it.

between DA, FA, inkbunny, pixiv, e621, danbooru, gelbooru, furpiled, wildcritters, 4chan, ychan, petichan, kemonochan pawsru, and all the independent personal websites out there that artists maintain, how the hell am I supposed to keep up with all this content?

why must it all be so good? :(

post #109279

Realize that 95% of your "good art" is just a poorly-drawn furry porn

Start your own imageboard.

Well, I can't help you with places that don't support the Danbooru API (yet), but we're still working on Danborganize...

Start living in a basement.

Jazz said:
Realize that 95% of your "good art" is just a poorly-drawn furry porn

^^ this. ive seen few furry porn images that could classify as art

Jazz said:
Realize that 95% of your "good art" is just a poorly-drawn furry porn

Blotch, BlackTeagan, Kacey, TaniDaReal, Klaus Doberman, Ebon Lupus, Washa, LoneWolf6666, Rukis, Rarakie, Adam Wan, Keovi, Blaze-Lupine, Tsaiwolf, GlowFox, SparkleMotion, TrueGrave, Taurin Fox, and hundreds more....

Art by those artists ARE NOT "poorly drawn furry porn"....

Well then maybe just submit what you can. ;D

One could download the E621 rip and have a solid start.( ﾟ ヮﾟ)

A problem that I have noticed is all of this old content suddenly being posted. There's a lot of stuff that I'd seen like 5 years ago just getting posted to e621. Do the mods run some sort of dupe check or is it 100% on the users to dupe check? I haven't the time to go through the rip, but I'm sure some of the recently posted things must be in there.

Images which are byte-for-byte identical are automatically detected as dupes. Otherwise it's the users' responsibility -- for example, if someone uploads a JPG version of an existing PNG, or a re-saved JPG, that isn't autodetected as a dupe. Ditto for slight cropping etc.

The most likely scenario is actually that the image dimensions are totally different (smaller, or larger).. in the case that they are larger, you really want to mark the previous upload rather than the new upload, as a dupe.

Jazz said:
Realize that 95% of your "good art" is just a poorly-drawn furry porn

you own, buddy

'good art' is terribly subjective.

'good food' should theoretically be fruits veggies and the occasional bit of meat. Yet we say 'good food' is what we enjoy. Frenchfies, fried chicken, steak... generally, the higher fat/sugar content, the more delicious a food is.

But 'good food' should be what keeps your body running optimally, not what's tastey. Yet, no. we define good food as what we enjoy. Even "it's not good for me, but it's so good". This is true even if your favorite food are apples.

Why should art be any different? Maybe Artist X's art isn't anything technically amazing... but if you enjoy it, isn't it good?

savageorange said:
Images which are byte-for-byte identical are automatically detected as dupes. Otherwise it's the users' responsibility -- for example, if someone uploads a JPG version of an existing PNG, or a re-saved JPG, that isn't autodetected as a dupe. Ditto for slight cropping etc.

The most likely scenario is actually that the image dimensions are totally different (smaller, or larger).. in the case that they are larger, you really want to mark the previous upload rather than the new upload, as a dupe.

Well the mods should invest in some software. I have a few programs I've written, I may be willing to bargain. I just can't stand dupes and incomplete collections, it's a boner thief. Nothin gets me hard like a complete collection without redundancy.

xanonx said:
Well the mods should invest in some software. I have a few programs I've written, I may be willing to bargain. I just can't stand dupes and incomplete collections, it's a boner thief. Nothin gets me hard like a complete collection without redundancy.

If you have software that can reliably fingerprint an image without bringing the server to a screeching halt every time an image is uploaded, we'd love to hear about it.

Char said:
If you have software that can reliably fingerprint an image without bringing the server to a screeching halt every time an image is uploaded, we'd love to hear about it.

Well, it would be up to a particular customer as to how fast work actually gets done. Most things I've written are passive, compromising to achieve the best possible performance of the server for the longest period of time; having a particular period of maintenance. Personally, I would back everything up, move to a remote system, check for dupes, and then remove the dupes later. This way you take care of your back-ups as well as the dupe checking.

The issue of how long a dupe check would take would be a matter of compromising how thorough said check is. A hybrid comparison using multiple histograms with key-point challenging would probably be the fastest and most accurate. Accuracy, here, is a direct correlation with how much human interaction is required: the more accurate, the less human required. This then depends on a disk space vs. memory trade-off; whether or not to bucket sort live in memory or offline on the hard drive.

Contrary to above, some things that I've written do a metric fuck-ton of work over large periods of time, allowing me to accept a pic of two people fucking and have the same pic of two people fucking with photoshopped cum shot in same collection without detecting a dupe. I run this on a desktop running quad SLI using the Nvidia SDK for their GPU's. Shit's on fire yo'. I'm assuming E621 does not have this kind of processing power?

I'm sure I could probably scale things down to work over a broader time period and use less cpu. It's just the initial check that's really gonna suck.

So basically, you want to be able to easily archive everything from various boorus?

Quite simple if you have the right add-on for firefox.

The Danbooru downloader is what I use and it comes with several boards pre-'installed'. You can easily add board though, but going to Add-ons - Danbooru Downloader - Options - and adding http://e621.net/post/show/ (also, in your profile make sure you don't have 'resize' checked. Load an image and once it's done loading, it'll automatically be saved to your designated folder.

It works with pretty much every booru I've tried. You need to have a profile and be logged into it though (because the display URL changes between logged in, and not logged in).

It sorts first by board (danbooru, this board, etc), then by Series, if there is no series, it will go by 'misc'. It's less useful on this board, but it works quite well on general image boards like Danbooru. In point of fact, my Danbooru board folder stands at over 550 folders for various series.

BTW: I just tested it on this site (just registered), et voila, it works.

EDIT: Also, that's just the sorting option I'm using. There are various sorting options you can use (including by artist and general tags).

Kage_No_Shi said:
So basically, you want to be able to easily archive everything from various boorus?

Quite simple if you have the right add-on for firefox.

The Danbooru downloader is what I use and it comes with several boards pre-'installed'. You can easily add board though, but going to Add-ons - Danbooru Downloader - Options - and adding http://e621.net/post/show/ (also, in your profile make sure you don't have 'resize' checked. Load an image and once it's done loading, it'll automatically be saved to your designated folder.

It works with pretty much every booru I've tried. You need to have a profile and be logged into it though (because the display URL changes between logged in, and not logged in).

It sorts first by board (danbooru, this board, etc), then by Series, if there is no series, it will go by 'misc'. It's less useful on this board, but it works quite well on general image boards like Danbooru. In point of fact, my Danbooru board folder stands at over 550 folders for various series.

BTW: I just tested it on this site (just registered), et voila, it works.

EDIT: Also, that's just the sorting option I'm using. There are various sorting options you can use (including by artist and general tags).

wut... we're talking about eliminating duplicate images from the collection. I've already done a complete site rip:
http://e621.net/forum/show/1625?page=2

xanonx said:
TL;DR (but I swear I at least skimmed)

Sounds hard. I don't want to be the one to add this to Danborganize.

xanonx said:
wut... we're talking about eliminating duplicate images from the collection. I've already done a complete site rip:
http://e621.net/forum/show/1625?page=2

It wouldn't be used just for duplicate images; it's primary function would be detecting and flagging artwork that has been deleted previously, and flagging artwork that an artist has informed us they don't want on the site.

The idea is that we have a separate uploader only for artists. Once an artist draws a new picture, but decides that they don't want it to appear on e621, then they upload it using this uploader, the image is analyzed and fingerprinted, and then stored away so that it can be reviewed by an admin (in case someone is trying to upload art to this system that isn't their own). At that point, the artwork will be prevented from appearing on e621; users can still upload it, but it won't actually appear unless an admin approves it.

With this system, protecting and managing copyrights becomes the responsibility of the copyright holder, which is where it belongs. If we give you a system to help protect your copyrights, and you're just too lazy to use it... I can't help you.

I see. So you want something that's online that takes a fingerprint and then compares fingerprints. This approach will require significantly more disk space, not like that's a problem since it is 2011.

So we would set up a series of buckets containing XML fingerprints and then compare a given finger print to those. Sounds like a challenge.

I think the biggest concern would simply be processing power, possibly to the point of needing to get another server so the site itself doesn't get bogged down.

Also kind of curious how much a hosted solution such as PixMatch would cost: http://www.ideeinc.com/products/pixmatch/ Though I think I remember e-mailing them a while ago, but never heard anything back.

Fingerprinting would almost be a constant. That aside, processing probably wouldn't be that big of a deal. Using the methods described, you'd be making like (base2) n*log(n) comparisons at best up to a certain interval for the buckets, followed by a trivial constant for key-point challenging (It depends if we want to bucket off of key-point challenging as well).

The problem gets to be the tree shape if one were to bucket everything so there would be as few images in each bucket as possible. While this would be a simple comparison based operation, the tree may be disproportionate and become just a singly linked list resulting in n^2 operations.

Idk, this is a trade-off that pros have to deal with and since I don't quite yet have my doctorates in Computer Science, I can't quite give you an application with out physically trying it first.

WolfieWolfie1992 said:
Blotch, BlackTeagan, Kacey, TaniDaReal, Klaus Doberman, Ebon Lupus, Washa, LoneWolf6666, Rukis, Rarakie, Adam Wan, Keovi, Blaze-Lupine, Tsaiwolf, GlowFox, SparkleMotion, TrueGrave, Taurin Fox, and hundreds more....

Art by those artists ARE NOT "poorly drawn furry porn"....

+ IPoke

Riversyde said:
post #109279

well said mate.