Danborganize

I was bored a few days ago browsing e621.. when I stumbled upon the API section of the site and thought it would be a neat idea to make a small application that would enable a user to download images and organize them by their tags, exactly like e621 does, but at a desktop-level with a customized gallery and the ability to import your own images.. upload them to e621 etc..

Blah blah blah... Danborganize. I'd like to say it's current state is pre-alpha and very buggy.. however it is stable enough for decent use and I don't plan on changing its system of storage so portability to newer versions should be easy!

http://code.google.com/p/danborganize/

Contribution

Any contribution is welcome! As I am new to the open source world.. Critique on code, ideas, changes are all welcome!

email: [email protected]

How badly will this effect the bandwidth?

From a quick look, I'd say about as badly as anyone who searches the booru through the web interface, with the exception that a user can quickly select a batch of images returned by the program's search feature and download them en masse into a "local gallery".

In the wrong hands, the whole API could be a terrible source of abuse to the server, if only because of what this tool demonstrates: retrieving a list of posts based on a tag search, and then grabbing all of those posts in a large batch, with no potential for ad revenue.

There doesn't appear to be any paging capabilities in the tool yet, and the tool defaults to 64 images in a search, so a person would be limited to the first 64 images under a given tag search.

For comparison, the e621 booru itself limits a search to 30 results in a page, and the API docs say there's a max of 100 posts returnable in a single request.

Bandwidth-wise, this tool is slightly more efficient than using the website, but what it gains in efficiently it probably more than loses in danger for abuse (searching for and grabbing far more previews and/or images than necessary). But then, the whole API struck me that way, as well. :/

ikdind said:
From a quick look, I'd say about as badly as anyone who searches the booru through the web interface, with the exception that a user can quickly select a batch of images returned by the program's search feature and download them en masse into a "local gallery".

In the wrong hands, the whole API could be a terrible source of abuse to the server, if only because of what this tool demonstrates: retrieving a list of posts based on a tag search, and then grabbing all of those posts in a large batch, with no potential for ad revenue.

There doesn't appear to be any paging capabilities in the tool yet, and the tool defaults to 64 images in a search, so a person would be limited to the first 64 images under a given tag search.

For comparison, the e621 booru itself limits a search to 30 results in a page, and the API docs say there's a max of 100 posts returnable in a single request.

Bandwidth-wise, this tool is slightly more efficient than using the website, but what it gains in efficiently it probably more than loses in danger for abuse (searching for and grabbing far more previews and/or images than necessary). But then, the whole API struck me that way, as well. :/

You're entirely correct. I was shocked about the 100 post limit and thought it would be more wise to limit my program to 64, however I do plan on improving it a lot , and adding a page system...

To compromise for ads I decided to add the option to copy the link address of the image, which directs the user to the /post/ page instead of a direct link to the image...

This is completely new and very underdeveloped, I accept all sorts of comments good/bad :3

When I go to tools - preferences, nothing happens.

I have no problems with people using this tool; there aren't any capacity issues at present, and as long as it doesn't go batshit insane and consume 10+mbit/sec per user, it's unlikely to have much of an effect on the bottom line.

One thing I would like to request is a unique User-Agent sent in each HTTP request, if possible containing the version number of the application. This would help me a lot if one version of the app turns out to be a complete resource monster, and results in me needing to throttle the throughput of that version or something.

If people are going to use the API heavily for day-to-day usage, I think a good way around the whole 'no ads' issue would be to simply include an ad image + link in each result (or as a seperately callable API function), for app builders to voluntarily support the site by displaying ads in their app. Of course, this isn't going to become mandatory - and I don't really mind if people wish to use the API as long as it doesn't take down the site :D

Considering the speed the application runs at (it's single threaded, and is pretty much limited to the round-trip request time for each image), I don't anticipate any significant load issues.

So far, really useful tool! It'd be really handy if the local gallery were seperated into the tag searched for rather than by image hash; this way you could search for specific things then hit 'download' and have it keep each search seperate on the local machine.

Varka

@JimmyJ: Nope, that isn't setup currently. Pre-alpha and stuff.

I'd like to become a contributor to this project if the admins don't quash it. You can see some of my ideas by grabbing modified source here:
http://rapidshare.com/files/437759000/Danborganize-ikdind.zip

This puts tag search in a separate thread so it doesn't hang the application during searching, adds reading and writing XML for preferences, and some initial ideas/implementation for a skin.

Good to hear, Varka. Personally, I'd be for all for setting up some way of injecting ads into the application if it helps to support the booru.

I'm not sure why the app tries to mimic the directory structure of the booru with locally stored images. I also figured it would be more useful (and thus I think it should be implemented) if there were some kind of database-like feature in the app that could store tag and other information about posts so that a user could search through their local gallery by tag as an addition to searching e621.

Adding a user agent string appears to be easy enough since the app uses standard System.Net.HttpWebRequest class.

Edit: I'd always assumed that the reason e621 and other boorus split images into multiple directories is so that they can be split across multiple mount points via softlinks, allowing the system to grow beyond filesystem volume size limitations.

ikdind said:
Good to hear, Varka. Personally, I'd be for all for setting up some way of injecting ads into the application if it helps to support the booru.

I'm not sure why the app tries to mimic the directory structure of the booru with locally stored images. I also figured it would be more useful (and thus I think it should be implemented) if there were some kind of database-like feature in the app that could store tag and other information about posts so that a user could search through their local gallery by tag as an addition to searching e621.

Adding a user agent string appears to be easy enough since the app uses standard System.Net.HttpWebRequest class.

Edit: I'd always assumed that the reason e621 and other boorus split images into multiple directories is so that they can be split across multiple mount points via softlinks, allowing the system to grow beyond filesystem volume size limitations.

You obviously know a lot more about keeping a database than I do xD.. I've actually never implemented multi-threading in an application before and had it work properly. I'm about to study your code and read up on your advice/fixes.

I love the input <3. I can easily be contacted through aim/msn/yim messengers they're listed on my FA: oskenso

@Varka: thanks for all that great info, i'll implement a user-agent and start designing a better/gallery way of organizing the gallery

@ikdind i'd love to have you on the project as a project committer if your interested :D

@ikdind I've checked all your improvements and they're remarkable. I'm guessing asynchronous calls are just like multi threaded procedures? they provide a callback function when completed?

Also mega ++ for fixing up the preferences class :3

Glad you appreciate my ideas. I've emailed you from a google account that you should be able to add to the project. As I indicate in my email, there's a little more that I need to do to clean up that first batch of changes, but then I can commit them to the repository.

Attempting to use search function with an empty gallery gives birth to Unhandled exception.
http://pastebay.com/112350

ktkr said:
Attempting to use search function with an empty gallery gives birth to Unhandled exception.
http://pastebay.com/112350

I haven't been able to reproduce this bug, however i've been trying to track it down in the latest revision. perhaps it's been fixed :o.

If that's the same thing as Bug ID 1 from the Google Code site, then I haven't been able to repro it, either. Not sure what's up with that, maybe its been fixed already or maybe there's something non-obvious going on.

Varka, I went ahead and added a User Agent string to the web requests. The latest version of Danborganize should be identifying itself as "Danborganize/0.1.0.0". Future versions should iterate the version number appropriately, as long as Osk and I can agree to when version numbers should be ticked.

ikdind said:
Future versions should iterate the version number appropriately, as long as Osk and I can agree to when version numbers should be ticked.

Heh, as i'm new to most of this i'm pretty sure i'd agree with almost anything that's sound.

Also we can safely assume that issue 1 was fixed.

And last for those of you that don't know. the program's state is like a little newborn baby it needs care and food (code) before it can start walking/talking. Don't expect it to start helping with your house payment just yet!