I tried to use login and password_hash to log in to e621 for a linux tool I'm developing, but I couldn't find a way to do so. What I'm looking to do is login to my account and apply my blacklist to the search that it conducts based on your entry to the textfield popup. Here's a snippet of the code that (attempts) to use the API:

tags="$(zenity --entry)";
ID="/home/user/.e6ID2.txt";
wget -O - 'https://e621.net/post/index.json?login=NikolaiTheFur&password_hash=(my_api_key)&tags='$tags | jq --raw-output '.[].ID' - > "$ID";

Running this portion of the code will successfully output a list of all the post IDs to /home/user/.e6ID2.txt, but the list of IDs it outputs will include posts with tags that I have blacklisted.

Does anybody have an idea of what I'm doing wrong?

Blacklisting is applied client side via javascript -- the server doesn't do anything except supply the blacklisting script.

That means you have to implement it, unfortunately. Of course, it also means E621 supplies an implementation that you can use as a reference.

Your login looks okay to me; if you are concerned you could always do something like submitting a tag edit via API -- which you must be logged in to do; if that returns 200 OK, then you are logging in ok.

savageorange said:
...Of course, it also means E621 supplies an implementation that you can use as a reference.

Do you know where this implementation is?

After doing a bit of research here in the forums, I found some other people that wanted to do the same thing, but I could only find examples for Python that would do what I'm looking to do. I have no background in Python at all, so I can either program it in shell scripts, or find a way to host a php tool on my server from a separate port than the one I'm using for my booru...

I don't know offhand exactly where to find it, only that it would be specified in the source of any post/index page as a <script> element, either inline or as a href. A quick scan of the source suggests that Post.init_blacklisted() is the function that handles it. (you can see this right at the bottom of the source.)

Python (or some other language that has set operations built in) would be the simplest implementation (I think it could be done in one line). Of course it would be possible to implement in shell or PHP -- they are both turing-complete languages. I think you could even do it entirely inside jq (but, that would be even crazier than doing it in PHP)

EDIT:
Yep, you can see the implementation most of the way through application-min.js
It starts with

init_blacklisted:function(options)

I think. I don't speak badly-formatted JavaScript very fluently -- help:blacklist may be of more help here;)

It's basically a very simple operation - you have a bunch of lines in the blacklist, and each blacklist line lists a set of boolean (TRUE or FALSE) conditions. If all the conditions are TRUE for a given blacklist line, then you know the post is blacklisted. If you get the whole way through the blacklist without encountering a line where all conditions are TRUE, then you know the post is not blacklisted.

I should also mention that you can grab a copy of your blacklist through the API now(which is also a good test that you're logging in successfully) by using the /user/blacklist.json/xml endpoint.

The blacklist is applied in two parts in the JavaScript Post.init_blacklist and Post.apply_blacklist, but what savageorange describes is accurate.

The relevant pieces are

var post = pair.value
      var has_tag = post.match_tags.member.bind(post.match_tags)
      post.blacklisted = []

      Post.blacklists.each(function(b) {
        if (b.require.all(has_tag) && !b.exclude.any(has_tag)) {
          b.hits++

          if (!b.disabled)
            post.blacklisted.push(b)
        }
      })

and

var bl_entries = Cookie.get("blacklisted_tags").split(/[&,\n]/);

    bl_entries.each(function (val) {
      var s = Cookie.unescape(val).replace(/(rating:[qes])\w+/, "$1");
      var tags = s.match(/\S+/g);

      if (!tags)
        return;

      var b = {tags: tags, require: [], exclude: [], disabled: false, hits: 0};

      tags.each(function (tag) {
        Post.blacklisted_user(tag);

        if (tag.charAt(0) == "-")
          b.exclude.push(tag.slice(1));
        else
          b.require.push(tag);
      });

      Post.blacklists.push(b);});

The magical match_tags field on each post is just a copy of the tags with some of the metadata tags shoved into it, like id:12832.

KiraNoot said:
I should also mention that you can grab a copy of your blacklist through the API now(which is also a good test that you're logging in successfully) by using the /user/blacklist.json/xml endpoint.

I was able to access my blacklist using the following code:

wget -O /home/user/blacklist.json 'https://e621.net/user/blacklist.json?login=NikolaiTheFur&password_hash=(my_password_hash)';

However, it formats it like this:

{"blacklist":"akaece\nakunim\nambiguous_gender ... n\u30e9\u30a4\u30ebid:980544"}

How many tags can I use in API post searches? Would I be able to pipe all the tags in my blacklist into my search script without getting a 500 error?
And if you know, how would I cut out the {"blacklist":", \n and "} parts?

You get the same number of tags per search as you do outside the API. Which I believe is either 6 or 8 tags. Not enough for you to meaningfully implement the blacklist in search. You should implement it on your side.

And a string replacement should get rid of that, and you can split on the \n

^ Actually, when I think about it there are really tiny blacklists -- like "fox -lion" -- that are not even possible to implement via search, since searching doesn't support parenthetical expressions. If I understand correctly, the lack of parenthetical expressions means any blacklist line with more than one term cannot be incorporated into a search even if it wouldn't exceed the number-of-terms limit.

^^ jq should do that for you. You just need to select the blacklist key, and output in raw mode (-r, as I've previously illustrated for you). After all, what you are getting back is just a JSON document; the \n are escaped newlines.

savageorange said:
^ Actually, when I think about it there are really tiny blacklists -- like "fox -lion" -- that are not even possible to implement via search, since searching doesn't support parenthetical expressions. If I understand correctly, the lack of parenthetical expressions means any blacklist line with more than one term cannot be incorporated into a search even if it wouldn't exceed the number-of-terms limit.

^^ jq should do that for you. You just need to select the blacklist key, and output in raw mode (-r, as I've previously illustrated for you). After all, what you are getting back is just a JSON document; the \n are escaped newlines.

So, in the script you demonstrated to me, you had me use jq --raw-output '.[0].id', but I never completely understood what the .[0] part meant. All I know is that I can replace .id with something like .file_url, and it'll just tell me where the source image file is. How would I modify the code above to only show the blacklisted tags shown in blacklist.json, each on a new line?

Nikolaithefur said:
So, in the script you demonstrated to me, you had me use jq --raw-output '.[0].id', but I never completely understood what the .[0] part meant.

Assuming that you understand basic JSON ,

read the jq man page, using man jq. (RTFM.. It means no worries, for the rest of your days.. It's a problem free philosophy... [well unless the documentation is terrible ;)])

Relevant part:

.[<string>], .[2], .[10:15]

You can also look up fields of an object using syntax like .["foo"] (.foo above is a shorthand version of this). This one works for arrays as well, if the
key is an integer. Arrays are zero-based (like javascript), so .[2] returns the third element of the array.

The .[10:15] syntax can be used to return a subarray of an array or substring of a string. The array returned by .[10:15] will be of length 5, containing
the elements from index 10 (inclusive) to index 15 (exclusive). Either index may be negative (in which case it counts backwards from the end of the
array), or omitted (in which case it refers to the start or end of the array).

The .[2] syntax can be used to return the element at the given index. Negative indices are allowed, with -1 referring to the last element, -2 referring to
the next to last element, and so on.

The .foo syntax only works for simply keys i.e. keys that are all alphanumeric characters. .[<string>] works with keys that contain special characters
such as colons and dots. For example .["foo::bar"] and .["foo.bar"] work while .foo::bar and .foo.bar would not.

The ? "operator" can also be used with the slice operator, as in .[10:15]?, which outputs values where the inputs are slice-able.

jq '.[0]'
              [{"name":"JSON", "good":true}, {"name":"XML", "good":false}]
           => {"name":"JSON", "good":true}

           jq '.[2]'
              [{"name":"JSON", "good":true}, {"name":"XML", "good":false}]
           => null

The above should illustrate both what .[0] does and what .id does, I've bolded the bits I think are most important. A bit of experimentation with JSON files you make up should help you understand any subtleties.

PS. I'm not sure whether you understood when I said before that \n are escaped newlines: this means that the blacklist is already returned 'one blacklist entry per line', the newlines are just encoded as \n. Outputting in raw mode causes that escaping not to be done (hence, you get -literal- newlines out , not the "symbolic" \n you would get in default non-raw mode)

savageorange said:
Assuming that you understand basic JSON ,

read the jq man page, using man jq. (RTFM.. It means no worries, for the rest of your days.. It's a problem free philosophy... [well unless the documentation is terrible ;)])

Relevant part:
The above should illustrate both what .[0] does and what .id does, I've bolded the bits I think are most important.

Heh, I dunno what I'd do without you guys... Thank you so much. Here's the code that worked:

ID="/home/user/e6blacklist.txt";
wget -O - 'https://e621.net/user/blacklist.json?login=NikolaiTheFur&password_hash=(my_api_key)' | jq --raw-output '.blacklist' - > "$ID";

Nikolaithefur said:
Heh, I dunno what I'd do without you guys... Thank you so much. Here's the code that worked:

ID="/home/user/e6blacklist.txt";
wget -O - 'https://e621.net/user/blacklist.json?login=NikolaiTheFur&password_hash=(my_api_key)' | jq --raw-output '.blacklist' - > "$ID";

Looks good. Clarity could be improved slightly by using truthful variable name (ie. ID really isn't an ID, is it? Isn't it BLACKLIST or something like that?)

savageorange said:
Looks good. Clarity could be improved slightly by using truthful variable name (ie. ID really isn't an ID, is it? Isn't it BLACKLIST or something like that?)

I suppose, I was just copying and pasting some code from https://github.com/Justawinterwolf2012/e621-md5-tagger (which is long due for an update to the local version I have tidied up on).

Just a reminder that you can get the un-minified (and therefore much easier to read, except for the libraries included at the very top) version of application-min.js at https://e621.net/javascripts/application.js. (The relevant sections for this question were posted by Kira though)