Bug overview description.
Using e621 site in Firefox (84.0.2) on Windows 10 (20H2 19042.685 with no extra keyboard inputs or languages enabled); pasted a string from Notepad++ into e6 pool description, text area js counter indicated 9980 characters, submitted change and received a server validation message it exceeded 10000 limit. (NP++ indicated length 10076 due to a few multibyte …“”’ & \r\n instead of \n, I didn't use any DText/BCode markup). Played around a bit to figure out why, determined that js counter counts per character (line breaks and non_ascii/multibyte characters each count as 1) which makes sense. Figured maybe the server validation was counting bytes instead so replaced anything outside common 7-bit ascii (&#20-&#7F) with equivalents in range and switched from CrLf(\r\n) to Lf(\n) line breaks and then js count matched NP++ indicated lengths. Submitted again and received notice that it exceeded 10k again, shortened it by removing 10 of the chars I added and a few line breaks and the server accepted it. (pool was: 25763)

What part(s) of the site page(s) are affected?
Pool editing (and probably pool creation page and a few other site pages too) description field.

What is the expected behavior?
Server validation and js character count use same criteria for count and allow for 10k (ascii) characters (or adjusted to compensate for multibyte characters if needed due to db field size limit).

What actual behavior is given instead?
JS character count shows actual individual character count, server validation rejects string as too long when close to limit

Time of incident (if applicable).
2021/12/29 ~1700 EST

Can you reproduce the bug every time?
Yes

What steps did you take to replicate this bug?
Copied txt from FA post pages to Notepad++, added some labels and copy/pasted into e621

Errors or other messages returned (if any).
"Error: Description is too long (maximum is 10000 characters)"

Interesting.

The JS side of things just gets the length property of the string to determine the character count.
Rails' validation is more complex, and must be counting bytes instead.

It's probably easier to fix the former, rather than the latter.

Can you provide the original in a text file somewhere? Having the original text will make it much more easier to look into.

Files posted here: https://ufile.io/f/opu1r (good for 30 days)

For pool: https://e621.net/pools/25763 (what's there is fine, just here for reference)

- multibyte.txt (\r\n & mb …“”’) - is what I tried to post at first (file character length 10018, bytes 10076, js indicated 9980 which is character length with \n, would not post)

- ascii.txt (\n & swapped mb …“”’ for ...""') - was trying to figure out discrepancy and also get bytes below 10k (character length, bytes & indicated: 9982, would not post)

- posted.txt (\n & mb …“”’) - what's currently there, went back to original (multibyte) and removed line 3 "The story:" and a few line breaks (character length & indicated 9964 & bytes 10022, posted fine)

Edit: I didn't know the site was open source, cool, I'm not a ruby/rails dev though, but now I know it's there, I'm going to take a look too; mystery, must figure out... :)

The problem doesn't seem to be multibyte handling, both js and rails report the same length for the text. Newlines are an issue though. It seems that every \n is transformed into \r\n during submit (at least on firefox), which we are both using. 10000 \n will appear as 10000 \r\n, effectivly doubling the length in this case and thus failing validation.

Interesting... (I think I've run into this on other sites in the past too)

Same behavior on MacOS/Firefox too; I also tried on Safari, but Dev tools didn't show newline characters on request data.

A bit of Googling and is sounds like "newline normalizations" are part of the browser specs these days... (I guess that's good, something to add to my notes though)

Like @bitWolfy mentioned, probably easier to fix in js side (and just count all newlines as 2x chars, worse case js reports something as too long that server side would accept after all) rather than overriding in Rails with a custom validator. Kinda hack-y but in dtext.js function build_charcounter changing lines 125 & 130 as follows might do... (untested)

(textarea.val() + "").replace(/\r/g, "").replace(/\n/g, "\r\n").length

-
Thank you both for all your work on the site :)