Bug #75592 for HTTP-Message: RFC Violation - Broken POST content

I found the problem. HTTP/Request/Common.pm:85: # HTML/4.01 says that line breaks are represented as "CR LF" pairs (i.e., `%0D%0A') HTTP/Request/Common.pm:86: $content =~ s/(?<!%0D)%0A/%0D%0A/g; That's a nifty little negative lookbehind substitution, but "%0A" is not a newline. In fact, it all three characters fit on one line. And modifying already-encoded binary data isn't being helpful at all.

Since the new behaviour is to encode "\n" as "\r\n" for application/x−www−form−urlencoded parts this is clearly not transparant if you try to pass through binary data. I can see that. The new behaviour only make sense to text content (where you emulate an HTML form). I don't know what you mean by RFC Violation. The patch that introduced this behaviour claimed the the old behaviour conflicted with the HTTP RFC. What RFC do you claim say otherwise? Binary parts would normally be handled by using the multipart/form−data format instead. Your patch is broken. It breaks the test suite and works for you simply be effectively disabling the statement it affects. Thus reverting to the old behaviour. I think it ought to be possible use this interface and get transparent binary encoding as well. Not quite sure how yet.

Gisle, Thank you for your response! I understand that HTTP RFC 2068 suggests that CRLF be used across the wire to fully comply. And I believe it might even be appropriate to convert multipart/form−data fields (that happen to be text/*) to CRLF format so that everything across the raw transport is standard. But there is nothing in RFC 1738 that suggests you can inject extra characters during the encoding process for application/x−www−form−urlencoded data. And "%0A" is just three digits in plain text without any special characters so should be safe across the wire as is, thus the violation referred to in my report. However, I have seen SOME real browsers such as IE encode textarea inputs with forced CRLF prior to encoding, which I believe your 6.03 will help emulate this behavior. I'm not sure what any RFC states about that or where that idea even comes from or if that is even proper or not. And if this behavior I've seen is proper, then yes, my patch would break that. Unfortunately, I'm not sure how to tell LWP::UserAgent or HTTP::Request when I know that my content is not text and should not be corrupted in this case. I just know that everything used to work fine for years posting binary inputs, and suddenly after upgrading to 6.03 it is broken. I guess I can double URL encode it on the client end and then double decode it on the server end to workaround any encoding issues with HTTP::Request, but that would require changes on both ends and doesn't provide a slow migration path very easily. I'm just not sure what the best solution is. And which RFC are you claiming that the OLD behavior does not comply? Or do you have any other suggestions?

Looking at http://www.w3.org/Protocols/rfc2616/rfc2616- sec14.html#sec14.15 I see: "Conversion of all line breaks to CRLF MUST NOT be done" in reference to calculating a Content-MD5 header, which suggested to me that messing with my line endings was probably a bad idea. Then, I saw: "HTTP allows transmission of text types with any of several line break conventions and not just the canonical form using CRLF." which prompted me to look at http://www.w3.org/Protocols/rfc2616/rfc2616- sec3.html#sec3.7.1 which is more explicit. I'm certainly an amateur at reading RFCs, but it looks like you're not required to convert text types in the body to CRLF. If that's the case, please consider reverting this change.

Plus one to reverting conversion. Not only prepending every 0A with 0D completely breaks binary data, it is also not clear that "x-www-form-urlencoded" content-type is not suited for it, as many cache servers and existing communications depend on it. Actually, any silent alteration (and this is alteration, not some sort of transfer-encoding - server gets data different from what was sent by client) of user data seems very unnatural to me, not to mention breaking compatibility with previous versions. Issue warning (it is at most; I'm not sure that you got whole idea right).

Ticket migrated to github as https://github.com/libwww-perl/HTTP-Message/issues/69