Tech: Seeing any funny characters?

Subject: Tech: Seeing any funny characters?
From: Sawzaw
Date: 20 Feb 10 - 12:05 AM

I am seeing square characters with FFFD inside a box where an apostrophe should be.

Sometimes I see something really weird with a euro symbol where a quotation mark should be.

Anybody see the same thing?

I think it has something to do with a character set statement in the header of a page.

Subject: RE: Tech: Seeing any funny characters?
From: Joe Offer
Date: 20 Feb 10 - 12:55 AM

Hi, Sawzaw -
Is this in a particular program, on the Internet, or in Windows or Mac in general? I get that every once in a while, too - but now I can't recall what I did to fix it.

-Joe-

Subject: RE: Tech: Seeing any funny characters?
From: Geoff the Duck
Date: 20 Feb 10 - 10:41 AM

Sometimes there are strange characters when someone has written a posting using a programme such as MS Word and then copied from the programme and pasted into the Mudcat Reply box. Somewhere in the copying, it carries hidden "formatting" codes which are interpreted differently when printed using HTML in the Mudcat posting.
Often you will find that postings from a particular individual show the same set of "wrong" characters.
Quack!
Geoff the Duck.

Subject: RE: Tech: Seeing any funny characters?
From: Geoff the Duck
Date: 20 Feb 10 - 10:42 AM

Anyway - Mudcat is full of strange characters!
Some of them are even stranger in real life...
Quack!
GtD.

Subject: RE: Tech: Seeing any funny characters?
From: Bill D
Date: 20 Feb 10 - 11:33 AM

Also, if the character specified is not supported by the font in use by your machine, you will see the box with the 'code' it is attempting to use.

Subject: RE: Tech: Seeing any funny characters?
From: Sawzaw
Date: 20 Feb 10 - 12:23 PM

It occurs when I cut and paste into a mudcat post using XP and Seamonkey or Firefox.

Some Quote marks are more slanted that others and some slant right and left which causes a strange character to appear so you have to hunt through the text and try to replace them.

Subject: RE: Tech: Seeing any funny characters?
From: JohnInKansas
Date: 20 Feb 10 - 09:24 PM

Some browsers may show the code for unrecognized characters, but others only show a box - or in some fonts another symbol - that's the "universal character not recognized" glyph.

For any of them, you can copy a passage of the text from the browser and paste into Word. Put your cursor immediately to the right of the character in question and hit Alt-X and the glyph will be changed to the Unicode Hexadecimal Code Number. Going the other direction, you can type a Hex Code, and with the cursor at the right of the code, Alt-X will translate the Hex Code to the glyph, if there is a glyph for it in the font you're using. (It's a toggle.)

A common cause for "strange glyphs" is a change in the "language" being used. Almost any PC program can be set to use any of dozens of languages, and will choose different font sets when the setting is changed. Occasionally a change can creep up on you inadvertently.

Keyboards regionalized to different areas may have different keys, strange key locations, and may use font sets in which "special characters" are mapped to different character codes. Most European keyboards have a key for the € (euro) character, but US keyboards generally don't. The decimal character 128 is undefined in the ANSI specification, so Microsoft has used it to display the euro, and turning on Num Lock and typing Alt-128 on the Num Pad is the simplest way for 'muricans to insert a Euro. Most browsers set to use UK-English probably will recognize the number code 128 as a €, but some other regional languages - some of which closely resemble a common "English" - may not.

Some browsers, when set to "text only" will not render "curly quotes," and since they're not "standard language" in html, a pure html site like Mudcat may do strange things with them. Some html interpreters can handle them, and some will display them okay in plain text but will mangle links/commands if the curlies are used in place of straight ones.

If you encounter text with lots of "curly quotes" and want to convert them to straight quotes, you can paste the text in Word and uncheck the "use curly quotes." If you [Replace-All " - Replace With " ] all the quotes will be converted to whichever form is set at the time of replacement. (and don't forget to Replace ' with ').

There are some times when strange characters are pretty much expected, but if you're getting strange characters unexpectedly a first step would be to check what language your browser is set to use, and choose a different one that might give a better font set mapping.

A second step would be to choose a different font that may have a more complete character set. Microsoft provides almost-full-Unicode fonts, but they're so large you may not want to have them loaded all the time. (Most people don't use the "full Unicode" fonts unless they're doing lots of multi-language correspondence.)

If changing language and/or fonts doesn't clear things up to your satisfaction, you can paste the text into Word as described above, and "read off" the actual Hex Codes being used (assuming your browser doesn't display them already), and then look up the characters intended in the Unicode tables or by scanning (in Windows) through likely fonts in the Character Map utility.

Once you find something that works, you'll probably know why it wasn't working with your original settings.

John

Subject: RE: Tech: Seeing any funny characters?
From: Sandra in Sydney
Date: 21 Feb 10 - 07:35 AM

I use a Mac & firefox & sometimes see the strange characters mentioned & sometimes not!

This thread is a perfect example - http://www.mudcat.org/thread.cfm?threadid=21508&messages=17 - On the Mudcat Discussion Forum page - mudcat.org - it has (the box with question mark) instead of (e with accent mark), but when I open the thread I see Armentières (which I just copied & pasted from the thread)

very strange that on 2 different Mudcat pages I see 2 different things.

sandra

Subject: RE: Tech: Seeing any funny characters?
From: Sawzaw
Date: 21 Feb 10 - 10:39 AM

I found this on the net but the problem is with Seamonkey too. I can't even stand to open up Explorer.

Still it does not explain the spidery looking mess with the euro symbol in it that you get from some quotation marks.

Here

And here also:

How To Fix 'FFFD' Thing In Mozilla Firefox 3.0?

Since I have installed the new version of Mozilla Firefox (v3.0) few weeks ago, I've seen several web pages that show 'FFFD thing' (or FF-FD) instead of letters and numbers.

Searching for a solution for this problem in my Internet browser, I've found it at one Firefox related help forum.
So, I'll present you the solution in this post, in hope that it will help to somebody that have the same problem with Mozilla Firefox v.3 .

The solution is simple.
Just follow this path in your Firefox and click at 'Universal' (activate it, because probably the 'Auto-detection' was turned off):

View >> Character Encoding >> Auto-detect >> Universal

Subject: RE: Tech: Seeing any funny characters?
From: Richard Bridge
Date: 21 Feb 10 - 10:43 AM

Well, some of my lodgers' friends are a bit odd - like his lead singer who with a bit of red paint cloven hooves and a tail and horns would strongly resemble a cartoon Satan, and then there's his other friend Mike aka "Captain Damage". But there again said lodger is a drummer in a metal band so it's probably what passes for normal in his life, a bit like here.

Subject: RE: Tech: Seeing any funny characters?
From: Bill D
Date: 21 Feb 10 - 12:08 PM

⚧

curious to know what various people see above.

Subject: RE: Tech: Seeing any funny characters?
From: Wolfhound person
Date: 21 Feb 10 - 12:20 PM

A circle, with a cross below (the symbol for female)and two arrows above at angles(male symbol twice over) the right hand one plain, the left with a line through it.

XP and Firefox if that helps....

Paws

Subject: RE: Tech: Seeing any funny characters?
From: Bill D
Date: 21 Feb 10 - 12:28 PM

I can't get it to show the symbol in any font I have installed. I suppose it require that huge font that is not practical for everyday use.

∞

but that one works fine.....for me...

Subject: RE: Tech: Seeing any funny characters?
From: Bill D
Date: 21 Feb 10 - 12:29 PM

great! It does work...thanks... now I have to figure out what setting *I* need to adjust.

Subject: RE: Tech: Seeing any funny characters?
From: Bill D
Date: 21 Feb 10 - 12:42 PM

⚐

I surrender...

Subject: RE: Tech: Seeing any funny characters?
From: Sawzaw
Date: 21 Feb 10 - 02:11 PM

“

â€

I think I captured them crazy critters,

It has something to do with smart or curly or magic quotes.

Subject: RE: Tech: Seeing any funny characters?
From: JohnInKansas
Date: 21 Feb 10 - 02:24 PM

All I see is the simple empty square that the majority of fonts display for any character not recognized.

My browser receives it as Unicode Character 26A7. In my copy of Unicode 4 character tables (downloaded some time ago) there is no character defined for that code number. The last (highest) number in the 26?? chart is 26A1, which should show you the "high voltage sign" - a zig-zag three segment "spark" representation ⚡- which does NOT display in my preview.

The series from 2600 through 26FF is defined as "Miscellaneous Symbols" and does NOT HAVE characters assigned for about 30% of the "numbers" in the range. Few of the characters from this range will work without the reader of your document having "special" fonts, although a few have been mapped in my IE7 - US English - Times New Roman setup. I'm not sure that my Unicode Char Tables are the latest, but since nobody who wants to communicate should be using these chars, I havent' bothered to check for updates. They're okay to play with; but unless you confirm that those reading your messages will be able to see them, it's a bit childish to try to post them.

ALL ASSIGNED CHARACTERS have names. If you can provide the name of a glyph or its Unicode Hex Nuber, for something that you need to be able to post to convey an important message, one or more of us may be able to help find a method that will work for most people, although it's impossible to guarantee that "out of range" chars will be seen by everyone.

The following chars in teh 2600 to 26FF range are defined and display in my preview, but may not show as anythings but a "char unknown" symbol in your browser. (These are just samples selected at random.)

☠ ♔ ♕ ♖ ♗

John

Subject: RE: Tech: Seeing any funny characters?
From: JohnInKansas
Date: 21 Feb 10 - 02:59 PM

I don't see a reason why smart or curly quotes should affect posting characters.

For chars in the ANSI range, you just hold down the Alt key and type the decimal number of the character on the Number Pad (NumPad must be turned on.)

128 is the decimal number used on US machines for the euro, so if you type Alt-0128 you should get €.

For characters in any range, you can use the decimal number for the character and type:

&#nnnn; where nnnn is the decimal number.

Entered this way, € produces €.

For characters in any range, you can use the hex number and type:

&#Xnnnn; where nnnn is the hex number.

You used hex number 20AC for the euro, so typing € gets €.

For some characters there are character names, and you can type ® to get ®, © to get © and such, but no quotes of any kind are needed.

Have you found a harder way to do it?

John

Subject: RE: Tech: Seeing any funny characters?
From: Bill D
Date: 21 Feb 10 - 03:01 PM

well John, I got the character from this page..one of two which list strange ones.

I have in the past managed to post...and see... the famous 'peace' symbol that was debated awhile back.

Now, I do NOT see the symbol that Wolfhoundperson saw, but I DO see the the little box...not empty, but with the relevant 4 numbers/letters in it.

I have no idea why it showed up fine for someone using the same operating system & browser I used, but not for me.... I tried ever font and every encoding system that seemed like they might be relevant, but *shrug*....

I kinda like exploring the thing... but it's even more fun when I can make it work and define the rules for ensuring it DOES work.

☺

I guess I'll keep trying unto ∞

Subject: RE: Tech: Seeing any funny characters?
From: Bill D
Date: 21 Feb 10 - 03:04 PM

Oh! and I DO see the chess symbols you posted...hmmmm.... gotta wonder

Subject: RE: Tech: Seeing any funny characters?
From: Bill D
Date: 21 Feb 10 - 03:08 PM

test

⚐

Subject: RE: Tech: Seeing any funny characters?
From: Bill D
Date: 21 Feb 10 - 03:10 PM

nope.. (tried copying John's code and substituting the numbers I wanted...didn't help)

Subject: RE: Tech: Seeing any funny characters?
From: JohnInKansas
Date: 21 Feb 10 - 03:17 PM

Bill D -

You forgot the ";" on the end.

& says what follows is a code.

# says the code is a number

X says the number is a hex number

; says this is the end of the code.


⚐

Unfortunately, hex char 2690 is not defined in Unicode 4, so I see only the "unk char" rectangle.

Maybe you could use 2622, or 2623, which are defined, instead:

☢     ☣

John

Subject: RE: Tech: Seeing any funny characters?
From: JohnInKansas
Date: 21 Feb 10 - 03:22 PM

Bill D -

Try hex 262E

☮ = ☮

John

Subject: RE: Tech: Seeing any funny characters?
From: Q (Frank Staplin)
Date: 21 Feb 10 - 03:30 PM

¤

Subject: RE: Tech: Seeing any funny characters?
From: Bill D
Date: 21 Feb 10 - 03:45 PM

Ok, in Firefox, I do see those defined ones. (in a different browser, they didn't work... need to compare settings)

one more test, though I don't have much hope.

⚧

Subject: RE: Tech: Seeing any funny characters?
From: Bill D
Date: 21 Feb 10 - 03:47 PM

and a different one ♬

Subject: RE: Tech: Seeing any funny characters?
From: Bill D
Date: 21 Feb 10 - 03:49 PM

so...some are just not defined....

♿...crippled, as it were,,,

Subject: RE: Tech: Seeing any funny characters?
From: Bill D
Date: 21 Feb 10 - 03:50 PM

tsk...too bad. That one was a wheelchair.

Subject: RE: Tech: Seeing any funny characters?
From: Q (Frank Staplin)
Date: 21 Feb 10 - 03:59 PM

☮

Subject: RE: Tech: Seeing any funny characters?
From: JohnInKansas
Date: 21 Feb 10 - 04:03 PM

Bill D

Hex 26A7 is NOT DEFINED in the Unicode Standard, and it's unlikely that it's going to be reliably displayed in any browser.

In the 2600 range, undifined values are

2618, 267E, 267F, 2692 through 269F, 26A2 through 26FF.

NONE OF THESE are likely to work no matter how you type them in.

The undefined chars can be used by a computer, by installing a special purpose character mapping, but it's useless to produce something that's only readable on your own machine (unless you're a person of rather narcisistic kind, which I suppose might be applicable here).

Microsoft got away with mapping the € euro to ANSI 128 since it was an unassigned number and they gave everybody with Windows the new char map(s). If you're able to make an undef char work, you've got a rather immense distribution problem getting your char def distributed to the rest of the world.

        ☮

John

Subject: RE: Tech: Seeing any funny characters?
From: JohnInKansas
Date: 21 Feb 10 - 04:58 PM

The box with FFFD or FF-FD that some report seeing quite probably is NOT GIVING YOU THE CHAR NUMBER.

Unicode 4 does NOT DEFINE a character for any numbers between 009F and 00FF, which includes both 00FF and 00FD.

(I have those tables and the individual cells for those numbers are shown as undefined.)

Unicode Code Charts will give you an authoritative list of the characters that are (likey to be) usable. Put a (hex) number in the search box, and you'll get a link to the page that includes that number. The actual chart is a pdf that can be saved. (There should be about 4,369 pages if you'd like to save them all.)

I haven't bothered to download the pdf charts for numbers above Hex 34FF, but as a special favor I've looked up FFFD and find:

"FFFD Replacement Character. Used to replace an incoming character whose value is unknown or unrepresentable in Unicode."

The "standard glyph" for FFFD is a black square, rotated 45 degrees with a white question mark (?) inside.

Technically, your browser is probably defective if it actually uses FFFD as a character number, since that code means the character doesn't exist in Unicode. A different "substitute" should be used to represent a valid Unicode char code that just isn't in the fonts available.

Side note: both FFFE and FFFF show, e.g.:

"FFFF (not a character) The value FFFF is guaranteed not to be a Unicode character at all."

In other words, FFFE and FFFF CANNOT BE LEGALLY assigned to any character by mapping something else to that value. I'm not sure these are the only characters that can't be mapped to an undefined glyph, but there aren't many of them so far as I know.

What you are seeing is a glyph that your browser has used to represent "not a recognized character," but it's only a dummy "picture." There are several chars commonly used for "clinkers" and it's customary for the font in use to define a "dummy" char to be used in that font. If you get a different picture, using the same font in different browsers, it probably means your browsers are using different versions of same-named fonts.

It is likely that if you copy the glyph from your browser and paste it into Word and toggle it to the Unicode Hex number (Alt-X as described above), you'll see what code was originally typed. With that code number, you likely can look up what the intended glyph was supposed to look like. You might, alternatively, get the hex code for a character that looks like a box with FFFF or FF-FD, but that's not what I'd expect.

(What is it that they say is "doing the same thing over and over and expecting a different result?")

John

Subject: RE: Tech: Seeing any funny characters?
From: Bill D
Date: 21 Feb 10 - 05:44 PM

wow... more information than I can shake a stick at! (old Kansas saying)

(I don't think I wll download all 4000+ pages)

I am absorbing the data on what I CAN do, and not trying to deal with the technical details of why I cannot for some.

I might actually make myself a list/chart of the characters that work in 'most' situations and save it as a local file which keep in my tabs for quick reference. I alread have some pages from sites copied, but as we see, they are a mixed bag of good & bad.

As usual, thanks for the wealth of information, John.

Subject: RE: Tech: Seeing any funny characters?
From: Q (Frank Staplin)
Date: 21 Feb 10 - 05:47 PM

€
€

Sorry, should go to practice for this.

Subject: RE: Tech: Seeing any funny characters?
From: Janie
Date: 21 Feb 10 - 05:51 PM

I am sooooooooo confused by the posts to this thread.

Subject: RE: Tech: Seeing any funny characters?
From: Bill D
Date: 21 Feb 10 - 06:00 PM

Why, it's easy, Janie...you just.... ummm... poosha lotsa buttons and remember if anything works!

Subject: RE: Tech: Seeing any funny characters?
From: Q (Frank Staplin)
Date: 21 Feb 10 - 06:13 PM

Sometimes pooshing da buttons no work.
A while back, I could make single-spaced reports using Word. Now everything comes out double-spaced although I poosha da single space check mark option.

Subject: RE: Tech: Seeing any funny characters?
From: Artful Codger
Date: 21 Feb 10 - 06:38 PM

The Unicode standard for Euro is hex 20AC. Don't encode the Euro as decimal 128 (hex 80). Even Microsoft uses 20AC when pasting to a Unicode-aware destination--the 128 mapping is only used for 8-bit locale-specific text files (such as those edited by Notepad), where 20AC is outside the representable range of a single byte.

FFFE is reserved because it is one of the possible permutations of the Byte Order Mark (BOM), whose value is U+FEFF. This mark is often added at the start of a file so that software can automatically detect the type of Unicode encoding (UTF-7, UTF-8, UTF-16, UTF-32, BOCU-1...) and byte ordering within the file; the byte ordering may be the reverse of that natively used on your system or by a particular protocol.

U+FEFF is technically a Zero-Width No-Break Space (ZWNBSP), but this is deemed to be nonsensical to place as the first character in a file, since its purpose is to serve as a non-visual separator. I suppose they could map U+FFFE to some other special character or control marker which shouldn't occur at the start of a file, but they haven't.

Subject: RE: Tech: Seeing any funny characters?
From: Bill D
Date: 21 Feb 10 - 06:52 PM

Q.. my wife uses Word, but I never do.... if I need a program with lots of options, I have two free ones which will handle .doc files.

AbiWord is one, and it will do single spacing, double...or 1½ by clicking a box in the toolbar....and it never argues with me!

Subject: RE: Tech: Seeing any funny characters?
From: JohnInKansas
Date: 21 Feb 10 - 07:42 PM

A while back, I could make single-spaced reports using Word. Now everything comes out double-spaced although ...

If you're using Word 2007, my experience has been that lots of stuff on the menu bars don't do much of anything useful, and even less often do what anyone but an idiot would expect.

Since Office 2007 obliterated all the useful menus, it's simpler just to use the "quick keys" (which are hidden in 2007 but easy to find in earlier versions). The quick keys still mostly work in all versions:

Hit Alt-o, P

The Alt o gets a Format menu which is not visible in Word 2007 but should pop up in earlier versions, P tells it you want the Paragraph format menu, and when you hit the P in Word 2007 you should see a menu.

On the "Indents and Spacing" tab you should find a box where you can select single or double (or other) line spacings. If you select "Exactly" in the box, you can use the roller arrows to put a specific number of "points" of separation between lines, or just type a number in the box. This might be thought of as getting "single spaced lines where the lines actually are a different distance apart."

You also will find a place to set "space before" and "space after." This lets you have single spaced lines but paragraphs separated by some different spacing. Set both of those to zero if you really want everything all cramped together.

There are additional settings in Word's Tools|Options or Tools|Customize menus (old fashioned versions where there are lots of useful things) - which in Word 2007 is at COWSPLAT "Word Options" where there are lots fewer things you can do.

John