momoi: When we store preferences do we need to store them as unicode? urgh; sorry i'm late good morning slept right through my alarm LDAP uses UTF-8 as the data code. In Comm 4.5 onward, we have used UTF-8 as the store code as well. It's simple that way. No change to the data received from the server. Easily storable as LDIF file which needs to be in UTF-8 also. momoi: i was talking to phil about this a bit, and he said that problem that they ran across is that LDAP v2 was defined to use "T.61" and that because very few people actually know what T.61 was, they just put Latin into their LDAP server which is unfortunately not a subset of UTF8 For preferences (i.e. prefs.js), it is already in UTF-8 for other data like e-mail user names. momoi: ah, good Right. But in Comm 4, we dealt with it by having a prefs.js item which could set a folder charset to be T.61. Normally it defaulted to UTF-8 for any LDAP servers but the user could designate that folder as T.61 in which case, we assume that server tobe sending T.61 data. what do you mean by a "folder" charset? Wel, in Address Book, each addressbook folder was assumed to have a charset associated with it. So the main address book folder, pab, would have a charset of the operating system, e.g. Shift_JIS for Japanese on a Japanese Windows but ISO-8859-1 if the opearting system was US Windows. For LDAP folders, we assumed it to be in UTF-8 no matter what the operating system was. In other words, each LDAP entry in prefs.js would have a line indicating that is in UTF-8. ok Just look in a 4.7x prefs.js and look for LDAP entries. You will find charset names like UTF-8 associated with it, I think. srilatha: so in our case, it sounds like it would make sense to use the preference that csaba was suggesting a week or two ago This is the csid preference srilatha: right. ie assume UTF8, unless that pref is set to something else. yes, but csaba was saying with -i option the user can set the language for input srilatha: that was just an example, it's how a client called "ldapsearch" works. I wonder hw important it is to support T.61 at ths point. We implemented T.61 for 4.5 but were never able to test that implementation because the customer in Canad never got back to us with the actual working server which had T.61 data. momoi: is T.61 some superset of ASCII that is a different superset than iso-8859-1 and a different superset than UTF8? or am i confused? In any case, if we need T.61, we should be able to re-use T.61 conversion table from 4.7x. I forgot the details but T.61 is somewhat like Latin 1. I have an ISO document which defines T.61. In any case, if we need T.61, we should file a request to the i18n group for a new conversion table to/from Unicode. Frank Tang put in that table for 4.5, I beleive, yeah, i wish we had more data on what people are actually using in their servers we should probably bug mitch green about that how does one put in a request to the i18n group? Just a feature request for an additional Unicode converter for T.61 in the bug report stating the need for it. ah, ok I would still look hard for any customer needing T.61, though. For 4.5, there was a stink about it and we implemented it but no customer ever got back to us sayign they use T.61. * dmose chuckles By the way, back to prefs.js, there are cases like Search Domain names in LDAP that might be using non-ASCI names. In fact we have an i18n LDAP server running now which has a Search Root name in Japanese. That will have to be stored in UTF-8 in the prefs.js. momoi: but it gets specified in a XUL text field, which means we get it in unicode, right? so, csid for this server would be shift jis What server? srilatha: well, depends if the server in question is running LDAPv2 or v3 srilatha: if v3, it would be UTF8 right As I recall LDAPv2 supported only Latin1 and there was a special case like T.61. I know that Netscape LDAP group never claimed to support Japanese or other non-ASCII languages until LDAP v3 was introduced. I think that it the right approahc for Mozilla also. It is futile to try to support anything other than ASCII or Latin 1 with LDAPv2. alright, that sounds like a fine plan so do we just assume Latin-1 since it's a superset of ASCII and skip the charset id preference? I don't know if the 4.x approach is the right one for Mozilla. Do we need to associate a charset ID for adbook dmose: I think we should. Remember, we're always gonna try using LDAP v3 first, so in a majority of the cases, UTF-8 is used and supported. folders in Mozilla? In 4.x we were not processing Unicode internally and so we needed to associate casid for folders. Considering the time crunch we have to deliver this, I'd punt on any kind of special feature which is for LDAP v2 only. ok, sounds like the right thing to me momoi: so XUL text fields will generate unicode, right? I think we might be able to do without for Mozilla since everything is in Unicode anyway. The only thing is that LDAP server data need to be assumed to be in UTF-8. I mentioned folder CSID only as a point of reference for 4.7x. I would talk to nhotta@netscape.com about Mozilla csid issue. momoi: ie we just need to convert from unicode to UTF8 before writing out to prefs.js? momoi: I know almost nothing about Unicode and UTF-8. But, assuming an LDAP server is v2, and uses regular ascii (or maybe latin1), what happens if we assume it's encoded in UTF-8 ? bad things if it's latin-1, accordding to phil\ Nothing bad. It will show ASCII because it is a transparent subset of ASCII. But any Latin 1 data in eth 8-bit range would be different between Latin 1 and UTF-8. that's what i meant latin-1 data wouldn't work, and it sounded as though there was some non-trivial number of folks using that at least at that time Would you not negotiate the LDAP version first anyway? yes So that you know not to assume UTF-8 unless it is v3 and up? yeah, that sounds like the right thing Do you know if the Mozilla AdBook pab currently stores in UTF-8 or in Unicode (which would be UCS-2 or UTF016)? afk for a minute not sure. we're actually focussing almost entirely on getting autocomplete to work right now, outside of the addressbook framework repeating dmose's question. Before writing out to prefs.js do we need to convert from unicode to UTF8? Yes if the data is not in UTF-8. prefs.js i sassumed to be in UTF-8 if it contains any 8-bit data. If the original data is not in UTF-8 but something like UCS-2 or UTF-16 used in the Mozilla layout. are UCS-2 and UTF-16 the same thing? They are sometimes used synonymously but technically different. I understand we use UTF-16 internally. but it's referred to as UCS2 sometimes in the APIs? I don't know for sure. But UTF-16 can cover what is known as the surrogate range of Unicode, which is what makes UTF-16 cover a bigger range of characters than UCS-2. But the surrogate range is rarely used and so sometimes, we loosely talk of UTF_16 an dUCS-2 in the same breath. ok, fair enough are URIs themselves assumed to be encoded in UTF-8? Basically UTF-16 allows 2 16 but sequences to desigante a single character but UCS-2 is always fixed 16-bit per one character. That is is the differebce. ah, got it As I recall, in 4.x if we used ldap:// protocol, we got back the results in escaped UTF-8 if the ldap url contained 8-bit UTF-8 characters. "escaped"? %A4%90 type of URL That is a single 2-byte character %-escaped. i'm confused. got it back from where? From an ldap server. Here is an example: ldap://polyglot.mcom.com:8000/uid%3Dnsasaki%2Co%3D%E3%83%8D%E3%83%83%E3%83%88%E3%82%B9%E3%82%B1%E3%83%BC%E3%83%97 but servers themselves don't hand back URIs This is my LDAP server and I just brough up an Japanese entry name and dispalyed it in the Browser window rather than in AdBook display area. The URL is what I got in the Browser location window. ok, so we should UTF8 escape any LDAP URL, it sounds like leif: maybe check to see what the LDAP URL RFC says about that one Anyway that was teh implementation in 4.x. ok, cool. > but servers themselves don't hand back URIs dmose: I'll check the rfc. Guys, I have to drive in to work now (I commute with Michelle). leif: ok say, is everyone here cool with us posting a log of this chat on www.mozilla.org I guess you're right. We are probably sending that to the server to access data if Search Roto contained 8-bit characters, which is what the above examples shows. dmose:yes sure, please post it. I would like to have an opportnity to correct factual errors if any. Do you mind? momoi: no problem; i'll forward it along first so a question about how content communicates charset encoding to layout Can you send me a log later -- I will look at it later today. sure i've got a channel, which writes its data down an AsyncStream through which it ultimately ends up in layout but it just treats the data being written as bytes any idea how i can either As I understand it, the content is always expecting UTF-16. but sending the content down in it's raw form (UTF8) seems to work fine and i would expect to see alternating skipped chars if layout were expecting UTF16 --> momoi1 (momoi@adsl-216-103-212-1.dsl.snfc21.pacbell.net) has joined #addressbook I got disconnected, sorry. ah, ok. i'll repaste my last comments but sending the content down in it's raw form (UTF8) seems to work fine and i would expect to see alternating skipped chars if layout were expecting UTF16 Hmm, I don't know if we are using anything other than UTF-16 in content. The onyl excpetion I know is Mail body display, which sends UTF-8 in HTML format but in that case we specify the UTF-8 charset in the HTML document in the form of HTTP Meta-equivalent charset tag. Plesase check with nhotta about this. ok, i'll do that, thanks <-- momoi has quit (Ping timeout: 361 seconds) i think that's all the questions I've got for the moment Is this it then? srilatha, do you know of any other info that kat might be able to help us with? Can i ask you about lcalization questions too? If so, please send the info via mail to momoi. I'll talk to nhotta and make sure that the questions get answered. Sure. abotu L10n. I'll be expecting mail, then. The strings that need to be translated have tobe in the properties file right? Yes. They need to be extracted out to .dtd files or .properties files. These .dtd and .properties files will be placed in locale directories. ok, how about the strings that we save in prefs.js like ldap_2.servername.description So if you add AdBook UI, make sure that all localizable strings are extracted out. ok Those are profile specific items. If so, they don't have to be extracted out unless some default settings in all.js or mailnews.js contain localizable strings, then they need to be extracted out. Actually we are not addinga anything to the adbook ui but will be adding to to mail/news accout settings srilatha: i assume that we could just save the description as unicode Well, in that case, you do need to add these strings to Account UI dialog .dtd files. > i assume that we could just save the description as unicode If the server name is input as Japanese, then it should written out in UTF-8 into prefs.js. oh, because otherwise the pref string might have nulls in it? I guess I'm using Unicode to mean UCS-2 or UTF-16. UTF-8 is a transfer format of Unicode and used in data tarnsmisison and storage. I want to know if the string"ldap_2.servers.servername.description" has to be localized or not. i am not talking about the value of hte preference momoi1: ah, ok So any time we write out to storable or network transferrable output, it is safe to assume that we would be using UTF-8. OK. I don't think. That is a place holder for a server and would not have to be exposed to the user, right? right. seems like localizing it would be overkill, since we're not supporting users who edit prefs.js by hand. We never did for 4.7x. ok I need to get going soon. Thanks momoi. yes, you've been a great help i'll mail you the log in a bit Thanks everyone for finally gettting to LDAP work!! :-) Bye. see ya