Functional components of Netlib

This document attempts to break down the current netlib functionality into smaller more maintainable libraries. The current Netlib (circa 7/96) can be broken down into the following parts:

See also: proposed changes to the URL struct

URL parsing library

URL_Type, MakeAbsoluteURL, SpaceToPlus, EscapeURL, etc.

URL Loader Library

Net_GetURL, ProcessNet, InterruptWindow, etc.

Protocol library

Low level HTTP, FTP, etc. + mime header parsing

stream registration and setup library

StreamBuilder, RegisterContentTypeConverter, etc.

file extension to MIME type mapper

the cinfo stuff borrowed from the server team

Cache library

Disk and memory cache code

Random stream converters

view source converter, plaintext converter, etc.

Random XP Utilities

Help file parser, ScanForURL's, EscapeHTML

Stuff that belongs in the XFE

Mailcap parser and UNIX external viewer code

Stuff that belongs in MSGlib

Newsgroup handling code

Below is each of the libraries and a list of all the current Netlib PUBLIC API's sorted under the new library headings

(note that this list will have to be extended to export previously MODULE_PUBLIC api's that were shared only within the Netlib)

URL Parsing Library Public API

All functions are prepended by "URL_"

• ParseURL (const char *url, int wanted)
• UnEscape (char * str);
• UnEscapeCnt (char * str)
• IsLocalFileURL(char *address)
• MakeAbsoluteURL(char * absolute_url, char * relative_url)
• Escape (const char * str, int mask)
• EscapedSize (char, int)
• SpaceToPlus(char)
• PlusToSpace(char *str)
• URL_Type(const char *URL)
• MakeRelativeURL( char *base_url, char *absolute_url, char **relative_url );
• MakeTargetURL( char *base_url, char *src_url, char **relative_url )
• URL_AddExternalURLType(char * type);
• URL_DelExternalURLType(char * type);
• URL_CheckForExternalURLType(char * url);
• URL_IsHTTP_URL(char *URL)
• URL_IsURLSecure(char * address);

URL Lib dependancies

• nspr

URL Struct Library Public API

This library will probably get folded into the URL loader library if it doesn't get larger.

All functions are prepended by "URLS_"

• NET_AddCoordinatesToURLStruct(URL_Struct * url_struct, int32 x_coord, int32 y_coord)
• AddLOSubmitDataToURLStruct(LO_FormSubmitData * sub_data, URL_Struct * url_struct)
• CreateURLStruct (const char *url, NET_ReloadMethod force_reload)
• MakeUploadURL( char **full_location, char *location, char *user_name, char *password )
• ParseUploadURL( char *full_location, char **location, char **user_name, char **password );

URLS Lib dependancies

• nspr
• URL struct

URL Loader Library Public API

All functions are prepended by "LOADURL_"

• GetURL (URL_Loader_Struct * URL_s, present_struct *output_format, NetContext * net_context, void* stream_closure, int32 group_id, Net_GetUrlExitFunc* exit_routine)
• The URL_Loader Struct is defined at the bottom of this document
• the present_struct is a structure that replaces the current pesentation type enum
• The NetContext is a structure that contains function pointers to just those callbacks necessary for netlib. The netcontext pointer will contain a group parent pointer that can used as a grouping ID and as a key for interrupting loads. The Netcontext also contains a pointer to the exoit routine
• stream_closure is an opaque structure pointer that will get passed into stream creation functions. It can be used to pass a display context to layout for instance.
• ProcessNet(void)
• event notification will be based on a condition variable that will be set TRUE whenever the netlib has data pending in one of it's thread queues.
• InterruptWindow(NetContext *)
• SilentInterruptWindow(NetContext *)
• InterruptStream (URL_Struct *nurl)
• InterruptSocket (NETSOCK socket)
• This one may be difficult to do, so if it isn't needed any more it will be deprecated
• AreThereActiveConnectionsForWindow(NETContext)
• HasNetworkActivity(NetContext *, Bool waiting, Bool background)
• InitNetLib(int socket_buffer_size, int max_number_of_connections)
• ShutdownNetLib(void)
• ChangeMaxNumberOfConnections(int max_number_of_connections)
• ChangeMaxNumberOfConnectionsPerContext(int max_number_of_connections)
• ChangeSocketBufferSize (int size)j
• SetCacheUseMethod(CacheUseEnum method)
• CheckForTimeBomb(????)
• PrintNetlibStatus(void)
• AddExternalURLType(char * type)
• DelExternalURLType(char * type)
• SetNewContext(URL_Loader_Struct *URL_s, NetContext* new_context, void * stream_closure, Net_GetUrlExitFunc *exit_routine)
• IsSafeForNewContext(URL_Loader_Struct *URL_s)

Load URL lib dependancies

• nspr
• url lib
• proto lib
• cache lib
• stream lib

Protocol Library Public API

This library will likely be broken down further into one library per protocol.

Each protocol will have it's own directory and will be modeled after the JAVA URL Connection API.

All functions are prepended by "PROTO_"

• WarnOnMailtoPost(PRBool warn)
• SendEmailAddressAsFTPPassword(Bool do_it)
• UsePASV(Bool do_it)
• SetPopPassword(const char *password)
• GetPopUsername(void)
• SetPopUsername(const char *username)
• GetPopMsgSizeLimit(void)
• LeavePopMailOnServer(Bool do_it)
• GetProxyConfigSource(void)
• SetNoProxyFailover(void)
• SetProxyServer(NET_ProxyType type, const char * host_port)
• SelectProxyStyle(NET_ProxyStyle style)
• ReloadProxyConfig(MWContext *window_id)
• SetNewsHost(const char * hostname)
• SetSocksHost(char * host)
• SetTCPConnectTimeout(uint32 seconds)
• ProxyAutoConfig(int fmt, void *data_obj, URL_Struct *URL_s, MWContext *w);
• SetMailRelayHost(char * host)
• SetIdentifyMeType(IdentifyMeEnum method)
• SetCookiePrefs(NET_CookiePrefsEnum pref_enum)
• SaveCookies(char *filename)
• ToggleTrace(void)
• SetNumberOfNewsArticlesInListing(int32 number)
• SetCacheXOVER(XP_Bool value)
• ParseMimeHeader(MWContext *context, URL_Struct *URL_s, char *name, char *value)

PROTO dependancies

• nspr
• ssl
• url lib
• mime type lib

These dependancies need to be cleaned up

• msglib
• fe's

Stream Library Public API

All functions are prepended by "NETSTREAM_"

• StreamBuilder ( FO_Present_Types format_out, URL_Struct * anchor, MWContext * window_id);
• NewStream(char *, MKStreamWriteFunc, MKStreamCompleteFunc, MKStreamAbortFunc, MKStreamWriteReadyFunc, void *, MWContext *)
• RegisterContentTypeConverter (char * format_in, FO_Present_Types format_out, void * data_obj, NET_Converter * converter_func)
• RegisterEncodingConverter (char * encoding_in, void * data_obj, NET_Converter * converter_func)
• ClearAllConverters(void)

Stream lib dependancies

• nspr
• url lib

MIME type Library Public API

All functions are prepended by "MIMETYPE_"

• cinfo_MasterListPointer(void)
• NET_cinfo_find_type(char *uri)
• cinfo_find_info_by_type(char *mime_type)
• cdataRemove(NET_cdataStruct *cd)
• cdataCreate(void)
• cdataCommit(char * mimeType, char * cdataString)
• cinfo_find_ext(char *mime_type)
• IsOldMimeTypes (XP_List *masterList)
• PrintFileType(int special_type)
• InitFileFormatTypes(char * personal_file, char *global_file)
• cdataAdd(NET_cdataStruct *cd)
• cdataExist(NET_cdataStruct *old_cd )
• cdataFree(NET_cdataStruct *cd)
• CleanupFileFormat(char *filename)

mime type mapping library dependancies

• nspr

Cache Library Public API

All functions are prepended by "CACHE_"

• CacheConverter (FO_Present_Types format_out, void *converter_obj, URL_Struct *URL_s, MWContext *window_id)
• DontDiskCacheSSL(XP_Bool set)
• CleanupCacheDirectory(char * dir_name, const char * prefix)
• ChangeCacheFileLock(URL_Struct *URL_s, Bool set)
• IsURLInDiskCache(URL_Struct *URL_s)
• IsURLInMemCache(URL_Struct *URL_s)
• WriteCacheFAT(char *filename, Bool final_call)
• DestroyCacheDirectory(char * dir_name, char * prefix)
• SetMemoryCacheSize(int32 new_size)
• SetDiskCacheSize(int32 new_size)
• GetMemoryCacheSize(void)
• GetMaxMemoryCacheSize(void)
• GetDiskCacheSize(void)
• NumberOfFilesInDiskCache()
• RemoveLastMemoryCacheObject(void)
• RemoveLastDiskCacheObject(void)

cache lib dependancies

• nspr
• url lib
• stream lib

Client Stream Converter Library Public API

All functions are prepended by "CSCL_"

• HTTPIndexFormatToHTMLConverter(int format_out, void *data_object, URL_Struct *URL_s, MWContext *window_id);
• HTTPIndexParserInit(void);
• HTTPIndexParserPut(HTTPIndexParserData *obj, char *data, int32 len)
• HTTPIndexParserSort(HTTPIndexParserData *data_obj, int sort_method)
• HTTPIndexParserGetTotalFiles(HTTPIndexParserData *data_obj)
• HTTPIndexParserGetFileNum(HTTPIndexParserData *data_obj, int32 num)
• HTTPIndexParserGetTextMessage(HTTPIndexParserData *data_obj)
• HTTPIndexParserGetHTMLMessage(HTTPIndexParserData *data_obj)
• ColorHTMLStream (int format_out, void *data_obj, URL_Struct *URL_s, MWContext *window_id)
• PlainTextConverter (FO_Present_Types format_out, void *data_obj, URL_Struct *URL_s, MWContext *window_id);
• CreateMochaConverter(FO_Present_Types format_out, void *data_object, URL_Struct *URL_struct, MWContext *window_id)

client stream converter lib dependancies

• nspr
• url lib
• stream lib

XP Utility Library Public API

All functions are prepended by "XPUL_"

• RegisterConverters(char * personal_file, char * global_file)
• GetHTMLHelpFileFromMapFile
• GetUniqueIdString(void)
• RegisterMIMEDecoders (void)
• ScanForURLS(MWContext *context, char * input, char *output, int output_size, XP_Bool urls_only_p)
• EscapeHTML(const char * string)
• PublishDirectory(MWContext *context, char *local_directory, char *remote_directory)
• PublishFiles(MWContext *context, char **files_to_publish, char *remote_directory)
• AssembleAllFilesInDirectory(MWContext *context, char * local_dir_name)
• UploadDirectory(MWContext *context, char *local_dir, char *remote_dir_url)

misc XP util dependancies

• nspr
• url lib
• stream lib

Functions that should be moved into the XFE

• mailcap_MasterListPointer(void);
• NET_mdataCreate(void);
• NET_mdataAdd(NET_mdataStruct *md);
• NET_mdataRemove(NET_mdataStruct *md);
• NET_mdataFree(NET_mdataStruct *md);
• NET_mdataStruct* NET_mdataExist(NET_mdataStruct *old_md );
• CleanupMailCapList(char* filename)
• RegisterExternalViewerCommand(char * format_in, char * system_command, unsigned int stream_block_size)
• RegisterExternalDecoderCommand (char * format_in, char * format_out, char * system_command)
• ProcessMailcapFile (char *file, Bool is_local)
• ClearExternalVieweraConverters(void)

Functions that should be moved into the Message Lib

• RegisterNewsrcFile(char *filename, char *hostname, Bool is_secure, Bool is_newsgroups_file)
• UnregisterNewsHost(const char *host, Bool is_secure)
• MapNewsrcHostToFilename(char *host, Bool is_secure, Bool is_newsgroups_file);
• SaveNewsrcFileMappings(void)
• ReadNewsrcFileMappings(void)
• FreeNewsrcFileMappings(void)

Changes to the URL struct

The URL struct can be broken down into two parts.

One part describes what is necessary to load the URL:

URL_Load_Struct
->address
->method
->post_data
->etc...

Another part describes the object after it is loaded:

URL_Desc_Struct
->content_type
->content_length
->last_modified
->etc...

The URL struct can then be expressed as a combination of the URL_Load_Struct and the URL_Desc_Struct.

Comments?
montulli@netscape.com