snapshot of project "lynx", label v2-8-5dev_17

3 files changed, 549 insertions, 7 deletions

diff --git a/docs/README.defines b/docs/README.defines
index 5b9cc92d..801a2533 100644
--- a/docs/README.defines
+++ b/docs/README.defines
@@ -88,6 +88,7 @@ Using curses or ncurses library:
                    for UNIX handled by ./configure, may be set in HTUtils.h
 # -DCOLOR_CURSES    whether curses lib supports color (in the right way).
                    for UNIX handled by ./configure
+# -DUSE_COLOR_STYLE (see INSTALLATION, Section II-1c) needs SVr4 curses
 # -DUNDERLINE_LINKS  Define if you want links underlined instead of bold.
 # -DHP_TERMINAL    For DIM workaround to REVERSE problems on HP terminals.
 # -DNO_SIZECHANGE (old) or HAVE_SIZECHANGE (new) whether using code to detect
@@ -104,18 +105,15 @@ Both:
 # 	if you have the reverse clear screen problem of some SUN4 systems, or
 # 	to work around some other "garbage on the screen" problems
 
-Related to experimental (unsupported) character style code
-------------------------------------------------------------
-# -DUSE_COLOR_STYLE (see INSTALLATION, Section II-1c) needs SVr4 curses
-
 Other general
 -------------
+# -DEXP_ASCII_CTYPES use POSIX locale for toupper/tolower/isupper
 # -DIGNORE_CTRL_C  Define if Control-C shouldn't exit lynx.
+# -DINACTIVE_INPUT_STYLE_VH  Different style for showing inactive text fields
+# -DMISC_EXP       Enable some truly experimental stuff
 # -DNO_DUMP_WITH_BACKSPACES Disables -with_backspaces code
-# -DUSE_VASPRINT   Use vasprintf() function
 # -DNO_RULES	   Disables code for CERN-style rules files
-# -DMISC_EXP       Enable some truly experimental stuff
-# -DINACTIVE_INPUT_STYLE_VH  Different style for showing inactive text fields
+# -DUSE_VASPRINT   Use vasprintf() function
 
 Untested
 --------
diff --git a/docs/README.rootcerts b/docs/README.rootcerts
new file mode 100644
index 00000000..fb294d11
--- /dev/null
+++ b/docs/README.rootcerts
@@ -0,0 +1,308 @@
+	DOS/Windows-oriented notes on Root Certificates
+
+To use certificates or a cert bundle within an SSL enabled
+application such as lynx you must place your certificate
+files into a known directory, and set the environment
+variables to a proper value (e.g. in CONFIG.SYS file).
+
+ set SSL_CERT_DIR=x:/usr/local/ssl/certs
+ set SSL_CERT_FILE=x:/usr/local/ssl/cert.pem
+
+(See "What are root certificates" below.)
+
+
+Q.  Why would I want to install openssl.exe?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+openssl.exe is used to manage certificates.  (See "What are root certificates"
+below.)
+
+Q.  How to install openssl.exe?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Put openssl.exe in a directory in your PATH and the DLLs to a directory
+in your LIBPATH.
+
+Copy conf\openssl.cnf.demoCA to a directory of your
+choice, rename it to openssl.conf and set the environment variable
+OPENSSL_CONF by putting
+
+SET OPENSSL_CONF=<your-directory>\openssl.cnf
+
+into CONFIG.SYS.
+
+
+Q. Why is this document so paranoid?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to use OpenSSL, then probably your Internet transactions have
+*real* monetary value embedded in them.  And as usual, the security is as good
+as the weakest link.  This document unravels only the tip of the iceberg
+of what can go wrong with improperly established "secure" connections.  And
+given the monetary value involved, "bad guys" have a high incentive to exploit
+the weakest links.  As experience shows, do not underestimate the intelligence
+of bad guys...
+
+Really, with security, a little knowledge is a dangerous thing; one can
+suspect that many people, if they really understood the trust structures
+associated with SSL, would be rather careful about checking the details
+of certificates.
+
+Q. What are root certificates?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Making a secure connection is like sending your valuables (for storage or
+consumption) to somebody who agreed to be at a prearranged place.  To
+guard the valuables on the way there, you can ask for a police escort; this is
+what https:// connections are about.  However, it does not make any sense to
+have an escort if the goods are transfered to a random person who happens to be
+at this place; one needs to certify the identity of the receiver as well.
+
+The certification process is a chain; when site A wants to certify that it is
+actually what it claims, it actually says "Check this certificate with site B";
+to proceed, one needs to certify that site B is what it claims, so B may
+redirect to site C etc.  For this process to stop, some sites claim
+"You must know my certificate, check it yourself".  These certificates are
+"root certificates"; one cannot verify such a site unless one has the
+certificate for the "end of its certification chain".  If you don't have the
+relevant root certificate in your local certificates file, it means that
+you don't trust anyone to vouch for the authenticity of the site.
+
+So one should have a collection of known certificates from several well-known
+sites known as "Root Certification Authorities".  Most sites for large-scale
+businesses have certificates which will eventually resolve to these places.
+Such certicates represent people like Verisign that are in the business of
+confirming the identity of servers, etc.
+
+Additionally, since having yourself certified through another site costs,
+some sites avoid this cost via presenting "end-of-chain certificates".
+One should have a way to obtain these certificates via other means than
+insecure Internet connection (e.g., one can walk into the office and copy
+the certificate file to a floppy).  These are so-called "Self-signed
+certificates"; they are "root certificates" as well.  The locally-installed
+securely obtained copies of such certificates are referred to as
+"local certificates".  (See 'What is "Snake Oil Ltd."' below.)
+
+If you are presented with a locally-unresolvable root certificate, and you
+*believe* that you are really talking to the site, and not someone
+in between (who is either completely simulating the site or relaying
+your requests onto the real site - called a "man in the middle" attack),
+you will still have an encrypted connection.  Otherwise, you should act
+as though the site was an impostor, unless and until you manage to get
+a root certificate from a trustworthy source, and that root certificate
+represents someone that you would trust to have vetted the site you
+want to connect to.
+
+Local certificates are stored in SSL_CERT_FILE (this "cert bundle", usually
+named cert.pem, contains several signatures for "Root Certification
+Authorities") and SSL_CERT_DIR (which has a signature per file, and usually
+contain local copies of self-signed certificates).
+
+There are three crucial considerations to be added to this picture:
+
+  a) While there are ways to ensure that the receivers are who they claim,
+     there is absolutely no technological way to verify how *trustworthy*
+     the receiving party is.  It does not make sense to secure-send your
+     valuables to a certified receiver if this receiver is a crook (or will
+     just keep them later in a publicly accessible place).
+
+  b) "VeriSign Syndrome".  For the above scheme of "a chain of trust" to work,
+     the "Root Certification Authorities" should be *very* trustworthy
+     high-integrity entities.  Unfortunately, there are certain doubts that
+     this is so.  E.g., fall 2003, VeriSign started an attack on DNS scheme
+     which could disrupt the whole architecture of Internet (hijacking *all*
+     unclaimed Internet addresses and redirecting them to a promotional site;
+     google for VeriSign DNS hijack).
+
+     One major company even issued a Microsoft certificate to a company
+     other than Microsoft, and there had to be a Windows critical update
+     to block that certificate.
+
+  c) Keep in mind that the "big 2 browsers" are adding an increasing
+     number of root certificates, and most users fail to realise that they
+     are putting a trust in the supply chain for the browser to give them
+     the certificates of reliable organisations (the browser suppliers could
+     make bad choices, or the browser could have been hacked before you got
+     it).
+
+     Incidentally, standard browsers come with certificates representing
+     very different levels of identity verification, but most people accept
+     all of those supplied with the big 2 as equally valid.
+
+Q. How to obtain root certificates?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Certificate files, such as cert.pem, are security critical; you have to
+trust whoever supplies it to you; all your certification process is no more
+trustworthy than the site you downloaded cert.pem from.  So you shouldn't just
+accept any offer.
+
+One way is to copy them from a machine which already obtained them in a secure
+way.  Another one is to extract them from a web browser which was itself
+obtained in a secure way (see "How to extract certificates from Internet
+Explorer" below).  If anything else fails, obtaining a privately-generated
+bundle from third-parties, such as
+
+  http://www.kfu.com/~nsayer/encryption/ca-bundle.crt.text
+
+is *not* much better than no certificates at all, but may avoid some warnings
+from applications.  One of the places which has a bundle is the mod_ssl site.
+
+Q. Should you trust this distribution system?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It is very hard to imagine a situation when the answer is different from
+"Absolutely not!".
+
+Indeed, obtaining the certificates is only half of the problem.
+The certificates are going to be checked by the SSL library.  Can you trust
+these executables (DLLs)?  Did you obtain the library via a secure connection?
+Are you sure that the place you obtained it from has reasonable security
+practice, so that the archive could not be tampered with?  The latter place
+most probably did not build the DLLs themselves; chances are they just
+store what a fourth-party supplied them.  Was *that* file transfer done via
+secure channels?  Can you trust this fourth-party so that it did not insert
+Trojans?
+
+Chances are that all of these questions are answered "No".  There are still
+major problems with bootstrapping security via the Internet...
+
+What about the application which uses these DLLs?  Do you have any reason to
+trust it?  What about the OS itself?  Did it come from a trustworthy source
+via trustworthy channels?  Are you sure it was not tampered with?
+
+Q.  How to compile and link with OpenSSL libraries?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Put the files from include and lib to your emx directory,
+or directories on C_INCLUDE_PATH and LIBRARY_PATH.
+Note that openssl should become a subdirectory of your include directory.
+If you need .lib files you can create them using emxomf.
+
+The supplied library files link against the new renamed dlls open_ssl and
+cryptsll.
+
+See the doc directory for some information and visit
+http://www.columbia.edu/~ariel/ssleay/ for more infos.
+
+
+Q. Why do you need your own keys and certificates?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are several situations: having a server which accepts secure connections;
+authenticating yourself to a server by means other than login/password,
+sending S-Mime crypto-mail, authenticating from a client browser to a server.
+In each of these situations one needs keys.
+
+The following sites may be useful:
+
+   http://www.pseudonym.org/ssl/ssl_cook.html#environment
+   http://the.earth.li/~sgtatham/putty/0.53b/htmldoc/Chapter8.html#8.2
+
+Q. How to generate your own keys and certificates?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are many ways. A good solution is to use sslRexx. It provides everything
+you need. PuTTYgen is a key generator that will work.
+
+Below is a short description of how I made my own Certification Authority,
+a Server Key for Apache and a client Key/Certificate for me, signed by my
+own CA.
+
+
+Q. Howto: Root CA (needed to self-sign all certificates)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Generate a CA-Key and store it in sub-directory private:
+
+  openssl genrsa -des3  -out private/MyOwnCA.pem 2048
+
+Make a selfsigned certificate based on above key.
+
+  openssl req -new -x509 -days 730 -key private/CAkey.pem -out CAcert.pem
+
+This certificate will expire in 2 years.
+
+Optional: generate text output of this certificate:
+
+  openssl x509 -in ./CAcert.pem -text > CAcert.txt
+
+Now you have a key and certificate for your own CA which can be used
+to sign user and server keys. The CAcert is also needed to configure
+Apache and Netscape. You can/should give away the CA certificate but
+never give the CA key to anybody.
+
+
+Q. Howto: Your Client Certificate/Key
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Generate a private key
+----------------------
+
+  openssl genrsa -des3 -out hrom-key.pem 2048
+
+
+Create a signing request (same command again)
+------------------------
+
+  openssl req -new -key hrom-key.pem -out hrom-req.pem
+
+Let the CA sign it (same command again)
+------------------
+
+  openssl ca -in hrom-req.pem -out hrom-cert.pem -outdir MyOwnCA/newcerts
+
+After you get back the certificate from the CA, combine it with
+your private key and store the result as p12 file. This file can
+be imported into your browser.  The browser will use this file to present
+to a server requiring it for access.
+
+  openssl pkcs12 -export -name Hromadka -in hrom-cert.pem -inkey hrom-key.pem -out hrom.p12
+
+
+Security Notes: Never give your private key to a CA, they only need the
+signing request. Never give away your p12 file. Always secure your private
+keys with a passphrase.
+
+
+Q. How to use c_rehash?
+~~~~~~~~~~~~~~~~~~~~~~
+
+One needs a working port of Perl and cp.exe to run this.  Set OPENSSL to the
+full name of openssl executable.  One may also need to change some ':' to
+$Config{path_sep}.  c_rehash finds certs from enviroment variables and allows
+them to be recognized by openssl.
+
+Q. How to extract certificates from Internet Explorer?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To make your own file of certificates, go to the
+"Tools/Internet Options/Content/Certificates/Trusted Root Certificates"
+section of IE. Select all the certificates, then "export" to a file.
+It will be saved as a PKCS#7 file, with suffix ".p7b". You can call
+it "ca_bundle.p7b". Then use openssl to convert it with the command:
+"openssl pkcs7 -inform DER -in ca_bundle.p7b -print_certs -text -out cert.pem".
+Ask your system administrator to put the file "cert.pem" in the openssl
+directory and c_rehash it. Then lynx can check the certificates against the
+set of certificates that you (or Microsoft) trusts, and you won't get the
+warning message any more.
+
+Q. How to install a self-signed certificate?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When you would like to trust a self-signed (non-commercial) certificate you will
+need to get hold of the actual file. If it's a cert local to your network you
+can ask the sysadmin to make it available for download as a link on a webpage.
+
+If such file is not human-readable it's probably DER formatted and will need to
+be converted to PEM format to allow openssl to use it.
+
+To convert DER formatted certificates into something openssl can deal with:
+
+Save the cert as site_name.crt in a directory. In that directory, type:
+
+  openssl x509 -inform DER -in site_name.crt -outform PEM -out site_name.pem
+
+You can now copy this individual cert into the directory for that and hash the
+cert by running c_rehash. A complete discussion of this procedure for unix is
+in the document README.sslcerts.
diff --git a/docs/README.sslcerts b/docs/README.sslcerts
new file mode 100644
index 00000000..09de6277
--- /dev/null
+++ b/docs/README.sslcerts
@@ -0,0 +1,236 @@
+	Lynx SSL support for certificates - README.sslcerts file
+
+BACKGROUND:
+
+The original README.ssl document for lynx stated:
+
+	Note that the server... may not have a valid certificate. Lynx will not 
+	complain, as it does not yet support certificates...
+
+Such lack of support is no longer the case.  Lynx now features excellent 
+certificate management through the openssl project.  There is almost no 
+online documentation available regarding how to use openssl's certificate 
+management with other programs, so this will accompany lynx and hopefully 
+encourage good practical security for unix clients. 
+
+Lynx relies on openssl to not only encrypt connections over https, but also to
+determine whether it should even accept a certificate and establish a secure 
+connection with a remote host. Because of this reliance upon openssl by lynx, 
+most of this tutorial deals with how to use openssl to "install" both
+commercial CA cert bundles as well as self-signed certs from trusted sources 
+and, most importantly, how to get them recognized by lynx. 
+
+While lynx on many systems will transparently accept valid certificates, not 
+all systems enjoy such functionality. Further, as noted above, older versions 
+of lynx do not perform any validity checks on a certificate. 
+
+There is also the common case of wanting to trust, use and install a 
+self-signed certificate from a known server source and have it be trusted by 
+client programs.
+
+Briefly, the procedure will involve confirming the default system location for
+certificates, possibly setting values for SSL_CERT_DIR and SSL_CERT_FILE in 
+the environment, and converting and hashing the certificates using openssl 
+utilities to enable recognition.
+
+THE CURRENT SITUATION:
+
+Prior to lynx2.8.5dev9, lynx did not check at all for certificate validity.
+
+Since lynx2.8.5.dev9, lynx has reported this openssl error:
+
+SSL error:unable to get local issuer certificate-Continue? (y)
+
+whenever an https connection was initiated and the certificate could not be
+found,  for whatever reason, by openssl, and therefore lynx. 
+
+This checking for a certificate is an enhancement to security, but rather 
+tediously generates errors at each https browser request. 
+
+The ability to turn off reporting of this error to the user was added to 
+lynx2.8.5dev16 as the FORCE_SSL_PROMPT setting in lynx.cfg as noted in the
+CHANGELOG:
+
+	This lets the user decide whether to ignore prompting for questionable 
+	aspects of an SSL connection.  
+
+While this is a convenient setting to employ when using lynx to script 
+https -dumps, it by definition ignores the issue of certificate validity 
+altogether. Those concerned with proper certificate management and 
+the maintenance of a store of updated CA certificates will be uncomfortable 
+with this relaxed security setting.
+
+PRELIMINARY PROCEDURES:
+
+It is assumed that openssl has been installed correctly, that the default 
+cert directory is /usr/local/ssl/certs, (it's often /etc/ssl/certs, but we 
+need a point of departure for the discussion) and that lynx has been compiled 
+--with-ssl. 
+
+The default location for certs on your system may be different, or there may not
+be one. You will have to substitute that location for /usr/local/ssl/certs in 
+the following instructions, and/or set environment variables.
+
+To determine the default location for certs on your system you may run the 
+following command: 
+
+strings libcrypto.a  | grep -in cert | less
+
+Look in this output for SSL_CERT_DIR and SSL_CERT_FILE, and the lines just
+above them. This is your default location, respectively, for certificates, 
+and the CA cert bundle, cert.pem. You will need to know where libcrypto.a is 
+found of course.
+
+Example output:
+
+<snip>
+7490:/etc/ssl/certs
+7491:/etc/ssl/cert.pem
+7492:SSL_CERT_DIR
+7493:SSL_CERT_FILE
+<snip>
+
+Other possible example output:
+
+<snip>
+31555:/usr/local/ssl/certs
+31556:/usr/local/ssl/cert.pem
+31557:SSL_CERT_DIR
+31558:SSL_CERT_FILE
+<snip>
+
+Note that when OpenSSL is installed, the c_rehash utility is installed in a 
+binary directory (default /usr/local/ssl/bin).  You will need to know where it 
+is on your system. The command:
+
+whereis c_rehash 
+
+will probably give useful results.
+
+Note also that there is no CA cert bundle distributed with OpenSSL. The 
+OpenSSL team specifically decided NOT to do that. Getting a set of trusted 
+certificates is left up to the installer. 
+
+It is a fairly trivial procedure to pull the bundle of trusted root certs out 
+of a recent version of Internet Explorer. The mod_ssl project also provides 
+them.  The procedure to convert and install them is detailed later in this 
+document, and if you simply need to have commercially provided certificates 
+trusted by lynx, you can skip down a few lines to the INSTALLING OR UPDATING 
+THE CA BUNDLE section.
+
+INSTALLING A SELF-SIGNED CERTIFICATE:
+
+When you would like to trust a self-signed (non-commercial) certificate you will
+need to get hold of the actual file. If it's a cert local to your network you
+can ask the sysadmin to make it available for download as a link on a webpage.
+
+If such file is not human-readable it's probably DER formatted and will need to 
+be converted to PEM format to allow openssl to use it.
+
+To convert DER formatted certificates into something openssl can deal with:
+
+Save the cert as site_name.crt in a directory. In that directory, type:
+
+openssl x509 -inform DER -in site_name.crt -outform PEM -out site_name.pem
+
+You can now copy this individual cert into the directory for that, usually
+/usr/local/ssl/certs.  The alternative is to concatenate the individual certs 
+to the cert.pem bundle in /usr/local/ssl. (Please see INSTALLING OR UPDATING 
+THE CA BUNDLE below). 
+
+The cert file will now be in an acceptable format to openssl, PEM encoded. 
+However, openssl, and by extension lynx, will not know about it until that 
+cert is present in a file named after the hash value of that cert, in the 
+default directory /usr/local/ssl/certs.
+
+So the next thing to do is to hash the cert using c_rehash.
+
+INSTALLING OR UPDATING THE CA BUNDLE:
+
+Now would be a good time to check to see if you have the bundle of CA certs
+/usr/local/ssl/cert.pem, or to update them.
+
+CA bundles are available in various places, such as the modssl distribution, 
+for those who want to take that route, or you can extract the current bundle 
+from a current version of Netscape or Internet Explorer (export them all from 
+IE and transfer it onto your system). 
+
+From IE it extracts as a PKCS7 file and needs to be converted with something 
+like:
+
+openssl pkcs7 -inform DER -in bundle.crt -outform PEM -out cert.pem \ 
+-print_certs -text
+
+The resulting cert.pem file should be copied to the default directory for 
+bundles (usually /usr/local/ssl) and renamed to "cert.pem". It will also 
+process just fine if it is present and hashed in /usr/local/ssl/certs.
+
+We now have all of the individual certs we wish to trust in our certs 
+directory, and the most recent bundle of CA certs as well. 
+
+Confirm that you have the script c_rehash (See PRELIMINARY PROCEDURES; if it is 
+not found, a copy is usually located in the tools directory of the openssl 
+source tree. If you use this copy, it needs the execute bit set or it will not
+run).
+
+Run:
+
+./c_rehash
+
+The c_rehash utility is a perl script that runs openssl commands which creates 
+the files named after the hash values of the certs in the default directory 
+for certs. 
+
+Its output looks like this:
+
+Doing /usr/local/ssl/certs
+vsignss.pem => f73e89fd.0
+vsign3.pem => 7651b327.0
+...more output
+<snip>
+
+All pem encoded certs in /usr/local/ssl/certs will now be recognized.
+
+SETTING AND EXPORTING ENVIRONMENT VARIABLES:
+
+If lynx is still not recognizing certs, environment variables may need
+to be set; if on a sh type shell, the variables also need to be exported.
+
+The environment variables SSL_CERT_DIR and SSL_CERT_FILE only need to be set 
+if a non-default location is used for certificates, or if certs just can't be
+found by lynx. They may be set as follows in /etc/profile, or a shell 
+initialization .profile or .*shrc, if we run a non csh type shell, according 
+to the results of the search for the default location for certs procedure 
+(See PRELIMINARY PROCEDURES):
+
+SSL_CERT_DIR="/usr/local/ssl/certs"
+SSL_CERT_FILE="/usr/local/ssl/cert.pem"
+export SSL_CERT_DIR SSL_CERT_FILE
+
+On csh type shells, you can use:
+setenv SSL_CERT_DIR "/usr/local/ssl/certs"
+setenv SSL_CERT_FILE "/usr/local/ssl/cert.pem"
+
+On some systems setting and exporting them makes all the difference. Apparently 
+this is often not an issue on other systems, but this might help someone 
+somewhere.
+
+Note that the environment variable SSL_CERT_FILE applies to the cert-bundle 
+if used outside of the default location (/usr/local/ssl/cert.pem) compiled 
+into OpenSSL. There are issues with SSL_CERT_FILE in 0.9.6x versions of openssl.
+
+Make sure you have FORCE_SSL_PROMPT set to PROMPT in lynx.cfg like so:
+
+FORCE_SSL_PROMPT:PROMPT
+
+You will now connect without error to https servers with trusted certs, but
+will still get this error for untrusted certs:
+
+SSL error:self signed certificate-Continue? (y)
+
+A quick check confirms that these procedures have the same effect with ssl 
+errors in the pine program.
+
+--Stef Caunter <cauntes@mail.mohawkc.on.ca>
+Mohawk College Department of Computer Science
+Hamilton Ontario Canada