distcache-users — For users of distcache tools and programmers using distcache APIs.

[distcache-users] Fw: surprisingly interesting

[shal <sh...@fr...>]

Dear, 

Here is an  article that  I found by chance, but it  turned out so interesting that I decided to share  it with you, you can  find it here  <http://razybupu.sustainableplay.biz/e6aasnnr>

See you around, shal

Re: [distcache-users] memory leak in distcache 1.5.1

[Bryan D. <med...@us...>]

Attached is a patch to correct a memory leak in distcache 1.5.1
triggered when dc_server_default::cache_add_session() determines that it
must forcibly expire some cache entries to make room for the new session
data.

We encountered this situation in a production environment due to high
volume session creation with a moderate TTL and a very small cache.

The leak is fairly easy to recreate when running distcache under
valgrind. I setup a test environment with an apache server using
dc_client to talk to a dc_server instance running under valgrind
monitoring. The apache server was configured with a 5 minute TTL for
each session. The dc_server was configured with the default max 512
sessions. I then used sslswamp to make 1k request to the apache server
where each request negotiated a new session. At the end of the test
valgrind would consistently report leakage of 180 byte blocks which
correspond to the size of my session_id and session data. I expect the
leaks would be larger if client certificates were in use.

Bryan

On 1/18/12 9:08 AM, Geoff Thorpe wrote:
> Please send me your patch. In the mean time, I have taken a checkout of
> the code and it appears I'll need to update the CVS scripts due to
> something sourceforge has changed in their backend since the last time I
> used their system. Actually, please CC your patch to the distcache-users
> mail list too, which hopefully hasn't been killed off by sourceforge due
> to lack of activity. :-)
> 
> Thanks,
> Geoff
> 
> On 12-01-10 03:27 PM, Bryan Davis wrote:
>> I have found a memory leak in distcache 1.5.1 related to
>> early eviction of data. When
>> dc_server_default::cache_add_session forces an early
>> eviction it uses int_force_expire without first calling
>> int_pre_remove_DC_ITEM on the items to be evicted. This
>> behavior leaks the session_id and data that are associated
>> with the cache item. Looking through svn I believe that this
>> bug effects the 1.4.x stable release as well.
>>
>> I have a patch that corrects this problem and was wondering
>> where to send it. The project on sourceforge seems to be
>> long dormant, but I know that distcache is packaged by
>> several distributions.
>>
>> Thanks,
>> Bryan

-- 
  "The awful thing about getting it right the first time is that
   nobody realizes how hard it was." -- Unknown. (via jw...@jw...)

[distcache-users] how to avoid dc_server being a spof?

[Mário B. <mar...@lo...>]

Hi,

Disclaimer:
I am a noob wrt distcache + mod_ssl on apache (even though I have been a 
sysadmin for some years now), and the following question comes from 
ignorance. Please kick me in the general direction of the appropriate 
documentation.

The context:
I have just had to setup a small ssl-protected site across two 
load-balanced webservers, and came across distcache as the readily 
available method (on CentOS 5.x) of sharing the ssl session cache across 
multiple webservers.
I have perused the most immediate documentation on the subject of having 
a few dc_client's connecting to a "backend" dc_server, but it was not 
obvious to me how I could, for instance, have the dc_client's 
automatically connect to a second dc_server in case of (hard) failure on 
the primary.

The questions:
Is this at all possible to do? If it is, by which method:
. multiple -server arguments to the dc_client binary?
. multiple SSLSessionCache directives on the httpd config?
. some sort of HA / loadbalancer tool?

Thanks in advance,
------------------------------------------
Mário Barbosa <mar...@lo...>
SysAdmin @ log
log <www.log.pt>
------------------------------------------

[distcache-users] distcache compiling with glibc2.8

[elliott <el...@el...>]

Hello,

I just tried compiling distcache with lib2.8 and found that the use of
LONG_MAX and LONG_MIN need the inclusion of the limits.h header.  Here
is what I added to the 1.4.5 version of distcache's libnal/proto_fd.c
to make things compile cleanly:
 > #ifndef LONG_MAX
 > #include <limits.h>
 > #endif

Thanks.

elliott

[distcache-users] test

[Geoff T. <ge...@ge...>]

Hi folks,

This list has either been *extremely* quiet for a long time, or I've had
a mail configuration glitch of the most stunning magnitude. Having
rewired my mail configuration lately, I felt compelled to test. Please
ignore (or by all means say anything you have on your mind, it's not
like we're drowning in list-mail ...)

Cheers,
Geoff

[distcache-users] Debian package

[Adrian D. <ad...@mi...>]

Hi there!

I've been trying to make a Debian package for distcache; for testing 
(very possible that unstable also) branch it works flowless. For stable 
branch, there is apache2-2.0.54 which needs to be patched (right?); I've 
downloaded the patch from SF site, patched the source, compiled, 
everything is ok, but trying to start apache, i get the foloowing:
-----8<---------------
Starting web server: Apache2Syntax error on line 1 of 
/etc/apache2/mods-enabled/ssl.load:
Cannot load /usr/lib/apache2/modules/mod_ssl.so into server: 
/usr/lib/apache2/modules/mod_ssl.so: undefined symbol: DC_CTX_add_session
-----8<---------------

What could be wrong? The distcache was built simply, with no extra 
configure options.

Best regards,
Ady Deac

Re: [distcache-users] add_session failed

[<sh...@fr...>]

Selon sh...@fr...:
> The visible error is: [error] distributed scache 'add_session' failed i=
n
> ssl_error apache log file.


Ok, I found the problem: just a problem of right on the file shared betwe=
en
apache and dc_client

[distcache-users] add_session failed

[<sh...@fr...>]

Hi,

I am new in distcache. I have problem to configure it.

The visible error is: [error] distributed scache 'add_session' failed in
ssl_error apache log file.

I use Fedora Core 6 : Apache/2.2.3 anddistcache-1.4.5

I use these command line:
 dc_server -listen IP:127.0.0.1:62001
 dc_client -listen UNIX:/tmp/dc_client   -server IP:127.0.0.1:62001

and in the ssl.conf apache configuration file:

#   Inter-Process Session Cache:
#   Configure the SSL Session Cache: First the mechanism
#   to use and second the expiring timeout (in seconds).
SSLSessionCache        dc:UNIX:/tmp/dc_client
SSLSessionCacheTimeout  30000


Any idea about the error ?

Thank
Olivier

[distcache-users] update in the near future

[Geoff T. <ge...@ge...>]

Hi all,

Not much has changed in distcache since 1.5.1, just a couple of twiddly 
build things mostly (thanks Joe:-)). However with Apache 2.2 out now 
(which includes distcache hooks), someone recently reminded me that it 
might be a good idea to roll a new release - if for no other reason to 
assure people that the package is alive. :-)

So, are there any patches anyone felt like getting upstream for 1.5.2? Any 
little useful distro-neutral stuff we could put in that would provide 
consistency across installations/platforms? Please give it some thought 
and drop a note to the list if you've any contributions for the release. 
Also, if anyone feels like contributing but has no ideas, the 
documentation could probably use a spring-cleaning (IIRC, sslswamp's man 
page was in dire need of work). I'll probably be busy all week but will 
look at throwing some time at this next weekend. TIA.

Cheers,
Geoff

-- 
Geoff Thorpe
ge...@ge...
http://www.geoffthorpe.net/

Self-interest and materialistic desire are parts of who we are, but
not all. To base a social and economic system on these traits is
dangerously fundamentalist.
  -- Joel Bakan

Re: [distcache-users] Re: A defect in distcache 1.5.1

[Geoff T. <ge...@ge...>]

Hi Priit,

On November 1, 2005 09:06 am, Priit Randla wrote:
>     Well, the problem is real but the patch isn't complete enough.
> One more macro with unsigned long overflowing.
> Hope this patch is better...

Yes it is, thanks for the patch - it's now in cvs. Please let us know how 
you get on with squid :-)

Regards,
Geoff

-- 
Geoff Thorpe
ge...@ge...
http://www.geoffthorpe.net/

Self-interest and materialistic desire are parts of who we are, but
not all. To base a social and economic system on these traits is
dangerously fundamentalist.
  -- Joel Bakan

[distcache-users] Re: A defect in distcache 1.5.1

[Priit R. <pri...@se...>]

Priit Randla wrote:

>
>    Hello,
>
> While writing  a squid authentication module (pin-calc with limited 
> lifetime one-time passwords) which supports sharing user 
> authentication status
> within squid cluster (using distcache, of course ;-)) I found out that 
> when I created distcache sessions with timeout of 10 hours,
> these sessions would be removed by dc_server in 1640 seconds instead. 
> I took a look at the source and spotted following problem in distcache 
> 1.5.1
> libsys/sys.c:


    Well, the problem is real but the patch isn't complete enough.
One more macro with unsigned long overflowing.
Hope this patch is better...

Regards,
Priit

[distcache-users] A defect in distcache 1.5.1

[Priit R. <pri...@se...>]

    Hello,

While writing  a squid authentication module (pin-calc with limited 
lifetime one-time passwords) which supports sharing user authentication 
status
within squid cluster (using distcache, of course ;-)) I found out that 
when I created distcache sessions with timeout of 10 hours,
these sessions would be removed by dc_server in 1640 seconds instead. I 
took a look at the source and spotted following problem in distcache 1.5.1
libsys/sys.c:
---------------------------
int SYS_expirycheck(const struct timeval *timeitem, unsigned long 
msec_expiry,
                const struct timeval *timenow)
{
        struct timeval threshold;
        unsigned long usec_expiry = msec_expiry * 1000;               
<-------------- When using timeouts long enough (>4294 seconds), this 
will overflow.
        SYS_memcpy(struct timeval, &threshold, timeitem);
        threshold.tv_sec = threshold.tv_sec + (usec_expiry / 1000000L);
        threshold.tv_usec += (usec_expiry % 1000000);
        if(threshold.tv_usec > 1000000) {
                threshold.tv_usec -= 1000000;
                threshold.tv_sec++;
        }
        if(timercmp(timenow, &threshold, <))
                return 0;
        return 1;
}
--------------------------------

Maybe this function should look something like this instead?

int SYS_expirycheck(const struct timeval *timeitem, unsigned long 
msec_expiry,
                const struct timeval *timenow)
{
        struct timeval threshold;
        SYS_memcpy(struct timeval, &threshold, timeitem);
        threshold.tv_sec = threshold.tv_sec + (msec_expiry / 1000);
        threshold.tv_usec += (msec_expiry % 1000) * 1000;
        if(threshold.tv_usec > 1000000) {
                threshold.tv_usec -= 1000000;
                threshold.tv_sec++;
        }
        if(timercmp(timenow, &threshold, <))
                /* Not expired yet */
                return 0;
        /* Expired */
        return 1;
}

Regards,
Priit

[distcache-users] regarding sslswamp load test

[Horthik <p_k...@ya...>]

hi,

 I am facing a peculiar problem, I searched the net
but i cant able to find the solution.

    I am using
Apache1.3.33+mod_ssl2.8.22+openssl0.9.8g with an
sslaccelerator card, It all workin fine, but if i try
to test the load for server form another client with
sslswamp,
  distchache version in clients is -
distcache-1.4.5.tar.bz2

sslswamp option

 sslswamp -connect IP:<sever IP>:443 -update 1 -nologo
-cipher RSA

 The problem is ,The number of ops/sec is not getting
constant, ie if i see the (update1) output it give
around 200 after 30 to 40 minutes it become idle,even
not printing the out put, I saw the cpu utilization of
client and
i infer it is going nerarly 60% in the working
condition and idle (4%)remaining time, after some time
again cpu suit and getting around 200
ops/sec , It is forming like a pattern.

 Also some time sslswamp showing -1.00 ops/sec in the
finalresult(sometimes in idle condition). Why sslswamp
not utilization full cpu in client.

 please help me to solve this,

-Karthikeyan


		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: [distcache-users] How-to for distcache + Apache

[Geoff T. <ge...@ge...>]

Hi Matt,

On May 23, 2005 08:58 pm, Matt Bostock :: RetroWeb.net wrote:
> Is there a how-to or guide that demonstrates step-by-step how to use
> distcache with Apache? I'm using Apache 1.33 and FreeBSD 5.3.

Sure, if you go to the distcache download page (see www.distcache.org) you 
can grab the "support-apache1.3-modssl" package which contains an 
explanatory README and a patch to mod_ssl. The "distcache" package itself 
provides the underlying libraries and tools this depends on, and also has 
reasonable information in the README. The support-apache1.3-modssl 
package hooks into mod_ssl 2.8.16, which in turn should hook into any 
appropriate 1.* apache release. Hmm, just noticed that modssl is now up 
to 2.8.22, and 2.8.16 was only known to work with apache 1.3.29 ... 
damn. :-( The hooks are extremely minor and so the 2.8.16 patch may very 
well just patch into newer mod_ssl's without change, especially 
considering mod_ssl has been in purely maintenance mode since it got 
moved to apache2. But if the patch needs updating, please let us know; if 
you succeed in fixing it yourself we'd love a copy :-), otherwise we'll 
know that one of us has to go in and update things when we find time.

If you're not using mod_ssl (eg. if you're using Ben Laurie's module), 
then you'd need to ask someone to hook it up. distcache exposes a pretty 
straightforward API and openssl has a pretty straightforward way to plug 
in session caching callbacks.

Cheers,
Geoff

-- 
Geoff Thorpe
ge...@ge...
http://www.geoffthorpe.net/

Greedy Genghis George, Guru of God and Guns.

[distcache-users] How-to for distcache + Apache

[Matt B. :: RetroWeb.n. <ma...@re...>]

Hello,

Is there a how-to or guide that demonstrates step-by-step how to use
distcache with Apache? I'm using Apache 1.33 and FreeBSD 5.3.

Best regards,
Matt Bostock

------------------------------------------------------------
RetroWeb.net Owner
ma...@re...

www.retroweb.net
Good value never goes out of style.
------------------------------------------------------------

[distcache-users] [PATCH] fix library dependencies

[Joe O. <jo...@re...>]

Howdy, I noticed that libdistcache and libdistcacheserver are not linked
against the libraries on which they depend (at least in 1.4.5) -- here's
a fix:

--- distcache-1.4.5/libdistcacheserver/Makefile.am.libdeps
+++ distcache-1.4.5/libdistcacheserver/Makefile.am
@@ -3,4 +3,5 @@
 lib_LTLIBRARIES			= libdistcacheserver.la
 libdistcacheserver_la_SOURCES	= dc_server.c dc_server_default.c
 libdistcacheserver_la_LDFLAGS	= -version-info 1:1:0
+libdistcacheserver_la_LIBADD	= ../libdistcache/libdistcache.la ../libnal/libnal.la 
 
--- distcache-1.4.5/libdistcache/Makefile.am.libdeps
+++ distcache-1.4.5/libdistcache/Makefile.am
@@ -3,4 +3,5 @@
 lib_LTLIBRARIES		= libdistcache.la
 libdistcache_la_SOURCES = dc_client.c dc_enc.c
 libdistcache_la_LDFLAGS	= -version-info 1:1:0
+libdistcache_la_LIBADD	= ../libnal/libnal.la
 

Re: [distcache-users] Re: Followup to Jake Appelbaum's report.

[Joe O. <jo...@re...>]

On Wed, Oct 27, 2004 at 01:47:57PM -0400, Geoff Thorpe wrote:
> > /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engin
> >e_io.c(1517): OpenSSL: I/O error, 5 bytes expected to read on
> > BIO#8223ea0 [mem: 8212268] [Tue Oct 26 12:14:33 2004] [debug]
> > /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engin
> >e_kernel.c(1793): OpenSSL: Exit: error in SSLv3 read client certificate
> > A [Tue Oct 26 12:14:33 2004] [debug]
> > /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engin
> >e_kernel.c(1793): OpenSSL: Exit: error in SSLv3 read client certificate
> > A [Tue Oct 26 12:14:33 2004] [info] (9)Bad file descriptor: SSL
> > handshake interrupted by system [Hint: Stop button pressed in
> > browser?!] [Tue Oct 26 12:14:33 2004] [info] Connection to child 0

Actually I'm not sure this is an expected failure case from a premature
connection close with mod_ssl in 2.0, at least.  If some read or write
call has *really* failed with EBADF that seems worrying.  A strace/truss
of a child process failing like this would be useful.

joe

Re: [distcache-users] Re: Followup to Jake Appelbaum's report.

[Geoff T. <ge...@ge...>]

Hi there,

Sorry for the delay in getting back to you (I'm cursed by a job that 
expects my undivided concentration to matters non-distcache?!?! :-)

On October 26, 2004 03:49 pm, C. Scott Ananian wrote:
> Here's the additional information.  Note that distcache is *not* turned
> on in this trace (ie, there is no SSLSessionCache dc:... line), but it
> seems to be being invoked anyway (note the output by ssl_scache_dc.c),
> which explains why turning distcache off doesn't solve the problem,
> only removing the patch to apache fixes it.

Erk?! Is there some other SSLSessionCache line or is this just magically 
selecting some default? (BTW, is it possible a session cache is 
configured somewhere else in the config file(s)?) I'll try to take a look 
at this later, perhaps something got clobbered by the patch ... hmmm...

> The key lines are the ones about 'bad file descriptor'; these are
> generated when apache is dropping connections.

These on their own wouldn't bother me, because this is quite normal 
whenever the connection breaks before the SSL logic has completely 
shutdown an SSL tunnel. Eg. if the browser has already seen all it cares 
about and decides to disconnect;

> /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engin
>e_io.c(1517): OpenSSL: I/O error, 5 bytes expected to read on
> BIO#8223ea0 [mem: 8212268] [Tue Oct 26 12:14:33 2004] [debug]
> /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engin
>e_kernel.c(1793): OpenSSL: Exit: error in SSLv3 read client certificate
> A [Tue Oct 26 12:14:33 2004] [debug]
> /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engin
>e_kernel.c(1793): OpenSSL: Exit: error in SSLv3 read client certificate
> A [Tue Oct 26 12:14:33 2004] [info] (9)Bad file descriptor: SSL
> handshake interrupted by system [Hint: Stop button pressed in
> browser?!] [Tue Oct 26 12:14:33 2004] [info] Connection to child 0
> closed with abortive shutdown(server voteprotect.org:443, client
> 128.30.5.114)

See the second to last note.

A couple of questions;

- what are you testing with? Ie. do you have some client testing tool or 
are you manually using browsers?

- what happens to the server after you've experienced this problem? Does 
it take a restart to restore good behaviour? If yes, can you retry but 
kill and restart the dc_client tool *instead* of restarting the web 
server and tell me if that fixes it?

- if you dramatically decrease or increase the session cache timeout does 
the problem alter? Ie. setting it so that you either never, or very 
easily, fill up the cache before it is able to expire sessions.

- finally, did you try the "-idle" switch to dc_client? This is used to 
detect if a client connection from the web-server is inactive for too 
long and automatically discards it.

> The short log excerpt is appended; a fuller log excerpt is attached.
> [Will be attached when the message gets moderated through.]

I saw the moderation note come through but haven't done anything about it 
yet (I'll go in when I have more time, but it's probably not needed).

Cheers,
Geoff

-- 
Geoff Thorpe
ge...@ge...
http://www.geoffthorpe.net/

[distcache-users] Re: Followup to Jake Appelbaum's report.

[C. S. A. <cs...@cs...>]

[Resending this without the complete log, in order to get it onto the list
in a timely fashion.]

Here's the additional information.  Note that distcache is *not* turned on in 
this trace (ie, there is no SSLSessionCache dc:... line), but it seems to be 
being invoked anyway (note the output by ssl_scache_dc.c), which explains why 
turning distcache off doesn't solve the problem, only removing the patch to 
apache fixes it.

The key lines are the ones about 'bad file descriptor'; these are generated 
when apache is dropping connections.

The short log excerpt is appended; a fuller log excerpt is attached.
[Will be attached when the message gets moderated through.]
I haven't done any internal edits to these.
--scott

Diplomat LCPANGS DTFROGS SECANT Noriega KMFLUSH TASS AMTHUG LITEMPO
   Register to vote!  http://www.yourvotematters.org/VerifiedVoting
                         ( http://cscott.net/ )

[Tue Oct 26 12:14:33 2004] [debug] /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_scache_dc.c(139): distributed scache 'get_session' MISS
[Tue Oct 26 12:14:33 2004] [debug] /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engine_kernel.c(1602): Inter-Process Session Cache: request=GET status=MISSED id=17F3196234E613FFB220F38A277407971FE5466447DE046039F0FA76E200EF18 (session renewal)
[Tue Oct 26 12:14:33 2004] [debug] /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engine_kernel.c(1764): OpenSSL: Loop: SSLv3 read client hello A
[Tue Oct 26 12:14:33 2004] [debug] /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engine_kernel.c(1764): OpenSSL: Loop: SSLv3 write server hello A
[Tue Oct 26 12:14:33 2004] [debug] /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engine_kernel.c(1764): OpenSSL: Loop: SSLv3 write certificate A
[Tue Oct 26 12:14:33 2004] [debug] /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engine_kernel.c(1148): handing out temporary 1024 bit DH key
[Tue Oct 26 12:14:33 2004] [debug] /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engine_kernel.c(1764): OpenSSL: Loop: SSLv3 write key exchange A
[Tue Oct 26 12:14:33 2004] [debug] /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engine_kernel.c(1764): OpenSSL: Loop: SSLv3 write server done A
[Tue Oct 26 12:14:33 2004] [info] (9)Bad file descriptor: core_output_filter: writing data to the network
[Tue Oct 26 12:14:33 2004] [debug] /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engine_kernel.c(1764): OpenSSL: Loop: SSLv3 flush data
[Tue Oct 26 12:14:33 2004] [debug] /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engine_io.c(1517): OpenSSL: I/O error, 5 bytes expected to read on BIO#8223ea0 [mem: 8212268]
[Tue Oct 26 12:14:33 2004] [debug] /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engine_kernel.c(1793): OpenSSL: Exit: error in SSLv3 read client certificate A
[Tue Oct 26 12:14:33 2004] [debug] /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engine_kernel.c(1793): OpenSSL: Exit: error in SSLv3 read client certificate A
[Tue Oct 26 12:14:33 2004] [info] (9)Bad file descriptor: SSL handshake interrupted by system [Hint: Stop button pressed in browser?!]
[Tue Oct 26 12:14:33 2004] [info] Connection to child 0 closed with abortive shutdown(server voteprotect.org:443, client 128.30.5.114)
[Tue Oct 26 12:14:44 2004] [debug] /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engine_io.c(1517): OpenSSL: I/O error, 5 bytes expected to read on BIO#81b2f18 [mem: 8249630]
[Tue Oct 26 12:14:44 2004] [info] (70007)The timeout specified has expired: SSL input filter read failed.
[Tue Oct 26 12:14:44 2004] [debug] /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engine_kernel.c(1774): OpenSSL: Write: SSL negotiation finished successfully
[Tue Oct 26 12:14:44 2004] [info] Connection to child 5 closed with standard shutdown(server www.voteprotect.org:443, client 128.30.5.114)
[Tue Oct 26 12:14:44 2004] [debug] /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engine_io.c(1517): OpenSSL: I/O error, 5 bytes expected to read on BIO#81b2f18 [mem: 827ed48]
[Tue Oct 26 12:14:44 2004] [info] (70007)The timeout specified has expired: SSL input filter read failed.
[Tue Oct 26 12:14:44 2004] [debug] /usr/src/APACHE/apache2-2.0.52/build-tree/apache2/modules/ssl/ssl_engine_kernel.c(1774): OpenSSL: Write: SSL negotiation finished successfully
[Tue Oct 26 12:14:44 2004] [info] Connection to child 3 closed with standard shutdown(server www.voteprotect.org:443, client 128.30.5.114)

[distcache-users] Followup to Jake Appelbaum's report.

[C. S. A. <cs...@cs...>]

So wish apache logging turned up to 'info' I managed to squeeze out some 
error messages.  When the machine stops accepting connections, it *does* 
so the cert handshake, and then drops the connection and puts the 
following two lines in the log:

[Tue Oct 26 11:13:04 2004] [info] (9)Bad file descriptor: SSL input filter read failed.
[Tue Oct 26 11:13:04 2004] [info] (9)Bad file descriptor: core_output_filter: writing data to the network

Which looks (to me) like a file descriptor leak.  The problem goes away 
with a /etc/init.d/apache restart, and then we can cause it to appear 
again by hitting the site repeatedly.  Interestingly enough, the two 
vhosts seem to fail at different times; maybe apache does a fork-per-vhost 
first so they are drawing from separate pools of file descriptors?

I downgraded one of the machines to apache-without-the-distcache patch and 
we've been pounding on it, but haven't been able to get it to start 
dropping connections.

I'm going to crank logging up to 'debug' and then try to trigger this 
again, and see if we get any more revealing messages.
  --scott

Milosevic ZRMETAL Indonesia PLO plastique LIONIZER PAPERCLIP PBPRIME
    Register to vote!  http://www.yourvotematters.org/VerifiedVoting
                          ( http://cscott.net/ )

Re: [distcache-users] multiple certificates?

[Geoff T. <ge...@ge...>]

Hi Jake,

On October 26, 2004 04:45 am, Jake Appelbaum wrote:
> So in the system I am working on, it's using a few different apache
> servers, each server has two IPs. Each IP has a different cert. So that
> means a total of two certs between any number of webhosts, but each
> webhost has two certs, one for each ip.
>
> Dist-cache is running as a client, tunneled over stunnel and dist-cache
> is running as a server on a non-webserver host.
>
> One dc_client per server, and one dc_server for all the clients.

This all sounds fine.

> However, SSL "breaks" sometimes. It just stops responding.
> I wonder if this means it has the session keys for another host using
> the system that's using a different cert. Is that possible?
>
> It seems to be tied to the same time length as the amount of time it
> takes to expire session keys from the cache, then the error goes away.

Hmm... a couple of suggestions.

(1) can you reproduce this problem with Apache's logging cranked right up 
(eg. set to "debug" level) and then check the log for "dc" notes at the 
time the problem is observed? The logging should include timestamps and 
so you should be able to observe pauses, etc. In fact, if you look at the 
ssl module source code, you'll see that the "dc" cache has its own C file 
and you should be able to add/instrument any additional logging you like. 
If you think a particular call might contain the hang/pause/bug, try 
putting some logging either side of the call and grep the resulting logs 
after running. It's a bit mundane, but it can be a useful way to dig for 
this kind of bug.

(2) try setting the "-idle" flag for dc_client and see if that changes the 
behaviour? That would help identify what is "hanging", or at least 
eliminate certain links in the chain.

BTW, what version of Apache are you running with?

> Oh and the error is less than verbose, it's just "cannot read data from
> server." Mozilla firefox, debian, etc

Yeah the browser certainly won't be able to say anything intelligent about 
this (not that browsers are given to saying intelligent things at other 
times, but that's another bleat for another time :-). The issue is 
between apache's SSL module code and backwards from there via the caching 
module that's active. Is this on unix with the standard fork() model, or 
are you using Apache's threaded abomination on win32? (Which is no more 
abominable than IIS mind you.)

> Strange stuff, but I think that's the issue?

Strange indeed ... :-( If you get no joy from the above and want something 
else to try, perhaps build the latest release of distcache-1.5 and 
rebuild+link your apache stuff against that - a lot of the networking 
underbelly was overhauled since 1.4.*, and though I doubt it, it's still 
possible the "hang" is an I/O condition getting deadlocked. More 
importantly, I haven't looked at 1.4.* code for ages and it would be 
easier to debug issues in 1.5 and back-port from there if applicable.

Cheers,
Geoff
-- 
Geoff Thorpe
ge...@ge...
http://www.geoffthorpe.net/

[distcache-users] multiple certificates?

[Jake A. <ja...@ap...>]

Hi,

So in the system I am working on, it's using a few different apache
servers, each server has two IPs. Each IP has a different cert. So that
means a total of two certs between any number of webhosts, but each
webhost has two certs, one for each ip.

Dist-cache is running as a client, tunneled over stunnel and dist-cache
is running as a server on a non-webserver host.

One dc_client per server, and one dc_server for all the clients.

However, SSL "breaks" sometimes. It just stops responding.
I wonder if this means it has the session keys for another host using
the system that's using a different cert. Is that possible?

It seems to be tied to the same time length as the amount of time it
takes to expire session keys from the cache, then the error goes away.

Oh and the error is less than verbose, it's just "cannot read data from
server." Mozilla firefox, debian, etc

Strange stuff, but I think that's the issue?

-- 
Jake Appelbaum <ja...@ap...>

Re: [distcache-users] distcache for apache

[Geoff T. <ge...@ge...>]

On October 21, 2004 09:26 pm, Cere M. Davis wrote:
> Hi again Geoff,
>
> Ok, I just wanted to let you know that I used the SSLSessionCache dbm:
> stanza and it seems to work fine without the apache distcache patch. 

Of course, it's the built in dbm-based session cache. This is low 
performance and single-server, but if that is all you need, great.

> Can you confirm that this is possible?  Apache does not seems to
> complain or have a problem with this.

Nor should it, that cache mode has been included in the source for ages. 
There is also a shared-memory cache (shmcb) included that is, like dbm, 
limited to the one web-server - however shmcb performs better and has 
smarter cache semantics than dbm.

You would want distcache if you need multiple web server machines to share 
the same session cache. And for that, you'd need to patch the distcache 
support into apache unless you're using a pre-release snapshot of apache 
2.1. If you don't need distcache (dist=distributed, hence the issue of 
multiple web-servers), then this is a non-issue.

Cheers,
Geoff

-- 
Geoff Thorpe
ge...@ge...
http://www.geoffthorpe.net/

Re: [distcache-users] distcache for apache

[Cere M. D. <cere@u.washington.edu>]

Hi again Geoff,

Ok, I just wanted to let you know that I used the SSLSessionCache dbm:
stanza and it seems to work fine without the apache distcache patch.  Can
you confirm that this is possible?  Apache does not seems to complain or
have a problem with this.

Thanks,
-Cere


On Wed, 20 Oct 2004, Geoff Thorpe wrote:

> Date: Wed, 20 Oct 2004 19:13:55 -0400
> From: Geoff Thorpe <ge...@ge...>
> To: dis...@li...
> Cc: Cere Davis <cere@u.washington.edu>
> Subject: Re: [distcache-users] distcache for apache
>
> On October 20, 2004 01:31 am, Cere Davis wrote:
> > So, just to make sure I understand.  It is not possible to run
> > distcache with apache without the distcache apache patch?
>
> If you're using apache 2.0.<something>, yeah. If you're using snapshots or
> checkouts from apache CVS called 2.1.<something>, then no - the apache
> developers committed distcache support into CVS (based on the existing
> 2.0.* patches) a while ago and this will turn up in 2.1 releases whenever
> they come out.
>
> Or was your last sentence just an attempt to generate a new
> tongue-twister? :-)
>
> Cheers,
> Geoff
>

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
		        Cere Davis
		Unix Systems Administrator - CSDE
            cere@u.washington.edu   ph: 206.685.5346
         https://staff.washington.edu/cere

GnuPG Key   http://staff.washington.edu/cere/gpgkey.txt
Key fingerprint = B63C 2361 3B9B 8599 ECC9  D061 3E48 A832 F455 9E7FA

Re: [distcache-users] distcache for apache

[Geoff T. <ge...@ge...>]

On October 20, 2004 01:31 am, Cere Davis wrote:
> So, just to make sure I understand.  It is not possible to run
> distcache with apache without the distcache apache patch?

If you're using apache 2.0.<something>, yeah. If you're using snapshots or 
checkouts from apache CVS called 2.1.<something>, then no - the apache 
developers committed distcache support into CVS (based on the existing 
2.0.* patches) a while ago and this will turn up in 2.1 releases whenever 
they come out.

Or was your last sentence just an attempt to generate a new 
tongue-twister? :-)

Cheers,
Geoff
-- 
Geoff Thorpe
ge...@ge...
http://www.geoffthorpe.net/