linkchecker-user — Help and discussion about LinkChecker

[Linkchecker-user] leave "access denied by robots.txt" off output?

[Daniel G. <eh...@ce...>]

Hi Bastian,

linkchecker is great. Here is a suggestion: Don't print results for 
files that are denied access by the robots.txt file. That would get rid 
of a lot of clutter, separate the wheat from the chaff. Maybe there is 
already some way to not print the robots excluded pages. I could not 
figure it out from reading the man page.

Thanks,
Dan

[Linkchecker-user] Seems like Linkchecker-8.4 ought to replace 8.2

[David F. <da...@da...>]

Seems like version 8.4 has been out a long time.

Last source update was...

    git log -1 --date=short --pretty=format:"%h-%cd"
    91cf5f5-2013-04-21

Since last update was 7 months, seems like releasing
8.4 might be in order.

Just a thought.

-- 
Love Living Well Doing What You Love?
http://DavidFavor.com/books can help!

[Linkchecker-user] Linkchecker-8.4-20-g91cf5f5 seems to ignore --no-warnings + --no-status

[David F. <da...@da...>]

linkchecker --no-status --no-warnings --threads=5 --user-agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/27.0.1453.93 Safari/537.36' domain-to-check

Prints all sorts of status + warnings.

Let me know the best way to fix this.

Using latest git pull.

-- 
Love Living Well Doing What You Love?
http://DavidFavor.com/books can help!

[Linkchecker-user] Error display

[Matt H. <mat...@gm...>]

Hi,

Love linkchecker, but I am wondering if there is a way to configure it to
only log 404 errors?

Thanks!
Matt

[Linkchecker-user] Changes for ignoring URLs

[Bastian K. <ca...@us...>]

Hello,

several people had problems or questions with ignoring URLs and the --ignore-
url option. In the upcoming release I changed the ignore rules and I hope this 
makes ignore URLs easier:

When using --ignore-url all URLs matching the given pattern are ignored 
completely. So they will not be part of any output (even with --verbose).

Before the change, the URLs matching the given pattern were checked for syntax 
errors and were printed in verbose output.


Regards,
  Bastian

Re: [Linkchecker-user] Need login page example

[Bastian K. <ca...@us...>]

Hi,

you need to add a user entry in the linkcheckerrc file in the [authentication]
section. See the documentation[1] for an example.
[1]
http://linkchecker.git.sourceforge.net/git/gitweb.cgi?p=linkchecker/linkchecker;a=blob;f=doc/examples/linkcheckerrc_loginurl;hb=HEAD

If you do not add a password in the entry, you have to enter it on the 
commandline.

Regards,
  Bastian

Am Dienstag, 10. Juli 2012, 13:28:18 schrieb Heinz-Alexander Fuetterer:
> Hi,
> 
> first of all i would like to say that Linkchecker is a great software.
> 
> I use version 7.9. I have a similar problem like Charlie Clark and John
> Gluck.
> 
> I don't understand where to put username and password for a loginurl.
> 
> loginurl = http://lab-dashboard.bitgravity.com/login.php # change
> loginuserfield = name_of_the_username_field
> loginuserpasswordfield = name_of_the_password_field
> 
> I get this, but where do i put the actual login information?
> 
> The site i want to enter has a form like this:
> 
> <div class="loginform">
> <div class="form-label">
> 	<label for="username">Benutzername</label>
> </div>
> <div class="form-input">
> 	<input id="username" type="text" value="" size="15" name="username">
> </div>
> <div class="clearer"></div>
> <div class="form-label">
> 	<label for="password">Passwort</label>
> </div>
> <div class="form-input">
> 	<input id="password" type="password" value="" size="15" name="password">
> 	<input type="submit" value="Login">
> 	<input type="hidden" value="1" name="testcookies">
> </div>
> <div class="clearer"></div>
> </div>
> 
> So i put this in the linkcheckerrc-file.
> 
> loginurl = https://example.com/login/index.php
> loginuserfield=username
> loginpasswordfield=password
> 
> What else do i have to enter? What about this part?
> 
> <input type="submit" value="Login">
> <input type="hidden" value="1" name="testcookies">
> 
> Help would be very much appreciated.
> 
> Greetings from Berlin,
> ALEX
> 
> --------------------------------------------------------------------
> 
> This relates to a post from last year before I was subscribed.
> 
> I don't know whether it has been resolved.
> 
> A couple of notes: logging in via a form requires the twill library to be
> installed. The error is coming from LinkChecker trying to do "simple" http
> authentication. I agree that the documentation could be better but based
> on config0.ini I can see what you're missing: you must have values in the
> entry key and the loginurl for LinkChecker to try and use the form.
> 
> [authentication]
> entry = http://lab-dashboard.bitgravity.com myusername it's_a_secret #
> change
> 
> loginurl = http://lab-dashboard.bitgravity.com/login.php # change
> loginuserfield = name_of_the_username_field
> loginuserpasswordfield = name_of_the_password_field
> 
> Note, I haven't cracked this myself currently stuck with an Exception from
> Twill twill.errors.TwillAssertionError: no matching forms!
> 
> @ Bastian: the docs for entry imply that you can have multiple logins but
> this is only true for http authentication and not form-based, correct?
> 
> ***
> 
> My site has the first page as a login,  In order to check the links I need
> to hit the first page and login.  Ideally, I'd like to pass the user
> password info in either through a cookie or through the linkcheckerrc so
> that I don't have to manually enter the password in everytime.
> 
> I can't figure out how to do this.  The examples aren't clear enough.  I've
> struggled with this for an hour.  Unfortunately, my python isn't strong
> enough for me to figure out how the code is parsing the conf file.
> 
> Here's what I'm executing
> https://<url> --config=linkcheckerrc --output=html --cookiefile=cookie -C>
> log.html
> 
> Here's the contents of cookie
> 
> Scheme: https
> Host: <url>
> Path: /
> Set-cookie: user_id="<value>";ahash="<value>"
> 
> Here's what's in linkcheckerrc
> [authentication]
> # Different user/password pairs for different URLs can be provided.
> # Entries are a triple (URL regular expression, username, password),
> # separated by whitespace.
> # If the regular expression matches, the given user/password pair is used
> # for authentication. The commandline options -u,-p match every link
> # and therefore override the entries given here. The first match wins.
> # At the moment, authentication is used for http[s] and ftp links.
> #entry=
> # Note that passwords are optional. If any passwords are stored here,
> # this file should not readable by other users.
> #  ^https?://www\.example\.com/~calvin/ calvin mypass
> #  ^ftp://www\.example\.com/secret/ calvin
> # if the website requires a login the URL and optionally the user and
> # password CGI fieldnames can be provided.
> loginurl=<URL> <username> <pass>
> # The name of the user and password CGI field
> loginuserfield=acct
> loginpasswordfield=pass
> # Optionally any additional CGI name/value pairs. Note that the default
> # values are submitted automatically.
> loginextrafields=
> cmd:login
> submit=Login
> 
> I get the following
> WARNING MainThread warning: no user/password authentication data found for
> login URL.
> WARNING MainThread warning: disabling login URL
> http://lab-dashboard.bitgravity.com/index.php.
> 
> And, of course, I don't get logged in.
> 
> Please help

Re: [Linkchecker-user] Need login page example

[Heinz-Alexander F. <hei...@ib...>]

Hi,

first of all i would like to say that Linkchecker is a great software.

I use version 7.9. I have a similar problem like Charlie Clark and John 
Gluck.

I don't understand where to put username and password for a loginurl.

loginurl = http://lab-dashboard.bitgravity.com/login.php # change
loginuserfield = name_of_the_username_field
loginuserpasswordfield = name_of_the_password_field

I get this, but where do i put the actual login information?

The site i want to enter has a form like this:

<div class="loginform">
<div class="form-label">
	<label for="username">Benutzername</label>
</div>
<div class="form-input">
	<input id="username" type="text" value="" size="15" name="username">
</div>
<div class="clearer"></div>
<div class="form-label">
	<label for="password">Passwort</label>
</div>
<div class="form-input">
	<input id="password" type="password" value="" size="15" name="password">
	<input type="submit" value="Login">
	<input type="hidden" value="1" name="testcookies">
</div>
<div class="clearer"></div>
</div>

So i put this in the linkcheckerrc-file.

loginurl = https://example.com/login/index.php
loginuserfield=username
loginpasswordfield=password

What else do i have to enter? What about this part?

<input type="submit" value="Login">
<input type="hidden" value="1" name="testcookies">

Help would be very much appreciated.

Greetings from Berlin,
ALEX

--------------------------------------------------------------------

This relates to a post from last year before I was subscribed.

I don't know whether it has been resolved.

A couple of notes: logging in via a form requires the twill library to be
installed. The error is coming from LinkChecker trying to do "simple" http
authentication. I agree that the documentation could be better but based
on config0.ini I can see what you're missing: you must have values in the
entry key and the loginurl for LinkChecker to try and use the form.

[authentication]
entry = http://lab-dashboard.bitgravity.com myusername it's_a_secret #
change

loginurl = http://lab-dashboard.bitgravity.com/login.php # change
loginuserfield = name_of_the_username_field
loginuserpasswordfield = name_of_the_password_field

Note, I haven't cracked this myself currently stuck with an Exception from
Twill twill.errors.TwillAssertionError: no matching forms!

@ Bastian: the docs for entry imply that you can have multiple logins but
this is only true for http authentication and not form-based, correct?

***

My site has the first page as a login,  In order to check the links I need
to hit the first page and login.  Ideally, I'd like to pass the user
password info in either through a cookie or through the linkcheckerrc so
that I don't have to manually enter the password in everytime.

I can't figure out how to do this.  The examples aren't clear enough.  I've
struggled with this for an hour.  Unfortunately, my python isn't strong
enough for me to figure out how the code is parsing the conf file.

Here's what I'm executing
https://<url> --config=linkcheckerrc --output=html --cookiefile=cookie -C>
log.html

Here's the contents of cookie

Scheme: https
Host: <url>
Path: /
Set-cookie: user_id="<value>";ahash="<value>"

Here's what's in linkcheckerrc
[authentication]
# Different user/password pairs for different URLs can be provided.
# Entries are a triple (URL regular expression, username, password),
# separated by whitespace.
# If the regular expression matches, the given user/password pair is used
# for authentication. The commandline options -u,-p match every link
# and therefore override the entries given here. The first match wins.
# At the moment, authentication is used for http[s] and ftp links.
#entry=
# Note that passwords are optional. If any passwords are stored here,
# this file should not readable by other users.
#  ^https?://www\.example\.com/~calvin/ calvin mypass
#  ^ftp://www\.example\.com/secret/ calvin
# if the website requires a login the URL and optionally the user and
# password CGI fieldnames can be provided.
loginurl=<URL> <username> <pass>
# The name of the user and password CGI field
loginuserfield=acct
loginpasswordfield=pass
# Optionally any additional CGI name/value pairs. Note that the default
# values are submitted automatically.
loginextrafields=
cmd:login
submit=Login

I get the following
WARNING MainThread warning: no user/password authentication data found for
login URL.
WARNING MainThread warning: disabling login URL
http://lab-dashboard.bitgravity.com/index.php.

And, of course, I don't get logged in.

Please help
-- 
Charlie Clark
Managing Director
Clark Consulting & Research
German Office
Kronenstr. 27a
Düsseldorf
D- 40217
Tel: +49-211-600-3657
Mobile: +49-178-782-6226

[Linkchecker-user] Linkcheck questions !! help !!

[조용두 <le...@na...>]

 
Hello~ hi~I can not speak English T_T 

I'm really sssorrry if difficult to interpret /

It is a GOOD program. thank you very much !! *^^*I got a question sends mail.

My command :linkchecker - ignore-url = ^ mailto: - no-warnings - output = html - recursion-level = 3 - timeout = 10 http://test.co.kr> C: \ temp \ test.htmlQuestion !!Results only "Error: 404 Not Found (404 Not Found)" What can I do to get to?

404error only want to report~

error = 404 only !!

 

... impossible? T_T?

This sends mail so I wondered.Thank you for the time to read mail.Have a nice day /////

Re: [Linkchecker-user] MainThread Could not import twill for login URL visit

[Bastian K. <ca...@us...>]

Hi,

the twill module in the Windows installer was incomplete.
The next release of LinkChecker will fix this.

Regards,
  Bastian

Am Thursday, 3. May 2012, 16:37:13 schrieb Thomas Smith:
> LinkChecker 7.7
> Windows 7 (64bit)
> 
> I am using the CMD Promt to try and  log into a form page on an internal
> HTTP site.  After I provide my password and hit enter I get a message
> "WARNING MainThread Could not import twill for login URL visit".
> 
> LinkChecker does test the front page but does not appear to log in.  Can
> someone assist me on this?
> 
> Thanks
> 
> 
> -Thomas

Re: [Linkchecker-user] MainThread Could not import twill for login URL visit

[Charlie C. <cha...@cl...>]

Am 03.05.2012, 20:32 Uhr, schrieb Thomas Smith <TS...@aw...>:

> Hi Charlie,
> Thanks for getting back to me.  Do you know where exactly I need to  
> install twill for this to work? Or if there is a certain command I now  
> need to run.  I am still getting the same warning.

Hi Thomas,

it's always a good to reply to this list as well.

On un*x systems it's pretty easy to install additional Python libraries  
using "pip" - the Python installer. The same is generally possible on  
Windows if a bit more of a challenge as the command line works differently.

In your Python installation you should have a folder called "Scripts" say,  
c:\python\scripts and this should contain "pip"

Either you type this out in full or add this and the path to the  
Python.exe to your environmental variables. Installing is then simply a  
question of typing c:\python\scripts\pip install -U <library name>. The  
"-U" will update the library to the latest version if it's already  
installed. Note, this approach doesn't work for all libraries including  
linkchecker because it needs additional files and also needs to compile  
them, which is difficult on Windows as it doesn't come with a compiler.

I noticed this bug myself the other day. I hope it will be fixed in the  
next release by including the dependency in the installer.

Hope this helps

Charlie
-- 
Charlie Clark
Managing Director
Clark Consulting & Research
German Office
Kronenstr. 27a
Düsseldorf
D- 40217
Tel: +49-211-600-3657
Mobile: +49-178-782-6226

Re: [Linkchecker-user] MainThread Could not import twill for login URL visit

[Charlie C. <cha...@cl...>]

Am 03.05.2012, 16:37 Uhr, schrieb Thomas Smith <TS...@aw...>:

> LinkChecker 7.7
> Windows 7 (64bit)
> I am using the CMD Promt to try and  log into a form page on an internal  
> HTTP site.  After I provide my password and hit enter I get a message  
> "WARNING MainThread Could not import twill for login URL visit".
> LinkChecker does test the front page but does not appear to log in.  Can  
> someone assist me on this?
> Thanks

Hi Thomas,

you'll need to install the Twill library: generally pip install twill will  
do the work.

Charlie
-- 
Charlie Clark
Managing Director
Clark Consulting & Research
German Office
Kronenstr. 27a
Düsseldorf
D- 40217
Tel: +49-211-600-3657
Mobile: +49-178-782-6226

[Linkchecker-user] MainThread Could not import twill for login URL visit

[Thomas S. <TS...@aw...>]

LinkChecker 7.7
Windows 7 (64bit)

I am using the CMD Promt to try and  log into a form page on an internal HTTP site.  After I provide my password and hit enter I get a message "WARNING MainThread Could not import twill for login URL visit".

LinkChecker does test the front page but does not appear to log in.  Can someone assist me on this?

Thanks


-Thomas

Re: [Linkchecker-user] Need login page example

[Charlie C. <cha...@cl...>]

This relates to a post from last year before I was subscribed.

I don't know whether it has been resolved.

A couple of notes: logging in via a form requires the twill library to be  
installed. The error is coming from LinkChecker trying to do "simple" http  
authentication. I agree that the documentation could be better but based  
on config0.ini I can see what you're missing: you must have values in the  
entry key and the loginurl for LinkChecker to try and use the form.

[authentication]
entry = http://lab-dashboard.bitgravity.com myusername it's_a_secret #  
change

loginurl = http://lab-dashboard.bitgravity.com/login.php # change
loginuserfield = name_of_the_username_field
loginuserpasswordfield = name_of_the_password_field

Note, I haven't cracked this myself currently stuck with an Exception from  
Twill twill.errors.TwillAssertionError: no matching forms!

@ Bastian: the docs for entry imply that you can have multiple logins but  
this is only true for http authentication and not form-based, correct?

***

My site has the first page as a login,  In order to check the links I need
to hit the first page and login.  Ideally, I'd like to pass the user
password info in either through a cookie or through the linkcheckerrc so
that I don't have to manually enter the password in everytime.

I can't figure out how to do this.  The examples aren't clear enough.  I've
struggled with this for an hour.  Unfortunately, my python isn't strong
enough for me to figure out how the code is parsing the conf file.

Here's what I'm executing
https://<url> --config=linkcheckerrc --output=html --cookiefile=cookie -C>
log.html

Here's the contents of cookie

Scheme: https
Host: <url>
Path: /
Set-cookie: user_id="<value>";ahash="<value>"

Here's what's in linkcheckerrc
[authentication]
# Different user/password pairs for different URLs can be provided.
# Entries are a triple (URL regular expression, username, password),
# separated by whitespace.
# If the regular expression matches, the given user/password pair is used
# for authentication. The commandline options -u,-p match every link
# and therefore override the entries given here. The first match wins.
# At the moment, authentication is used for http[s] and ftp links.
#entry=
# Note that passwords are optional. If any passwords are stored here,
# this file should not readable by other users.
#  ^https?://www\.example\.com/~calvin/ calvin mypass
#  ^ftp://www\.example\.com/secret/ calvin
# if the website requires a login the URL and optionally the user and
# password CGI fieldnames can be provided.
loginurl=<URL> <username> <pass>
# The name of the user and password CGI field
loginuserfield=acct
loginpasswordfield=pass
# Optionally any additional CGI name/value pairs. Note that the default
# values are submitted automatically.
loginextrafields=
cmd:login
submit=Login

I get the following
WARNING MainThread warning: no user/password authentication data found for
login URL.
WARNING MainThread warning: disabling login URL
http://lab-dashboard.bitgravity.com/index.php.

And, of course, I don't get logged in.

Please help
-- 
Charlie Clark
Managing Director
Clark Consulting & Research
German Office
Kronenstr. 27a
Düsseldorf
D- 40217
Tel: +49-211-600-3657
Mobile: +49-178-782-6226

[Linkchecker-user] Cookies with special characters

[Charles J. <cb...@ch...>]

I've got a site sending a cookie whose value looks like a JSON object, e.g., {"name1":"val 1","name2":"val 2"};
This violates the cookie specs in many ways, but browsers don't seem to have any problem with it. It looks like linkchecker is already allowing commas and whitespace in cookie values, so only double-quotes are causing parsing issues in this example. 

If we allow quotes, however, linkchecker escapes them while wrapping the entire value in a pair of double quotes. I was not able to make the site in question happy until I removed the quoting/escaping logic while sending cookies. 

It appears that Firefox does not quote cookie values either. I did not check any other browsers. 

Any thoughts?

Charles

[Linkchecker-user] Configuration file not recognized

[Paige B. <pai...@gm...>]

I'm working with linkchecker 7.5 on a Windows 7 64 bit system. I found the
sample configuration file, but it doesn't seem to be recognized by the
program. Am I supposed to move it from its current location? If so, where
to put it?
Paige

-- 
^..^
Programs are never finished, merely abandoned. ~a programmer's truism

Re: [Linkchecker-user] Problem with options in configuration file

[Bastian K. <ca...@us...>]

Hi,

I added the missing config file options pause and cookiefile.
Wait until the next release to use them.

Regards,
  Bastian

Am Thursday, 15. March 2012, 03:18:46 schrieb Markus Santa:
> Hello everyone
> 
> I have been playing around with Linkchecker and noticed that some options
> don't seem to get used if provided via the configuration file.
> 
> For example if I put "pause=5" into the configuration file (under the
> checking section) and then run linkchecker there is no pausing between url
> checks.
> However if I type into the command line "linkchecker.exe --pause=5
> someurl", then it pauses 5 seconds between checks.
> 
> I have noticed the same thing with specifying a cookie file. If I put
> "cookiefile=C:\\Users\\Mark\\.linkchecker\\cookies" into the config under
> the checking section it does nothing. If I provide it through the command
> line: "linkchecker.exe --cookiefile=C:\\Users\\Mark\\.linkchecker\\cookies
> someurl" then it works.
> 
> Now, the pause and cookiefile options aren't in the default linkcheckerrc
> file, but I assumed I could still use those options as documented here:
> http://linkchecker.sourceforge.net/man1/linkchecker.1.html
> Especially since it says:  "Configuration files can specify all options
> above. They can also specify some options that cannot be set on the command
> line. See
> *linkcheckerrc<http://linkchecker.sourceforge.net/man5/linkcheckerrc.5.htm
> l> *(5) for more info."
> 
> So my question is, has anyone managed to get 'pause' and 'cookiefile'
> options to work when put into the configuration file? I can change other
> options in the config file and they are used, so it isn't a matter of
> linkchecker not finding the config file.
> 
> 
> Thanks for you help.

[Linkchecker-user] Problem with options in configuration file

[Markus S. <mar...@gm...>]

Hello everyone

I have been playing around with Linkchecker and noticed that some options
don't seem to get used if provided via the configuration file.

For example if I put "pause=5" into the configuration file (under the
checking section) and then run linkchecker there is no pausing between url
checks.
However if I type into the command line "linkchecker.exe --pause=5
someurl", then it pauses 5 seconds between checks.

I have noticed the same thing with specifying a cookie file. If I put
"cookiefile=C:\\Users\\Mark\\.linkchecker\\cookies" into the config under
the checking section it does nothing. If I provide it through the command
line: "linkchecker.exe --cookiefile=C:\\Users\\Mark\\.linkchecker\\cookies
someurl" then it works.

Now, the pause and cookiefile options aren't in the default linkcheckerrc
file, but I assumed I could still use those options as documented here:
http://linkchecker.sourceforge.net/man1/linkchecker.1.html
Especially since it says:  "Configuration files can specify all options
above. They can also specify some options that cannot be set on the command
line. See *linkcheckerrc<http://linkchecker.sourceforge.net/man5/linkcheckerrc.5.html>
*(5) for more info."

So my question is, has anyone managed to get 'pause' and 'cookiefile'
options to work when put into the configuration file? I can change other
options in the config file and they are used, so it isn't a matter of
linkchecker not finding the config file.


Thanks for you help.

Re: [Linkchecker-user] Real url not copied from cache

[Bastian K. <ca...@us...>]

Hi,

I guess the reason is outdated now that anchor handling works different.
Will test implement this change for the next release.

Thanks,
  Bastian

Am Friday, 2. March 2012, 20:57:45 schrieb Charles Jones:
> Is there any reason why the url field is not copied over when copying a
> cached result? I have a site with about 400 links to the same url, which
> is being redirected to another url. Only one of the error reports shows
> the redirected url, the rest show the original one.
> 
> Here is a proposed patch, but I wanted to make sure I wasn't missing
> anything obvious before submitting it:
> 
> diff --git a/linkcheck/checker/urlbase.py b/linkcheck/checker/urlbase.py
> index fb7bf7b..98e91a1 100644
> --- a/linkcheck/checker/urlbase.py
> +++ b/linkcheck/checker/urlbase.py
> @@ -301,6 +302,7 @@ class UrlBase (object):
>          """
>          Fill attributes from cache data.
>          """
> +        self.url = cache_data["url"]
>          self.result = cache_data["result"]
>          self.has_result = True
>          anchor_changed = (self.anchor != cache_data["anchor"])
> @@ -326,7 +328,8 @@ class UrlBase (object):
> 
>      def get_cache_data (self):
>          """Return all data values that should be put in the cache."""
> -        return {"result": self.result,
> +        return {"url": self.url,
> +                "result": self.result,
>                  "warnings": self.warnings,
>                  "info": self.info,
>                  "valid": self.valid,
> ---------------------------------------------------------------------------
> --- Virtualization & Cloud Management Using Capacity Planning
> Cloud computing makes use of virtualization - but cloud computing
> also focuses on allowing computing to be delivered as a service.
> http://www.accelacomm.com/jaw/sfnl/114/51521223/
> _______________________________________________
> Linkchecker-user mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linkchecker-user

Re: [Linkchecker-user] The --scan-virus option

[Bastian K. <bas...@go...>]

Hi,

you have to install the clamav-daemon package.
I improved the error message for the next release.

Regards,
  Bastian

Am Monday, 12. March 2012, 09:52:52 schrieb wout:
> Hi all,
> 
> one of the interesting options of Linkchecker is the --scan-virus
> switch, using ClamAV to check for malware. On my Ubuntu 10.04 computer,
> I have installed Linkchecker version 5.1 and ClamAV version 0.96.5. Both
> are working fine. However, when I would like to scan a web site for
> malware with linkchecker, I get this error (reproducible):
> 
> Start checking at 2012-03-12 09:42:59+002
> WARNING CheckThread-http://www.nu.nl Double result u'error: [Errno 2] No
> such file or directory' (previous u'200 OK') for http link
> base_url=u'http://www.nu.nl'
> parent_url=None
> base_ref=None
> recursion_level=0
> url_connection=<HttpConnection state=Idle
>   method=GET
>   sock=<socket._socketobject object at 0x97823ac>
>   buffer=[]
>   response=<HTTPResponse status=200 reason=OK chunked=0 length=0>>
> line=0
> column=0
> name=u''
> 
> ID         1
> URL        `http://www.nu.nl'
> Real URL   http://www.nu.nl
> Check Time 0.383 seconds
> D/L Time   0.079 seconds
> Size       75.75KB
> Info       Last modified Mon, 12 Mar 2012 08:41:09 GMT.
> Result     Error: error: [Errno 2] No such file or directory
> 
> Without --scan-virus, there is no error message. I looked for similar
> bugs or messages on this mailing list, but I could not find any. I would
> like to know whether other users have the same problem and whether there
> is a solution for this.
> 
> Thanks in advance.
> 
> Wout
> 
> ---------------------------------------------------------------------------
> --- Try before you buy = See our experts in action!
> The most comprehensive online learning library for Microsoft developers
> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> Metro Style Apps, more. Free future releases when you subscribe now!
> http://p.sf.net/sfu/learndevnow-dev2
> _______________________________________________
> Linkchecker-user mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linkchecker-user

[Linkchecker-user] Real url not copied from cache

[Charles J. <cb...@ch...>]

Is there any reason why the url field is not copied over when copying a cached result? I have a site with about 400 links to the same url, which is being redirected to another url. Only one of the error reports shows the redirected url, the rest show the original one.

Here is a proposed patch, but I wanted to make sure I wasn't missing anything obvious before submitting it:

diff --git a/linkcheck/checker/urlbase.py b/linkcheck/checker/urlbase.py
index fb7bf7b..98e91a1 100644
--- a/linkcheck/checker/urlbase.py
+++ b/linkcheck/checker/urlbase.py
@@ -301,6 +302,7 @@ class UrlBase (object):
         """
         Fill attributes from cache data.
         """
+        self.url = cache_data["url"]
         self.result = cache_data["result"]
         self.has_result = True
         anchor_changed = (self.anchor != cache_data["anchor"])
@@ -326,7 +328,8 @@ class UrlBase (object):
 
     def get_cache_data (self):
         """Return all data values that should be put in the cache."""
-        return {"result": self.result,
+        return {"url": self.url,
+                "result": self.result,
                 "warnings": self.warnings,
                 "info": self.info,
                 "valid": self.valid,

[Linkchecker-user] LinkChecker support for sitemaps (urlset and sitemapindex)

[Sven H. <har...@gm...>]

Hi all.

Sitemaps are becoming more and more popular. Hence, my question:
Does LinkChecker support sitemaps
(as specified on http://www.sitemaps.org/protocol.html)?
Web pages containing an urlset element would be primary,
while the sitemapindex element would probably be secondary
(in importance).

I tried to run LinkChecker 7.1 on a sitemapindex and an urlset;
it did not crash, but it did not follow any links contained in <loc> elements.

Greetings
Sven

[Linkchecker-user] Need login page example

[john g. <joh...@gm...>]

My site has the first page as a login,  In order to check the links I need
to hit the first page and login.  Ideally, I'd like to pass the user
password info in either through a cookie or through the linkcheckerrc so
that I don't have to manually enter the password in everytime.

I can't figure out how to do this.  The examples aren't clear enough.  I've
struggled with this for an hour.  Unfortunately, my python isn't strong
enough for me to figure out how the code is parsing the conf file.

Here's what I'm executing
https://<url> --config=linkcheckerrc --output=html --cookiefile=cookie -C>
log.html

Here's the contents of cookie

Scheme: https
Host: <url>
Path: /
Set-cookie: user_id="<value>";ahash="<value>"

Here's what's in linkcheckerrc
[authentication]
# Different user/password pairs for different URLs can be provided.
# Entries are a triple (URL regular expression, username, password),
# separated by whitespace.
# If the regular expression matches, the given user/password pair is used
# for authentication. The commandline options -u,-p match every link
# and therefore override the entries given here. The first match wins.
# At the moment, authentication is used for http[s] and ftp links.
#entry=
# Note that passwords are optional. If any passwords are stored here,
# this file should not readable by other users.
#  ^https?://www\.example\.com/~calvin/ calvin mypass
#  ^ftp://www\.example\.com/secret/ calvin
# if the website requires a login the URL and optionally the user and
# password CGI fieldnames can be provided.
loginurl=<URL> <username> <pass>
# The name of the user and password CGI field
loginuserfield=acct
loginpasswordfield=pass
# Optionally any additional CGI name/value pairs. Note that the default
# values are submitted automatically.
loginextrafields=
cmd:login
submit=Login

I get the following
WARNING MainThread warning: no user/password authentication data found for
login URL.
WARNING MainThread warning: disabling login URL
http://lab-dashboard.bitgravity.com/index.php.

And, of course, I don't get logged in.

Please help

[Linkchecker-user] Discreapency in the output

[praveena vp <vpp...@gm...>]

Hi,

When Linkchecker was run against this URL "http://automobiles.honda.com/",
the output was "Error: gaierror: [Errno 8] nodename nor servname provided,
or not known", though the URL does not have any problem.

Thanks,
Praveena

[Linkchecker-user] Linkchecker - statistics

[praveena vp <vpp...@gm...>]

Hi,

Linkchecker output displays the "Statistics" info. Can anyone tell me what
"Application" means in that.

Thanks,
Praveena

[Linkchecker-user] issue installing LinkChecker 7.0 using setup.py

[vishal b. <cr...@gm...>]

Hi

I got the new tar file and unzipped it to install using setup.py so that i
get command line options.
I am running into errors like this when i build.
vishalbhatmac:LinkChecker-7.0 vishalbhat$ python setup.py install
creating
/Users/vishalbhat/Downloads/LinkChecker-7.0/_LinkChecker_configdata.py
running install
running bdist_egg
running egg_info
creating LinkChecker.egg-info
writing LinkChecker.egg-info/PKG-INFO
writing top-level names to LinkChecker.egg-info/top_level.txt
writing dependency_links to LinkChecker.egg-info/dependency_links.txt
writing manifest file 'LinkChecker.egg-info/SOURCES.txt'
reading manifest file 'LinkChecker.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'doc/html/*.qhp'
warning: no files found matching 'doc/html/*.qhcp'
writing manifest file 'LinkChecker.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.6-universal/egg
running install_lib
running build_py
running build_ext
building 'linkcheck.HtmlParser.htmlsax' extension
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -Os
-Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch ppc -arch x86_64
-pipe -DYY_NO_INPUT -Ilinkcheck/HtmlParser
-I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6
-c linkcheck/HtmlParser/htmllex.c -o
build/temp.macosx-10.6-universal-2.6/linkcheck/HtmlParser/htmllex.o
-pedantic -std=gnu99
/usr/libexec/gcc/powerpc-apple-darwin10/4.2.1/as: assembler
(/usr/bin/../libexec/gcc/darwin/ppc/as or
/usr/bin/../local/libexec/gcc/darwin/ppc/as) for architecture ppc not
installed
Installed assemblers are:
/usr/bin/../libexec/gcc/darwin/x86_64/as for architecture x86_64
/usr/bin/../libexec/gcc/darwin/i386/as for architecture i386
htmllex.l:1158: fatal error: error writing to -: Broken pipe
compilation terminated.
lipo: can't open input file:
/var/folders/KS/KS2O5YHeHXenZXOTJlmYLE+++TM/-Tmp-//ccMu4KyO.out (No such
file or directory)
error: command 'gcc-4.2' failed with exit status 1


Any idea how to resolve the issue

Thanks
Vishal