Crawler commands help for "WTM Crawler" by webtoolmaster.com
Example Scripts: - bold are commands
Script 1:
; check if a website work with a special script - search copyright information
; Remove data from result box
ResetResultBox
; Surf to a website
SurfToFollowUrlOnly "http://www.webtoolmaster.com/crawler-examples.htm"
; Check if the copyright information of the script exists on site
; Extract link when yes to the result box
SearchTextInSourceToResult "Powered by Script"
; Check if the copyright information of the script exists on site
; Extract link when yes to the follow box
SearchTextInSourceToFollow "Powered by Script"
Script 2:
; Extract mails from website
; Remove data from result box
ResetResultBox
; Surf to a website
SurfToFollowUrlOnly "http://www.webtoolmaster.com/crawler-examples.htm"
; Extract every mail address to result box
ExtractEMailsToResult
When you include a ";" in the first place of a line, than you can
use it for texts and information.
; Here you can give comments
SurfToFollowUrlOnly "http://www.yoururl.com/index.htm"
Surf to this url and follow every same root link found on this site. Important is that the url is between "URL".
ExtractLinksToFollow
This extract all links (not mailto and javascript) to the follow box.
ExtractAllLinksToFollow
This extract all links (with mailto and javascript) to the follow box.
FollowAllLinksInFollow "5"
// Here are commands that to do on every visit website. 5 is the deep of every
link in follow line.
FollowAllLinksInFollowEnd
This follow all Links in the follow bow. You can give a deep follow setting from 0 to 999999... every link. Standard is often 5. On every site will be do that commands are between this and FollowAllLinksInFollowEnd command on every vistis website.
ExtractEMailsToResult
Extract all founded mails on the visit website to the result box. Example: test@test.co
ExtractEMailLinkToResult
Extract the complet EMail link to the result field. Example: mailto:test@test.co
ExtractEMailTagToResult
Extract the complet EMail link without the "mailto:" to the result
field. Example: test@test.co?subject=Comments
GenerateLinksToFollow "http://www.yoururl.com/index.htm?FromPage:$\1:5:1\$&ToPage:$\2:6:1\$"
This command generate different links to the follow box. You can use variables
with counter. The command "$\1:5:1\$" replace from number 1 to 5.
The same is with $\2:6:1\$. So it replace from number 2 to 6. It generate 5
links in the follow box.
Variable: $\[StartNumber]:[EndNumber]:[StepsBetweenNumbers]\$
Example:
http://www.yoururl.com/index.htm?FromPage:1&ToPage:2
http://www.yoururl.com/index.htm?FromPage:2&ToPage:3
http://www.yoururl.com/index.htm?FromPage:3&ToPage:4
http://www.yoururl.com/index.htm?FromPage:4&ToPage:5
http://www.yoururl.com/index.htm?FromPage:5&ToPage:6
Variable: $\10:50:10\$ - http://www.yoururl.com/index.htm?PageResults:$\10:50:10\$
http://www.yoururl.com/index.htm?PageResults:10
http://www.yoururl.com/index.htm?PageResults:20
http://www.yoururl.com/index.htm?PageResults:30
http://www.yoururl.com/index.htm?PageResults:40
http://www.yoururl.com/index.htm?PageResults:50
GenerateLinksToResult "http://www.yoururl.com/index.htm?FromPage:$\1:5:1\$&ToPage:$\2:6:1\$"
The same then GenerateLinksToFollow, but put the result to the result box.
SearchTextInSourceToResult "My text to search."
This command extract every link that found the text "My text to search."
on the website in the result box. You can search links, tags and other information
too.
with if variable support / 1 = text found - 0 = text not found
SearchTextInSourceToFollow "My text to search."
This command extract every link that found the text "My text to search."
on the website in the follow box. You can search links, tags and other information
too.
with if variable support / 1 = text found - 0 = text not found
SearchNoTextInSourceToFollow "My text to search."
This command extract every link that the text "My text to search."
do not found on website in the follow box. You can search links, tags and other
information too.
with if variable support / 1 = text found - 0 = text not found
SearchNoTextInSourceToResult "My text to search."
This command extract every link that the text "My text to search."
do not found on website in the result box. You can search links, tags and other
information too.
with if variable support / 1 = text found - 0 = text not found
GetTextBetweenInSourceToResult "My text ":" search."
Extract the text between "My text " and " search." in the
same html line to the result box. On "My text to search." is the result
"to".
It show then exact the texts between this. When you leave the secound empty
then it show to the html line end. "My text":""
ExcludeTextInLink ".com"
This command is a text filter. So it do not visits domains that have .com inlcuded.
OnlyTextInLink "myDomain.com"
This command allowed links on site myDomain.com only. You can use every word you like.
ResetExcludeTextBox
This delete all entries in the exclude text box.
ResetOnlyTextBox
This delete all entries in the only text box.
FillTextInBrowser "FieldName":"InlcudeThisText"
Filled a Text in the named field in the webbrowser
ClickOnButtonInBrowser "FieldName"
Click on a Button in the webbrowser
CheckIfWebsiteChangeToFollow "http://www.yoursites3.com/tocheck.htm":""
Check if the website is changed since last visit. It show every changed url
to the follow box.
The program include a checksum at the end after the url name "http://www.yoursites3.com/tocheck.htm":"D3GA26432"
CheckIfWebsiteChangeToResult "http://www.yoursites3.com/tocheck.htm":""
Check if the website is changed since last visit. It show every changed url to the result box.
AddLinkToFollow "http://www.yoursites3.com"
Add this link to the follow box.
ResetFollowBox
Clear all links in the follow box.
MoveLastLineBeforeInResult " /% "
Move the last line to one line before and inlcude " /% " between this to lines in results. When you leave this empty than there are no free space between this both.
ResetResultBox
Delete all data in the result box
RemoveLinesFromFileInResult "c:\myfile.txt"
Remove every line in a selected file that found in the result box.
RemoveLinesFromFileCSInResult "c:\myfile.txt"
Remove every line in a selected file that found in the result box (case sensitive).
CheckDoubleLinesIn "result"
CheckDoubleLinesCSIn "follow"
Check double lines in a selected field. CheckDoubleLinesIn is without case sensitive
and CheckDoubleLinesCSIn with.
result = Result Box
follow = Follow Box
exclude = Exclude Box
only = Only in text box
Wait "1000"
The program wait for a select time in ms. 1000 = 1 sec.
AddTextInResult "AddMyText: "
Here you can add a selected text to the result field. Here you can add a title
or use the MoveLastLineBeforeInResult "" to get the next command
in result in the same line.
SetEdit "object-name":"AddMyText"
Here you can fill the edit field named "object-name" with the text
AddMyText in the filler box.
ClearFiller
Delete and clear every filling in the filler box.
SetText "object-name":"AddMyText"
Here you can fill the memo field named "object-name" with the text
AddMyText in the filler box.
SetSubmit "object-name":"AddMyText"
Here you can fill the button named "object-name" with the text AddMyText
in the filler box.
SetCheckBox "object-name":"1"
Here you can fill the checkbox named "object-name" and checked it
in the filler box. 0 = disabled, 1 = enabled
SetFillerBoxToAutomatic "1"
Enable the automatic filler. So every command in the filler box will be check
every loaded website. 0 = disabled, 1 = enabled
SetSubmit "object-name":"Submit"
Here you can fill the checkbox named "object-name" and with the caption
Submit in the filler box.
SetBrowserToFillerBox "1"
Enable the browser to filler when every website visit. 0 = disabled, 1 = enabled
If "1"
// Here you can enter your commands
IfEnd
With this command you can check if a result is true or false that if start some
commands only. An example is if you check if a text is exists then you fill
the form only. This command support now: SearchTextInSourceToResult, SearchTextInSourceToFollow,
SearchNoTextInSourceToFollow and SearchNoTextInSourceToResult.
if variable support / 1 = YES, TRUE - 0 = NO, FALSE
example: 1 = text found - 0 = text not found
InsertTextInResult "My Text"
Here you can include texts in the result box without start a new line.
SurfToFirstFollowUrl
Surf to the first url in the follow box.
DeleteFirstUrlInFollow
Delete the first line in the follow box.
SetFillerClickerToAutomatic "1"
Enable the clicker from filler box.