Crawler commands help for "WTM Crawler" by webtoolmaster.com


Example Scripts: - bold are commands

Script 1:
; check if a website work with a special script - search copyright information

; Remove data from result box
ResetResultBox

; Surf to a website
SurfToFollowUrlOnly "http://www.webtoolmaster.com/crawler-examples.htm"

; Check if the copyright information of the script exists on site
; Extract link when yes to the result box
SearchTextInSourceToResult "Powered by Script"

; Check if the copyright information of the script exists on site
; Extract link when yes to the follow box
SearchTextInSourceToFollow "Powered by Script"


Script 2:
; Extract mails from website
; Remove data from result box
ResetResultBox

; Surf to a website
SurfToFollowUrlOnly "http://www.webtoolmaster.com/crawler-examples.htm"

; Extract every mail address to result box
ExtractEMailsToResult


Script 3:
; Extract all link to follow it
; Remove data from follow box
ResetFollowBox

; Surf to a website
SurfToFollowUrlOnly "http://www.webtoolmaster.com/crawler-examples.htm"

; Extract every link to the follow box
ExtractAllLinksToFollow

; Visit every link in follow
FollowAllLinksInFollow "0"
FollowAllLinksInFollowEnd


; here you can make a command of every followed url


bold are commands

SurfToFollowUrlOnly
ExtractLinksToFollow
ExtractAllLinksToFollow
FollowAllLinksInFollow
FollowAllLinksInFollowEnd
ExtractEMailsToResult
ExtractEMailLinkToResult
ExtractEMailTagToResult
GenerateLinksToFollow
GenerateLinksToResult
SearchTextInSourceToResult
SearchTextInSourceToFollow
SearchNoTextInSourceToFollow
SearchNoTextInSourceToResult
GetTextBetweenInSourceToResult
ExcludeTextInLink
OnlyTextInLink
ResetExcludeTextBox
ResetOnlyTextBox
FillTextInBrowser
ClickOnButtonInBrowser
CheckIfWebsiteChangeToFollow
CheckIfWebsiteChangeToResult
AddLinkToFollow
ResetFollowBox
MoveLastLineBeforeInResult
ResetResultBox
RemoveLinesFromFileInResult
RemoveLinesFromFileCSInResult
CheckDoubleLinesIn
Wait
AddTextInResult
SetEdit
ClearFiller
SetText
SetSubmit
SetCheckBox
SetFillerBoxToAutomatic
SetBrowserToFillerBox
If
IfEnd
InsertTextInResult
SurfToFirstFollowUrl
DeleteFirstUrlInFollow
SetFillerClickerToAutomatic

; Remember Texts

When you include a ";" in the first place of a line, than you can use it for texts and information.
; Here you can give comments


SurfToFollowUrlOnly "http://www.yoururl.com/index.htm"

Surf to this url and follow every same root link found on this site. Important is that the url is between "URL".


ExtractLinksToFollow

This extract all links (not mailto and javascript) to the follow box.


ExtractAllLinksToFollow

This extract all links (with mailto and javascript) to the follow box.


FollowAllLinksInFollow "5"

// Here are commands that to do on every visit website. 5 is the deep of every link in follow line.
FollowAllLinksInFollowEnd

This follow all Links in the follow bow. You can give a deep follow setting from 0 to 999999... every link. Standard is often 5. On every site will be do that commands are between this and FollowAllLinksInFollowEnd command on every vistis website.


ExtractEMailsToResult

Extract all founded mails on the visit website to the result box. Example: test@test.co


ExtractEMailLinkToResult

Extract the complet EMail link to the result field. Example: mailto:test@test.co


ExtractEMailTagToResult

Extract the complet EMail link without the "mailto:" to the result field. Example: test@test.co?subject=Comments


GenerateLinksToFollow "http://www.yoururl.com/index.htm?FromPage:$\1:5:1\$&ToPage:$\2:6:1\$"

This command generate different links to the follow box. You can use variables with counter. The command "$\1:5:1\$" replace from number 1 to 5. The same is with $\2:6:1\$. So it replace from number 2 to 6. It generate 5 links in the follow box.
Variable: $\[StartNumber]:[EndNumber]:[StepsBetweenNumbers]\$

Example:
http://www.yoururl.com/index.htm?FromPage:1&ToPage:2
http://www.yoururl.com/index.htm?FromPage:2&ToPage:3
http://www.yoururl.com/index.htm?FromPage:3&ToPage:4
http://www.yoururl.com/index.htm?FromPage:4&ToPage:5
http://www.yoururl.com/index.htm?FromPage:5&ToPage:6

Variable: $\10:50:10\$ - http://www.yoururl.com/index.htm?PageResults:$\10:50:10\$
http://www.yoururl.com/index.htm?PageResults:10
http://www.yoururl.com/index.htm?PageResults:20
http://www.yoururl.com/index.htm?PageResults:30
http://www.yoururl.com/index.htm?PageResults:40
http://www.yoururl.com/index.htm?PageResults:50


GenerateLinksToResult "http://www.yoururl.com/index.htm?FromPage:$\1:5:1\$&ToPage:$\2:6:1\$"
The same then GenerateLinksToFollow, but put the result to the result box.


SearchTextInSourceToResult "My text to search."

This command extract every link that found the text "My text to search." on the website in the result box. You can search links, tags and other information too.
with if variable support / 1 = text found - 0 = text not found


SearchTextInSourceToFollow "My text to search."

This command extract every link that found the text "My text to search." on the website in the follow box. You can search links, tags and other information too.
with if variable support / 1 = text found - 0 = text not found


SearchNoTextInSourceToFollow "My text to search."

This command extract every link that the text "My text to search." do not found on website in the follow box. You can search links, tags and other information too.
with if variable support / 1 = text found - 0 = text not found

SearchNoTextInSourceToResult "My text to search."

This command extract every link that the text "My text to search." do not found on website in the result box. You can search links, tags and other information too.
with if variable support / 1 = text found - 0 = text not found


GetTextBetweenInSourceToResult "My text ":" search."

Extract the text between "My text " and " search." in the same html line to the result box. On "My text to search." is the result "to".
It show then exact the texts between this. When you leave the secound empty then it show to the html line end. "My text":""


ExcludeTextInLink ".com"

This command is a text filter. So it do not visits domains that have .com inlcuded.


OnlyTextInLink "myDomain.com"

This command allowed links on site myDomain.com only. You can use every word you like.


ResetExcludeTextBox

This delete all entries in the exclude text box.


ResetOnlyTextBox

This delete all entries in the only text box.


FillTextInBrowser "FieldName":"InlcudeThisText"

Filled a Text in the named field in the webbrowser


ClickOnButtonInBrowser "FieldName"

Click on a Button in the webbrowser


CheckIfWebsiteChangeToFollow "http://www.yoursites3.com/tocheck.htm":""

Check if the website is changed since last visit. It show every changed url to the follow box.
The program include a checksum at the end after the url name "http://www.yoursites3.com/tocheck.htm":"D3GA26432"


CheckIfWebsiteChangeToResult "http://www.yoursites3.com/tocheck.htm":""

Check if the website is changed since last visit. It show every changed url to the result box.


AddLinkToFollow "http://www.yoursites3.com"

Add this link to the follow box.


ResetFollowBox

Clear all links in the follow box.


MoveLastLineBeforeInResult " /% "

Move the last line to one line before and inlcude " /% " between this to lines in results. When you leave this empty than there are no free space between this both.


ResetResultBox

Delete all data in the result box

RemoveLinesFromFileInResult "c:\myfile.txt"

Remove every line in a selected file that found in the result box.


RemoveLinesFromFileCSInResult "c:\myfile.txt"

Remove every line in a selected file that found in the result box (case sensitive).


CheckDoubleLinesIn "result"
CheckDoubleLinesCSIn "follow"


Check double lines in a selected field. CheckDoubleLinesIn is without case sensitive and CheckDoubleLinesCSIn with.
result = Result Box
follow = Follow Box
exclude = Exclude Box
only = Only in text box


Wait "1000"

The program wait for a select time in ms. 1000 = 1 sec.


AddTextInResult "AddMyText: "

Here you can add a selected text to the result field. Here you can add a title or use the MoveLastLineBeforeInResult "" to get the next command in result in the same line.


SetEdit "object-name":"AddMyText"


Here you can fill the edit field named "object-name" with the text AddMyText in the filler box.


ClearFiller


Delete and clear every filling in the filler box.


SetText "object-name":"AddMyText"


Here you can fill the memo field named "object-name" with the text AddMyText in the filler box.


SetSubmit "object-name":"AddMyText"


Here you can fill the button named "object-name" with the text AddMyText in the filler box.


SetCheckBox "object-name":"1"


Here you can fill the checkbox named "object-name" and checked it in the filler box. 0 = disabled, 1 = enabled


SetFillerBoxToAutomatic "1"


Enable the automatic filler. So every command in the filler box will be check every loaded website. 0 = disabled, 1 = enabled


SetSubmit "object-name":"Submit"


Here you can fill the checkbox named "object-name" and with the caption Submit in the filler box.


SetBrowserToFillerBox "1"

Enable the browser to filler when every website visit. 0 = disabled, 1 = enabled


If "1"
// Here you can enter your commands
IfEnd

With this command you can check if a result is true or false that if start some commands only. An example is if you check if a text is exists then you fill the form only. This command support now: SearchTextInSourceToResult, SearchTextInSourceToFollow, SearchNoTextInSourceToFollow and SearchNoTextInSourceToResult.
if variable support / 1 = YES, TRUE - 0 = NO, FALSE
example: 1 = text found - 0 = text not found


InsertTextInResult "My Text"

Here you can include texts in the result box without start a new line.


SurfToFirstFollowUrl

Surf to the first url in the follow box.


DeleteFirstUrlInFollow

Delete the first line in the follow box.


SetFillerClickerToAutomatic "1"

Enable the clicker from filler box.




Parameter:

/articletool
Open the article tool wizard only

/keywordtool
Open the keyword tool wizard only