Parse-O-Matic: Positional Commands for Parsing Data Files

Cols

Format	v1 =Cols v2 v3 [v4 ]
Example	MyVar =Cols OtherVar 10 20 ;Columns 10 to 20
Purpose	Copies a range of columns (i.e.character positions)
Parameters	v1 -Variable being set v2 -Value (usually a variable)being copied v3 -Starting column v4 -Ending column
Defaults	v4 =v3 (i.e.copy one character)
Similar Cmds	Equals (Set Variable)with a range specified
Notes	If v3 is less than or equal to 0,it is treated as 1. If v3 points to a position beyond the end of v2,v1 will be null. If v4 points to a position beyond the end of v2,it is treated as if it was the same as the length of v2.

FindPosn

Parse-O-Matic Scripting supports several types of comparators:

Format	v1 =FindPosn v2 d3 [v4 ]
Example	MyVar1 =FindPosn 'ABC''BC';Set MyVar1 to '2' MyVar2 =FindPosn 'ABCC''>*C';Set MyVar2 to '4'
Purpose	Find the character position of text
Parameters	v1 -Variable being set v2 -Value being searched d3 -Decapsulator v4 -Decapsulator control settings
Controls	Exclude/Include;IgnoreCase/MatchCase
Defaults	v4 ='Include MatchCase'
Similar Cmds	ScanPosn
Notes	If nothing is found,v1 is set to '0'(zero). If the “Exclude ” decapsulator setting is used,FindPosn will point to the character position after the string it finds.

ScanPosn

Format	ScanPosn v1 v2 v3 v4 [v5 ]
Example	See below
Purpose	Searches v3 for the start and end columns (character positions) for one of the strings or patterns listed in v4.
Parameters	v1 -Variable being set:“From ” column v2 -Variable being set:“To ” column v3 -The value being searched v4 -The list of strings or patterns for which to search v5 -Control settings
Controls	Any/First/Last;IgnoreCase/MatchCase;RegExp
Defaults	v5 ='Any IgnoreCase'
Similar Cmds	FindPosn,Parse
Notes	Sets $Success ('Y'=something was found). If nothing is found,v1 and v2 are both set to '0'(zero). If RegExp is included in the control settings,each string is treated as a regular expression pattern rather than an actual string.

When you are analyzing data,a common requirement is to find out if one of several strings can be found inanother string.For example,you might want to find out if a name starts with a salutation (Mr.,Mrs.,Ms.). ScanPosn lets you perform such a search with a single command.

For example,to search for a salutation in a string:

ScanPosn from to MyVar '/Mr./Mrs./Miss/Ms.'

If MyVar contains one of the scanterms (e.g.'Mrs.')in the scanlist,ScanPosn will set the appropriate “From ” and “To ” variables..Thus,if MyVar contains 'Ms.Mary Jones',the “From ” variable is set to '1'and the “To ” variable is set to '3'(since 'Ms.'goes from positions 1 to 3 in MyVar).

If none of the scanterms is found,the “From ” variable is set to ''0'and the special variable $Success is set to 'N'.Thus,if MyVar contains 'John Smith',no salutation is found,and the ScanPosn command shown above will set the “From ” variable to ''0'.

The Scanlist

The scanlist can contain one or more scanterms.The first character in the scanlist is interpreted as the delimiter (separator)for the scanterms.Thus,the following scanlists are all valid:

'/Mr./Mrs./Miss/Ms.'	;Delimiter is:/
'xMr.xMrs.xMissxMs.'	;Delimiter is:x
'@Library@School@Gymnasium@Clinic/Hospital'	;Delimiter is:@
'/Cow.'	;Delimiter is:/

The first example ('/Mr./Mrs./Miss/Ms.')has already been demonstrated.The second example uses the letter 'x'as a delimiter.This might be a bad choice for a delimiter;it would cause a problem if one of the scanterms contained an 'x',since it would be treated as two scanterms.For example:

'xJohnxTrixiexFred'

The name 'Trixie'contains an 'x',so it would be broken down into two scanterms ('Tri'and 'ie').You should
always choose a scanlist delimiter that does not appear in the list of scanterms.

Accommodating Variation

When you design a scanlist,you should take into account the possibility that the input might contain strange variations.Consider this command:

ScanPosn x y 'Mr John Smith''/Mr./Mrs./Ms.'

This search will fail because the 'Mr'is followed by a space,not a period.A more forgiving command would be:

ScanPosn x y 'Mr John Smith''/Mr./Mrs./Ms./Mr /Mrs /Ms '

This would successfully locate the 'Mr 'string,and set x to '1'and y to '3'.(The '3'points to the space.)

Handling Prefixes and Suffixes

When designing a scanlist,you should consider that a scanterm might be part of a word.For example:

ScanPosn x y 'Mississippi Sue''/Mr./Mrs./Miss/Ms.'

This will find the 'Miss'in Mississippi,even though this is not part of a salutation.A more appropriate command would be:

ScanPosn x y 'Mississippi Sue''/Mr./Mrs./Miss /Ms.'

The space after 'Miss'in the scanlist ensures that if it is found,it will be separate from any word following it.

The trailing space is not necessary for the scanterm 'Mr.',since no word contains a period.However,if you do include spaces after the periods (as in '/Mr./Mrs./Miss /Ms.')the consistency of rationale may simplify your subsequent script code.

You must also take suffixes into account.For example:

ScanPosn x y 'Zinc Enterprises''/Inc/Co/Enterprises'

This will find the 'inc'in 'Zinc'.You can add a space in front of each scanterm to ensure that it is separated from any other word:

ScanPosn x y 'Zinc Enterprises''/Inc/Co/Enterprises'

You may be tempted to always put spaces on both sides of a word,to handle both prefixes and suffixes. However,consider this example:

ScanPosn x y 'Wazoo Inc''/Inc /Co /Enterprises '

None of the scanterms is found,because the 'Inc'in the source string does not end in a space.The control settings (described next)can help you address this kind of problem.

Control Settings

Unless otherwise instructed,ScanPosn will find the first scanterm that appears anywhere in the source string, and return its start and end positions.It will also ignore text case (e.g.'CAT'='Cat').You can modify this behaviour by using the optional control setting.

Last,First and Any

The 'Last'(i.e.rightmost)control setting tells ScanPosn to find the scanterm that has the highest “To ” value with the lowest “From ” value.This means that all of the scanterms are evaluated.Consider this command:

ScanPosn x y 'SHREWxxxCATxxxMOUSExxx''/CAT/DOGGY/MOUSE/ELK''Last'

ScanPosn finds 'CAT',but continues looking to see if there are any better matches to the right.Eventually it finds MOUSE and sets x to '15'and y to '19'(pointing at 'MOUSE').

If you use the 'First'(i.e.leftmost)parameter,ScanPosn will check all the scanterms to find out which one has the lowest “From ” position with the highest “To ” value..For example:

ScanPosn x y 'SHREWxxxCATxxxMOUSExxx''/CAT/DOGGY/MOUSE/ELK''First'

This will set x to ''and y to '11'(pointing at 'CAT').

If you do not specify 'First'or 'Last',ScanPosn assumes you mean to use the 'Any'control setting.It finds the
first scanterm it can,and ignores the rest.Here is an example.

ScanPosn x y 'SHREWxxxCATxxxMOUSExxx''/CAT/DOGGY/MOUSE/ELK'

The first scanterm is 'CAT',and this can be found at positions 9 to 11.ScanPosn will return those values, and ignore the rest of the scanterms.

The 'Any'technique is useful if you want to know if one of the scanterms appears in the source string,but you are not interested in finding out which one.(You can specify 'Any'explicitly,but since it is the default control setting,this is not necessary.)

The “Best Match ”Principle

Note:The “Best Match ” principle does not apply to the ''Any'control setting.It applies only to 'First'and 'Last'searches.

To use the ScanPosn command effectively,you must understand the concept of 'the best match'.This can be illustrated with an example:

ScanPosn x y 'MegaWhizco International''/CO/WHIZCO/MEGAWHIZ''Last'

The ScanPosn command finds the scanterm 'CO'at positions 5 to 6.However,it continues looking for an
even better match.

It finds that 'WHIZCO'is just as far to the right (i.e.it ends at position 6),but has a lower starting position.
This makes it a better match.

The next scanterm ('MEGAWHIZ')has a lower starting position,but its ending position is not as good for a
'Last'search because it is not as far to the right.

As a result of all this,ScanPosn will set x to '5'and y to '10'— pointing to the “From ” and “To ” columns for 'WHIZCO'.

In other words,when ScanPosn is looking for the 'Last'scanterm,it will first identify the found scanterms which have the highest ending position,and then choose the longest one.

Here is an example using a 'First'search:

ScanPosn x y 'Our catalog is enclosed''/CAT/MOOSE/CATALOG/DOG''First'

ScanPosn finds 'CAT'at positions 5 to 7,but as it continues checking the scanterms,it finds that 'CATALOG'is just as far to the left (i.e.it starts at position 5),but it is a better match since it has a higher ending position.

As a result,ScanPosn will set x to '5'and y to '11'.

The “Best Match ” principle does not affect ''Any'searches.For example:

ScanPosn x y 'Our catalog is enclosed''/CAT/MOOSE/CATALOG/DOG'

This sets x to '5'and y to '7'.Since this is a 'Any'search,ScanPosn stops looking as soon as it has found a match.

When doing an 'Any'search,you cannot be sure if any of the other scan terms appear in the source string.
For example:

ScanPosn x y 'Our cat and dog are upstairs''/CAT/DOG'

This will find 'CAT'and stop looking for additional matches.If you change the order of the scanlist,you will get different values:

ScanPosn x y 'Our cat and dog are upstairs''/DOG/CAT'

This would give different values for the “From ” and “To ” variables..This is normal behaviour;an 'Any'
search is useful only for detecting if one of the scanterms appears in the source string.After doing an 'Any'
search,you will typically check the special variable $Success to see if a string was found.

Finding Patterns with ScanPosn

You can include the control setting “RegExp ” (meaning “Regular Expression ”)to indicate that ScanPosn should look for a pattern of characters rather than specific characters.For example:

;Scale ----+----1----+----
Source ='Kitty Cats Are Cool'
ScanList ='/c.t/co*l'
ScanPosn p1 p2 Source ScanList 'First RegExp'
ScanPosn p3 p4 Source ScanList 'Last RegExp'

This would set the following values:

p1 =7
p2 =9
p3 =16
p4 =19

Regular Expressions are explained in the “Comparators ” section of the user manual..

(This page is part of the online user manual for Parse-O-Matic. Parse-O-Matic is a programmable parsing tool that can extract, manipulate, convert or mine existing data sources and turn them into importable data. For more information on Parse-O-Matic products and conversion services, please visit www.ParseOMatic.com)