June 23, 2011

The Fuss About Programming Anti-patterns

Andrew Koenig started talking about programming anti-patterns more than 15 years ago. It probably never got as popular as it should have been because it addresses the mistakes in syntactically correct software and we can imagine that the number of wrong ways to do something might be an inexhaustible list compared to the right ways. I still think it is a useful concept. Even experts are known to make mistakes, all the time. Majority of the performance enhancements I came across in my limited experience in fact fall under bug fixes, whether it is because the solution wasn't designed well enough or wasn't implemented well enough.

Given a system, improving its performance in general corresponds to improving the performance of its slowest component. This needs the identification of the slowest component to begin with using profiling at some level. Profiling, being time-consuming, isn't always done until a system (an operation on the system) feels poorly performant, and such feelings get mixed up and get slower to perceive as the thickness (complexity) of a system increases.

For this reason I like the approach of checking for known programming anti-patterns using static code analysis. Static code analysis may get slower than build times, but is faster than test suite execution on built images. For now. The day the list of anti-patterns grows infinitely long, and along with it the static code analysis cycle time, the approach won't be feasible (though tiering of anti-patterns based on severity and history of occurrences seems promising).

Hence my interest in static code analyzers like Coverity and FindBugs, though I am yet to explore them well enough to actually know them. I am not aware of any major work along similar lines for Shell and Perl. I think it's worth exploring because there's a significant amount of software written in them (definitely in Symantec Corporation) and they're already slower than compiled languages.

Shell Scripts
The famous todo software is fully written in Bash. It has a lot of anti-patterns, a few of them being various usages of "echo | grep", "echo | sed", "echo | tr" itself. The longest piping anti-pattern in todo.sh is of the form: "echo | sed | eval sort | sed | awk | sed | eval cat". It takes a list of todo items (echo), pads them appropriately with leading zeroes (sed), sorts them (eval sort), color codes the done items (sed), does something more related to color coding (awk), nullifies a few strings (sed), and then gets the final list (eval cat usually). Doesn't seem like the most natural way to do. If nothing, and if my superficial understanding isn't totally off, all the seds can be merged into the awk.

The Shell scripts that are part of the Cygwin installation on my computer have over a hundred anti-patterns of type "echo | sed" alone, and another hundred of "expr", without having all the packages. Bash Completion is one of the packages which might benefit significantly from fixing Shell anti-patterns, and it might translate into better response times. e.g. Command completion for "gcc --" with gcc v3.4.4-999 and bash-completion v1.3-1 creates 12 new processes, 6 of them due to a sed, sort and tr and their prerequisite bashes.

Perl Scripts
Fedora users will be familiar with dvd::rip, largely written in Perl. Even if one-tenth of its $command lines refer to Unix commands, they account a large number of anti-patterns. I'm not familiar with the command-line utilities related to multimedia, but there seem to be several avoidable usages of "cd", "convert", "echo", "ffmpeg", "ls", "mkdir", "rm", "umask", "which". (I've not gone through the source code, but glanced at the parsed code -- which I'll soon come to -- so I'm likely to be mistaken.)

On Solaris, the SUNWwebminu package (11.10.0 version that shipped with Solaris 10) has a surprising number and a wide variety of Perl anti-patterns using -- cat, chown, cd, cp, echo, find, grep, hostname, mv, ps, pwd, rm, rsh, sed, ssh, uname -- you name it. It could be the package that benefits the most from an overhaul in this direction.

The above mentions are only examples of programming anti-patterns out there in the vast universe of software that is being written, shipped, used. That is understandably because even experts are known to make mistakes, all the time. What we need to work on are mechanisms that can minimize those.

Below is a table of counts of common Unix commands found in scripts across various OS installations that I had access to. They are incomplete and likely full of false negatives, and the OSs were not full installations. I'm sure you understand my preference to not hit you with the versions, package names, their versions, etc. They don't mean much. They don't mean nothing either.

AIX PerlAIX ShellCygwin PerlCygwin ShellHPUX PerlHPUX ShellRHEL PerlRHEL ShellSolaris PerlSolaris Shell
Packages25Packages103Packages19Packages95Packages27Packages56Packages86Packages175Packages42Packages227
Total Files2880Total Files1832Total Files2361Total Files1563Total Files7630Total Files7533Total Files1262Total Files2518Total Files5600Total Files5271
Size (MB)27Size (MB)13.5Size (MB)25Size (MB)8.5Size (MB)57.75Size (MB)84.25Size (MB)11.5Size (MB)13Size (MB)28.25Size (MB)20.5
Files421Files759Files285Files403Files796Files692Files313Files673Files684Files963
Total1935Total20684Total257Total2370Total841Total23137Total255Total9733Total1566Total13907
CommandCountCommandCountCommandCountCommandCountCommandCountCommandCountCommandCountCommandCountCommandCountCommandCount
/dev/null410/dev/null5281/dev/null89/dev/null531/dev/null248echo5776/dev/null105echo2539/dev/null1088echo2823
grep326cat5105cat41sed381cat105cat3948cat26/dev/null2037rsh190/dev/null2605
echo137grep2607pwd30echo371echo85sed3869ifconfig22sed1582mount167grep1431
rm99echo2414echo27cat226rm81/dev/null3628echo20grep1232cat161sed1419
mkdir93awk2383cp15basename193grep50grep2349find18cat1112cd131cat1224
hostname92cut1348hostname12grep185hostname46awk1011uname15awk281umount100expr742
ls89sed1087grep12awk137pwd44eval977ls14uname257uname73awk733
awk85rm579date12expr116cp44expr607grep14expr244df72cut528
cp81expr466cd12dirname68find41uname401pwd12eval237ps61basename466
uname71ls344chmod11sort62ps31basename389ps12sort211rm59uname406
date69sort290bc10uname59uname29rm377cp12ls181ifconfig59cd327
cat62egrep290ls9ssh52cd27cut363date11cut175hostname59dirname310
cd58basename277rm8date50ls18egrep324sort8basename174pwd39ls305
find55cp233find8cut48eval18ps262hostname7find161touch35pwd277
egrep49wc228tr6pwd44date18cp249mount6egrep148echo35rm272
pwd43tail212eval6eval42bc18ssh193mkdir6date142ssh34egrep268
ifconfig40head198uname5ps40ssh16ls182basename5tail105find34eval259
basename38cd186ps5hostname27tr15pwd161mv4pwd87cp31mount251
mount35tr172sed4cd27netstat15find157eval4cd81ping25sort197
rsh34ps167netstat4egrep26kill15sort154cd4dirname75grep18date192
head34mount165cut4wc22sed14tr146wc3ps74tr17find120
sed30hostname161wc3chmod19awk13date142tr3tr72ls16hostname119
cut28uname155sort3ls18sort10wc122ping3rm71date16ps91
tail26df146head3tr17touch9cd104ln3wc65chown15tr89
netstat26date144dirname2kill17ln9tail98df3head54pkginfo13head73
umount25mv135chown2rm15chown9head96cut3uniq49mv12pkginfo70
mv25mkdir123ping1find15mv8dirname82awk3kill36chmod11mkdir70
ps21fgrep122mv1tail13mkdir7kill70rsh2hostname31tail8wc65
chmod21dirname121df1umask12cut7hostname68rm2mount30sort8tail65
kill16kill118cp12head4bc65head2netstat27prtvtoc8fgrep65
dirname16find94touch9du4fgrep49chown2mkdir26kill7cp60
ping13pwd93mkdir8dirname4ln48umount1ln21sed6kill55
ln12umount76head8chmod4uniq45touch1umask19eval6df49
wc11eval71mount4tail3mount36kill1cp19mkdir5mv48
sort11ln65fgrep4ifconfig3mkdir35egrep1mv17ln5svccfg42
tr10uniq57uniq3egrep3touch26dirname1touch16head5chmod40
touch10du43sleep3scp2sleep24chmod1pkginfo15dirname5pgrep39
df9chmod43rsh3ping2mv21ssh14cut5ssh38
ssh7netstat42pgrep3mount2chmod17scp10wc4uniq36
eval7umask24ln3df2umask13chmod7netstat4ln32
chown6touch21bc3cksum2ifconfig11umount6expr2umask29
rcp5ifconfig19mv2basename2chown11ifconfig5du2ifconfig27
vmstat2rsh13chown2wc1scp10sleep4uniq1chown23
uniq2ping12ifconfig1umount1ping9fgrep4scp1touch22
scp2ssh7netstat9psrinfo3basename1svcadm18
fgrep2sleep7du6ping3umount16
bc2bc6umount5pgrep2ping13
svcs4pkginfo4df2svcs11
pkginfo4df4vmstat1cksum11
cksum3cksum4sar1bc10
chown3svcs3isainfo1sleep8
vmstat1psrinfo1du1isainfo7
bc1cksum1netstat5
basename1chown1du4
awk1bc1iostat3
/dev/null1prtvtoc2
sar1
psrinfo1

Anti-patterns Parser
Symantec Corporation gave me permission to share this study with the community. Here I am with a hope to widen this discussion and learn something. As part of it, anti-patterns.pl is a dirty parser that I wrote and extensively used along the way, and somehow embarrassed of. I'm sharing it only to convey a better idea of how easy it is to catch some of the programming anti-patterns. Before using it, understand that the parser is very incomplete, incorrect (defined how?), without any warranties or guarantees, and all that cal. I won't even recommend using it, but do take a look.

From all the code that I've read so far, I can see this barely scratches the surface. Apart from continuing to find and add programming anti-patterns in Shell and Perl, my next steps are to move on to Java and C in line with my personal and company interests. Please point me in possible directions and reach out to me if you share my interests.

June 10, 2011

Programming Anti-patterns in Perl

I will spare you with another round of the first half of this post and jump straight to the table of anti-patterns and alternatives in Perl. Check out alternatives.pl for detailed examples. These barely scratch the surface for a rich language like Perl, whose motto itself is TMTOWTDI. My Perl knowledge is still amateurish and these concentrate almost exclusively on the "Accessing memory << Calling a function << Forking a process" guideline. This might be useful for anybody switching from Shell to Perl. So I hope you'll have a lot more to contribute.

30th June, 2011: TimeRatio is the ratio of time taken by Alternative to that by Anti-pattern, as taken from two different trials, using Perl 5.8 on Solaris 10 (Sun T5120). I hope these time ratios will highlight better why especially some of the anti-patterns are to be avoided. My suggestion is to not take these numbers on face value.

Anti-patternAlternativeTimeRatio1TimeRatio2
awkopen, split, close0.780.79
basenames/.*\///859.47872.42
catopen, close28.4828.72
cdchdir1588.341577.63
cpuse File::Copy "cp"1.851.87
chmodchmod (Perl)335.75343.30
cutsplit306.35307.05
datelocaltime200.32214.04
dirnames/\/^\/*$//325.41324.56
echoprint4008.824006.22
finduse File::Find0.550.55
grepopen, grep (Perl), close2.562.57
headopen, break, close42.1242.60
hostnameuse Sys::Hostname10361.9410451.25
idgetpwnam43.3843.28
killkill (Perl)55.0450.84
ln -ssymlink1010.70995.73
lsopendir, closedir32.8724.17
mkdirmkdir (Perl)13.2112.93
mkdir -puse File::Path9.859.87
mvuse File::Copy "move"32.5332.65
pinguse Net::Ping1.040.65
ps -elfuse Proc::ProcessTable3.843.84
pwd$ENV{'PWD'}3575.623742.53
rmunlink75.0574.07
rmdirrmdir (Perl)20.1319.88
rm -ruse File::Path9.009.02
sedopen, s/find/replace/, close25.5425.76
sleepsleep (Perl)2825.822824.30
sortopen, sort (Perl), close4.314.18
tailopen, close2.712.69
touchopen, close65.4165.41
umaskumask (Perl)7749.437742.40
unameuse Config13662.5613652.62
uniqopen, unless seen, close2.332.35
wc -lopen, close6.796.82

June 09, 2011

Programming Anti-patterns in Shell Scripts

How to create a new file from inside a Shell script?
touch newfile1
# OR
echo > newfile2
#OR
: > newfile3

Which of these is better? "touch" is a Unix command, "echo" and ":" (No-Op) are built-in Shell functions. If you "time" the above operations, you will see that the first is very much slower than the second is slightly slower than the third. I'll spare you the reasons.

In my opinion this is a difference that is not highlighted enough. I have seen thousands of lines of "shell script" which are indistinguishable from "one-liners" typed at the command-line. The reason anyone writes something inside a script and not at the command-line is the assumption that it will be used more than once, and that alone is reason enough for "efficiency" to be a criterion. Nothing slows down a script like a bunch of one-liners in it. And scripts are slow to start with.

Contrary to popular belief a Shell script is not a file with a bunch of lines each with some combination of Unix commands (called "commands text"). Shells are like all other languagues, with many useful features. I have a few unoriginal guidelines for writing scripts (Shell, Perl, Batch, ...):
1. Use environment and internal variables. (Like $HOSTNAME, $MACHTYPE, $PWD in Bash.)
2. Use built-ins and libraries. (Shells have a type command to verify the existence of an eqivalent built-in.)
3. Self-parse text. (Shells have built-in string manipulation and substring extraction functions with regexp support.)
4. Combine operations. (Many commands, internal and external, can take multiple arguments.)
5. Check fork rampage and /dev/null redirections. (They are used far more than necessary.)
6. Refer the manual. (Again.)

Moving on to the more useful part. Below is a table containing many anti-patterns that can be found -- I came across each of them numerous times -- in Shell scripts along with suggested alternatives. It is a work in progress. I hope you find it useful and I hope you will help add more rows ot it. Check out alternatives.sh for detailed examples of the anti-patterns and alternatives. It should work with various other Shells as well with minor changes. Combinations of these anti-patterns are common, and alternatives aren't necessarily the most efficient, nor are they efficient all the time.

30th June, 2011: TimeRatio is the ratio of time taken by Alternative to that by Anti-pattern, as taken from two different trials, using Korn Shell 93 (dtksh) on Solaris 10 (Sun T5120). I hope these time ratios will highlight better why especially some of the anti-patterns are to be avoided. My suggestion is to not take these numbers on face value.

Anti-patternAlternativeTimeRatio1TimeRatio2
awk '{print}'read -A1.311.32
awk | grepawk1.091.09
awk | sedawk1.531.53
awk | sortawk2.182.18
basename${string##*/}2.442.44
cat | awkawk1.11.1
cat | grepgrep1.381.36
cat | headhead21.5921.46
cat | readread1.451.47
cat | sedsed1.171.17
cat | tailtail1.61.5
cat |wcwc15.2115.45
/usr/bin/cdcd407.87413.4
dirname${string%/*}2.182.3
/usr/bin/echoecho99.28130.35
echo | awk($string)212.72159.63
echo | cut($string)100.83100.25
echo | cut | cutsubstrings187.91185.18
echo | grepif [[ $string == regexp ]]381.61378.55
echo | sed${string/find/replace}341.36329.14
echo | wc -m${#string}161.39165.7
echo | wc -w($string); ${#array[*]}85.6985.9
grep | awkgrep1.071.07
grep | grepsed11.01
grep | sedsed1.021.03
grep | grep -vsed1.011
grep | wcgrep -c0.990.99
headread, break1.21.21
head | awkread -A, break66.7268.4
/usr/bin/killkill3.755.79
ls | awk(ls)1.041.23
/usr/bin/pwd$PWD305.72271.46
sed | sedsed1.091.05
tail | awk(tail)1.481.44
/usr/bin/testtest210.81203.97
/usr/bin/trueTRUE496.71452.64
/usr/bin/ulimitulimit180.74167.1
/usr/bin/umaskumask180.16166.78
uname$ENV3.243.21
wc | awk(wc)13.5311.67

June 07, 2011

Programming Anti-patterns

TMTOWTDI is often a trait of the problem that one is trying to solve, and not just the programming language being used. Solutions can be simple or complicated, efficient or inefficient and an invisible constraint of solving problems (implementing) is to find a solution that is in the neighborhood of simple and efficient. Internet helps in finding a solution that is popular or common, and not necessarily efficient.

When I started noticing lots of inefficient code and identified a few patterns across them, I embarked on a study of programming anti-patterns. The definition I follow is, "Commonly used inefficient programming practices to which better alternatives are known".

For my study I also need anti-patterns to be easily identifiable so that dummies like yours truly can identify them, for comparisons between anti-patterns and alternatives to be easily made, for static code analyzers to fish them out, and for changes to be relatively easy to implement. Bill Pugh once said (probably in this) that the goal of FindBugs is to find bugs that are one step away from compilers. My goal is similar, only my focus is on efficiency more than correctness. (There are other kinds of anti-patterns, most famously design anti-patterns, but my focus at present is on code efficiency itself.)

The problem of looking for a new anti-pattern (source) is non-trivial. A few guidelines that I am considering are:
1. Accessing memory << Calling a function << Forking a process: There are exceptions to this, but to my understanding it's mostly true. Smaller the operation that needs to be performed, larger the overhead of doing it in round-about methods. It's easier to check the forking of processes as explained earlier. Re-calling functions is apprently not very uncommon, but I'm yet to scratch the surface of calling functions vis-a-vis reusing their return values.
2. Legacy code: In my experience code is more often added to, and old code isn't as much "updated" as is "fixed". Resultantly, "legacy code" to a great extent contains code written ages ago. Languages on the other hand constantly evolve, and the things of interest here are newly introduced, say functions, that can do something better. (The first example that comes to my mind is the rampant presence of += for string concatenation in Java instead of StringBuffer or StringBuilder.)
3. n00bs: Bill Pugh mentioned in the same video that, not surprisingly, people new to a programming language are a great source of learning how not to do something (even though it is not syntactically wrong). I have personally seen this to be true. A trickier subset of this is anti-patterns brought in when one migrates from one programming language to another.

June 03, 2011

NewProcs: A High-level System Activity Indicator

Operating systems are fairly multi-tasking and processes are their major, most visible components. In line with Heisenberg's Uncertainty Principle, which I see everywhere these days, observing processes can be more accurate than observing any smaller components. In my job I mostly observe process-level behaviors; at least I begin there. Recently I started staring from one more level zoomed out -- the lifetime of a process -- and insensitively asking, "Should this process have been created?"

It's a question that is sometimes easy to decide, but not always. It is like an Age of Empires player asking herself, "Should I create a new villager?". It depends on the tasks pending, the resources available, the world and the neighborhood into which he might be born, the greater purpose his long-term life could serve, and even the nature of the creator.

One small step towards deciding this is the knowledge of process lifetimes, their births and deaths, and so a measurement that I now routinely take is "NewProcs". On Windows, Process Monitor through filters "Operation is Process Create" and "Operation is Process Exit" does a beautiful job of this. On Solaris the following DTrace one-liner will do: dtrace -qn 'proc:::exec-success, proc:::exit {printf("%d\t%d\t%Y\t%s\n", pid, ppid, walltimestamp, curpsinfo->pr_psargs)}'. On OSs with SystemTap the following one-liner will do: stap forktracker.stp (the file should be located in the SystemTap examples directory). On Unix-based OSs, I have a feeling GNU Acct can serve as a workaround.

One major area that can take advantage of this is "scripts" - be they Batch scripts or Shell scripts or Perl scripts. Especially in enterprise software, perepheral operations like installations and configurations use a lot of scripts. I will try to elaborate on that in the near future.

The number of processes being created and killed itself isn't necessarily a good indicator, but the detailed list can occasionally throw a lot of light on the high-level system activity during various operations. Below are a few examples (naïve layman's curiosities):

1. On Windows 7 and Windows Server 2008 (probably Windows Vista as well), Explorer can navigate the calendar, connect to various networks, switch battery power plans, browse and search for various programs all without creating any new processes. However the Speakers/Headphones icon in the system tray needs to create a new process, SndVol, for any basic volume change operation.

2. Windows Media Player Network Sharing Service Configuration Application (wmpnscfg) is one inexplicable troll. It keeps appearing during scenarios where its existence isn't all that clear. e.g. Whenever one connects to some networks, several short-lived wmpnscfg processes get created. A disconnect is usually associated with another instance.

3. When one looks for the version of Google Chrome (at least with 11.0.696.71), the "About Google Chrome" dialog creates a GoogleUpdateOnDemand process which goes on to create two more processes: GoogleUpdate /ondemand, GoogleUpdate -Embedding. The browser also has a nice feature of Gmail notifications. Unlike notifications from Microsoft Office Outlook, each Gmail notification comes through a separate procress (chrome -type=renderer, the tab process). Google Chrome is known to be a "multi-processed" application for reasons of stability and security, but.

Cygwin (and MinGW) earnestly takes up its share of process creation as explained here. I'm sure everyone has their reasons, but high birth rates and infant mortality rates always concern me. Should these processes have been created? What do you think?

June 01, 2011

vxperf2 Examples

1st July, 2011: GitHub is the new home of vxperf2. Download vxperf2.

Let us take a sar log which was first collected as a binary using -A and -o options and then dumped as a text database using -A and -f options. The log is almost in a text database format, with several tables in it. It has one OS-specific header and many "Average:" lines that are not of our interest because we're interested in subsets of the duration in which the logging was done, and not the entire duration.

So we first convert the log to a table using log2db.pl from the command-line or log2db module (modifySysstat subroutine) inside a Perl script. sardb is the text database file that we'll use from here onwards.

Example 1
Suppose we want a summary of CPU utilization (just %usr, %sys), load averages (ldavg-1, ldavg-5, ldavg-15), faulting (fault/s, majflt/s) and context switching (cswch/s). Our rules file would be something like:

# rules1: Rules file for Example 1.
# Any line in the rules file that does not start with a keyword is ignored.
y-axis: %usr %sys ldavg-1 ldavg-5 ldavg-15 fault/s majflt/s cswch/s
# End of Example 1 rules file.

We get the results from sardb and rules1 using vxperf2.pl from the command-line or vxperf2 module (parseLogFile and applyRules subroutines) inside a Perl script. In the results directory, a summary.txt (name hardcoded) is created that contains one line each for %usr, %sys, ldavg-1, ldavg-5, ldavg-15, fault/s, majflt/s and cswch/s reporting their functions (currently min, max, sum, count and average).

Example 2
I glossed over one aspect of the CPU utilization summary in the previous example, which is that -A option gave per-processor statistics as well as an "all", and we went ahead with clubbing them together. It isn't exactly a mistake in this case, but suppose we do want to separate the per-processor statistics. Our rules file would be something like:

# rules2: Rules file for Example 2.
y-axis: %usr %sys ldavg-1 ldavg-5 ldavg-15 fault/s majflt/s cswch/s
z-axis: CPU
# End of Example 2 rules file.

There can be only one z-axis rule, and only one field for it. vxperf2 takes the field following the z-axis keyword, and divides the CPU utilization table into sub-tables, one for each distinct value of "CPU". Along with all the lines of the previous summary.txt, the new file will now contain a few additional lines each for %usr and %sys, lines corresponding to the processors and "all".

Example 3
Suppose we want a couple of graphs, one for CPU utilization and one for load averages. Our rules file would be something like:

# rules3: Rules file for Example 3.
y-axis: %usr %sys ldavg-1 ldavg-5 ldavg-15 fault/s majflt/s cswch/s
z-axis: CPU
plot: %usr %sys
plot: ldavg-1 ldavg-5 ldavg-15
# End of Example 3 rules file.

vxperf2 takes the fields following the plot keyword on one line and plots them using Gnuplot, naming it $field1-$field2-...-$fieldn.png. As many images as there are plot keywords in the rules file are created. The CPU utilization image will have per-processor and "all" plots because of the z-axis keyword. In this particular case, however, the second image will be empty as there are no per-processor load averages. (Instead, dropping the z-axis keyword will allow plotting of load averages. A limitation.)

Example 4
In the %usr-%sys.png created in the above examples, there are per-processor %usr plots and one "all", and similarly for %sys plots. What if we are interested in timelines of only a few of the processors, or not interested in "all"? A similar question of more significance would be when looking into block device or process-level statistics. Our rules file would be something like:

# rules4: Rules file for Example 4.
y-axis: %usr %sys ldavg-1 ldavg-5 ldavg-15 fault/s majflt/s cswch/s
z-axis: CPU
plot: %usr %sys
only: 0 all
# End of Example 4 rules file.

There can be only one only rule, and it makes sense only in the presence of a z-axis rule (z-axis has a field name and its corresponding values are a superset of only). vxperf2 this time only plots %usr and %sys corresponding to processor 0 and "all", though it summarizes for other values of CPU as well.

Example 5
It is common to start logging a few intervals before the period of interest, and to end logging a few intervals after. Suppose we want to ignore the first 10 intervals, and then we are only interested in the next 100 intervals. Our rules file would be something like:

# rules5: Rules file for Example 5.
y-axis: %usr %sys ldavg-1 ldavg-5 ldavg-15 fault/s majflt/s cswch/s
z-axis: CPU
plot: %usr %sys
only: 0 all
offset: 10
points: 100
# End of Example 5 rules file.

vxperf2 does what it did exactly in the previous example, except that it does it over a subset starting at the 11th reading and ending at the 110th.

Simple enough?

I mentioned a couple of limitations above and there are many others, most of which I discover only when using vxperf2, and there are often workarounds to these. vxperf2 module exposes two subroutines parseLogFile and applyRules. parseLogFile parses a text database and stores it as an in-memory database. applyRules parses a rules file and applies the rules to multiple databases at once. One parseLogFile invocation per log file, followed by one applyRules invocation per rules file.