Troff Text Processor for Typesetters

"The great strength of troff is the flexibility of the basic language and its programmability -- it can be made to do almost any formatting task."
-- Brian W. Kernighan, Rob Pike - The UNIX Programming Environment

Troff is a document processing system, and it's part of the default UNIX text processing tools.

• troff - for output device that can handle variable-width fonts and different character sizes
• nroff - produces output suitable for simple fixed-width terminals or line printers
• groff - a GNU licensed implementation of the ditroff and all the troff preprocessors
• unroff - a free flexible implementation of troff written in Scheme
  

Even though now it has been superseded by T_EX and L^AT_EX, troff is still quite common, as UNIX manual pages for the "man" command are written in troff, using the "man macros".

"Since bare troff is unusable by humans, a race of gods now gone created macro packages for mortals to use."
-- Brian Kernighan on the troff macro package.

On some systems manpages are stored in compressed formats, so, before we operate on such a manpage, we need to uncompress it.

Let's use groff and the man macros to convert a manpage to ps, and send it to the printer spooler.

% cat /usr/share/man/man7/bootparam.7.gz | gunzip | groff -man | lpr

We could also send it to ps2pdf to publish the manpage as pdf.

% cat /usr/share/man/man1/groff.1.gz | gunzip | groff -man > groff.ps && ps2pdf groff.ps && xpdf groff.pdf"

If you plan on doing anything serious with troff, you should take a look at Bell Labs Computing Science Technical Report #54, Troff User's Manual.

Cisco ASA issues: Configuration line too long

When trying to write the configuration to startup-config or doing something like copying the running-config to a file or remote system (tftp and the like) an error like "Configuration line too long" might occur.

When you issue a command that uses the running-config, like:

# copy running-config startup-config
# wr

or anything that uses the running config, a similar error could occur:

# wr
Cryptochecksum: deadbeef cafebabe feedface %Error
reading system:/running-config (Configuration line too long
Error executing command [FAILED]

This is actually a bug that occurs in version 7.2(1)25 or greater (includes 7.2(2)) of the firewall.

This is actually caused by "snmp-server" commands. One workaround is to remove such lines (you can add them later, it might "trick" it). Not all "snmp-server" lines seem to affect it either. Just remove them one at a time, and wr. See if it works.

Another workaround would be to downgrade to something pre-7.2(1)25 or upgrade to anything more recent than 7.2(2.2) or 8.0(0.141).

Either way, it can be a time consuming issue, and, although it's supposed to be a corner case, it's already happened to me on several occasions... and it took me a while to shotgun-debug my way to the offending line (as you can probably realize, it's NOT the longest line that causes the issue... in fact, it was one of the shorter lines... so it took a while to realize it's caused by the "snmp-server" lines. This was later confirmed by the Cisco bug database).

Problem solving with awk

"It seemed like a good idea at the time."
--Brian Kernighan

Nothing beats good old fashion awk when it comes to problem solving :-). Here is one such example:

[ircuser] is there any command or quick way to do like,say i have two strings, "tes**ng" and "lkmtide" . need to replace the '*' in the first string with the exact characters in the second string. so, "tes**ng" changes to "testing"
[ircuser] cmihai, here the issue is * is not fixed to two chars... it can be from one to len-2

# Author: Criveti Mihai, 2007.

# This program is free software.
# It comes without any warranty, to the extent permitted by applicable law.
# You can redistribute it and/or modify it under the terms of the
# Do What The Fuck You Want To Public License, Version 2
# as published by Sam Hocevar.
# See http://sam.zoy.org/wtfpl/COPYING for more details.

# Usage: cat file | awk -f change_stars_and_stuff.awk > output

BEGIN {
FS = " ";
}

{
string1 = $1;
string2 = $2;

do {
position = index( string1,"*");
finalstring = substr(string2,position,1);
sub(/\*/,finalstring,string1);
} while (position != 0)
}

{
print string1;
}

The input file would look something like this (from what I gather anyway):

cmihai@alap ~/scripts
$ cat file
tes**ng lkmtide
tes***g lkmtinaaa
testing lkmtide
abcdef* abcdefghijklmn

And now, let's take our little script for a test drive:

$ cat file | awk -f change_stars_and_stuff.awk
testing
testing
testing
abcdefg

Well, that did it :-). Good old fashion AWK to the rescue!

OpenBSD RAIDframe mirror - Software RAID

By popular demand, Software RAID mirrors in OpenBSD, using RAIDframe.

For software RAID, OpenBSD has RAIDframe, softraid(4) (but only in 4.2) and ccd(4) (the concatenated disks driver - part of the GENERIC kernel, suppors a mirroring feature and a "striping" effect, but has limitations).

I should warn you that RAIDframe has quite a few problems. It doesn't seem to be properly maintained, code quality is down, it can be quite slow, the configuration process is a PITA and quite low level (as you can probably see), it makes upgrades a difficult and you'd get the same kind of downtime from a backup / restore process (dump/restore). Still, if you're up for it, read the raid(4) manpage, and give it a shot.

The new softraid(4) mirroring is simpler, but it's far from complete or tested as of OpenBSD 4.2.


Step 1. Installing OpenBSD:

The installation is pretty straight forward. You only need to pay attention to the disk labeling part. Create a 512M /, a small swap (128m is fine) and give the rest (a d) to RAID (type RAID in the FS type). Make sure you install compiling tools (comp).

Step 2. Install and boot a RAIDframe kernel

The problem is OpenBSD doesn't ship a RAIDframe enabled kernel by default (in GENERIC or otherwise). And you can't enabled it via config(8) either. So you'll have to build yourself a custom kernel and enable RAIDframe. Read 5.7 - Building a custom kernel of the FAQ for that. Once built, you can use that on all your machines, so it's a good idea to use GENERIC as a template for your RAIDframe kernel. Just uncomment the "pseudo-device raid" and "option RAID_AUTOCONFIG" lines and you're set.

You can use this as an excuse to move to OpenBSD-stable. See http://www.openbsd.org/stable.html - just copy GENERIC to GENERICRAID and edit / config that.

Also remember to turn on SoftUpdates (softdep in /etc/fstab), or it will take forever ;-).

Technically, you could do this on another machine. Once the kernel is built, you can use it to master your own media kits for use in future installations.

Step 3. Configure the mirror disk

You need to clone the disklabel from disk wd0 (or whatever your first disk is) and initialize the second disk (fdisk -i).

# disklabel wd0 > disklabel0
# fdisk -i wd1
# disklabel -R -r wd1 disklabel0

Step 4. Configure the RAID array:

See raidctl(8) and raid(4) for this step. Just setup your /etc/raid0.conf.

An example for a mirror:

# cat /etc/raid0.conf

START array
1 2 0
START disks
/dev/wd0d
/dev/wd1d
START layout
128 1 1 1
START queue
fifo 100

Step 5. Clone the disk

See http://unixsadm.blogspot.com/2007/08/cloning-disk-in-openbsd.html

Step 6. Enable the RAID

# raidctl -C /etc/raid0.conf raid0
# raidctl -I 100 raid0
# raidctl -iv raid0

Step 6. Disklabel and newfs the RAID disk:

# disklabel -E raid0
# newfs /dev/rraid0a

(Newfs created disklabels)

Step 7. Copy the installed OpenBSD on your RAID array:

You should use dump / restore for this process.

# mount /dev/raid0a /mnt
# cd /mnt
# mkdir usr
# mount /dev/raid0d /mnt/home
# dump -0f - / | restore -rf -

(make sure you create mount points and mount all the disk labels you've created).


Step 8. Activate the RAID config:

# raidctl -A root raid0

Remember to edit /etc/fstab to reflect these changes. You should have at least the / line similar to:

/dev/raid0a / ffs rw 1 1

Step 9. Reboot, enjoy, check status:

# raidctl -s raid0

Read the raidctl(8) manpage for details.

Windows Crash Dump and Hang Analysis - BSOD keyboard shortcut

How to debug Windows crashes and hangs:

1) Configure Windows to take a complete memory dump, log the event and make sure it won't automatically restart in case of system failure (so you can see the BSOD).



2) Configure Windows to crash and memory dump on a hot key. In case Windows freezes, it may become necessary to force Windows to crash. Note: the hot key only works with a PS/2 keyboard, not USB, since it uses i8042prt driver that controls the keyboard and PS/2-compatible mouse for the Intel 8042 controller. The system may not respond to the shortcut in some scenarios.

Start regedit, go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\i8042prt\Parameters and add a DWORD value called "CrashOnCtrlScroll" and give it a value other than 0 (like say, 1).

3) Reboot.

4) Wait for the system to hang.

5) Press and hold the _right_ Ctrl key and press ScrollLock twice to force the system to crash and take a memory dump. If you've done everything right, you should get a BSOD similar to this one (the one in the picture is a Windows 2003 R2 / SP2 BSOD):



Now you have a memory dump you can use to debug the system freeze.

If you want to see how to analyze an actual memory dump, see: Windows Crash Dump Analysis - Pinpointing Faulty Drivers with Driver Verifier and WinDbg -
http://unixsadm.blogspot.com/2007/10/windows-crash-dump-analysis-pinpointing.html

Using KrView (KernView) to profile CPU usage by kernel-mode and user-mode processes and drivers

KrView (kernrate viewer) is a profiling tool for tracking CPU utilization by kernel-mode and user-mode processes that can be used to fine tune device drivers.

It can be hard to find out the real resource usage on a system. Take the System process for example, that has Interrupts, DPC's (Deferred Procedure Calls) and such, all calculated by subtracting the total usage from 100.

Also, due to context switches "bellow the radar" that happen between the system clock interval checks (system clock resolution - On Windows it's usually 15 ms on 64bit systems, 10ms on 32 bit systems) the numbers in task manager are hardly accurate.

There are other tools that use the Context Switch Deltas to print out more accurate numbers, but they still don't give you enough details about kernel mode drivers and such.


While other operating systems such as OpenBSD or Solaris have "systat vmstat" and the DTrace script intrstat respectively, Windows has KernRate (part of Windows 2003 rc) and KrView (to interpret and view kernrate data).

One tool that does give you a better idea of what happens in kernel mode and with drivers is KrView - the Kernrate Viewer.

I've got the system pushing quite a lot of data through the network, so, as you can see from the following:

b57nd60x 2214 67227 4 % 823329
tcpip 1128 67227 2 % 419474
ndis 1105 67226 2 % 410927

b57nd60x.sys is the Broadcom NetXtreme Gigabit Ethernet NDIS6.0 Driver. tcpis is the TCP/IP stack and ndis is the Network Driver Interface Specification, a standard API for NICs.

Start - run - cmd ->

"c:\Program Files\KrView\Kernrates\Kernrate_i386_XP.exe" > output.txt && notepad output.txt

C:\Program Files\KrView\Kernrates>Kernrate_i386_XP.exe
/==============================\
<>
\==============================/
Date: 2007/10/05 Time: 14:23:38
Machine Name: ALAP
Number of Processors: 2
PROCESSOR_ARCHITECTURE: x86
PROCESSOR_LEVEL: 6
PROCESSOR_REVISION: 0f0a
Physical Memory: 2046 MB
Pagefile Total: 5032 MB
Virtual Total: 2047 MB
PageFile1: \??\C:\pagefile.sys, 3067MB
OS Version: 6.0 Build 6000 Service-Pack: 0.0
WinDir: C:\Windows

Kernrate User-Specified Command Line:
Kernrate_i386_XP.exe


Kernel Profile (PID = 0): Source= Time,
Using Kernrate Default Rate of 25000 events/hit
Starting to collect profile data

***> Press ctrl-c to finish collecting profile data
===> Finished Collecting Data, Starting to Process Results

------------Overall Summary:--------------

P0 K 0:00:28.251 (42.0%) U 0:00:09.157 (13.6%) I 0:00:29.796 (44.3%) DPC 0:00:10.623 (15.8%) Interrupt 0:00:01.778 ( 2
%)
Interrupts= 370214, Interrupt Rate= 5509/sec.

P1 K 0:00:17.425 (25.9%) U 0:00:09.219 (13.7%) I 0:00:40.560 (60.4%) DPC 0:00:10.436 (15.5%) Interrupt 0:00:01.544 ( 2
%)
Interrupts= 370229, Interrupt Rate= 5509/sec.

TOTAL K 0:00:45.677 (34.0%) U 0:00:18.376 (13.7%) I 0:01:10.356 (52.3%) DPC 0:00:21.060 (15.7%) Interrupt 0:00:03.322 ( 2
%)
Total Interrupts= 740443, Total Interrupt Rate= 11018/sec.


Total Profile Time = 67205 msec

BytesStart BytesStop BytesDiff.
Available Physical Memory , 200392704, 549707776, 349315072
Available Pagefile(s) , 4235825152, 4270665728, 34840576
Available Virtual , 2122371072, 2122371072, 0
Available Extended Virtual , 0, 0, 0

Total Avg. Rate
Context Switches , 949913, 14135/sec.
System Calls , 2540971, 37809/sec.
Page Faults , 469998, 6993/sec.
I/O Read Operations , 15073, 224/sec.
I/O Write Operations , 37020, 551/sec.
I/O Other Operations , 22403, 333/sec.
I/O Read Bytes , 378399597, 25104/ I/O
I/O Write Bytes , 605689075, 16361/ I/O
I/O Other Bytes , 1259301, 56/ I/O

-----------------------------

Results for Kernel Mode:
-----------------------------

OutputResults: KernelModuleCount = 184
Percentage in the following table is based on the Total Hits for the Kernel

Time 45690 hits, 25000 events per hit --------

Module	Hits	msec	%Total	Events/Sec
ntkrnlpa	37371	67226	81%	13897524
b57nd60x	2214	67227	4%	823329
tcpip	1128	67227	2%	419474
ndis	1105	67226	2%	410927
hal	1101	67226	2%	409439
win32k	662	67227	1%	246180
dne2000	651	67227	1%	242090
afd	317	67227	0%	117884
iaNvStor	256	67226	0%	95201
VMNetSrv	147	67227	0%	54665
NETIO	131	67226	0%	48716
nvlddmkm	108	67227	0%	40162
fltmgr	93	67226	0%	34584
Ntfs	92	67226	0%	34212
CVPNDRVA	91	67227	0%	33840
dxgkrnl	76	67227	0%	28262
USBPORT	30	67227	0%	11156
rspndr	20	67227	0%	7437
iaStor	17	67226	0%	6321
pacer	13	67227	0%	4834
sptd	11	67226	0%	4090
usbuhci	7	67227	0%	2603
fileinfo	7	67226	0%	2603
MpFilter	6	67227	0%	2231
ecache	5	67226	0%	1859
HIDPARSE	4	67227	0%	1487
usbhub	4	67227	0%	1487
HIDCLASS	3	67227	0%	1115
CLASSPNP	3	67226	0%	1115
luafv	2	67227	0%	743
fvevol	2	67226	0%	743
cdd	1	67227	0%	371
ATMFD	1	67227	0%	371
rdbss	1	67227	0%	371
hidusb	1	67227	0%	371
mouhid	1	67227	0%	371
NETw4v32	1	67227	0%	371
usbehci	1	67227	0%	371
watchdog	1	67227	0%	371
volsnap	1	67226	0%	371
ksecdd	1	67226	0%	371
ataport	1	67226	0%	371
volmgr	1	67226	0%	371
acpi	1	67226	0%	371

======== END OF RUN =======
======== NORMAL END OF RUN =======


So, is this all? Nope, you can "zoom in" on individual modules (use multiple -z entries if you need to).

kernrate -xk -u -z intelppm -j srv**http://msdl.microsoft.com/download/symbols > krlog.txt

So, how does zooming in work?

C:\Users\cmihai>kernrate -xk -u -z hal

----- Zoomed module hal.dll (Bucket size = 16 bytes, Rounding Down) --------
Percentage in the following table is based on the Total Hits for this Zoom Module

Time 5 hits, 25000 events per hit --------
Module Hits msec %Total Events/Sec
READ_PORT_ULONG 1 4726 16 % 5289
READ_PORT_USHORT 1 4726 16 % 5289
KeReleaseQueuedSpinLock 1 4726 16 % 5289
KeAcquireInStackQueuedSpinLockRaiseToSynch 1 4726 16 % 5289
KfAcquireSpinLock 1 4726 16 % 5289
HalpGenerateInterrupt 1 4726 16 % 5289

================================= END OF RUN ==================================
============================== NORMAL END OF RUN ==============================

Other Debugging Links

rzip - great compression rates for logs and such

The rzip program is huge-scale data compression software designed around initial LZ77-style string matching on a 900 MB dictionary window, followed by Bzip2-based Burrows-Wheeler transform (BWT) and entropy coding (Huffman) on 900 kB output chunks. The first stage of rzip is similar to that of rsync, and, no wonder: they're both authored by Andrew Tridgell, author of Samba (and lots more).

Here are some "benchmarks" I did of rzip on a collection of binary logs and transaction logs. Compress ratio would be even better on plain text logs.

If you have to store huge amounts of log files, maybe archive them for undetermined periods of time, sometimes on write once media (WORM) such as DVD-R, BD-R or HP UDO to provide compliance with unalterable retention policies or being able to easily destroy media (just pop in the DVD in the microwave for 3 seconds on 700W) then you really should look at rzip. Just set your logs to rotate at 1-2 GB, maybe even trying to compensate for compression(to avoid filesystem limitations such as ISO vs. UDF, FAT32 and even some tools that can't handle big files) then compress them with rzip!



While on the system itself you can store them on a compressed filesystem (and I've gotten quite good compression ratios using ZFS compression) once you dump them to tape or write once media, it's a different story.

rzip -9 logs.tar 155.00s user 2.17s system 99% cpu 2:37.79 total

gzip -9 logs.tar 138.90s user 4.32s system 99% cpu 2:23.79 total

bzip2 -9 logs.tar 311.03s user 2.17s system 99% cpu 5:14.61 total

7za a logs.tar.7za logs.tar 805.75s user 11.16s system 165% cpu 8:12.84 total


-rw-r--r-- 1 cmihai sysadmin 899M Mar 6 14:33 logs.tar
-rw-r--r-- 1 cmihai sysadmin 528M Mar 6 14:43 logs.tar.7za
-rw-r--r-- 1 cmihai sysadmin 576M Mar 6 14:12 logs.tar.bz2
-rw-r--r-- 1 cmihai sysadmin 587M Mar 6 14:09 logs.tar.gz
-rw-r--r-- 1 cmihai sysadmin 223M Mar 6 14:30 logs.tar.rz

But this great compression ration does come at a price: memory usage. rzip uses a history buffer of 900MB compared to 32kb for gzip and 900k for bzip. And again, memory is cheap and plenty now, and it's not really that big of an issue. The other major issue is: you can't pipe.

Rzip uses a two stage process. The first stage of rzip is very similar to that of rsync. It finds and encodes large data segments using a 900MB history buffer. The second stage is basically bzip2.

Still, as you can see, it can be faster than bzip at times. It's speed is actually comparable to that of gzip.

Memory usage:
13960 cmihai 1520K 1048K cpu0 0 0 0:00:17 28% gzip/1
13965 cmihai 8848K 7584K cpu0 0 19 0:00:22 32% bzip2/1
13967 cmihai 643M 642M cpu0 0 0 0:00:14 24% rzip/1

Now for a larger quantity of log files:

~/rzip -9 biglogs.tar 1815.89s user 69.63s system 98% cpu 31:48.57 total

-rw-r--r-- 1 cmihai sysadmin 2.7G Mar 6 16:00 biglogs.tar.rz
-rw-r--r-- 1 cmihai sysadmin 9.5G Mar 6 15:27 biglogs.tar

As you can see, rzip proves to be quite the disk space saver :-). If you have to archive logs on a regular basis, consider giving rzip a spin. Though I can't stress enough that you should do a couple of benchmarks _yourself_, using the kind of data you're trying to archive. Just take a couple of samples and tar them up (1GB, 10GB sound like fair values), then do a simple time rzip / time gzip / time bzip and check the results. But, like I've said, expect memory usage for rzip to be around 500-900MB .

If it's logs you're archiving, decompression times shouldn't be an issue, but you should and least try to decompress the archive on a couple machines. Also, try testing with large files, see if there are any filesystem or application limitations you need to worry about.

Note: these tests have been done with an older version of rzip, on a Solaris machine (so yeah, it is quite portable). Newer versions of rzip are faster, and have an even better compression ratio.

There are also other implementations of rzip that use LZMA (Lempel-Ziv-Markov chain algorithm) as as the second stage, instead of bzip. Long Range ZIP or Lzma RZIP is such an implementation.

Oracle 10g on Solaris 10 - SUNWsprox dependency

Oracle 10g seems to depend on SUNWsprox - Sun workshop Bundled 64-bit make library. The x termination stood for 64 bit libs in the pre Solaris 10 days, but this no longer applies. The package you actually want is SUNWsprot.

# pkginfo | grep SUNWsprot
system SUNWsprot Solaris Bundled tools

Oracle actually needs libmakestate.so.1. As we can see, that is provided by the SUNWsprot package (part of Solaris 10).

# grep libmakestate.so.1 /var/sadm/install/contents
/usr/lib/amd64/libmakestate.so.1 f none 0755 root bin 12432 31159 1172201465 SUNWsprot
/usr/lib/libmakestate.so.1 f none 0755 root bin 9080 44832 1172201465 SUNWsprot

During the install, it will print out:

SUNWsprox package not installed

If you have the SUNWsprot package installed, you can safely ignore this message.

Sysinternals Process Monitor - mplayer GL output on Vista

Mplayer is my favourite movie player. It's got handy keyboard shortcuts, virtually all codecs, it's fast and without bloat. Problem is some video output drivers don't work well on all platforms. Some require you to use "-vo xv" (for fast scalable graphics on UNIX systems), some only support -vo gl and so on. In the case of Windows Vista, you need to use "-vo gl" if you're using Aero.

Ever wondered where a programs looks for default configurations files, registry entries and such? You can use truss or DTrace on Solaris for example, but on Windows, you have ProcMon. Mplayer usually looks for default configuration in ~/.mplayer/config or mplayer.conf in the default dir and such. Well, on Vista, it looks for the default config in C:\Users\cmihai\AppData\Roaming\mplayer (or whatever your home may be), but it really depends on your setup. A quick way to find out is to use Process Monitor:

Process Monitor - procmon is a Sysinternals tool that combines FileMon and RegMon.

Just filter by process name to include mplayer, and you've got yourself a nice list of files and registry keys it opens or tries to access. Simple as that.



All that remains is to edit that file C:\Users\cmihai\AppData\Roaming\mplayer\config and to add "vo=gl" to it, and we've "fixed" mplayer so that it works with the Aero interface in Vista.

Antigen SPAM Gateway

Have you ever had the feeling that all your spam (millions of mails a week, usually up to 95% of all mail traffic) are caused by a very small number of people or viruses? Makes you wonder if it wouldn't be easier to just eliminate spam at the source. Sadly, laws are sketchy and are "impotent" (at best) and in need of "v1eagr4" when it comes to the issue of spammers.

Still, SMTP gateways, greylisting, tarpitting, stuttering SMTP responses, RBLs, antivirus software and such do a pretty good job, keeping spam under 1% (better results are achievable at the risk of eliminating actual requested or regular messages).