When you create a loop device from a disk image with losetup, it doesn't bother reading the partition table from the disk image so you don't get the nice and easy access to, for example, /dev/loop0p1 for partition 1.

FreeBSD seems to get this right, as I recall, but Linux does not.

fdisk outputs these devices, but they don't exist:

% sudo fdisk -l /dev/loop0 | grep '^/'
/dev/loop0p1   *           1        1043     8377866    7  HPFS/NTFS
/dev/loop0p2            1044        2088     8393962+   7  HPFS/NTFS

Linux's mount(8) command gives you the '-o offset=XXX' option. The offset is a byte offset, and lets you decide how far into your disk image you want to start. However, fdisk doesn't output in bytes, it outputs in cylinders or sectors.

Not to worry, it helpfully outputs the conversion between the units and bytes:

% sudo fdisk -l /dev/loop0 | grep Units
Units = cylinders of 16065 * 512 = 8225280 bytes

Knowing this, let's use awk to generate the offsets for us:

% sudo fdisk -l /dev/loop0 
  | awk '/^Units/ { bytes=$(NF-1) } /^\// { print $1 "[" $NF "]: mount -o offset=" $3 * bytes }'
/dev/loop0p1[HPFS/NTFS]: mount -o offset=8225280
/dev/loop0p2[HPFS/NTFS]: mount -o offset=17174384640

Now simply mount them with 'mount -t ntfs -o loop,offset=XXXX mydiskimage /mnt' or whatever you want :)

Previous posts about screen have shown a few new tools for searching your list of open screen sessions.o

Today, I finally sat down and worked on the next installment: Being able to query any screen window and the window list. The difference between the previous script is that we can now grep screen windows other than the 0th one. Additionally, we can now grep the screen window list (which, by the way, has some excellent information).

To that end, I present now two scripts:

• screen-session-hardcopy.sh
• screen-session-search.sh

You need both for this to work optimally, but they exist separately because the functionality is somewhat distinct.

The 'hardcopy' script takes a single argument, a screen session. It will hardcopy all windows in that screen session including the window list. If you specify OUTDIR in your environment, the screen hardcopies will be put in that directory; otherwise, the output directory is printed to stdout for consumption by another script.

The 'search' script runs the hardcopy script on all active screen sessions (in parallel, yay xargs). Once it has all of the copies, it will grep through the output for your query string (regular expression). It supports 3 flags:

• -t - only search 'window titles' (ie; only window list output)
• -w - only search window contents (ie; exclude window list output)
• -l - only search the 'location' field of the window list

Now, with a single command, I can find out where that ssh session to 'foo' disappeared to. Here's an example screen window list capture (accessed with Ctrl+A " (doublequote))

Num Name                                                              Location Flags

  0 zsh                                                                        syn $
  1 zsh                                                                      scorn $

Now, I want to find all sessions open to 'scorn':

% screen-session-search.sh -t 'scorn'
sty 18210.pts-8.snack window 1
sty 18556.pts-0.snack window 0

It found 2 sessions. I can attach to the first one with:

screen -x 18210.pts-8.snack -p 1
          ^ screen session     ^ window

caveat: I've been hacking on things all night, so the code may or may not be very readable. Apologies if you go blind while trying to read it ;)

Press 'f' when keynav is active and instantly jump to my firefox window.

# in .keynavrc
f sh "activate-firefox.sh", end

Now, in activate-firefox.sh:

#!/bin/sh
# activate-firefox.sh
xdotool windowactivate $(xdotool search -title -- '- Mozilla Firefox')

Extend that and press 'g' to jump to gmail, assuming that tab is open. (Requires my firefox tabsearch plugin)

# in .keynavrc
g sh "activate-gmail.sh"

Now, in activate-gmail.sh:

#!/bin/sh
# activate-gmail.sh

./activate-firefox.sh
xdotool key ctrl+apostrophe
xdotool type gmail
xdotool key Return

Hop on over to the keynav project page and download the new version.

The changelist from the previous announced release is as follows:

20080614.01:
  - Several bug fixes and feature additions suggested by Yuri D'Elia.
  - Sync xdotool library to 20080606
  - Added default key binding Ctrl+[ as 'end' (requested by Luke Macken)
  - New command: 'sh' - Executes shell commands.
    Example keynavrc: ctrl+x sh "xterm -bg black -fg white"
  - New command: 'history-back' - Undo a window change operation
    Example keynavrc: a history-back
    + Such operations include: cut-*, grid, cell-select, move-*
    + The history size is currently hard-coded at 100 entries.
    + If you exceed 100 moves, the oldest entry will be removed.
    + Every time keynav is activated, the history is wiped.
  - Fix: Any command starting with "start" is now bound globally.
  - Fix: All rendering is delayed until after the end of the current command
    sequence. This fixes (in order of annoyance, worst first):
    1) Crash when a 'start' and 'end' exist in the same command sequence.
    2) Visible 2x2 grid first, before a 3x3 grid when the start command is
       'start, grid 3x3'
    3) Rendering blinking a full white window on the screen before clipping to
       the grid.
    4) Visible blink when "cut-left,cut-up" and such are run simultaneously.
  - Fix: If the 'start' command is invoked again while keynav is active, then
    the default arrangement is set (full screen and 2x2 grid). Previously, the
    'start' command was a no-op if keynav was active.

Tonight's fun was spent learning bpf's internals (the pseudo-machine code it uses). The point was to find out exactly how much effort it would take to add secure bpf support to jails. Ideally, we'd want to expose the bpf(4) device to any jail but only make available the traffic that is actually destined for the jail (or broadcast traffic).

It seems like you could get away with this, if you prefixed all jailed bpf filters with: (ip and (host [jail_ip] or multicast or broadcast)). I've got userland-code that does exactly this. Once I knew how to inject my own bpf code into an existing bpf_program struct, I was basically ready to go. The only other thing left was to figure out how, in the FreeBSD kernel, to figure out if you were in a jail and what that jail's IP was - turns out this is a trivial operation :)

Userland example code: pcapinject.c

Working Patch: bpf-jail.patch

The code in the patch is crappyish and has a pile of debug statements, but it does appear to work as intended.

I finished the first function in pcre-grok's predicate feature.

% ifconfig | ./grokre '%{IP}' IP
Entry: IP => 192.168.0.5
Entry: IP => 127.0.0.1

# Now, with a predicate:
% ifconfig | ./grokre '%{IP =~ /^192/}' IP
Entry: IP => 192.168.0.5

The eventual plan is to allow users to register their own predicates. The first target of this will be a python module wrapping grok allowing you to use grok and additionally write predicate functions in python, executed inside the regular expression.

So far, PCRE has not let me down.

Hop on over to the xdotool project page and download the new version.

I've also put up an HTML version of the manpage: xdotool.xhtml.

The changelist from the previous announced release is as follows:

20080606:
  * Correctly report an error if xdotool is invoked with an invalid command.
  * Fix invalid error code reporting from 'windowraise'

Like I said, I run screen in all of my xterms...

xterm sets an environment variable in child processes: WINDOWID. This is the X window id of the xterm window. Using this, we can extend upon my last post and come up with a much neater solution. Knowing what screen session you want to bring forward (assuming it's running in an xterm), we can run a command inside that session that grabs the $WINDOWID variable in the shell and uses xdotool to activate the window.

session=$(sh screen-find.sh "#freebsdhelp")
STY=$session screen -X screen sh -c 'xdotool windowactivate $WINDOWID'

Running this causes my IRC window to be activated by xdotool, which means it is now active, focused, and on top.

This isn't entirely optimal, because it assumes the xterm attached to that screen session is the xterm that launched it. If you run 'xterm -e screen -RR' and close the xterm (don't log out of the shell), then rerun 'xterm -e screen -RR' it will attach to that previous screen session, but the WINDOWID will understandably not be correct any longer.

So what do we do? Using the screen session given, we create a new window in that session and set the title of that window to a generated string. We then use xdotool to search for that string and activate the window. Once we get there, we can kill that new screen window we created and we are left with the terminal holding our screen session sitting in front of us.

I wrote a script to do just that tonight: screen-activate.sh. Example usage: screen-activate.sh 24072.pts-25.snack

This has a great benefit of supporting every terminal program that understands how to set the terminal window title when screen changes it's title. I have tested my .screenrc in Eterm, Konsole, gnome-terminal, and xterm - all know when screen changes it's title if you put this in your .screenrc:

hardstatus string "[%n] %h - %t"
termcapinfo xterm 'hs:ts=\E]2;:fs=\007:ds=\E]2;screen (not title yet)\007'

# Might need this:
termcapinfo  * '' 'hs:ts=\E_:fs=\E\\:ds=\E_\E\\'

Scenario: I run lots of xterms. Each xterm runs a single screen session(*). At any given time, I can only see some of the xterm windows (the others are hidden).

(*) All my xterms run with: 'xterm -e screen -RR'. This causes them to attach to the first-found detached screen, and if none exist creates a new screen session. See run-xterm.sh for my pleasant, random-colored xterm script.

Problem: I forget where I put things. I can't find that terminal where I'm editing foo.c!

Possible Solutions:

1. Bad: Kill the vim session that's editing the file, and rerun vim somewhere else.
2. Good: Use xdotool to search window titles for 'foo.c'
3. Great: Find the screen STY variable for the process 'vim foo.c'
4. Great: Ask each open screen session about what it is on screen

Today, we'll cover the two 'great' solutions. I wrote both of these a while ago, but I totally forgot to post about them. Here you go :)

Find a screen by it's child processes

Tool: screenps.sh

This tool takes a regexp pattern as the only argument and will output a list of screen sessions having child process commands that match that pattern. This is useful for finding what screen is running 'vim foo.c'

% ./screenps.sh 'vim foo.c'
23464.pts-0.snack

Find a screen by what is being displayed

Tool: screen-find.sh

This tool takes a regexp pattern as the only argument. It uses screen's hardcopy command to save the on-screen buffer and then applies the regexp given to the buffer. If it matches, the screen session is output. There is special behavior if only one screen session is found: If the screen session is currently attached, it will flash that screen session giving you a visual clue about where it is; if it is not attached, it will attach to it.

% ./screen-find.sh "keynav"
28504.pts-27.snack

In case you still aren't clear, the two tools help you find your lost screen sessions. Maybe they aren't lost, but certainly it's easier to search for them by text than by eyeballs if you know what's in them.

A short summary: screenps.sh will search for commands running in a screen session and screen-find.sh will search for literal text displayed in a screen session. Both are super useful.

Note: Currently, screen-find.sh can only capture the contents of the 0th screen window (screen sessions can have multiple windows). I worked for a while on solving this, but for whatever reason I couldn't get it working properly.

Perl grok was great. I learned a lot about how far beyond normal I could take regular expressions. I ported much of perl grok to C++ using Boost Xpressive, but Boost has a lot of baggage with it. I didn't like the feel of Xpressive, Boost is huge, compiling takes forever and a day (thanks C++ Templates!), and binaries are almost guaranteed to be 1meg or more.

That said, I think I might be reinventing the wheel again by trying to see what grok in C with libpcre feels like. Sample code line:

  re = pcre_compile("([0-9]+)(?C1)", 0, &errptr, &erroffset, NULL);

(?C1) is PCRE-syntax for "call callback #1" - the callback I wrote converts the last capture into a number and only succeeds if the value is greater than 5. It'll succeed once that precondition passes:

% ./a.out "foo 2 4 6 8"     
Trying: 2
Trying: 4
Trying: 6
Found: 6

All with a single regular expression + callouts. This feature (called callouts by PCRE) is what allows me (and you) to use predicates in grok. PCRE passes the first test.

A few hours later, I had pattern injection working (Turning %FOO% into it's regular expression) and could parse logs with ease.

I couldn't help pitting the boost and pcre versions against eachother, even though the feature set isn't the same, yet. pcregrok processed 37000lines/sec of apachelog (the most complex regexp I have), versus 6200/sec from c++/boost grok.